TL;DR : How the READ/WRITE requests for KEYS present in failed node handled during rehashing/rebalancing ?
Suppose in Consistent Hashing setup of database, one node in database goes down. Now , only the keys on that particular node will be rehashed/re-balanced across the remaining nodes in the hash ring.
My question is , how the READ/WRITE requests for KEYS present in failed node handled during rehashing/rebalancing ?
One simplistic answer can be that READ/WRITE requests are handled by replicas of the failed node during rebalancing. However consider below case :
Suppose, my replication factor(N) = 3 , WRITE quorum(W) = 3 and READ QUORUM(R) = 3
Now , if one node fails then during rebalancing , my replication factor will become 2 i.e. there are only two copies of data present on failed node.
Now , if a READ request comes , since my READ quorum is set at 3 , my database will be unavailable to that request since there is no way it can find 3 copies corresponding to READ request.
However , as far as I know Consistent Hashing helps in designing ALWAYS-ON systems.
Does , it mean that during rebalancing/rehashing , it is quite possible that my R/W quorum may not be satisfied if I desire high Aavailability ??
