126 questions
1
vote
1
answer
129
views
Ingress-NGINX consistent-hash for WebSocket routing across multiple controller pods is inconsistent (hash by query param)
I’m trying to route WebSocket connections deterministically to the same backend pod(in a k8s deployment) based on a query parameter (room/league id). This works with a single ingress-nginx pod, but ...
- 11
1
vote
0
answers
72
views
Does Nginx uses consistent hashing with Virtual nodes for load balancing using hash / ip_hash
Under ngx_http_upstream_module module we have hash and ip_hash directive which is used to route requests from same client to a single pod every time using client's ip or any other key. These ...
- 49
0
votes
1
answer
178
views
How failures and restore operations in sharding (consistent hashing)
In consistent hashing, suppose we are using username as for hashing
hashFunction(username) = nodeA
Now from what I understand, if there is any failure or a node is removed requests will be directed to ...
- 47
0
votes
0
answers
203
views
How to establish a consistent hash ring with 300 million virtual nodes within 10 seconds with C++
As stated in the title I received an optimization task. I want to create a hash ring but I need to add too many virtual nodes on it(Distributed storage requirements), and I hope the entire ...
- 13
1
vote
0
answers
208
views
Applying modulo function on MD5 hash
I have a use case to evenly distribute input strings into ‘n’ buckets. If I apply MD5 algorithm to generate a hash of the string and then apply modulo function will it result in even distribution of ...
- 11
2
votes
1
answer
846
views
What is the lookup time complexity of Consistent Hashing Design? [closed]
While reading Alex Xu's book on designing large systems, I had a question about the part about consistent hashing.
The book says that in a stable hash, the server where the key will be stored is a ...
- 23
1
vote
1
answer
682
views
How to combine sharding and consistent hashing within a distributed system?
Sharding, or say partitioning, is a technique widely used in distributed systems which logically splits data into partitions. Each node is assigned a set of partitions and hence the read/write ...
- 21
0
votes
1
answer
221
views
DynamoDB What are coordinator nodes in context of replication?
So i was going through this paper :- http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
There in the part of replication they have mentioned that Each key k is assigned to a ...
- 88
1
vote
1
answer
433
views
Akka Consistent-hashing routing
I have developed an application using Typed Akka 2.6.19.
I want to route events from a certain source to the SAME routee/worker based on IP address. So, I have planned to use Consistent-hashing ...
- 167
0
votes
1
answer
436
views
consistent hashing where you want a key mapped to multiple servers
I'm wondering if I'm missing a concept here somewhere, and wondering if someone can explain how this might work.
My understanding of consistent hashing makes perfect sense where I want to map a ...
- 509
1
vote
1
answer
375
views
Does a Cassandra node get assigned a new token every time it restarts?
From my limited knowledge, Cassandra assigns a random token for every new node in the ring. The ring position is important because data is replicated in the SimpleStrategy according to the position. ...
2
votes
1
answer
3k
views
Using custom hash function with HAProxy LoadBalancer
I have configured a load balancer using HAProxy on frontend and on the backend I have 4 server serving the request using Apache web server.
#HAProxy for web servers
frontend web-frontend
bind ...
user10798329
1
vote
2
answers
584
views
Can Rendezvous hashing add a node efficiently?
The wikipedia article for Rendezvous hashing (https://en.wikipedia.org/wiki/Rendezvous_hashing) doesn't explain what happens when you add a node to the hash table. The way I understand it, if you add ...
- 166
1
vote
0
answers
159
views
How to use consistent hashing to allocate multiple servers for the same key?
I'm reading a system design article about design ticketmaster. It mentions:
We can use the Consistent Hashing to allocate application servers for both ActiveReservationService and WaitingUserService ...
- 11
1
vote
0
answers
194
views
Can I use routing key for binding to exchange when using RabbitMQ consistent hashing?
I need to achieve pub/sub messaging where one publisher publishes an event to a single consistent hashing exchange and other subscribers use routing key (event name) to indicate interest in a specific ...
- 11
0
votes
1
answer
441
views
Autoscaling Backend and RabbitMQ Queues
I have an IoT system around 100k devices, publishing their state every second to the backend written in Java/Spring Boot. Until now, I was using gRPC but I see excessive CPU usage so I was planning to ...
- 474
3
votes
1
answer
502
views
Consistent hashing collisions?
Probably an implementation detail of consistent hashing, IIUC, under ideal conditions, with N servers and K hash functions, we could expect to see NK evenly distributed nodes on the hash ring.
but ...
- 10.7k
12
votes
2
answers
7k
views
Consistent hashing, why are Vnodes a thing?
My understanding of consistent hashing is that you take a key space, hash the key and then mod by say 360, and place the values in a ring. Then you equally space nodes on that ring. You pick the node ...
- 2,185
8
votes
1
answer
877
views
How to solve hot partition problem in consistent hashing Load Balancing
I am trying to understand how different Load Balancing strategy works.
One way is to use consistent hashing algorithm where we divide the entire space into multiple virtual nodes and each physical ...
- 453
4
votes
1
answer
758
views
what is the difference between consistent hashing and sticky sessions?
My understanding of consistent hashing is that it is used to minimize rehashing overhead and that request is served to the next nearest worker in the ring to the hash value of the request. So this ...
- 817
0
votes
1
answer
352
views
Does it make sense to use zookeeper in masterless architecture?
In distributed set up using consistent hashing,eg. distributed cache implementation using consistent hashing, how can we manage the multiple nodes? By managing I mean, monitoring health and adjusting ...
- 493
2
votes
1
answer
3k
views
Zookeeper for Consistent hashing?
Can we use zookeeper for implementing consistent hashing? I want to maintain my hashring of servers with zookeeper so that their health can be monitored and the logic of mapping a given key to ...
- 135
0
votes
1
answer
200
views
Is it necessary to hash data POSTed if using SSL?
In some sensitive apps, like payment systems, code is set up as such: when user sends info to server, for example:
{money_amount: 5, receiver_id: 2}
It's also required to send over a hashed string, ...
- 1,124
4
votes
2
answers
1k
views
Rendezvous vs consistent hashing
I read on Wikipedia:
Unlike consistent hashing, HRW (Highest Random Weight, aka Rendezvous Hashing) requires no precomputing or storage of tokens.
Why?
My understanding is that:
In consistent ...
- 12.9k
1
vote
1
answer
125
views
Is consistent hashing useless if our server uses RESTful APIs? [closed]
So RESTful APIs are stateless wherein the server does not store any state about the client session on the server side.
And consistent hashing in load balancing is used to associate a client with a ...
- 21
4
votes
1
answer
441
views
Consistent hashing domino effect
I'm trying to figure out how a system like Cassandra which uses consistent hashing handles cascading node failures. I know there's this concept of virtual nodes and keys are mapped to virtual nodes. ...
- 1,571
3
votes
1
answer
1k
views
Consistent Hashing to a Dynamic Set of Servers Based on TCP Payload
I have a scenario where I need to route TCP traffic to a dynamic set of backend servers (Kubernetes pods to be exact but we can ignore that for purposes of this post) through a proxy like HAProxy or ...
- 460
9
votes
1
answer
3k
views
How to add partition to Kafka topic and keep same-key message in same partition？
It is common to require ordering in same partition of given Kafka topic. That is, messages with same key should go to same partition. Now, if I want to add new partition in a running topic, how to ...
- 76.4k
1
vote
1
answer
336
views
How to reliably shard data across multiple servers
I am currently reading up on some distributed systems design patterns. One of the designs patterns when you have to deal with a lot of data (billions of entires or multiple peta bytes) would be to ...
- 2,449
0
votes
0
answers
47
views
Exclude a specific node for a specific input in consistent hashing?
If my input key hashes to anywhere between machine1 and machine2 in the circle, and our algorithm goes clockwise, then we know machine2 will be returned.
consistentHash.getMachine(myInput) = machine2;...
- 2,355
1
vote
1
answer
1k
views
Nginx Consistent Hashing for Multiple Upstreams
The Upstream server is wowza. There are two upstreams
upstream wowza {
hash $arg_streamKey consistent;
server x.x.x.x:8087;
server x.x.x.y:8087;
}
upstream wowza_thumbnail {
hash $...
- 2,300
0
votes
1
answer
232
views
Dynamically add shards to DynamoDB and remap old data
I know that dynamoDB supports shards. I wanted to know that is it possible to add shards dynamically.
Suppose I provisioned 4 shards and shardkey would be customerID.
Now in the future I want to ...
- 1,164
0
votes
2
answers
369
views
how to implement consistent hashing in django using redis database?
I want to implement consistent hashing in django with redis as my cache DB.
Firstly I tried utilising uhashing(https://pypi.org/project/uhashring/) package to implement consistent hashing with redis ...
- 1
8
votes
4
answers
12k
views
why do we need consistent hashing when round robin can distribute the traffic evenly
When the load balancer can use round robin algorithm to distribute the incoming request evenly to the nodes why do we need to use the consistent hashing to distribute the load? What are the best ...
- 161
1
vote
1
answer
385
views
Does load distribution with consistent hashing provide an advantage over standard hashing for stateless servers while scaling a system?
I understand that using consistent hashing for load distribution in case of cache servers or (sharded) database servers offer a significant advantage over usual key-based hashing, as while adding/...
- 297
0
votes
1
answer
95
views
How to distribute (card game table) dealers across servers in a balanced way?
I am currently working on an online card game, similar to blackjack, which will consist of a series of tables where each table has a "dealer" and multiple human players. The dealer (a computer bot) is ...
- 20.1k
2
votes
0
answers
87
views
How are READ/WRITE requests for failed node handled in Consistent Hashing based datastore during REHASHING?
TL;DR : How the READ/WRITE requests for KEYS present in failed node handled during rehashing/rebalancing ?
Suppose in Consistent Hashing setup of database, one node in database goes down. Now , only ...
- 931
3
votes
1
answer
4k
views
Zookeeper-Kafka and Consistent hashing
I am learning Zookeeper and I was stuck in middle with some confusion. I gone through various forums and questions and none clear my confusion and came to SO finally to get some clarification on the ...
- 63
0
votes
1
answer
23
views
ConsistentHashingGroup - detecting failing node in cluster
I am using ConsistentHashingGroup and I would like to be notified when Akka node in the cluster that had routees for the group went down.
Is there any way to do so without subscribing to cluster ...
- 1
1
vote
1
answer
865
views
Unavailable nodes in consistent hashing
From everything I have read, in consistent hashing, if a node crashes, the keys handled by that node will be re-mapped to the adjacent node in the hash ring. This conceptually makes sense to me.
What ...
- 739
11
votes
2
answers
5k
views
does redis cluster use consistent hashing
I'm using redis cluster 3.0.1.
I think redis cluster use consistent hashing. The hash slots are similar to virtual nodes in consistent hashing. Cassandra's data distribution is almost the same as ...
- 776
0
votes
1
answer
130
views
Two questions about OpenStack Swift ring function
I'm new in Swift and I'm trying to learn its functionality. I have two question for you regarding the ring and the consistent hashing algorithm.
When we want to store an object, we take its path (for ...
- 144
2
votes
1
answer
392
views
Classic Cassandra and Coordination
I am curious about coordination in classic Cassandra. I read the Facebook paper written by Avinash Lakshman and Prashant Malik called Cassandra - A Decentralized Structured Storage System
An excerpt ...
- 5,832
1
vote
1
answer
128
views
Custom hashing function in infinispan
I am using Infinispan version 8.2 .I already knew when an Node puts an entry to an Cache.It uses its internal hashing on the key and chooses a primary node in an cluster.
I know that we can override ...
- 253
11
votes
2
answers
1k
views
Memcached consistent hashing not working with 3 of 4 servers down
Story
I have 3 memcached servers running where I shutdown the one or the other to investigate how PHP-memcached behaves upon a server not beeing reachable.
I have defined 4 servers in PHP, 1 to ...
- 32.6k
0
votes
0
answers
133
views
Is there any advantage of Simple Hashing over Consistent Hashing?
Can anyone possible let me know any advantages of Simple Hashing over Consistent Hashing. I was looking over internet and couldn't found some parameters like time to add a new node in consistent ...
- 8,937
4
votes
1
answer
1k
views
Why does elasticsearch still use simple routing value using modulo?
Just wondering why elasticsearch still use that simple routing value approach for deciding which shard the data must be stored to. Actually this approach is limiting us to change the number of shards ...
- 143
1
vote
2
answers
216
views
Is it possible to add a node to a cluster, where data is replicated synchronously?
If I have system which writes data to a storage.
In cluster setup if the data is replicated synchronously, that is when I get request to change data that is updated across all the nodes and then the ...
- 11
0
votes
2
answers
3k
views
Implement Consistent Hash Algorithm in Java
I am trying to implement a consistent hash based algorithm in java using the following reference for sharding keys to redis -
Stanford Theory on CH
I am trying to understand the best way to generate ...
- 459
2
votes
0
answers
33
views
How to independently scale M web app servers and N cache servers?
I have M web app servers (with the same app on them, shared-nothing), behind a load balancer.
The app running requires heavy caching, so I would like to have N cache servers behind (using Memcached).
...
- 171