Kafka - Message with different key stored in same partition

Question

I am trying to store Messages with different key to different partition.

For example:

ProducerRecord<String, String> rec1 = new ProducerRecord<String, String>("topic", "key1", line);
ProducerRecord<String, String> rec2 = new ProducerRecord<String, String>("topic", "key2", line);
producer.send(rec1);
producer.send(rec2);

But when i try to run my Producer class, it always stored in single partition.

As per documentation, DefaultPartitioner uses message key hash code to find the partition. I also saw this question Kafka partition key not working properly‏, but i cannot find ByteArrayPartitioner class in 0.9.x version of Kafka Client library.

props.put("partitioner.class", "kafka.producer.ByteArrayPartitioner")

Update: I am creating the topic on the fly using code.

If i create a topic with partitions manually, then its working fine.

Matthias J. Sax · Accepted Answer · 2016-10-07 19:10:22Z

3

If topics are created "on the fly", the are created with number of partitions according to num.partitions parameters (with default value 1). And if you have only one partitions, all data will go to this single partitions.

However, keep in mind, even if you have multiple partitions, a partitions can still get different keys assigned! Even if you have num-partitions == num-distinct-keys there might be hash collisions, assigning two different keys to the same partitions (and leaving some partitions empty).

If you want to ensure that different keys always go to different partitions, you need to use a consumer partitioner or specify the partition number directly.

answered Oct 7, 2016 at 19:10

Matthias J. Sax

62.8k8 gold badges128 silver badges148 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Shankar Over a year ago

Thanks so much for your answer.

Collectives™ on Stack Overflow

Kafka - Message with different key stored in same partition

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related