Kafka: How to fix TimeoutException error?

Question

I am using Kafka With Nifi and when I am ingesting a large file (over 100MB). I am getting a TimeOutException error. Kafka does not crash. Reading about it, i need to increase the following property 'request.timeout.ms' that is set by default to 30ms. What should this property be set for with bigger files that takes longer time to be ingested? How can I calculate it? i am using Confluent Kafka 5.3.1 in a production environment.

Thank you

maxime G · Accepted Answer · 2020-05-27 12:45:21Z

1

With publishKafka and ConsumeKafka processors you can add dynamic properties to add kafka configuration to your consumer or producer. so you can add a property like this :
key : request.timeout.ms
value : the value you want

you can also configure back pressure in nifi connection to secure your environement.

you should check nifi app-logs, maybe your kafka is down.

answered May 27, 2020 at 12:45

maxime G

1,7981 gold badge17 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Kafka: How to fix TimeoutException error?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related