logstash kafka input not working

Question

I am trying to get the data from Kafka and push it to ElasticSearch.

Here is the logstash configuration I am using:

input {
kafka {
zk_connect => "localhost:2181"
topic_id => "beats"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "elasticse"
}
}

Can anyone help here with the logstash configuration? If I run this I am getting invalid configuration error.

D:\logstash-5.0.0\bin>logstash -f log-uf.conf
Sending Logstash logs to D:\logstash-5.0.0\logs\logstash-plain.txt which is now
configured via log4j2.properties.
[2016-11-11T16:31:32,429][ERROR][logstash.inputs.kafka    ] Unknown setting 'zk_
connect' for kafka
[2016-11-11T16:31:32,438][ERROR][logstash.inputs.kafka    ] Unknown setting 'top
ic_id' for kafka
[2016-11-11T16:31:32,452][ERROR][logstash.agent           ] fetched an invalid c
onfig {:config=>"input {\n   kafka {\n   zk_connect => \"localhost:2181\"\n   to
pic_id => \"beats\"\n   consumer_threads => 16\n  }\n}\noutput {\nelasticsearch
{\nhosts => [\"localhost:9200\"]\nindex => \"elasticse\"\n}\n}\n", :reason=>"Som
ething is wrong with your configuration."}

can anyone help here?

Val · Accepted Answer · 2016-11-11 11:35:23Z

8

You're running Logstash 5 with a config for Logstash 2.4.

zk_connect (Zookeeper host) was replaced by bootstrap_servers (Kafka broker) and topic_id by topics in 5.0

Try this config instead:

input {
  kafka {
    bootstrap_servers => "localhost:9092"
    topics => ["beats"]
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "elasticse"
  }
}

answered Nov 11, 2016 at 11:35

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Soundarya Thiagarajan Over a year ago

Val, I need one more help. If I want Kafka as Output, is it fine if I mention like the answer I have mentioned below : input { kafka { bootstrap_servers => "localhost:9092" topics => ["beatsab"] } } output { kafka { bootstrap_servers => "localhost:9092" topics => ["sound"] } }

Val Over a year ago

No in the output, you need topic_id instead of topics

Collectives™ on Stack Overflow

logstash kafka input not working

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related