logstash output to kafka - topic data in message

Question

I want to create a conf file for logstash that loads data from a file and send it to kafka.

the file is in json format and has the topicId in it.

This is what I have so far..

input {
   file {
    path => "~/file1.json"
    start_position => "beginning"
    codec => "json"
  }
}
filter {
    json {
        source => message
    }
}
output {
  kafka { 
    bootstrap_servers => "localhost"
    codec => plain {
        format => "%{message}"
    }
    topic_id =>  "???"
  }
}

can this be done?

Regards, Ido

Arijeet Saha · Accepted Answer · 2016-08-03 08:19:56Z

3

Yes it can be done.

For example if the message json contains a topic_id key like:

"topicId": "topic1"

Then in logstash kafka output plugin:

output {
  kafka { 
    bootstrap_servers => "localhost"
    codec => plain {
        format => "%{message}"
    }
    topic_id =>  "%{topicId}"
  }
}

answered Aug 3, 2016 at 8:19

Arijeet Saha

1,17812 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

logstash output to kafka - topic data in message

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related