Dynamic elasticsearch index_type using logstash

Question

I am working on storing data on elasticsearch using logstash from a rabbitmq server.

My logstash command looks like

logstash -e 'input{
rabbitmq {
    exchange => "redwine_log"
    key => "info.redwine"
    host => "localhost"
    durable => true
    user => "guest"
    password => "guest"
}
}

output {
  elasticsearch {
    host => "localhost"
    index => "redwine"
  }
}
filter {
  json {
    source => "message"
    remove_field => [ "message" ]
  }
}'

But I needed logstash to put the data into different types in elasticsearch cluster. What i meant by type is:

"hits": {
      "total": 3,
      "max_score": 1,
      "hits": [
         {
            "_index": "logstash-2014.11.19",
            "_type": "logs",
            "_id": "ZEea8HBOSs-QwH67q1Kcaw",
            "_score": 1,
            "_source": {
               "context": [],
               "level": 200,
               "level_name": "INFO",

This is the part of search result, where you can see the logstash by defualt creates a type named "logs" (_type : "logs"). In my project i needed the type to be dynamic and should be created based on the input data. For example: my input data looks like

{
    "data":"some data",
    "type": "type_1"
}

and i need the logstash to create a new type in elasticsearch with a name "type_1"..

I tried using grok..But couldnt able to get this specifc requirment.

Ysak · Accepted Answer · 2014-11-19 17:27:07Z

5

Its worked for me in this way

elasticsearch {
    host => "localhost"
    index_type => "%{type}"
  }

answered Nov 19, 2014 at 17:27

Ysak

2,7757 gold badges35 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Helge Klein Over a year ago

index_type has been renamed to document_type in 1.5 (see github.com/elastic/logstash/issues/3151). Current documentation: elastic.co/guide/en/logstash/current/…

Ysak Over a year ago

Thanks..i will take care while we migrate to new es

Collectives™ on Stack Overflow

Dynamic elasticsearch index_type using logstash

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related