Logstash: send different json fields to different types in Elasticsearch

Question

I have the following format of JSON data that I send to a logstash instance listening on a http endpoint

{
    client: "c",
    pageInfo: ["a","b","c"],
    restInfo: ["r","s","t"]
}

My goal is to send this input to an elasticsearch endpoint as two different types in the same index; for example

 PUT elasticsearchhost:port/myindex/pageInfo  
       { client: "c", pageInfo: ["a","b","c"] }

 PUT elasticsearchhost:port/myindex/restInfo  
       { client: "c", restInfo: ["r","s","t"] }

I have tried with some filters in logstash (split, mutate, grok) but I cannot understand how to perform this very specific split or if I have to modify my configuration also in the output section

Alcanzar · Accepted Answer · 2017-02-09 16:11:28Z

1

You will need to use clone to clone the events and then modify the clones.

For example:

filter { 
  clone { clones => ["pageInfo", "restInfo" ]  }
  if [type]=="pageInfo" {
     mutate {
        remove_field => "restInfo"
     }
  }
  if [type] == "restInfo" {
     mutate {
        remove_field => "pageInfo"
     }
  }
}

And then on your elasticsearch output, be sure to include document_type => "%{type}"

answered Feb 9, 2017 at 16:11

Alcanzar

17.2k7 gold badges45 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

alexlipa Over a year ago

thank you, it works! In this way it creates three entities related to the same event; is there a way to have just one clone and refer to the original message in the 'if' statement? what would the original type value be for the original message?

Alcanzar Over a year ago

The type for the original event is whatever you set on input.

Alcanzar Over a year ago

You can also do else if and then a final else {drop{}}

Collectives™ on Stack Overflow

Logstash: send different json fields to different types in Elasticsearch

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related