Logstash, json array into single event with fields generated from element fields?

Question

I am using http_poller as input and get back a json array. I'm using default json codec. The json looks like this:

[{"name":"UsedMemory","value": {"value":"222932448","type":"java.lang.Long"}}, {"name":"FreeMemory","value":{"value":"308630048","type":"java.lang.Long"}}, {"name":"Heap","value":{"value":"531562496","type":"java.lang.Long"}},{"name":"UpTime","value":{"value":"29900897","type":"java.lang.Long"}},{"name":"ProcessCPU","value":{"value":"0.15846178794662266","type":"java.lang.Double"}}, {"name":"GcCount","value":{"value":"2198","type":"java.lang.Long"}},{"name":"GcTime","value":{"value":"35658","type":"java.lang.Long"}}]

I see that logstash treats each array element as a new event by splitting up the array. Eg

{
    "name" => "UpTime",
    "value" => {
    "value" => "29579549",
    "type" => "java.lang.Long"
},
    "@version" => "1",
    "@timestamp" => "2016-11-17T04:12:45.492Z"
}
{
    "name" => "ProcessCPU",
    "value" => {
    "value" => "0.16146030555740817",
    "type" => "java.lang.Double"
},
    "@version" => "1",
    "@timestamp" => "2016-11-17T04:12:45.492Z"
}
...

How can I get logstash (2.4) to combine the elements, treat it as one event and output it like this

{
    "Uptime" => "29579549"
    "ProcessCPU" => ""0.16146030555740817"
    ....

}

I think I have to generate a new field like this but go through all the elements of the array to make one complete event. { "%{name}" => "%{[value][value]}" ... }

CharekC · Accepted Answer · 2016-11-24 21:53:50Z

1

After reading a bit more about logstash and experimenting, I found the answer:

In the http_poller, change the codec to "plain"
In the filter section, use a json filter. eg

json {
  source => "message"
  target => "doc"
  }

Add a ruby code filter to process the json doc field as follows:

ruby {
code => "
event['doc'].each { |elem|
event[ elem['name']] = elem['value']['value']}
"
}

Remove doc and message fields using mutate filter

answered Nov 24, 2016 at 21:53

CharekC

637 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Logstash, json array into single event with fields generated from element fields?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related