Parsing JSON objects with arbitrary keys in Logstash

Question

Consider a subset of a sample output from http://demo.nginx.com/status:

{
  "timestamp": 1516053885198,
  "server_zones": {
    "hg.nginx.org": {
      ...  // Data for "hg.nginx.org"
    },
    "trac.nginx.org": {
      ...  // Data for "trac.nginx.org"
    }
  }
}

The keys "hg.nginx.org" and "track.nginx.org" are quite arbitrary, and I would like to parse them into something meaningful for Elasticsearch. In other words, each key under "server_zones" should be transformed into a separate event. Logstash should thus emit the following events:

[
    {
        "timestamp": 1516053885198,
        "server_zone": "hg.nginx.org",
        ...  // Data for "hg.nginx.org"
    },
    {
        "timestamp": 1516053885198,
        "server_zone": "trac.nginx.org",
        ...  // Data for "trac.nginx.org"
    }
]

What is the best way to go about doing this?

Polynomial Proton · Accepted Answer · 2018-01-15 22:52:49Z

You can try using the ruby filter. Get the server zones and create a new object using the key value pairs you want to include. From the top of my head, something like below should work. Obviously you then need to map the object to your field in the index. Change the snipped based on your custom format i.e. build the array or object as you want.

filter {    
  ruby {
       code => "  time = event.get('timestamp')
                  myArr = []
                  event.to_hash.select {|k,v| ['server_zones'].include?(k)}.each do |key,value| 
                     myCustomObject = {}                        
                     #map the key value pairs into myCustomObject 
                     myCustomObject[timestamp] = time 
                     myCustomObject[key] = value
                     myArr.push(myCustomObject) #you'd probably move this out based on nesting level
                  end 
                  map['my_indexed_field'] = myArr
                 "
    }
}

In the output section use rubydebug for error debugging

output {
  stdout { codec => rubydebug }
}

Collectives™ on Stack Overflow

Parsing JSON objects with arbitrary keys in Logstash

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related