Parsing nested JSON string in Logstash

Question

I am logging to logstash,in json format, my logs have the following fields, each field is a string and the atts field is a stringified json (note: atts sub fields are different each time)

here is an example:

{"name":"bob","last":"builder", "atts":"{\"a\":111, \"b\":222}"}

I would like to parse it to something like this:

   {
     "name" => "bob",
     "last" => "builder"
     "atss" => {
          "a" => 111,
          "b" => 222}
   }

here is my configuration:

input { stdin { } }  

filter {
  json {
    source => "message"
    target => "parsed"
  }
}
 output { stdout { codec => rubydebug  }}

ok, so now I get this:

{
    "@timestamp" => 2017-04-05T12:19:04.090Z,
    "parsed" => {
        "atss" => "{\"a\":111, \"b\":222}",
        "name" => "bob",
        "last" => "the builder"
    },
        "@version" => "1",
          "host" => "0.0.0.0"
}

how can I parse the atts field to json so I receive:

{
    "@timestamp" => 2017-04-05T12:19:04.090Z,
    "parsed" => {
        "atss" => 
           {"a" => 111,
            "b" => 222},
        "name" => "bob",
        "last" => "the builder"
    },
        "@version" => "1",
          "host" => "0.0.0.0"
}

dina · Accepted Answer · 2017-04-05 14:41:12Z

26

thanks to @Alcanzar here is what I did

input { 
  stdin { } 
}  

filter {
  json {
    source => "message"
    target => "message"
  }
  json {
    source => "[message][atts]"
    target => "[message][atts]"
  }

}
 output { stdout { codec => rubydebug  }}

answered Apr 5, 2017 at 14:41

dina

4,3798 gold badges50 silver badges76 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Alcanzar · Accepted Answer · 2017-04-05 14:25:53Z

6

There is a json filter. Just pass it the field you want to parse and a target where you want it.

Something like:

json {
  source => "[parsed][atss]"
  target => "[parsed][newfield]"
}

I'm not sure if you can put atss as new field. It might or might not work. If it doesn't, use the mutate filter to remove_field and rename_field.

answered Apr 5, 2017 at 14:25

Alcanzar

17.2k7 gold badges45 silver badges60 bronze badges

1 Comment

dina Over a year ago

thanks! can you see my posted answer, is there a way to do this writing the filter all under json filter once? something like

filter {   json {     source => "message"     target => "message"     source => "[message][atts]"     target => "[message][atts]"   } }

ranvit · Accepted Answer · 2020-08-05 16:32:39Z

1

As of August 5 2020, In logstash 7.8.0, the JSON filter now parses nested strings.
I had the following string stored in a Postgres column, lets call it "column-1"

{ "XYZ" : {"A" : "B", "C" : "D", "E" : "F"} }

My json filter looked like this

json {
    source => "column-1"
    target => "column-1"
}

And it was parsed as a nested dictionary in my Elasticsearch Index

The difference from the accepted answer is that now we just need a single json filter instead of multiple ones.

answered Aug 5, 2020 at 16:32

ranvit

1296 bronze badges

Collectives™ on Stack Overflow

Parsing nested JSON string in Logstash

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related