Logstash - import nested JSON into Elasticsearch

Question

I have a large amount (~40000) of nested JSON objects I want to insert into elasticsearch an index.

The JSON objects are structured like this:

    {
    "customerid": "10932"
    "date": "16.08.2006",
    "bez": "xyz",
    "birthdate": "21.05.1990",
    "clientid": "2",
    "address": [
        {
            "addressid": "1",
            "tile": "Mr",
            "street": "main str",
            "valid_to": "21.05.1990",
            "valid_from": "21.05.1990",
        },
        {
            "addressid": "2",
            "title": "Mr",
            "street": "melrose place",
            "valid_to": "21.05.1990",
            "valid_from": "21.05.1990",
        }
      ]
    }

So a JSON field (address in this example) can have an array of JSON objects.

What would a logstash config look like to import JSON files/objects like this into elasticsearch? The elasticsearch mapping for this index should just look like the structure of the JSON. The elasticsearch document id should be set to customerid.

input {
  stdin {
    id => "JSON_TEST"
  } 
}
filter {
    json{
        source => "customerid"
        ....
        ....    
    }

}
output {
       stdout{}
       elasticsearch {
          hosts => "https://localhost:9200/"
          index => "customers"           
          document_id => "%{customerid}"
       }                                               
}

Talisman · Accepted Answer · 2020-01-22 18:10:23Z

2

If you have control of what's being generated, the easiest thing to do is to format you input as single line json and then use the json_lines codec.

Just change your stdin to:

stdin { codec => "json_lines" }

and then it'll just work:

cat input_file.json | logstash -f json_input.conf

where input_file.json has lines like:

{"customerid":1,"nested": {"json":"here"}}
{"customerid":2,"nested": {"json":"there"}}

and then you won't need the json filter.

edited Jan 22, 2020 at 18:10

Talisman

3884 silver badges19 bronze badges

answered Mar 29, 2018 at 14:58

Alcanzar

17.2k7 gold badges45 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Logstash - import nested JSON into Elasticsearch

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related