Logstash Update a document in elasticsearch

Question

Trying to update a specific field in elasticsearch through logstash. Is it possible to update only a set of fields through logstash ?

Please find the code below,

    input {
  file {
        path => "/**/**/logstash/bin/*.log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
        type => "multi"
  }
}

filter {
        csv {
                separator => "|"
                columns => ["GEOREFID","COUNTRYNAME", "G_COUNTRY", "G_UPDATE", "G_DELETE",  "D_COUNTRY", "D_UPDATE", "D_DELETE"]

}
elasticsearch {
         hosts => ["localhost:9200"]
         index => "logstash-data-monitor"
         query => "GEOREFID:%{GEOREFID}"
         fields => [["JSON_COUNTRY","G_COUNTRY"],
                    ["XML_COUNTRY","D_COUNTRY"]]
           }

if [G_COUNTRY] {
mutate {
    update => { "D_COUNTRY" => "%{D_COUNTRY}"
 }
  }
}
}
output {
elasticsearch {
        hosts => ["localhost:9200"]
        index => "logstash-data-monitor"
        document_id => "%{GEOREFID}"
  }
}

We are using the above configuration when we use this the null value field is getting removed instead of skipping null value update.

Data comes from 2 different source. One is from XML file and the other is from JSON file.

XML log format : GEO-1|CD|23|John|892|Canada|31-01-2017|QC|-|-|-|-|- JSON log format : GEO-1|AS|33|-|-|-|-|-|Mike|123|US|31-01-2017|QC

When adding one log new document will get created in the index. When reading the second log file the existing document should get updated. The update should happen only in the first 5 fields if log file is XML and last 5 fields if the log file is JSON. Please suggest us on how to do this in logstash.

Tried with the above code. Please check and can any one help on how to fix this ?

You only provide a filter definition, the update/insert/upsert operation however is done via the output to elasticsearch. You can read about it here: elastic.co/guide/en/logstash/2.4/… — pandaadb
– pandaadb, Commented Feb 6, 2017 at 14:46
@pandaadb edited the question. Can you please look at the question and help on this ? — banu
– banu, Commented Feb 6, 2017 at 15:05

sysadmin1138 · Accepted Answer · 2017-02-14 14:16:35Z

8

For the Elasticsearch output to do any action other than index you need to tell it to do something else.

elasticsearch {
        hosts => ["localhost:9200"]
        index => "logstash-data-monitor"
        action => "update"
        document_id => "%{GEOREFID}"
}

This should probably be wrapped in a conditional to ensure you're only updating records that need updating. There is another option, though, doc_as_upsert

elasticsearch {
        hosts => ["localhost:9200"]
        index => "logstash-data-monitor"
        action => "update"
        doc_as_upsert => true
        document_id => "%{GEOREFID}"
}

This tells the plugin to insert if it is new, and update if it is not.

However, you're attempting to use two inputs to define a document. This makes things complicated. Also, you're not providing both inputs, so I'll improvise. To provide different output behavior, you will need to define two outputs.

input {
  file {
    path => "/var/log/xmlhome.log"
    [other details]
  }

  file {
    path => "/var/log/jsonhome.log"
    [other details]
  }
}

filter { [some stuff ] }

output {
  if [path] == '/var/log/xmlhome.log' {
    elasticsearch {
      [XML file case]
    }
  } else if [path] == '/var/log/jsonhome.log' {
    elasticsearch {
      [JSON file case]
      action => "update"
    }
  }
}

Setting it up like this will allow you to change the ElasticSearch behavior based on where the event originated.

edited Feb 14, 2017 at 14:16

answered Feb 6, 2017 at 15:13

sysadmin1138

1,30311 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

banu Over a year ago

i have tired the above option document is not getting updated the null value field is getting removed instead of skipping null value update. elasticsearch { hosts => ["localhost:9200"] index => "logstash-data-monitor" action => "update" doc_as_upsert => true document_id => "%{GEOREFID}" }

banu Over a year ago

Any suggestion for the above comments @sysadmin1138

sysadmin1138 Over a year ago

@banu If the logic you're looking for is, IF NOT null-value THEN update, then I suggest you wrap that update output in an if conditional that tests for that null value.

banu Over a year ago

could you please provide me the sample condition on this ?

sysadmin1138 Over a year ago

@banu I've updated with an example of a conditional.

|

Collectives™ on Stack Overflow

Logstash Update a document in elasticsearch

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related