Update records in ElasticSearch

Question

I had like to update the logdate column for ALL records in a specific index. From what I have read so far, it seems that it is not possible? I am correct?

Here's a sample of a document:

{
            "_index": "logstash-01-2015",
            "_type": "ufdb",
            "_id": "AU__EvrALg15uxY1Wxf9",
            "_score": 1,
            "_source": {
               "message": "2015-08-14 06:50:05 [31946] PASS  level2      10.249.10.70    level2     ads       http://ad.360yield.com/unpixel.... GET",
               "@version": "1",
               "@timestamp": "2015-09-24T11:17:57.389Z",
               "type": "ufdb",
               "file": "/usr/local/ufdbguard/logs/ufdbguardd.log",
               "host": "PROXY-DEV",
               "offset": "3983281700",
               "logdate": "2015-08-14T04:50:05.000Z",
               "status": "PASS",
               "group": "level2",
               "clientip": "10.249.10.70",
               "category": "ads",
               "url": "http://ad.360yield.com/unpixel....",
               "method": "GET",
               "tags": [
                  "_grokparsefailure"
               ]
            }
         }

What do you mean by "update"? Change the value to some other string? Convert it to a date object instead of a string? Copy the value to @timestamp? or ? — Alain Collins
– Alain Collins, Commented Sep 25, 2015 at 17:27
I had like to change the logdate field to another date for ALL documents. In MySQL, I would have done something like this: UPDATE logstash SET logdate= "2015-09-20T04:50:05.000Z" but it ElasticSearch it seems that I can only update one document by supplying its _id. — Hyder B.
– Hyder B., Commented Sep 27, 2015 at 6:43

Mario Trucco · Accepted Answer · 2018-05-02 15:20:13Z

2

You are correct, that is not possible.

There's been an open issue asking Update by Query for long time, and I'm not sure it's going to be implemented anytime soon since it is very problematic for the underlying lucene engine. It requires deleting all documents and reindexing them.

An Update by Query Plugin is available on github, but it's experimental and I never tried it.

UPDATE 2018-05-02

The original answer is quite old. Update By Query is now supported.

edited May 2, 2018 at 15:20

answered Sep 27, 2015 at 8:33

Mario Trucco

2,0213 gold badges34 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Sloan Ahrens · Accepted Answer · 2015-09-25 19:26:16Z

You can use the partial update API.

To test it, I created a trivial index:

PUT /test_index

Then created a document:

PUT /test_index/doc/1
{
   "message": "2015-08-14 06:50:05 [31946] PASS  level2      10.249.10.70    level2     ads       http://ad.360yield.com/unpixel.... GET",
   "@version": "1",
   "@timestamp": "2015-09-24T11:17:57.389Z",
   "type": "ufdb",
   "file": "/usr/local/ufdbguard/logs/ufdbguardd.log",
   "host": "PROXY-DEV",
   "offset": "3983281700",
   "logdate": "2015-08-14T04:50:05.000Z",
   "status": "PASS",
   "group": "level2",
   "clientip": "10.249.10.70",
   "category": "ads",
   "url": "http://ad.360yield.com/unpixel....",
   "method": "GET",
   "tags": [
      "_grokparsefailure"
   ]
}

Now I can do a partial update on the document with:

POST /test_index/doc/1/_update
{
    "doc": {
        "logdate": "2015-09-25T12:20:00.000Z"
    }
}

If I retrieve the document:

GET /test_index/doc/1

I will see that the logdate property has been updated:

{
   "_index": "test_index",
   "_type": "doc",
   "_id": "1",
   "_version": 2,
   "found": true,
   "_source": {
      "message": "2015-08-14 06:50:05 [31946] PASS  level2      10.249.10.70    level2     ads       http://ad.360yield.com/unpixel.... GET",
      "@version": "1",
      "@timestamp": "2015-09-24T11:17:57.389Z",
      "type": "ufdb",
      "file": "/usr/local/ufdbguard/logs/ufdbguardd.log",
      "host": "PROXY-DEV",
      "offset": "3983281700",
      "logdate": "2015-09-25T12:20:00.000Z",
      "status": "PASS",
      "group": "level2",
      "clientip": "10.249.10.70",
      "category": "ads",
      "url": "http://ad.360yield.com/unpixel....",
      "method": "GET",
      "tags": [
         "_grokparsefailure"
      ]
   }
}

Here is the code I used to test it:

http://sense.qbox.io/gist/236bf271df6d867f5f0c87eacab592e41d3095cf

Thanks for your answer. I think I have not explained correctly. I have checked the update API but from what I can see is that you have to supply the _id field in order to run the update. In my case, I have like to run an update on all documents to make the 'logdate' the same.

Collectives™ on Stack Overflow

Update records in ElasticSearch

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related