Elasticsearch: update existing document by inserting elements to its array fields

Question

Consider the following document

{
  "title":  "My first blog entry",
  "text":   "Starting to get the hang of this...",
  "tags": [ "testing" ], 
  "views":  0 
}

I need to run kind of an upsert operation. If I encounter data like

{
    "id": 1,
    "tags": [ "new tag" ]
}

I want to update the existing document with same id. So result should be :

{
    "id": 1,
    "title":  "My first blog entry",
    "text":   "Starting to get the hang of this...",
    "tags": [ "testing", "new tag" ], 
    "views":  0 
}

If the document with same id does not exist, I want to create a new one.

Now in databases like mongoDB, I could use update with $addToSet or $push operation. I could not find similar operation in Elasticsearch.

I read that it can be done by writing scripts in groovy. However, this needs to be done on a file containing 200 million records. I am not sure if I can use groovy in combination with bulk API. Is it possible ?

Have a look at the HttpBuilder project (github.com/jgritman/httpbuilder) — divesh premdeep
– divesh premdeep, Commented Sep 25, 2015 at 1:33
Can you show an excerpt of your file containing the records? — Val
– Val, Commented Sep 25, 2015 at 3:24

Vineeth Mohan · Accepted Answer · 2015-11-01 16:38:07Z

8

You dont need to use bulk API for this. You can use an upsert request. Upsert request can ALSO be embedded in the bulk request.

curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
  "script": "if (ctx._source.tags.contains(\"tags\")) {ctx._source.tags += tag;} else {ctx._source.tags = [tag]}",
  "params": {
    "tag": "newTag"
  },
  "upsert": {
    "title": "My first blog entry",
    "text": "Starting to get the hang of this...",
    "tags": [
      "newTag"
    ],
    "views": 0
  }
}'

edited Nov 1, 2015 at 16:38

answered Sep 25, 2015 at 3:44

Vineeth Mohan

19.4k9 gold badges70 silver badges81 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Mandeep Singh Over a year ago

Thanks Vineet. This worked well with bulk API. However, there was a slight change in my requirement. The upsert document should contain the same tag in its tag array as the tag in params. Also, if condition should be modified to include the condition of if (!ctx._source.tags.contains(tag)). Please modify your answer so that I can accept it and close it.

Vineeth Mohan Over a year ago

I have made the change

Pavan Kumar Varma Over a year ago

in the same scenario if i want to update array of documents. how can i pass params. please help me out..thank you

Vineeth Mohan Over a year ago

@PavanKumarVarma - can you post that as a separate question and ping me

manojchowdary27 Over a year ago

@VineethMohan Is there any example on how to use upsert with bulk api?

Collectives™ on Stack Overflow

Elasticsearch: update existing document by inserting elements to its array fields

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related