elasticsearch create or update document using python

Question

I am using elasticsearch-py for elasticsearch operation.

I am trying for elasticsearch.helpers.bulk to create or update multiple records.

from elasticsearch import Elasticsearch
from elasticsearch import helpers
es = Elasticsearch()

data = [
    {
        "_index": "customer",
        "_type": "external",
        "_op_type": "create",
        "_id": 3,
        "doc" : {"name": "test"}
    },
    {
        "_index": "customer",
        "_type": "external",
        "_op_type": "create",
        "_id": 4,
        "doc" : {"name": "test"}
    },
    {
        "_index": "customer",
        "_type": "external",
        "_op_type": "create",
        "_id": 5,
        "doc" : {"name": "test"}
    },
    {
        "_index": "customer",
        "_type": "external",
        "_op_type": "create",
        "_id": 6,
        "doc" : {"name": "test"}
    },
]


print helpers.bulk(es, data)

Is there any way to perform this operation?

Now we can give only _op_type as create or update. If we give update and record is not exist, then it will raise error.

Traceback (most recent call last):
  File "/tmp/test.py", line 37, in <module>
    print helpers.bulk(es, data)
  File "/local/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 182, in bulk
    for ok, item in streaming_bulk(client, actions, **kwargs):
  File "/local/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 155, in streaming_bulk
    raise BulkIndexError('%i document(s) failed to index.' % len(errors), errors)
elasticsearch.helpers.BulkIndexError: ('4 document(s) failed to index.', [{u'update': {u'status': 404, u'_type': u'external', u'_id': u'3', u'error': u'DocumentMissingException[[customer][-1] [external][3]: document missing]', u'_index': u'customer'}}, {u'update': {u'status': 404, u'_type': u'external', u'_id': u'4', u'error': u'DocumentMissingException[[customer][-1] [external][4]: document missing]', u'_index': u'customer'}}, {u'update': {u'status': 404, u'_type': u'external', u'_id': u'5', u'error': u'DocumentMissingException[[customer][-1] [external][5]: document missing]', u'_index': u'customer'}}, {u'update': {u'status': 404, u'_type': u'external', u'_id': u'6', u'error': u'DocumentMissingException[[customer][-1] [external][6]: document missing]', u'_index': u'customer'}}])

have you tried using index as op_type instead of create and update ? — Val
– Val, Commented Aug 21, 2015 at 6:16
@Val, as per helpers.bulk document, we have to give index, I also tried your solution, its give ValidationError, elasticsearch.exceptions.TransportError: TransportError(500, u'ActionRequestValidationException[Validation Failed: 1: no requests added;]') — NPatel
– NPatel, Commented Aug 21, 2015 at 6:22
You can check docs for this method elasticsearch-py.readthedocs.org/en/master/… — NPatel
– NPatel, Commented Aug 21, 2015 at 6:30
Also have you tried without specifying _op_type at all, I think it will default to index by itself. — Val
– Val, Commented Aug 21, 2015 at 6:32

Val · Accepted Answer · 2015-08-21 06:51:46Z

10

According to the _bulk endpoint documentation, you can and should use the index action for this, provided your documents always have the same identifiers.

create is useful when creating documents the first time, and update is more meant for doing partial and/or scripted updates.

You can also not specify any _op_type at all and index will be taken by default.

edited Aug 21, 2015 at 6:51

answered Aug 21, 2015 at 6:39

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Mayank · Accepted Answer · 2015-08-21 06:42:20Z

5

I tried solution suggested by @Val and it works as charm.

from elasticsearch import Elasticsearch
from elasticsearch import helpers
es = Elasticsearch()

data = [
    {
        "_index": "customer",
        "_type": "external",
        "_id": 3,
        "doc" : {"name": "test"}
    },
    {
        "_index": "customer",
        "_type": "external",
        "_id": 4,
        "doc" : {"name": "test"}
    },
    {
        "_index": "customer",
        "_type": "external",
        "_id": 5,
        "doc" : {"name": "test"}
    },
    {
        "_index": "customer",
        "_type": "external",
        "_id": 6,
        "doc" : {"name": "test"}
    },
]


print helpers.bulk(es, data)

answered Aug 21, 2015 at 6:42

Mayank

5,5891 gold badge19 silver badges16 bronze badges

Collectives™ on Stack Overflow

elasticsearch create or update document using python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related