python elasticsearch timeout error when indexing large json

Question

i'm getting timeout exception while i indexing huge json text.

sample code:

es.index(index="test", doc_type="test", body=jsonString)

So i tried to increase the timeout using

es.index(index="test", doc_type="test", body=jsonString, timeout=60)

But is this the only way to fix the problem? sometimes my JSON String will be 40 MB to 60 MB in size.

Update:

I tried below way to index using bulk, it failed.

helpers.bulk(es, jsonOutput, index="test-las", doc_type="test-las")

Log:

Traceback (most recent call last):
  File "LasioParser.py", line 46, in <module>
    helpers.bulk(es, jsonOutput, index="test-las", doc_type="test-las")
  File "/usr/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 257, in bulk
    for ok, item in streaming_bulk(client, actions, **kwargs):
  File "/usr/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 192, in streaming_bulk
    raise_on_error, **kwargs)
  File "/usr/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 137, in _process_bulk_chunk
    raise BulkIndexError('%i document(s) failed to index.' % len(errors), errors)
elasticsearch.helpers.BulkIndexError: (u'500 document(s) failed to index.', [{

try to use the bulk api elasticsearch-py.readthedocs.io/en/master/helpers.html . If you provide more sample of your code, i could help you — Lupanoide
– Lupanoide, Commented Apr 27, 2018 at 9:47
@Lupanoide: please find updated question, is this correct way of doing it, i got some error.. i will try your solution now.. — Shankar
– Shankar, Commented Apr 27, 2018 at 10:42
is your jsonString a dict or you have just converted it with json.dumps()? It must be a dict — Lupanoide
– Lupanoide, Commented Apr 27, 2018 at 10:43

Jon Vaughan · Accepted Answer · 2018-09-25 08:03:09Z

1

Are you using http compression?

from elasticsearch import Elasticsearch es = Elasticsearch(hosts, http_compress = True)

https://elasticsearch-py.readthedocs.io/en/master/#compression

answered Sep 25, 2018 at 8:03

Jon Vaughan

2,0581 gold badge18 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

python elasticsearch timeout error when indexing large json

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related