How to use Elasticsearch bulk index with single JSON file in Python

Question

I have a single JSON file formatted like:

    {"index":{"_type": "summary", "_id": "0001-1200-2015-12-31-0007", "_index": "transaction"}}
    {"data_load_user_id": "nathan", "tender_time": 5, "ring_time": 8, "operator_id": "0111"}

for about 12,000 rows. As far as I know, this is the correct way to format a JSON file for the bulk API in elasticsearch. I'm still confused on how to use es.bulk() with this single file as input. I already have the index and id specified for each row.

If you're using the official module check out elasticsearch.helpers.bulk(). I'm pretty sure you can pass a list containing your rows and it will do the right thing. Edit: here's the documentation: elasticsearch-py.readthedocs.org/en/master/… — Cfreak
– Cfreak, Commented Jun 2, 2015 at 13:43
@Cfreak would it have to be a list of JSON objects or a list of strings? I tried opening the file and creating an iterator from it, but when I passed the iterator in, it gave the error AttributeError: 'str' object has no attribute 'copy' — Nathan Miranda
– Nathan Miranda, Commented Jun 2, 2015 at 13:47
@Cfreak the documentation says "it consumes an iterator of actions and sends them to elasticsearch in chunks"; will the list count as an iterator? — Nathan Miranda
– Nathan Miranda, Commented Jun 2, 2015 at 13:55
True. It needs to be a dict (which gets converted back to JSON). — Cfreak
– Cfreak, Commented Jun 2, 2015 at 13:55
Yes. A list or an iterator will work. I'm digging through some of my code to see if I've ever done it straight from JSON instead of decoding and encoding back. — Cfreak
– Cfreak, Commented Jun 2, 2015 at 13:58

Nathan Miranda · Accepted Answer · 2015-06-02 14:53:36Z

2

Ah okay, so the helpers.bulk() module takes the list of dicts and my elasticsearch client as parameters and instead of having the 2 row per entry JSON file, I just needed to add the "_index" "type" and the "_id" to each JSON dict and it worked!

answered Jun 2, 2015 at 14:53

Nathan Miranda

791 silver badge14 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to use Elasticsearch bulk index with single JSON file in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related