Trying to load json docs in elasticsearch

Question

I am trying to follow this answer, but I am getting error in the actions parameter of bulk method. I am able to use next to generate array of json objects but when I pass it to helpers.bulk I get error.

This is my code:

from elasticsearch import Elasticsearch, helpers
import sys, json
import os

es = Elasticsearch("localhost:9200")
filename = "path_to_file.json"
def load_json(filename):
#     " Use a generator, no need to load all in memory"
print(filename)
with open(filename,'r') as open_file:
    yield json.load(open_file)`

helpers.bulk(es, load_json(filename), index='cllimaster_test', doc_type='clli_data_test')

Error:

Jozef - Spatialized.io · Accepted Answer · 2020-04-07 21:36:49Z

1

This python (not ES) error would occur if .pop() had been called on a list instead of a dict. I haven't seen your json file but here's your linted code which works perfectly fine:

from elasticsearch import Elasticsearch, helpers
import sys
import json
import os

es = Elasticsearch("localhost:9200")
filename = "path_to_file.json"


def load_json(filename):
    with open(filename, 'r') as open_file:
        yield json.load(open_file)


helpers.bulk(es, load_json(filename), index='cllimaster_test',
             doc_type='clli_data_test')

print(es.count(index='cllimaster_test'))

answered Apr 7, 2020 at 21:36

Jozef - Spatialized.io

17k4 gold badges29 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Murtaza Haji Over a year ago

Thanks, turns out my json was returning a list, fixed it by adding key at the beginning of array of json objects. I have new issue of read timeout now, i might post new question if not able to figure that out. Please be on the lookout :)

Murtaza Haji Over a year ago

My json file: {rows:[{obj1},{obj2},{obj3}]}. Kibana is considering rows as field and storing my objects as nested field. Thats why I am getting read timeout coz its considering the whole file as single object within rows.

Jozef - Spatialized.io Over a year ago

extract the rows like so yield json.load(open_file)['rows'].

Murtaza Haji Over a year ago

I had edited the file in Notepad++ , but thanks for the solution.

Collectives™ on Stack Overflow

Trying to load json docs in elasticsearch

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related