Index JSON files in elasticsearch using Python?

Question

I have a bunch of JSON files(100), which are named as merged_file 1.json, merged_file 2. json and so on.

How do I index all these files into elasticsearch using python(elasticsearch_dsl) ?

I am using this code, but it doesn't seem to work:

from elasticsearch_dsl import Elasticsearch
import json
import os
import sys

es = Elasticsearch()

json_docs =[]

directory = sys.argv[1]

for filename in os.listdir(directory):
    if filename.endswith('.json'):
        with open(filename,'r') as open_file:
            json_docs.append(json.load(open_file))

es.bulk("index_name", "type_name", json_docs)

The JSON looks like this:

{"one":["some data"],"two":["some other data"],"three":["other data"]}

What can I do to make this correct ?

You're missing the command line before each document. See here for more details. — Val
– Val, Commented May 15, 2017 at 13:53

Honza Král · Accepted Answer · 2017-05-15 15:05:47Z

12

For this task you should be using elasticsearch-py (pip install elasticsearch):

from elasticsearch import Elasticsearch, helpers
import sys, json

es = Elasticsearch()

def load_json(directory):
    " Use a generator, no need to load all in memory"
    for filename in os.listdir(directory):
        if filename.endswith('.json'):
            with open(filename,'r') as open_file:
                yield json.load(open_file)

helpers.bulk(es, load_json(sys.argv[1]), index='my-index', doc_type='my-type')

answered May 15, 2017 at 15:05

Honza Král

3,03216 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

anshaj Over a year ago

How do I get the id's of the jsons that are indexed ?

Honza Král Over a year ago

If you care about the ids (elasticsearch will create random ones for you otherwise) just have an _id field in you json either directly or maybe put the filename there or something

Murtaza Haji Over a year ago

This throws error in the action parameter of bulk. """ ~\Anaconda3\lib\site-packages\elasticsearch\helpers\actions.py in expand_action(data) 25 # make sure we don't alter the action 26 data = data.copy() ---> 27 op_type = data.pop("_op_type", "index") 28 action = {op_type: {}} 29 for key in ( TypeError: pop() takes at most 1 argument (2 given)"""

Collectives™ on Stack Overflow

Index JSON files in elasticsearch using Python?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related