Import/Index a JSON file into Elasticsearch

Question

I am new to Elasticsearch and have been entering data manually up until this point. For example I've done something like this:

$ curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elastic Search"
}'

I now have a .json file and I want to index this into Elasticsearch. I've tried something like this too, but no success:

curl -XPOST 'http://jfblouvmlxecs01:9200/test/test/1' -d lane.json

How do I import a .json file? Are there steps I need to take first to ensure the mapping is correct?

Possible duplicate of is there any way to import a json file(contains 100 documents) in elasticsearch server.? — shailendra pathak
– shailendra pathak, Commented Dec 3, 2015 at 17:57

Community · Accepted Answer · 2019-07-11 06:51:06Z

105

The right command if you want to use a file with curl is this:

curl -XPOST 'http://jfblouvmlxecs01:9200/test/_doc/1' -d @lane.json

Elasticsearch is schemaless, therefore you don't necessarily need a mapping. If you send the json as it is and you use the default mapping, every field will be indexed and analyzed using the standard analyzer.

If you want to interact with Elasticsearch through the command line, you may want to have a look at the elasticshell which should be a little bit handier than curl.

2019-07-10: Should be noted that custom mapping types is deprecated and should not be used. I updated the type in the url above to make it easier to see which was the index and which was the type as having both named "test" was confusing.

edited Jul 11, 2019 at 6:51

CommunityBot

11 silver badge

answered Apr 10, 2013 at 21:37

javanna

60.4k14 gold badges147 silver badges125 bronze badges

Sign up to request clarification or add additional context in comments.

14 Comments

Konrad Over a year ago

I does't work for me, when I type Your command the console does't provide any data.

Ehtesh Choudhury Over a year ago

@Konrad you replaced jfblouvmlxecs01 with localhost, right?

Oliver Over a year ago

clwen - the "@" tells curl to load the data from the json file.

sarah w Over a year ago

hi i am also new in elastic search can anyone please gudie me where to store these .json files?

AV94 Over a year ago

Where to store json file?

|

Conrado · Accepted Answer · 2019-10-16 04:56:42Z

31

Per the current docs, https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html:

If you’re providing text file input to curl, you must use the --data-binary flag instead of plain -d. The latter doesn’t preserve newlines.

Example:

$ curl -s -XPOST localhost:9200/_bulk --data-binary @requests

edited Oct 16, 2019 at 4:56

Conrado

1,41117 silver badges23 bronze badges

answered Nov 13, 2014 at 0:36

KenH

5305 silver badges5 bronze badges

1 Comment

Steve Tarver Over a year ago

Note that the _bulk load json file is not valid a valid json file; the syntax is provided in the _bulk API link. Also, you do not have to provide an _id as indicated in these examples; an auto-generated _id will be provided when _id is omitted.

Evan · Accepted Answer · 2014-11-18 03:32:50Z

20

We made a little tool for this type of thing https://github.com/taskrabbit/elasticsearch-dump

answered Nov 18, 2014 at 3:32

Evan

3,3264 gold badges32 silver badges25 bronze badges

4 Comments

jgr0 Over a year ago

The given examples do not cover the question asked here. Will it work if we give the json file as an input and the elastic search url as the output?

Krishna Chaitanya Gopaluni Over a year ago

I am using this to export the index into json. Thanks.

Krishna Chaitanya Gopaluni Over a year ago

Use the following command. elasticdump --input=/path/to/file.json --output=http://'username:password'@localhost:9200/indexname --type=data. Remove 'username:password@' if you don't need.

Noumenon Over a year ago

This tool raises not_x_content_exception for plain line-delimited JSON and Bulk API JSON. It expects its own dump format, which looks like {"_index":"my_index","_type":"_doc","_id":"abc","_score":1,"_source":{"my":"json"}}.

Greg Dougherty · Accepted Answer · 2017-05-01 14:40:08Z

20

One thing I've not seen anyone mention: the JSON file must have one line specifying the index the next line belongs to, for every line of the "pure" JSON file.

I.E.

{"index":{"_index":"shakespeare","_type":"act","_id":0}}
{"line_id":1,"play_name":"Henry IV","speech_number":"","line_number":"","speaker":"","text_entry":"ACT I"}

Without that, nothing works, and it won't tell you why

answered May 1, 2017 at 14:40

Greg Dougherty

3,5218 gold badges39 silver badges62 bronze badges

Comments

MosheZada · Accepted Answer · 2018-03-16 20:29:06Z

14

I'm the author of elasticsearch_loader
I wrote ESL for this exact problem.

You can download it with pip:

pip install elasticsearch-loader

And then you will be able to load json files into elasticsearch by issuing:

elasticsearch_loader --index incidents --type incident json file1.json file2.json

answered Mar 16, 2018 at 20:29

MosheZada

2,4291 gold badge17 silver badges17 bronze badges

6 Comments

dr0i Over a year ago

This is nice! It adds the mandatory index line before every document.

Chiel Over a year ago

2018-10-04 11:51:40.395741 ERROR attempt [1/1] got exception, it is a permanent data loss, no retry any more 2018-10-04 11:51:40.395741 WARN Chunk 0 got exception (ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='localhost', port=9200): Read timed out. (read timeout=10.0))) while processing

Chiel Over a year ago

Apart from the fact that it doesn't work, where do you specify the URL and port?

MosheZada Over a year ago

You can visit the GitHub page or run elasticsearch_loader --help in order to view the full help message. You can specify the host:port with --es-host http://hostname:port

Vlad T. Over a year ago

Nice. Except that --type becomes redundant as Elasticsearch removes types in 6 version elastic.co/guide/en/elasticsearch/reference/6.0/…

|

Gajendra D Ambi · Accepted Answer · 2018-05-05 11:42:32Z

10

I just made sure that I am in the same directory as the json file and then simply ran this

curl -s -H "Content-Type: application/json" -XPOST localhost:9200/product/default/_bulk?pretty --data-binary @product.json

So if you too make sure you are at the same directory and run it this way. Note: product/default/ in the command is something specific to my environment. you can omit it or replace it with whatever is relevant to you.

answered May 5, 2018 at 11:42

Gajendra D Ambi

4,27933 silver badges35 bronze badges

Comments

filhit · Accepted Answer · 2016-05-18 17:50:04Z

9

Adding to KenH's answer

$ curl -s -XPOST localhost:9200/_bulk --data-binary @requests

You can replace @requests with @complete_path_to_json_file

Note: @is important before the file path

edited May 18, 2016 at 17:50

filhit

2,1941 gold badge22 silver badges34 bronze badges

answered May 18, 2016 at 15:51

Ram Pratap

1,11911 silver badges8 bronze badges

3 Comments

Piyush Mittal Over a year ago

can u give some example for path. i am giving "@c:\accounts.json" and placing it there even then, its not able to locate it

Ram Pratap Over a year ago

it should be @"c:\accounts.json"

Shady Kip Over a year ago

add a header flag like so -H "Content-Type: application/json"

MLS · Accepted Answer · 2017-06-14 05:33:27Z

You are using

$ curl -s -XPOST localhost:9200/_bulk --data-binary @requests

If 'requests' is a json file then you have to change this to

$ curl -s -XPOST localhost:9200/_bulk --data-binary @requests.json

Now before this, if your json file is not indexed, you have to insert an index line before each line inside the json file. You can do this with JQ. Refer below link: http://kevinmarsh.com/2014/10/23/using-jq-to-import-json-into-elasticsearch.html

Go to elasticsearch tutorials (example the shakespeare tutorial) and download the json file sample used and have a look at it. In front of each json object (each individual line) there is an index line. This is what you are looking for after using the jq command. This format is mandatory to use the bulk API, plain json files wont work.

Piyush Mittal · Accepted Answer · 2016-10-12 04:35:40Z

6

just get postman from https://www.getpostman.com/docs/environments give it the file location with /test/test/1/_bulk?pretty command.

answered Oct 12, 2016 at 4:35

Piyush Mittal

1,8881 gold badge23 silver badges40 bronze badges

2 Comments

Chiel Over a year ago

{ "error": "no handler found for uri [/test/test/1/_bulk?pretty] and method [POST]" }

X. L Over a year ago

{ "error": "Content-Type header [text/plain] is not supported", "status": 406 }

thSoft · Accepted Answer · 2020-06-08 09:19:37Z

3

As of Elasticsearch 7.7, you have to specify the content type also:

curl -s -H "Content-Type: application/json" -XPOST localhost:9200/_bulk --data-binary @<absolute path to JSON file>

answered Jun 8, 2020 at 9:19

thSoft

22.8k6 gold badges96 silver badges105 bronze badges

Comments

Eric Leschinski · Accepted Answer · 2019-01-10 21:01:53Z

2

I wrote some code to expose the Elasticsearch API via a Filesystem API.

It is good idea for clear export/import of data for example.

I created prototype elasticdriver. It is based on FUSE

edited Jan 10, 2019 at 21:01

Eric Leschinski

155k96 gold badges423 silver badges337 bronze badges

answered Dec 14, 2018 at 9:11

Yaroslav Gaponov

2,10715 silver badges12 bronze badges

Comments

waseem khan · Accepted Answer · 2020-10-01 13:20:57Z

If you are using the elastic search 7.7 or above version then follow below command.

curl -H "Content-Type: application/json" -XPOST "localhost:9200/bank/_bulk? pretty&refresh" --data-binary @"/Users/waseem.khan/waseem/elastic/account.json"
On above file path is /Users/waseem.khan/waseem/elastic/account.json.
If you are using elastic search 6.x version then you can use the below command.

curl -X POST localhost:9200/bank/_bulk?pretty&refresh --data-binary @"/Users/waseem.khan/waseem/elastic/account.json" -H 'Content-Type: application/json'

Note: Make sure in your .json file at the end you will add the one empty line otherwise you will be getting below exception.

"error" : {
"root_cause" : [
  {
    "type" : "illegal_argument_exception",
    "reason" : "The bulk request must be terminated by a newline [\n]"
  }
],
"type" : "illegal_argument_exception",
"reason" : "The bulk request must be terminated by a newline [\n]"
},
`enter code here`"status" : 400

sudarshan · Accepted Answer · 2014-02-24 10:35:14Z

0

if you are using VirtualBox and UBUNTU in it or you are simply using UBUNTU then it can be useful

wget https://github.com/andrewvc/ee-datasets/archive/master.zip
sudo apt-get install unzip (only if unzip module is not installed)
unzip master.zip
cd ee-datasets
java -jar elastic-loader.jar http://localhost:9200 datasets/movie_db.eloader

answered Feb 24, 2014 at 10:35

sudarshan

92 bronze badges

Comments

Mahan · Accepted Answer · 2020-12-09 08:56:52Z

-1

If you want to import a json file into Elasticsearch and create an index, use this Python script.

import json
from elasticsearch import Elasticsearch

es = Elasticsearch([{'host': 'localhost', 'port': 9200}])
i = 0
with open('el_dharan.json') as raw_data:
    json_docs = json.load(raw_data)
    for json_doc in json_docs:
            i = i + 1
            es.index(index='ind_dharan', doc_type='doc_dharan', id=i, body=json.dumps(json_doc))

answered Dec 9, 2020 at 8:56

Mahan

4411 gold badge6 silver badges11 bronze badges

1 Comment

fraank Over a year ago

highly not recommended with large number of documents. as it does one single insert request per document, this one is incredibly unperformant.

Collectives™ on Stack Overflow

Import/Index a JSON file into Elasticsearch

14 Answers 14

14 Comments

1 Comment

4 Comments

Comments

6 Comments

Comments

3 Comments

Comments

2 Comments

Comments

Comments

Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

14 Answers 14

14 Comments

1 Comment

4 Comments

Comments

6 Comments

Comments

3 Comments

Comments

2 Comments

Comments

Comments

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related