1

Elasticsearch and command line programming noobie question.

I have elasticsearch set up locally on my computer and want to pull documents from a server that uses a different version of es using the scan and scroll api and add them into my index. I am having trouble figuring out how to do this with the bulk api for es.

Right now in my testing phase I am just pulling a few documents from the server using the following code (which works):

   http MY-OLD-ES.com:9200/INDEX/TYPE/_search?size=1000 | jq   .hits.hits[] -c | while read x; do id="`echo "$x" | jq -r ._id`"; index="`echo "$x" | jq -r ._index`"; type="`echo "$x" | jq -r ._type`"; doc="`echo "$x" | jq ._source`"; http put "localhost:9200/junk-$index/$type/$id" <<<"$doc"; done

Any tips on how scan and scroll works (noob and a bit confused). So far know I can scroll and get a scroll id, but I'm unclear what to do with the scroll id. If I call 

http get http://MY-OLD-ES.com:9200/my_index/_search?scroll=1m&search_type=scan&size=10

I'll receive a scroll id. Can this be piped in and parsed the same way? Additionally, I believe I'll need a while loop to tell it to keep requesting. How exactly should I go about this?

Thanks!

Improve this question

1 Answer 1

Reset to default
3

The scan and scroll documentation explains it pretty clearly. After you get the scroll_id (a long base64 encoded string), you pass it in with the body of the request. With curl the request would looks something like this:

curl -XGET 'http://MY-OLD-ES.com:9200/_search/scroll?scroll=1m' -d '
c2Nhbjs1OzExODpRNV9aY1VyUVM4U0NMd2pjWlJ3YWlBOzExOTpRNV9aY1VyUVM4U0 
NMd2pjWlJ3YWlBOzExNjpRNV9aY1VyUVM4U0NMd2pjWlJ3YWlBOzExNzpRNV9aY1Vy
UVM4U0NMd2pjWlJ3YWlBOzEyMDpRNV9aY1VyUVM4U0NMd2pjWlJ3YWlBOzE7dG90YW
xfaGl0czoxOw==
'

Notice that while the first request to open the scroll was to /my_index/_search, the second request to read the data was to /_search/scroll. Each time you call that, passing the ?scroll=1m querystring, it refreshes the timeout before the scroll is automatically closed.

There are two more things to be aware of:

  1. The size you pass when opening the scroll applies to each shard, so you will get size multiplied by the number of shards in your index on each request.
  2. Each request to /_search/scroll will return a new scroll_id which you must pass on the next call to get the next batch of results. You can't just keep calling with the same scroll_id.

It is complete when no hits are returned in the scroll request.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Thanks! So something like this should work? scroll_id= http get MY-OLD-ES.com:9200/my_index/…; while $scroll_id; do $scroll_id = http 'MY-OLD-ES.com:9200/_search/scroll?scroll=1m' -d ' SCROLLIDSTRING '| jq .hits.hits[] -c | while read x; do id="echo "$x" | jq -r ._id"; index="echo "$x" | jq -r ._index"; type="echo "$x" | jq -r ._type"; doc="echo "$x" | jq ._source"; http put "localhost:9200/junk-$index/$type/$id" <<<"$doc"; done; done
I'm not gonna lie, bash scripting isn't a strong point for me, but I hope this set you on the right track :) If it helped I would appreciate if you mark it as the correct answer!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.