Elasticsearch Reindex

Question

Theres an index that I want to apply updated mappings to, I have done my best to follow the documentation on ES and Stackoverflow but I am now stuck.

The original index: logstash-index-YYYY.MM with data in it
I created index: logstash-index-new-YYYY.MM (which has a template for the new mapping)

Using the following query:
/logstash-index-YYYY.MM/_search?search_type=scan&scroll=1m
{ "query": { "match_all": {} }, "size": 30000 }

I get a _scroll_id and I have less than 30k docs so I should only need to run once.

How do I use that id to push the data into the new index?

the elasticsearch guid has a useful example: elasticsearch.org/guide/en/elasticsearch/guide/current/…. That allows you to retrieve the data in whatever page size you want using multiple requests. — Jilles van Gurp
– Jilles van Gurp, Commented Feb 16, 2015 at 17:08
Ok, thats actually up to the point that I got stuck, I have the data - my question is how do i put it into the new index? — Discorick
– Discorick, Commented Feb 16, 2015 at 17:24
Are you looking for the bulk API? If not I don't understand what you're asking. — Magnus Bäck
– Magnus Bäck, Commented Feb 16, 2015 at 19:12

rkrahl · Accepted Answer · 2015-02-16 19:18:16Z

1

You are not using scrollid to push the data into the new index. You use it to get another portion of data from the scroll query.

When you run scan query, first pass doesn't return any results, it scans through shards in your cluster and returns scrollid. Another pass (using scrollid from first one) will return actual results.

If you want to put that data into new index you should write some kind of simple program in language of your choice that will get this data and then put it into your new index.

There is a very good article on elasticsearch blog how to change mappings of your indices on the fly. Unfortunately, reindexing itself is not covered there.

edited Feb 16, 2015 at 19:18

answered Feb 16, 2015 at 19:12

rkrahl

1,1771 gold badge12 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Discorick Over a year ago

I was initially under the impression Elasticsearch had a 'magic' command to make this work, as you suggested I needed to write a script (I chose ruby) to search and scroll each result and push the data to the new index. Thanks!

Collectives™ on Stack Overflow

Elasticsearch Reindex

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related