ElasticSearch Multiple Scrolls Java API

Question

I want to get all data from an index. Since the number of items is too large for memory I use the Scroll (nice function):

client.prepareSearch(index)
    .setTypes(myType).setSearchType(SearchType.SCAN)
    .setScroll(new TimeValue(60000))
    .setSize(amountPerCall)
    .setQuery(MatchAll())
    .execute().actionGet();

Which works nice when calling:

client.prepareSearchScroll(scrollId)
    .setScroll(new TimeValue(600000))
    .execute().actionGet()

But, when I call the former method multiple times, I get the same scrollId multiple times, hence I cannot scroll multiple times - in parallel.

I found http://elasticsearch-users.115913.n3.nabble.com/Multiple-scrolls-simultanious-td4024191.html which states that it is possible - though I don't know his affiliation to ES.

Am I doing something wrong?

Related question of stackoverflow.com/questions/25453872/… — Danielson
– Danielson, Commented Jun 20, 2015 at 7:08

Community · Accepted Answer · 2017-05-23 10:32:31Z

3

After searching some more, I got the impression that this (same scrollId) is by design. After the timeout has expired (which is reset after each call Elasticsearch scan and scroll - add to new index).

So you can only get one opened scroll per index.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html states:

Scrolling is not intended for real time user requests, but rather for processing large amounts of data, e.g. in order to reindex the contents of one index into a new index with a different configuration.

So it appears what I wanted is not an option, on purpose - possibly because of optimization.

Update
As stated creating multiple scrolls cannot be done, but this is only true when the query you use for scrolling is the same. If you scroll for, for instance, another type, index, or just another query, you can have multiple scrolls

edited May 23, 2017 at 10:32

CommunityBot

11 silver badge

answered Jun 20, 2015 at 13:28

Danielson

2,7002 gold badges31 silver badges53 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Thomas Decaux · Accepted Answer · 2016-10-13 19:00:56Z

0

You can scroll the same index in same time, this is what elasticsearch-hadoop does.

Just, don't forget that under the hood, an index is composed of multiple shards that own data, so you can scroll each shards in parallel by using:

.setPreference("_shards:1")

answered Oct 13, 2016 at 19:00

Thomas Decaux

22.8k3 gold badges129 silver badges139 bronze badges

Collectives™ on Stack Overflow

ElasticSearch Multiple Scrolls Java API

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related