Elasticsearch incorrect document count after `scan` & `scroll` and then `refresh` & `flush`

Question

I use scan and scroll to reindex from old_index to new_index. Right after the reindex completes, I do a refresh and flush, hoping that the data will persist into the disk. I then immediately read the document counts of both indices. The new index will usually have zero documents. I have to repeatedly read the document counts in a loop (with a 1-second pause in each iteration). Only after 20 seconds or so, I will see equal document counts. (The refresh rate of both indices were set to 30 seconds).

From my understanding, calling either refresh or flush, I should see the actual document counts. But it is not the case.

chengpohi · Accepted Answer · 2015-08-06 02:27:16Z

2

In Elasticsearch1.6 You can use:

curl -XPOST localhost:9200/index/_flush/synced

to sync flush execution to be finished.

answered Aug 6, 2015 at 2:27

chengpohi

14.2k1 gold badge28 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

foresightyj Over a year ago

Thanks. It worked. But the official doc does not mention that this api will block even though the API name suggests so. A synced flush performs a normal flush, then adds a generated unique marker (sync_id) to all shards. From my understanding, a synced flush is merely a flush + a marker.

chengpohi Over a year ago

@foresightyj You can check Synced Flush Api to get more details.

foresightyj Over a year ago

Yes. I did read them (refresh, flush, synced flush) a few times. I am not concerned if all data is persisted. I just want to be sure that the new index has the same number of data as the old index (be it in the memory or the translog). Those three apis' documentations do not mention if they are synchronous or asynchronous. From the results I observed, I guess refresh and flush are asynchronous and synced flush, as only the name suggests, is synchronous.

foresightyj Over a year ago

originally I thought wait_if_ongoing in the flush api was the way to make it synchronous, but a closer look at the doc suggests that it will only block until it can start flushing, but will not wait until the flushing completes.

Collectives™ on Stack Overflow

Elasticsearch incorrect document count after `scan` & `scroll` and then `refresh` & `flush`

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related