Elasticsearch reindex only missing documents

Question

I am trying to reindex an index of 200M of documents from cluster A to cluster B. I used the Reindex API with a remote source and everything worked fine. In the menwhile of my reindex some documents were added into the cluster A so I want to add them as well into the cluster B.

I launched again the reindex request but it seems that the reindex process is taking a lot, like if it was reindexing everything again.

My question is, is the cluster reindexing from scratch all the documents, even if they didn't change ?

My elasticsearch version is the 5.6

you index is of type daily or you store all in the same index ? — Bouraoui KACEM
– Bouraoui KACEM, Commented Oct 20, 2020 at 10:53

Saeed Nasehi · Accepted Answer · 2020-10-20 12:38:43Z

5

The elasticsearch does not know there is a change in the documents or not. So it tries to have each document completely in both indices. If you have a field like insert_time in your data, you can use reindex with query to limit the part of index of A to become reindex on B. This will let you use your older reindex and finish it faster. Reindex by query would be something like this:

POST _reindex
{
  "source": {
    "index": "A",
    "query": {
       "range": {
          "insert_time": {
              "gt": "time you want"
      }
    }
  },
  "dest": {
    "index": "B"
  }
}

edited Oct 20, 2020 at 12:38

answered Oct 20, 2020 at 11:41

Saeed Nasehi

1,0001 gold badge14 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Bouraoui KACEM Over a year ago

@Alessio Trivisonno : You can use it by modifying the query to point to the added data

ale_tri Over a year ago

Thank you for your answers, as I don't have a timestamp value I couldn't use this solution. For the future it would be a good idea to add a default _timestamp as suggested here: stackoverflow.com/questions/17136138/…

Saeed Nasehi Over a year ago

You're Welcome my friend!

Collectives™ on Stack Overflow

Elasticsearch reindex only missing documents

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related