elasticsearch documents missing in the new index created from existing index

Question

I have an existing index named "prod.custom.historical.actions-2021-03" and created new index named "PUT /prod.custom.historical.actions-2021-04-19"

Executed below commands in the same order. PUT /prod.custom.historical.actions-2021-04-19

POST _reindex
{
  "source": {
    "index": "prod.custom.historical.tasks-2021-03"
  },
  "dest": {
    "index": "prod.custom.historical.tasks-2021-04-19"
  }
}

GET /prod.custom.historical.actions-2021-03/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "terms": {
            "projectId": [
              "73ef9fa8-6665-4cbb-881c-4663030a7d45"
            ]
          }
        }
      ]
    }
  }
}

returns

"took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 93,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [


GET /prod.custom.historical.actions-2021-04-19/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "terms": {
            "projectId": [
              "73ef9fa8-6665-4cbb-881c-4663030a7d45"
            ]
          }
        }
      ]
    }
  }
}

returns

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

Which clearly indicates that there are missing documents, someone can please help me to understand the issue here.

Bhavya · Accepted Answer · 2021-04-19 11:48:04Z

1

In reindex API, only the documents are copied from one index to another. You need to configure the mapping of the new index, as mentioned in the official documentation

The destination should be configured as wanted before calling _reindex. Reindex does not copy the settings from the source or its associated template.

Mappings, shard counts, replicas, and so on must be configured ahead of time.

If you have not explicitly defined any mapping for the index "prod.custom.historical.actions-2021-04-19" then you need to add .keyword to the projectId field. This uses the keyword analyzer instead of the standard analyzer (notice the ".keyword" after projectId field).

GET /prod.dhl.historical.actions-2021-04-19/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "terms": {
            "projectId.keyword": [            // note this
              "73ef9fa8-6665-4cbb-881c-4663030a7d45"
            ]
          }
        }
      ]
    }
  }
}

edited Apr 19, 2021 at 11:48

answered Apr 19, 2021 at 11:43

Bhavya

16.2k3 gold badges23 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Sunil Yerra Over a year ago

Applied the changes, but same issue.

Bhavya Over a year ago

@SunilYerra can you share the index mapping of your new index for the particular field projectId? Have you explicitly set any index mapping of the new index ?

Bhavya Over a year ago

@SunilYerra and can you please confirm whether the documents are indexed in the new index by hitting the match_all query elastic.co/guide/en/elasticsearch/reference/current/…

Sunil Yerra Over a year ago

I m new to this technology and honestly speaking donot know how to retrieve the mapping for the new index

Sunil Yerra Over a year ago

thanks for the quick help, the problem was while creating the new index, I did not map the filelds as per the old index.

|

Collectives™ on Stack Overflow

elasticsearch documents missing in the new index created from existing index

1 Answer 1

9 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Related