Re-Index Elasticsearch, ignore fields not in mapping

Question

Trying to test out re-index API in elasticsearch and running into issues where existing data contains fields not present in the new index's strict mapping. Is there a way to tell elasticsearch to simply ignore those fields and carry on?

Edit: To clarify, by ignore I meant not to include those fields during the re-index process.

can you adjust the mapping settings before starting the reindex? or you don't have access to it? — Andrey Borisko
– Andrey Borisko, Commented Sep 11, 2019 at 21:33
@AndreyBorisko I can but I want the fields ignored, or i guess removed, during the re-index. Is that even possible? — MarkII
– MarkII, Commented Sep 11, 2019 at 21:50
it's possible. might look dirty though. depending on how many properties you need to remove. — Andrey Borisko
– Andrey Borisko, Commented Sep 11, 2019 at 23:16

Andrey Borisko · Accepted Answer · 2019-09-13 03:00:37Z

2

If you have access to the index settings before running reindex you can just do:

PUT test/_mapping
{
    "dynamic": "false"
}

then change it back to strict once reindexing is done.

UPDATE based on your comment

POST _reindex
{
  "source": {
    "index": "src"
  },
  "dest": {
    "index": "dst"
  },
  "script": {
    "lang": "painless",
    "source": """
    ctx['_source'].remove('email');
    ctx['_source'].remove('username');
    ctx['_source'].remove('name');
    // removing from nested:
    for(item in ctx['_source'].Groups){
        item.remove('GroupName');
        item.remove('IsActive');
    }
    """
  }
}

edited Sep 13, 2019 at 3:00

answered Sep 11, 2019 at 21:49

Andrey Borisko

4,6193 gold badges24 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

MarkII Over a year ago

That lets me insert the data, however I would like any excess fields removed.

MarkII Over a year ago

It seems you have to specify the individual fields you want versus remove anything not in the schema.

sjsj15 · Accepted Answer · 2022-03-07 13:19:57Z

0

While reindexing you can include or exclude source fields according to your destination index mapping.

To exclude some specific fields while reindexing:

POST _reindex
{
  "source": {
    "index": "source-index",
    "_source": {
      "excludes": ["exclude_a", "exclude_b"]
    }
  },
  "dest": {
    "index": "dest-index"
  }
}

To include any specific field while reindexing:

POST _reindex
{
  "source": {
    "index": "source-index",
    "_source": ["include_a", "include_b"]
  },
  "dest": {
    "index": "dest-index"
  }
}

answered Mar 7, 2022 at 13:19

sjsj15

8159 silver badges13 bronze badges

Collectives™ on Stack Overflow

Re-Index Elasticsearch, ignore fields not in mapping

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related