Elasticsearch remove a field from an object of an array in a dynamically generated index

Question

I'm trying to delete fields from an object of an array in Elasticsearch. The index has been dynamically generated.

This is the mapping:

{
  "mapping": {
    "_doc": {
      "properties": {
        "age": {
          "type": "long"
        },
        "name": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "result": {
          "properties": {
            "resultid": {
              "type": "long"
            },
            "resultname": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          },
        "timestamp": {
          "type": "date"
        }
      }
    }
  }
}
}

this is a document:

{
    "result": [
        {
            "resultid": 69,
            "resultname": "SFO"
        },
        {
            "resultid": 151,
            "resultname": "NYC"
        }
    ],
    "age": 54,
    "name": "Jorge",
    "timestamp": "2020-04-02T16:07:47.292000"
}

My goals is to remove all the fields resultid in result in all the document of the index. After update the document should look like this:

{
    "result": [
        {
            "resultname": "SFO"
        },
        {
            "resultname": "NYC"
        }
    ],
    "age": 54,
    "name": "Jorge",
    "timestamp": "2020-04-02T16:07:47.292000"
}

I tried using the following articles on stackoverflow but with no luck: Remove elements/objects From Array in ElasticSearch Followed by Matching Query remove objects from array that satisfying the condition in elastic search with javascript api Delete nested array in elasticsearch Removing objects from nested fields in ElasticSearch

Hopefully someone can help me find a solution.

Luc Ebert · Accepted Answer · 2020-04-03 15:22:34Z

2

You should reindex your index in a new one with _reindex API and call a script to remove your fields :

POST _reindex
{
  "source": {
    "index": "my-index"
  },
  "dest": {
    "index": "my-index-reindex"
  },
  "script": {
    "source": """
     for (int i=0;i<ctx._source.result.length;i++) {
        ctx._source.result[i].remove("resultid")
     }
     """

  }
}

After you can delete your first index :

DELETE my-index

And reindex it :

POST _reindex
{
  "source": {
    "index": "my-index-reindex"
  },
  "dest": {
    "index": "my-index"
  }
}

edited Apr 3, 2020 at 15:22

answered Apr 3, 2020 at 14:28

Luc Ebert

1,25511 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Mark Over a year ago

Thank you so much! I used your script with an "update by query" so that I don't have reindex

Luc Ebert Over a year ago

Perfect ! it's just safer to reindex in a new index. Can you mark my answer as resolved

Mark Over a year ago

I agree that it is safer to reindex to a new index, so that if something goes wrong you'll always have your old index.

Mark · Accepted Answer · 2020-04-04 15:55:06Z

1

I combined the answer from Luc E with some of my own knowledge in order to reach a solution without reindexing.

POST INDEXNAME/TYPE/_update_by_query?wait_for_completion=false&conflicts=proceed

{
"script": {
    "source": "for (int i=0;i<ctx._source.result.length;i++) { ctx._source.result[i].remove(\"resultid\")}"
    },
"query": {
    "bool": {
      "must": [
        {
          "exists": {
            "field": "result.id"
          }
        }
      ]
    }
  }
}

Thanks again Luc!

edited Apr 4, 2020 at 15:55

answered Apr 3, 2020 at 16:02

Mark

4135 silver badges11 bronze badges

Comments

Marian Kosaniak · Accepted Answer · 2021-07-23 18:54:37Z

1

If your array has more than one copy of element you want to remove. Use this: ctx._source.some_array.removeIf(tag -> tag == params['c'])

answered Jul 23, 2021 at 18:54

Marian Kosaniak

1911 silver badge8 bronze badges

Collectives™ on Stack Overflow

Elasticsearch remove a field from an object of an array in a dynamically generated index

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related