elasticsearch reindex nested object's element to keyword

Question

I have an index structured like below:

"my_index": {
  "mappings": {
    "my_index": {
      "properties": {
        "adId": {
          "type": "keyword"
        },
        "name": {
          "type": "keyword"
        },
        "title": {
          "type": "keyword"
        },
        "creativeStatistics": {
          "type": "nested",
          "properties": {
            "clicks": {
              "type": "long"
            },
            "creativeId": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

I need to remove the nested object in a new index and just save the creativeId as a new keyword (to make it clear: I know I will loose the clicks data, and it is not important). It means the final new index scheme would be:

"my_new_index": {
  "mappings": {
    "my_new_index": {
      "properties": {
        "adId": {
          "type": "keyword"
        },
        "name": {
          "type": "keyword"
        },
        "title": {
          "type": "keyword"
        },
        "creativeId": {
          "type": "keyword"
        }
      }
    }
  }
}

Right now each row has exactly one creativeStatistics. and therefore there is no complexity in selecting one of the creativeIds.

I know it is possible to reindex using painless scripts, but I don't know how can I do that. Any help will be appreciated.

Val · Accepted Answer · 2018-11-24 09:01:46Z

1

You can do it like this:

POST _reindex
{
  "source": {
    "index": "my_old_index"
  },
  "dest": {
    "index": "my_new_index"
  },
  "script": {
    "source": "if (ctx._source.creativeStatistics != null && ctx._source.creativeStatistics.size() > 0) {ctx._source.creativeId = ctx._source.creativeStatistics[0].creativeId; ctx._source.remove('creativeStatistics')}",
    "lang": "painless"
  }
}

answered Nov 24, 2018 at 9:01

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Kamal Kunjapur · Accepted Answer · 2018-11-24 09:52:09Z

1

You can also create a Pipeline by creating a Script Processor as follows:

PUT _ingest/pipeline/my_pipeline
{
  "description" : "My pipeline",
  "processors" : [
    { "script" : {
        "source": "for (item in ctx.creativeStatistics) { if(item.creativeId!=null) {ctx.creativeId = item.creativeId;} }"
      }  
    },
    {
      "remove": {
        "field": "creativeStatistics"
      }
    }
  ]
}

Note that if you have multiple nested objects, it would append the last object's creativeId. And it would only add creativeId if a source document has one in its creativeStatistics.

Below is how you can then use reindex query:

POST _reindex
{
  "source": {
    "index": "creativeindex_src"
  },
  "dest": {
    "index": "creativeindex_dest",
    "pipeline": "my_pipeline"
  }
}

edited Nov 24, 2018 at 9:52

answered Nov 24, 2018 at 9:42

Kamal Kunjapur

8,9182 gold badges26 silver badges34 bronze badges

Collectives™ on Stack Overflow

elasticsearch reindex nested object's element to keyword

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related