Mapping a Map with Dynamic Keys in ElasticSearch

Question

I'm trying to store this kind of data inside an Elasticsearch index

{
  "id": "5644596f9bf67301645999d9",
  "headline": "Scientists Look Beyond Solar System to Study Planet",
  "renditions": {
    "baseImage": {
      "height": 933,
      "href": "www.imgur.com/animage",
      "mimetype": "image/jpeg",
      "width": 1400
    },
    "preview": {
      "height": 500,
      "href": "www.imgur.com/animage",
      "mimetype": "image/jpeg",
      "width": 400
    },
    "thumbnail": {
      "height": 150,
      "href": "www.imgur.com/animage",
      "mimetype": "image/jpeg",
      "width": 125
    }
  }
}

However the mapping keeps adding unique blocks for all the different children of "renditions".

Ideally, I would define a type of child for "renditions" then assert that all objects under it should be like that.

Is this possible?

I've looked into Dynamic Mappings but I've got a bit confused with it all...

Can you change the structure of your data? Or do you have to use that form? — Sloan Ahrens
– Sloan Ahrens, Commented Jan 15, 2016 at 15:29
Ideally it would stay this way, as it's stored in Mongo too, so sharing the same model structure would the be the easiest. What would you suggest if I did change the model? Storing it as an array? — Stickboy
– Stickboy, Commented Jan 15, 2016 at 15:31
Yeah, probably. Something like, "renditions":[{ "key_name": "thumbnail", "value": {...} },...], maybe. In that case you would probably want to use a nested type. — Sloan Ahrens
– Sloan Ahrens, Commented Jan 15, 2016 at 15:34

Mandeep Singh · Accepted Answer · 2016-01-16 08:16:33Z

4

You can try this mapping with dynamic templates:

PUT /test

PUT /test/_mapping/test
{
   "properties": {
      "id": {
         "type": "string"
      },
      "headline": {
         "type": "string"
      }
   },
   "dynamic_templates": [
      {
         "renditions_objects": {
            "mapping": {
               "dynamic": "strict",
               "type": "object",
               "properties": {
                  "height": {
                     "type": "integer"
                  },
                  "href": {
                     "type": "string"
                  },
                  "mimetype": {
                     "type": "string"
                  },
                  "width": {
                     "type": "integer"
                  }
               }
            },
           "match_mapping_type": "object",
           "path_match": "renditions.*"            
         }      
      }
   ]
}

The line

"dynamic": "strict",

makes sure that you cannot index any document where extra fields are present. The process fails if you try to index something like this:

POST /test/test
{
  "id": "5644596f9bf67301645999d9",
  "headline": "Scientists Look Beyond Solar System to Study Planet",
  "renditions": {
    "baseImage": {
      "height": 933,
      "href": "www.imgur.com/animage",
      "mimetype": "image/jpeg",
      "width": 1400,
      "dummy": 12
    },
    "preview": {
      "height": 500,
      "href": "www.imgur.com/animage",
      "mimetype": "image/jpeg",
      "width": 400
    },
    "thumbnail": {
      "height": 150,
      "href": "www.imgur.com/animage",
      "mimetype": "image/jpeg",
      "width": 125
    }
  }
}

Notice the extra dummy key in baseImage. It will fail with error:

{
   "error": "StrictDynamicMappingException[mapping set to strict, dynamic introduction of [dummy] within [renditions.baseImage] is not allowed]",
   "status": 400
}

If you want to make it a little relaxed and allow such documents to be indexed, but don't want the extra fields to be indexed, then set dynamic to false. This will not cause any failure while indexing such documents, also the extra field will be present in the _source field. However, that field will not be indexed and you won't be able to perform any query or filter on that field. Hope this helps

answered Jan 16, 2016 at 8:16

Mandeep Singh

8,25421 gold badges69 silver badges106 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Stickboy Over a year ago

Thanks, that works! However when I look at the mapping it still have 3 blocks, one for baseImage, preview & thumbnail. My original plan was to make it so it wouldn't need to create a new block for each Key name. Is this possible?

Stickboy Over a year ago

Mapping out, for reference - gist.githubusercontent.com/petersow/c1a926064d36f88dbc82/raw/…

Collectives™ on Stack Overflow

Mapping a Map with Dynamic Keys in ElasticSearch

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related