Storing nested objects in elastic search

Question

I have a 3 level nested object something like below, I want to index these objects in elastic search. Requirement here is users will write a search query keywords like "keyword1 keyword2 ..." and I want to return objects that has all of these keywords in it (at any level i.e. AND operation).

   [  
   {  
      "country":[  
         {  
            "name":"India",
            "ext_code":"91",
            "states":[  
               {  
                  "name":"Karnataka",
                  "ext_code":"04",
                  "cities":[  
                     {  
                        "name":"Bangalore",
                        "ext_code":"080"
                     }
                  ]
               }
            ]
         }
      ]
   }
]

Currently, I store them in nested format using the following mapping:

{
    "mappings":{
        "doc":{
            "properties":{
                "name": {"type":"text"},
                "ext_code": {"type":"text"}
                "state" {
                    "type": "nested",
                    "properties": {
                        "name": {"type":"text"},
                        "ext_code": {"type":"text"}
                        "city" {
                            "type": "nested",
                            "properties": {
                                "name": {"type":"text"}
                                "ext_code": {"type":"text"}
                            }
                        }
                    }
                }
            }
        }
    }
}

And while searching, I pass elastic search a nested query to search on all levels like below:

{
    "query": {
        "bool": {
            "should": [
                {
                    "multi_match": {
                        "query": "keyword1 keyword2 ...",
                        "fields": ['name'] 
                    }
                },
                {
                    "nested": {
                        "path": 'state',
                        "query": {
                            "multi_match": {
                                "query": "keyword1 keyword2 ...",
                                "fields": ['state.name']
                            }
                        }
                    }
                },
                {
                    "nested": {
                        "path": 'state.city',
                        "query": {
                            "multi_match": {
                                "query": "keyword1 keyword2 ...",
                                "fields": ['state.city.name']
                            }
                        }
                    }
                }
            ]
        }
    }
}

When sending multiple tokens to search, it applies an OR operation returning documents containing any of the search tokens.

Is there a way to configure Elastic Search to perform an AND operation for multiple tokens in search query?

Hi Lokesh, what is technically possible will always receive a solution, but sometimes you think that some solutions are harder if you don't know the purpose of what you are trying to accomplish, why did you choose nested types for this? — panchicore
– panchicore, Commented Sep 27, 2018 at 11:42
yep, but offcourse, all depends on the types of queries you want to run, one ES engineer once told me, nested types result eventually optimized (cpu & memory, hence costs) by converting the structure to flat and adding a little bit of work on the client side. — panchicore
– panchicore, Commented Sep 28, 2018 at 9:06
Is there any impact on analysis (using Kibana) by flattening out the data? — ThinkGeek
– ThinkGeek, Commented Sep 29, 2018 at 4:31
huge, kibana is designed for a "time-series first" data schema, flat is king in time-series, so yes. — panchicore
– panchicore, Commented Oct 2, 2018 at 8:27

Val · Accepted Answer · 2018-10-01 04:42:24Z

One solution would be to index all the values of the name fields inside a custom all field. First define your index and mappings like this:

PUT index
{
  "mappings": {
    "doc": {
      "properties": {
        "all": {                 <-- all field that will contain all values
          "type": "text"
        },
        "name": {
          "type": "text",
          "copy_to": "all"       <-- copy value to all field
        },
        "ext_code": {
          "type": "text"
        },
        "state": {
          "type": "nested",
          "properties": {
            "name": {
              "type": "text",
              "copy_to": "all"       <-- copy value to all field
            },
            "ext_code": {
              "type": "text"
            },
            "city": {
              "type": "nested",
              "properties": {
                "name": {
                  "type": "text",
                  "copy_to": "all"       <-- copy value to all field
                },
                "ext_code": {
                  "type": "text"
                }
              }
            }
          }
        }
      }
    }
  }
}

Then index your documents:

POST index/doc
{
  "name": "India",
  "ext_code": "91",
  "state": [
    {
      "name": "Karnataka",
      "ext_code": "04",
      "city": [
        {
          "name": "Bangalore",
          "ext_code": "080"
        }
      ]
    }
  ]
}

Finally, using a simple match query, you can search for any value anywhere in the document:

POST index/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "all": {
              "query": "bangalore india",
              "operator": "and"
            }
          }
        }
      ]
    }
  }
}

nitzien · Accepted Answer · 2018-09-27 14:38:55Z

Please try following

{
    "query": {
        "bool": {
            "should": [
                {
                    "simple_query_string": {
                        "query": "keyword1 keyword2 ...",
                        "fields": ['name'] ,
                        "default_operator": "and"
                    }
                },
                {
                    "nested": {
                        "path": 'state',
                        "query": {
                            "simple_query_string": {
                                "query": "keyword1 keyword2 ...",
                                "fields": ['state.name'],
                                "default_operator": "and"
                            }
                        }
                    }
                },
                {
                    "nested": {
                        "path": 'state.city',
                        "query": {
                            "simple_query_string": {
                                "query": "keyword1 keyword2 ...",
                                "fields": ['state.city.name'],
                                "default_operator": "and"
                            }
                        }
                    }
                }
            ]
        }
    }
}

I don't think multi_match is good for this requirement. Simple query string or query string query is better suited for this purpose.

Collectives™ on Stack Overflow

Storing nested objects in elastic search

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related