Elasticsearch aggregation doesn't work with nested-type fields

Question

I can't make elasticsearch aggregation+filter to work with nested fields. The data schema (relevant part) is like this:

"mappings": {
  "rb": {
    "properties": {
      "project": {
        "type": "nested",
        "properties": {
          "age": {
            "type": "long"
          },
          "name": {
            "type": "string",
            "index": "not_analyzed"
          }
        }
      }    
    }
  }
}

Essentially "rb" object contains a nested field called "project" which contains two more fields - "name" and "age". Query I'm running:

"aggs": {
  "root": {
    "aggs": {
      "group": {
        "aggs": {
          "filtered": {
            "aggs": {
              "order": {
                "percentiles": {
                  "field": "project.age",
                  "percents": ["50"]
                }
              }
            },
            "filter": {
              "range": {
                "last_updated": {
                  "gte": "2015-01-01",
                  "lt": "2015-07-01"
                }
              }
            }
          }
        },
        "terms": {
          "field": "project.name",
          "min_doc_count": 5,
          "order": {
            "filtered>order.50": "asc"
          },
          "shard_size": 10,
          "size": 10
        }
      }
    },
    "nested": {
      "path": "project"
    }
  }
}

This query is supposed to produce top 10 projects (project.name field) which match the date filter, sorted by their median age, ignoring projects with less than 5 mentions in the database. Median should be calculated only for projects matching the filter (date range).

Despite having more than a hundred thousands objects in the database, this query produces empty list. No errors, just empty response. I've tried it both on ES 1.6 and ES 2.0-beta.

Ian · Accepted Answer · 2017-02-18 16:04:25Z

8

I've re-organized your aggregation query a bit and I could get some results showing up. The main point is type since you are aggregating around a nested type, I took out the filter aggregation on the last_updated field and moved it up the hierarchy as the first aggregation. Then comes the nested aggregation on the project field and finally the terms and the percentile.

That seems to work out pretty well. Please try.

{
  "size": 0,
  "aggs": {
    "filtered": {
      "filter": {
        "range": {
          "last_updated": {
            "gte": "2015-01-01",
            "lt": "2015-07-01"
          }
        }
      },
      "aggs": {
        "root": {
          "nested": {
            "path": "project"
          },
          "aggs": {
            "group": {
              "terms": {
                "field": "project.name",
                "min_doc_count": 5,
                "shard_size": 10,
                "order": {
                  "order.50": "asc"
                },
                "size": 10
              },
              "aggs": {
                "order": {
                  "percentiles": {
                    "field": "project.age",
                    "percents": [
                      "50"
                    ]
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

edited Feb 18, 2017 at 16:04

Ian

25.6k24 gold badges62 silver badges98 bronze badges

answered Sep 2, 2015 at 3:57

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

extesy Over a year ago

It works! I just had to increase shard_size to 100 to actually get 10 values back. Do you mind explaining why my original variant did not work? Is it a restriction in ES that it doesn't support filters inside nested fields?

Val Over a year ago

Inside a nested aggregation you can only reference nested fields. Besides, it makes more sense to first filter out data based on last_updated and only then to operate on the nested data.

Jay Shah Over a year ago

@Val - Can you please tell me how can I do that for multivalue bucket ? Seperate question - stackoverflow.com/questions/43451667/…

Collectives™ on Stack Overflow

Elasticsearch aggregation doesn't work with nested-type fields

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related