Elasticsearch filter document ids with same field values count

Question

I need to return ids of documents that have the same myField value repeated [range] times. i.e. I have clientName field and want to find ids of all documents where clientName = 'John' but only if this name is found in 2-3 documents.

I have the following query

_search?filter_path=hits.total,hits.hits._id


"query": {
    some filters that return matching document ids
 },
"post_filter": {
    "bool": {
        "should": [
            {
                "range": {
                    "myField": {
                        "lte": 3,
                        "gte": 2
                    }
                }
            }
        ],
        "minimum_should_match": 1
    }
}

but it returns the range of myField values and not myField values count. I tried using aggregations but aggregation filters do not apply to the main query results. How do I modify my post_filter to get the results I require?

jaspreet chahal · Accepted Answer · 2020-03-15 06:50:34Z

Post_filter:

The post_filter is applied to the search hits at the very end of a search request, after aggregations have already been calculated

You cannot filter documents based on number of occurrence of a field in query or post_filter and neither can you use an aggregation result to filter documents in query.

Solution to the problem can be achieved in two ways

1. Get the terms which occur given number of times and search the documents for these terms (2 calls to the elastic search)

2. Get documents in aggregation itself using top_hits

Mapping:

{
  "index48" : {
    "mappings" : {
      "properties" : {
        "name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}

Query:

{
  "size": 0,
  "aggs": {
    "NAME": {
      "terms": {
        "field": "name.keyword",
        "size": 10
      },
      "aggs": {
        "documents": {
          "top_hits": {
            "size": 10
          }
        },
        "bucket_count": {
        "bucket_selector": {
          "buckets_path": {
            "path": "_count"
          },
          "script": "if(params.path>=1 && params.path<=3) return true"
        }
      }
      }
    }
  }
}

Result:

"aggregations" : {
    "NAME" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "John",
          "doc_count" : 3,
          "documents" : {
            "hits" : {
              "total" : {
                "value" : 3,
                "relation" : "eq"
              },
              "max_score" : 1.0,
              "hits" : [
                {
                  "_index" : "index48",
                  "_type" : "_doc",
                  "_id" : "ZRPC3HABF9RqmGpImxj_",
                  "_score" : 1.0,
                  "_source" : {
                    "name" : "John"
                  }
                },
                {
                  "_index" : "index48",
                  "_type" : "_doc",
                  "_id" : "ZhPC3HABF9RqmGpIpBh4",
                  "_score" : 1.0,
                  "_source" : {
                    "name" : "John"
                  }
                },
                {
                  "_index" : "index48",
                  "_type" : "_doc",
                  "_id" : "ZxPC3HABF9RqmGpIqRhj",
                  "_score" : 1.0,
                  "_source" : {
                    "name" : "John"
                  }
                }
              ]
            }
          }
        },
        {
          "key" : "Doe",
          "doc_count" : 2,
          "documents" : {
            "hits" : {
              "total" : {
                "value" : 2,
                "relation" : "eq"
              },
              "max_score" : 1.0,
              "hits" : [
                {
                  "_index" : "index48",
                  "_type" : "_doc",
                  "_id" : "aBPC3HABF9RqmGpIyhhU",
                  "_score" : 1.0,
                  "_source" : {
                    "name" : "Doe"
                  }
                },
                {
                  "_index" : "index48",
                  "_type" : "_doc",
                  "_id" : "aRPC3HABF9RqmGpIzhh7",
                  "_score" : 1.0,
                  "_source" : {
                    "name" : "Doe"
                  }
                }
              ]
            }
          }
        }
      ]
    }
  }

Collectives™ on Stack Overflow

Elasticsearch filter document ids with same field values count

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related