2

I'm using NEST 2.3.2 to query ElasticSearch and need to create a conditional filter for FollowUpDateTime field based on optional boolean. If that boolean

  • is true - want to see all records with non-blank date,
  • is false - records without date,
  • is null - show all records.

Here's what I have in Nest: 

                    var containers = new QueryContainer[7];
                    //Size is based on hte number of search parameters.. 
                    var descriptor = new QueryContainerDescriptor<EsNoteModel>();

                    var filters = new QueryContainer[1];
                    var filterdescriptor = new QueryContainerDescriptor<EsNoteModel>();  

       /*...... some code for other parameters here .....*/

        if (hasFollowUpDate.HasValue)
                    {
                        if ((bool) hasFollowUpDate)
                        {
                            //If true shows only with a non-blank follow-up date
                            filters[0] = filterdescriptor.Exists(ff => ff.Field("FollowUpDateTime"));
                        }
                        else
                        {
                            //If false shows only notes with a blank follow-up date
                            filters[0] = filterdescriptor.Missing(ff => ff.Field("FollowUpDateTime"));
                        }
                    }

/*...... CALL to ES below .....*/    

                    var result = ElasticSearchClient.Search<EsNoteModel>(body => body
                        .From(offset - 1)
                        .Size(rows)
                        .Query(q => q
                            .Bool(b => b
                                  .Must(containers)
                                  .Filter(f => f.Bool(t => t.Must(filters)))
                                  )
                                )

                        .Sort(s => s
                            .Field(f => f
                                .Field(p => p.NoteDate)
                                .Order(SortOrder.Descending)))

                        );

The problem is that the query above generated the error. Please see below:

Invalid NEST response built from a unsuccessful low level call on POST: /_search?pretty=true

Audit trail of this API call:

  • BadResponse: Node: http://**********.es.amazonaws.com/ Took: 00:00:00.1381301

    OriginalException: System.Net.WebException: The remote server returned an error: (400) Bad Request. at

System.Net.HttpWebRequest.GetResponse() at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData)

# Request:
{
  "from": 0,
  "size": 25,
  "sort": [
    {
      "NoteDate": {
        "order": "desc"
      }
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "CustomerId": {
              "query": "2810855"
            }
          }
        }
      ],
      "filter": [
        {
          "bool": {
            "must": [
              {
                "exists": {
                  "field": "FollowUpDateTime"
                }
              }
            ]
          }
        }
      ]
    }
  }
}

Response below:

# Response:
{
  "error" : "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[-t7SwBo_TjeH_98h_56ANw][customernote][0]: RemoteTransportException[[Neophyte][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[customernote][0]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@2d55231>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"28_10855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"bool\": {\r\n            \"must\": [\r\n              {\r\n                \"exists\": {\r\n                  \"field\": \"FollowUpDateTime\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[TOOugCq3ROqF9CGnHIay2g][customernote][1]: SearchParseException[[customernote][1]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@562f97b5>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"28_10855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"bool\": {\r\n            \"must\": [\r\n              {\r\n                \"exists\": {\r\n                  \"field\": \"FollowUpDateTime\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[TOOugCq3ROqF9CGnHIay2g][customernote][2]: SearchParseException[[customernote][2]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@394cf46e>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"28_10855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"bool\": {\r\n            \"must\": [\r\n              {\r\n                \"exists\": {\r\n                  \"field\": \"FollowUpDateTime\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[-t7SwBo_TjeH_98h_56ANw][customernote][3]: RemoteTransportException[[Neophyte][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[customernote][3]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@46d07ba8>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"28_10855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"bool\": {\r\n            \"must\": [\r\n              {\r\n                \"exists\": {\r\n                  \"field\": \"FollowUpDateTime\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[TOOugCq3ROqF9CGnHIay2g][customernote][4]: SearchParseException[[customernote][4]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@74fb3710>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"28_10855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"bool\": {\r\n            \"must\": [\r\n              {\r\n                \"exists\": {\r\n                  \"field\": \"FollowUpDateTime\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }]",
  "status" : 400
}

As you see error reports "nested: QueryParsingException[[customernote] bool query does not support [filter]];" Any ideas what I'm doing wrong in this example? Is this a proper use of filter clause?

**** UPDATE

changed query 

 var result = ElasticSearchClient.Search<EsNoteModel>(body => body
                .From(offset - 1)
                .Size(rows)
                .Query(q => q
                    .Bool(b => b
                          .Must(containers)
                          .Filter(filters)
                          )
                        )

                .Sort(s => s
                    .Field(f => f
                        .Field(p => p.NoteDate)
                        .Order(SortOrder.Descending)))

                );

request to 

# Request:
{
  "from": 0,
  "size": 25,
  "sort": [
    {
      "NoteDate": {
        "order": "desc"
      }
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "CustomerId": {
              "query": "2810855"
            }
          }
        }
      ],
      "filter": [
        {
          "exists": {
            "field": "FollowUpDateTime"
          }
        }
      ]
    }
  }
}

getting response error:

# Response:
{
  "error" : "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[TOOugCq3ROqF9CGnHIay2g][customernote][0]: SearchParseException[[customernote][0]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@4839a60c>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"2810855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"exists\": {\r\n            \"field\": \"FollowUpDateTime\"\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[-t7SwBo_TjeH_98h_56ANw][customernote][1]: RemoteTransportException[[Neophyte][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[customernote][1]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@21d845ae>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"2810855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"exists\": {\r\n            \"field\": \"FollowUpDateTime\"\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[-t7SwBo_TjeH_98h_56ANw][customernote][2]: RemoteTransportException[[Neophyte][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[customernote][2]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@6124d214>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"2810855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"exists\": {\r\n            \"field\": \"FollowUpDateTime\"\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[TOOugCq3ROqF9CGnHIay2g][customernote][3]: SearchParseException[[customernote][3]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@f884425>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"2810855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"exists\": {\r\n            \"field\": \"FollowUpDateTime\"\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }{[-t7SwBo_TjeH_98h_56ANw][customernote][4]: RemoteTransportException[[Neophyte][inet[/x.x.x.x:y]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[customernote][4]: from[0],size[25],sort[<custom:\"NoteDate\": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@77228f64>!]: Parse Failure [Failed to parse source [{\r\n  \"from\": 0,\r\n  \"size\": 25,\r\n  \"sort\": [\r\n    {\r\n      \"NoteDate\": {\r\n        \"order\": \"desc\"\r\n      }\r\n    }\r\n  ],\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"match\": {\r\n            \"CustomerId\": {\r\n              \"query\": \"2810855\"\r\n            }\r\n          }\r\n        }\r\n      ],\r\n      \"filter\": [\r\n        {\r\n          \"exists\": {\r\n            \"field\": \"FollowUpDateTime\"\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}]]]; nested: QueryParsingException[[customernote] bool query does not support [filter]]; }]",
  "status" : 400
}

Basically same error

nested: QueryParsingException[[customernote] bool query does not support [filter]];

Improve this question
5
  • Bool queries can only have a filter from elasticsearch 2 onwards. Commented Jun 10, 2016 at 20:55
  • using Elasticsearch.Net.2.3.2 Commented Jun 10, 2016 at 21:28
  • 1
    That's the client you're using but the server is 1.5.2 (since that is the only version provided by the aws elasticsearch service, which you appear to be using) Commented Jun 10, 2016 at 21:34
  • Frederick - you deserve a star! thank you Commented Jun 10, 2016 at 21:43
  • 1
    The major versions of NEST are compatible with the major versions of Elasticsearch so for Elasticsearch 1.5.2, you should use the latest NEST 1.x, which at this time is 1.8.2. Check out the compatibility matrix on github - github.com/elastic/elasticsearch-net Commented Jun 12, 2016 at 0:20

2 Answers 2

Reset to default
1

Apparently we have in out .NET 4.0 project the following references to

  1. Elasticsearch.Net - official low level elasticsearch client packages\Elasticsearch.Net.2.3.2\lib\net45\Elasticsearch.Net.dll
  2. NEST - official high level elasticsearch client packages\NEST.2.3.2\lib\net45\Nest.dll

interesting fact they still use deprecated queries. as mentioned by Frederick here

the client you're using but the server is 1.5.2 (since that is the only version provided by the aws elasticsearch service, which you appear to be using)

see below - it works:

            var query = new QueryContainer[8];//Size is based on hte number of search parameters
            var descriptor = new QueryContainerDescriptor<EsNoteModel>();

/*........... skipping here some code for other parameters ..........*/

 if (hasFollowUpDate.HasValue)
            {
                if ((bool) hasFollowUpDate)
                {
                    //If true shows only with a non-blank follow-up date
                    query[7] = descriptor.Filtered(p => p.Filter(f => f.Exists(r => r.Field(u => u.FollowUpDateTime))));
                }
                else
                {
                    ////If false shows only notes with a blank follow-up date
                    query[7] = descriptor.Filtered(p => p.Filter(f => f.Missing(r => r.Field(u => u.FollowUpDateTime))));
                }
            }

            var result = ElasticSearchClient.Search<EsNoteModel>(body => body
                .From(offset - 1)
                .Size(rows)
                .Query(q => q
                    .Bool(b => b
                          .Must(query)
                          )
                        )

                .Sort(s => s
                    .Field(f => f
                        .Field(p => p.NoteDate)
                        .Order(SortOrder.Descending)))

                );

So simply replacing word BOOL to Filtered - did the trick

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

how did you connect aws ES using nest? Simple post doesnt work I thinkt. It requires some security to be added
0

You can do this dynamically using a QueryContainer or FilterContainer. Here is a very similar one from our solution:

var query = new List<QueryContainer>
{
    Query<AlertForReporting>.Range(r => r.OnField(f => f.AlertDate)
                                          .Greater(start.ToEPCISFormat())
                                          .Lower(end.ToEPCISFormat())
        )
};


var result = ElasticClient.Search<AlertForReporting>(s =>
                        s.Size(0)
                            .Query(q =>
                                q.Bool(b =>
                                        b.Must(query.ToArray())
                                    )
                            ));
Improve this answer

2 Comments

we already using QueryContainer for some other parameters for DateRange, Match and QueryStrings searches. (code updated above). do we need to have use both: FilterContainer and QueryContainer? and if so - how? I am searching for answer.. reading elastic.co/guide/en/elasticsearch/reference/current/…
@NataliaZolotareva In NEST 1.x, there is FilterContainer and QueryContainer, the former is used where a filter is expected and the latter, where a query is expected. In NEST 2.x which works with Elasticsearch 2.x, there is only QueryContainer to reflect the change in Elasticsearch where filters and queries merged. In NEST 2.x, you use QueryContainer where you would have used QueryContainer and FilterContainer in NEST 1.x

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.