Elasticsearch inline script filter not working

Question

I am trying to execute the below query on ES-2.3.4. If you remove the inline script at the end the query is working as expected. But if I include the script the query is supposed to return results but it doesn't. It is a groovy script. Where "bio" is a nested object. Can anyone verify the query and suggest me if any changes are required.

    {
  "bool" : {
    "must" : [ {
      "nested" : {
        "query" : {
          "term" : {
            "bio.cl" : "Position"
          }
        },
        "path" : "bio"
      }
    }, {
      "nested" : {
        "query" : {
          "terms" : {
            "bio.type" : [ "SV" ]
          }
        },
        "path" : "bio"
      }
    }, {
      "nested" : {
        "query" : {
          "terms" : {
            "bio.node" : [ "XX" ]
          }
        },
        "path" : "bio"
      }
    }, {
      "terms" : {
        "domain" : [ "YY" ]
      }
    } ],
    "filter" : [ {
      "nested" : {
        "query" : {
          "term" : {
            "bio.chromo" : 1
          }
        },
        "path" : "bio"
      }
    }, {
      "nested" : {
        "query" : {
          "range" : {
            "bio.start" : {
              "from" : null,
              "to" : 1000140.0,
              "include_lower" : true,
              "include_upper" : true
            }
          }
        },
        "path" : "bio"
      }
    }, {
      "nested" : {
        "query" : {
          "range" : {
            "bio.stop" : {
              "from" : 1000861.0,
              "to" : null,
              "include_lower" : true,
              "include_upper" : true
            }
          }
        },
        "path" : "bio"
      }
    }, {
          "script" : {
            "script" : {
              "inline" : "percent <= ([stop,_source.bio.stop.value].min() - [start,_source.bio.start.value].max())/[length,_source.bio.stop.value-_source.bio.start.value+1].max()",
              "params" : {
                "stop" : 1001100,
                "start" : 999901,
                "length" : 1200,
                "percent" : 0.8
              }
            }
          }
    } ]
  }
}

Mapping:

"mappings": {
  "XX": {
    "properties": {
      "bio": {
        "type": "nested",
        "properties": {
          "alt": {
            "type": "string",
            "index": "not_analyzed"
          },
          "ann": {
            "type": "string",
            "index": "not_analyzed"
          },
          "chromo": {
            "type": "string",
            "index": "not_analyzed"
          },
          "cod": {
            "type": "string"
          },
          "conseq": {
            "type": "string",
            "index": "not_analyzed"
          },
          "contri": {
            "type": "string",
            "index": "not_analyzed"
          },
          "created": {
            "type": "string",
            "index": "not_analyzed"
          },
          "createdDate": {
            "type": "date",
            "format": "strict_date_optional_time"
          },
          "domain": {
            "type": "string",
            "index": "not_analyzed"
          }"id": {
            "type": "long"
          },
          "name": {
            "type": "string",
            "index": "not_analyzed"
          },
          "node": {
            "type": "string",
            "index": "not_analyzed"
          },
          "position": {
            "type": "string",
            "index": "not_analyzed"
          },
          "level": {
            "type": "string",
            "index": "not_analyzed"
          },
          "start": {
            "type": "long"
          },
          "stop": {
            "type": "long"
          }
        }
      }
    }
  }
}

Sample document:

_source" : {
        "id" : 25,
        "bio" : [ {
          "creation" : "2018-03-05T20:26:46.466Z",
          "updateDate" : "2018-03-05T20:26:46.466Z",
          "createdBy" : "XX",
          "type" : "SV",
          "creationDate" : "2018-03-05T20:26:46.472Z",
          "updateDate" : "2018-03-05T20:26:46.521Z",
          "createdBy" : "XX",
          "updatedBy" : "XX",
          "domain" : "YY",
          "node" : "XX",
          "ann" : "1.6",
          "gen" : "37",
          "level" : "Position",
          "chromo" : "1",
          "start" : 999901,
          "stop" : 1001100
      }]
    }

Since you're accessing the values directly from the _source, I think you need _source.bio.stop instead of _source.bio.stop.value, i.e. remove the .value — Val
– Val, Commented Mar 7, 2018 at 8:02
@Val Thanks for the suggestion. I tried this option but I see the below error in logs.groovy.lang.GroovyRuntimeException: Cannot compare java.util.ArrayList with value '[1001100]' and java.lang.Integer with value '1,001,100' — Abhi.G
– Abhi.G, Commented Mar 7, 2018 at 8:08
Can you show a sample document please? as well as your mapping. Thanks — Val
– Val, Commented Mar 7, 2018 at 8:09
@Val I have updated the mapping and document. data is morphed. Pardon for any inconsistency. But this should give you a basic idea. — Abhi.G
– Abhi.G, Commented Mar 7, 2018 at 8:41
Ok, one problem I see is that bio is nested, hence it is an array and you need to access a specific element within that array, so _source.bio.stop should be _source.bio[0].stop.value — Val
– Val, Commented Mar 7, 2018 at 8:48

Val · Accepted Answer · 2018-03-07 09:35:32Z

1

Following up from our discussion in the comments above...

You need to concat the arrays correctly, i.e.

[stop] + _source.biomarkers.collect{it.stop}

will create an array with [stop, bio[0].stop, bio[1].stop, etc] and then we can take the max() of that array.

So I suggest something like this should work (untested though)

percent <= (([stop] + _source.biomarkers.collect{it.stop}).min() - ([start] + _source.biomarkers.collect{it.start}).max()) / ([length] +_source.biomarkers.collect{it.stop - it.start + 1}).max()

answered Mar 7, 2018 at 9:35

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Abhi.G Over a year ago

Thanks, Val. Let me try this. But can I know why are you using "+" instead of "," in between the comparison variables(stop and _source)? I am new to Groovy. Don't mind if this is a silly question

Abhi.G Over a year ago

The solution you have provided works. Let me test it thoroughly and match the expected results.@Val Thanks.

Abhi.G Over a year ago

Perfect! I have marked this as a solution. Thank you very much as always.

Collectives™ on Stack Overflow

Elasticsearch inline script filter not working

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related