python elasticsearch: how to query a string on all fields

Question

I have the following python code which works fine, bringing me exactly 50 results as expected:

elastic = settings.ELASTIC
indexes = u'nginx-access-2769z-2018.11.26.16'
filter_by_client = [
    {'match_phrase': {'client_id': '2769z'}},
]
range_for_search = {
    'gte': str(1543248611),
    'lte': str(1543249511),
    'format': 'epoch_second',
}
query_body = {
    'from': 0,
    'size': 50,
    'query': {
        'bool': {
            'must': filter_by_client,
            'filter': {'range': {'@timestamp': range_for_search}},
        },
    }
}
search_result = elastic.search(index=indexes, body=query_body)
results = [result['_source'] for result in search_result['hits']['hits']]

And I now if I add another filter such as

...
filter_by_client = [
    {'match_phrase': {'client_id': '2769z'}},
    {'match': {'remote_address': '181.220.174.189'}}
]
...

It also works fine! Narrowing it down to 5 results.

My problem is: how do I query that string over all fields? Doesn't matter to me if that string is at the start/end of the field, if it is uppercase, if the field is actually an integer/float and not a string, ...

Already tried using the "_all" keyword like this

...
filter_by_client = [
    {'match_phrase': {'client_id': '2769z'}},
    {'match': {'_all': '181.220.174.189'}}
]
...

but it gives me 0 results. Trying to reproduce the same behaviour that happen over Kibana interface.

As mentioned in elasticsearch documentation _all is deprecated. Use copy_to instead and query on the custom field indexed by using copy_to parameter. — Nishant
– Nishant, Commented Nov 26, 2018 at 17:11

Kamal Kunjapur · Accepted Answer · 2018-11-26 19:19:42Z

What Nishant mentioned is the best solution using copy_to field, however if you don't have a control in changing your mapping, then you can try and see if any of the below approaches help.

Using Query String Query

You can make use of Query String Query where your query would be as follows:

...
filter_by_client = [
    {'match_phrase': {'client_id': '2769z'}},
    {'query_string': {'query': '181.220.174.189'}}
]
...

One important note is that query_string searches by default all the fields. The link I've mentioned states the below:

The default field for query terms if no prefix field is specified. Defaults to the index.query.default_field index settings, which in turn defaults to *. * extracts all fields in the mapping that are eligible to term queries and filters the metadata fields.

Also I am mentioning this because I want you to understand the difference in using query_string vs simple match Match vs Query-String before you decide to go for query_string.

The match family of queries does not go through a "query parsing" process. It does not support field name prefixes, wildcard characters, or other "advanced" features. For this reason, chances of it failing are very small / non existent, and it provides an excellent behavior when it comes to just analyze and run that text as a query behavior (which is usually what a text search box does). Also, the phrase_prefix type can provide a great "as you type" behavior to automatically load search results.

Using multi-match

The below another possible solution, if you are not wanting to change the mapping, which makes use of multi-match queries

...
filter_by_client = [
    {'match_phrase': {'client_id': '2769z'}},
    {'multi_match': {'query': '181.220.174.189', 'fields': ['url', 'field_2']}}
]
...

See how you need to explicitly mentioned the fields to be considered while querying. But do make sure you validate/test it thoroughly.

Let me know if this helps!

I'll look into this option of using copy_to, but for know I'll go with your suggestion. Thanks!!

Collectives™ on Stack Overflow

python elasticsearch: how to query a string on all fields

1 Answer 1

Using Query String Query

Using multi-match

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Using Query String Query

Using multi-match

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related