Performing filtering in Elasticsearch

Question

I am new to elastic search and i have been trying to get a grip on how it works but unfortunately didn't find a clear cut tutorial so far.

What i have been trying to achieve was to index some data and perform filtering on them.

I indexed the data by passing the data as JSON and sending them over HttpWebRequest . A sample data is given below

{"changeset":"2015-12-01 12:06+05:30","id":"1","registerid":"1"}

Now what puzzles me is the data type of each of the document fields. As some would be data another string and another one int. So when i filter data , say i want to get all index whose changeset date is greater than 2015-01-01 (yyyy-mm-dd) or i want to get all index whose registerid is less than 100.

Since all data is saved in a schema less format is JSON , how can we handle it.

Then after repeated searching (google) i found a concept called "mapping" Ref Link.

So do I have to create an index using the mapping to achieve my filtering things?

Val · Accepted Answer · 2015-12-21 07:40:56Z

1

Mapping types is the way to go, indeed. A mapping type allows you to define what fields you have in your document and what type they have. If you don't provide a mapping type, Elasticsearch will do its best to infer your fields and their types, but you might end in situations where its guesses were not accurate.

The common best practice is to always provide your own custom mapping so that you can ensure how your data is going to be processed.

For your sample data above, you can simply create an index by specifying a mapping type with a date field, a string field and an int field:

curl -XPUT localhost:9200/your_index -d '{
    "mappings": {
        "your_type": {
            "properties": {
                "changeset": {
                    "type": "date",
                    "format": "yyyy-MM-dd HH:mmZ"
                },
                "id": {
                    "type": "string"
                },
                "registerid": {
                    "type": "integer"
                },
            }
        }
    }
}'

With that mapping type specified, Elasticsearch will now know how to interpret the string data present in your JSON documents. With that done, it is now easy to create queries such as the two ones you've expressed, namely:

All docs whose changeset date is greater than 2015-01-01 can be queried like this:

curl -XPOST localhost:9200/your_index/_your_type/_search -d '{
    "query": {
        "filtered": {
            "filter": {
                "range": {
                    "changeset": {
                        "gt": "2015-01-01"
                    }
                }
            }
        }
    }
}'

All docs whose registerid date is less than 100 can be queried like this:

curl -XPOST localhost:9200/your_index/_your_type/_search -d '{
    "query": {
        "filtered": {
            "filter": {
                "range": {
                    "registerid": {
                        "lt": 100
                    }
                }
            }
        }
    }
}'

answered Dec 21, 2015 at 7:40

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Arjun Menon Over a year ago

Val,Thanks a lot for explaining in detail. Thanks a lot. Let me try this out and will get back to you. One simple doubt, when creating a mapping , you have provided the name your_type , it can be anything rt , also after the mapping is set , do we refer it for some querying afterwards???

Val Over a year ago

your_index and your_type can be freely chosen to describe your data, of course. Then you can use those in the path when querying, i.e. localhost:9200/your_index/_your_type/_search

Arjun Menon Over a year ago

There seems to be a problem i forgot to mention. The indexing is done programmatically, like i am fetching data in cassandra to elastic, so how can i know which column is of which type?

Val Over a year ago

That's something you need to prepare before indexing, of course. You need to know your data in order to make the best of it. if you know what you have in Cassandra, you know what you have to put in ES. Otherwise, you have a "garbage in, garbage out" situation

Val Over a year ago

Try it out and let us know what you find out.

|

Collectives™ on Stack Overflow

Performing filtering in Elasticsearch

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related