How to retrieve elasticsearch data from index based on timestamp?

Question

I want to retrieve data from elasticsearch based on timestamp. The timestamp is in epoch_millis and I tried to retrieve the data like this:

{
  "query": {
    "bool": {
      "must":[ 
              {
                "range": {
                  "TimeStamp": {
                    "gte": "1632844180",
                    "lte": "1635436180"
                  }
                }
              }
      ]
    }
  },
  "size": 10
}

But the response is this:

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

How can I retrieve data for a given period of time from a certain index?

The data looks like this:


    {
        "_index" : "my-index",
        "_type" : "_doc",
        "_id" : "zWpMNXcBTeKmGB84eksSD",
        "_score" : 1.0,
        "_source" : {
          "Source" : "Market",
          "Category" : "electronics",
          "Value" : 20,
          "Price" : 45.6468,
          "Currency" : "EUR",
          "TimeStamp" : 1611506922000        }

Also, the result has 10.000 hits when using the _search on the index. How could I access other entries? (more than 10.000 results) and to be able to choose the desired timestamp interval.

Tan Vu · Accepted Answer · 2021-11-01 23:04:39Z

2

For your first question, assume that you have the mappings like this:

{
    "mappings": {
        "properties": {
            "Source": {
                "type": "keyword"
            },
            "Category": {
                "type": "keyword"
            },
            "Value": {
                "type": "integer"
            },
            "Price": {
                "type": "float"
            },
            "Currency": {
                "type": "keyword"
            },
            "TimeStamp": {
                "type": "date"
            }
        }
    }
}

Then I indexed 2 sample documents (1 is yours above, but the timestamp is definitely not in your range):

[{
    "Source": "Market",
    "Category": "electronics",
    "Value": 30,
    "Price": 55.6468,
    "Currency": "EUR",
    "TimeStamp": 1633844180000
},
{
    "Source": "Market",
    "Category": "electronics",
    "Value": 20,
    "Price": 45.6468,
    "Currency": "EUR",
    "TimeStamp": 1611506922000
}]

If you really need to query using the range above, you will first need to convert your TimeStamp field to seconds (/1000), then query based on that field:

{
    "runtime_mappings": {
    "secondTimeStamp": {
      "type": "long",
      "script": "emit(doc['TimeStamp'].value.millis/1000);"
    }
  },
    "query": {
        "bool": {
            "must": [
                {
                    "range": {
                        "secondTimeStamp": {
                            "gte": 1632844180,
                            "lte": 1635436180
                        }
                    }
                }
            ]
        }
    },
    "size": 10
}

Then you will get the first document.

About your second question, by default, Elasticsearch's max_result_window is only 10000. You can increase this limit by updating the settings, but it will increase the memory usage.

PUT /index/_settings

{
   "index.max_result_window": 999999
}

You should use the search_after API instead.

answered Nov 1, 2021 at 23:04

Tan Vu

2991 silver badge5 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Ian Kurtis Over a year ago

I tried the first query you gave but the error is

{     "error": {         "root_cause": [             {                 "type": "parsing_exception",                 "reason": "Unknown key for a START_OBJECT in [runtime_mappings].",                 "line": 2,                 "col": 25             }         ],         "type": "parsing_exception",         "reason": "Unknown key for a START_OBJECT in [runtime_mappings].",         "line": 2,         "col": 25     },     "status": 400 }

Ian Kurtis Over a year ago

Also, the timestamp mapping format is like this ``` "TimeStamp": { "type": "date", "format": "epoch_millis" } ```

Tan Vu Over a year ago

which version of Elasticsearch are you using? I have tried that query and it's fine from my side. My query only works from version 7.12 upward

Tan Vu Over a year ago

If you are using older version of elasticsearch, why don't you just multiply your range by 1000 by adding 3 zeros in the timestamp in the range query? i.e., 1632844180000 instead of 1632844180. So you don't need that timestamp in second calculation.

Ian Kurtis Over a year ago

It is version 7.5.0. I managed to get the results by using the 000 at the end, thanks

Collectives™ on Stack Overflow

How to retrieve elasticsearch data from index based on timestamp?

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related