Elasticsearch Aggregation Performance

Question

In our elastic index we have daily news documents and we are running aggregations for these documents. But, after 2 consecutive run elasticsearch returns not enough memory exception. Now we have increased the heap size for elastic but is there any solution other then increasing the ram for elastic?

The field properties which is used for aggregations;

"detail_stop": {
          "type": "string",
          "store": true,
          "analyzer": "stop_analyzer"
        }

The query for aggregation;

 {
        "from": 0,
        "size": 5000,
        "query": {
            "bool": {
                "must": [
                    {
                        "range": {
                            "date": {
                                "gte": "now-0d/d"
                            }
                        }
                    }

                ]
            }
        },
        "aggs": {
            "words": {
                "terms": {
                    "size": 5000,
                    "field": detail_stop,
                    "min_doc_count": 3
                }
            }
        }
    }

Currently we have an elastic cluster with 1 node(8core 2.5ghz , 32gb) and ES_HEAP_SIZE = 16g(elastic have 16gb memory). How can we reduce usage of memory and increase performance?

bittusarkar · Accepted Answer · 2016-01-05 11:44:58Z

0

If you are using Field Cache, then you have to provide more memory for aggregations to work. You can try out the tips mentioned here to monitor and limit memory consumption.

With newer versions of Elasticsearch, doc values are becoming the norm instead of field data cache. They are slightly slower compared to field data but will solve the memory problem. Mind you, to use doc values you will have to re-index all your data again. Read more about it here.

answered Jan 5, 2016 at 11:44

bittusarkar

6,3674 gold badges36 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Fatih Aktepe Over a year ago

I know that I have to re-index when I use doc_val but doc values do not support analyzed fields so that is not an option. Thanks for the answer.

Fatih Aktepe Over a year ago

I've checked the link you mentioned above a dozen times and already made the tunning about heapspace, cpu usage etc. But couldn't get useful result so far. I'm asking for other solutions if exist.

bittusarkar Over a year ago

There aren't any. If you are using field cache, you have to make space in the heap memory.

Collectives™ on Stack Overflow

Elasticsearch Aggregation Performance

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related