ElasticSearch sliced scroll limit (python)

Question

I'm working with a huge (5 million documents) ElasticSearch database and I need to fetch data using sliced scroll in python. Question is: if there is some way to limit (set size param) the sliced scroll? I tried to set size param by [search obj].param(size=500000) or [:500000] but it doesn't seem to work - sliced scroll gives me all documents.

In my script, I'm using sliced scroll with python multiprocessing like in here: https://github.com/elastic/elasticsearch-dsl-py/issues/817

Is there some way to get for example 500000 documents using sliced scroll?

Thanks in advance.

Does this answer your question? In Elasticsearch scan-and-scroll, is there a way to control both the batch size and limit the number of documents in the search? — Aleksi
– Aleksi, Commented Nov 3, 2020 at 8:29

Jogi Michał · Accepted Answer · 2020-01-21 06:09:12Z

1

Answer from github:

"There is no limit on scroll, it always returns all documents. To only get a subset simply stop consuming the iterator after you get the number you wanted to retrieve by using a break statement or similar."

https://github.com/elastic/elasticsearch-dsl-py/issues/817

answered Jan 21, 2020 at 6:09

Jogi Michał

739 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

ElasticSearch sliced scroll limit (python)

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related