how to do pagination with elasticsearch? from vs scroll API

Question

I'm using elasticsearch as DB to store a large batch of log data. I know there are 2 ways to do pagination:

Use size and from API
Use scroll API

Now I'm using 'from' to do pagination.Get page and size parameters from front end,and at back end(Java)

searchSourceBuilder.size(size);
searchSourceBuilder.from(page * size);

However, if page*size > 10000, an exception thrown from ES.

Can I use scroll API to do pagination?

I know that if I use scroll API, the searchResponse object will return me a _scroll_id, which looks like a base64 string.

How can I control page and size?

It seems Scroll API only support successive page number?

Hey, Can you please mention what approach you took to solve this problem. I also implement similar pagination to show <1,2,3,4,5> button so that use can jump to any specific page. — Deepankar Singh
– Deepankar Singh, Commented Aug 5, 2019 at 12:37

TechnocratSid · Accepted Answer · 2018-06-20 17:40:52Z

5

There is nothing in Elasticsearch which allows direct jump to a specific page as the results have to be collected from different shards. So in your case search_after will be a better option. You can reduce the amount of data returned for the subsequent queries and then once you reach the page which is actually requested get the complete data.

Example: Let's say you have to jump to 99th page then you can reduce the amount of data for all 98th pages request and once you're at 99 you can get the complete data.

answered Jun 20, 2018 at 17:40

TechnocratSid

2,4451 gold badge19 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Mamen Over a year ago

what do you mean reduce amount of data? is it size param

Nur Zico · Accepted Answer · 2017-11-14 05:55:49Z

1

What you said is correct!

You can't do traditional pagination by using scroll API.

I might suggest you to look the Search After API

This might not help you to cover your requirement!

The From / Size default max result size is 10,000.

As it is mentioned here

Note that from + size can not be more than the index.max_result_window index setting which defaults to 10,000

So if you somehow increase the index.max_result_window setting persistent then it will increase the max number of search result! But this might not be the solution but decrease the limitation.

Remember this above solution might hamper the performance of your ES server. Read through all the posts here.

My suggestion is to use Scroll API and change the pagination style

answered Nov 14, 2017 at 5:55

Nur Zico

2,4472 gold badges23 silver badges26 bronze badges

3 Comments

Neilson3r Over a year ago

Thanks for you help,I've learned a lot from your suggestion.I'll consider that.Like you said 'Using scroll API and change the pagination style' ,so ,I'd like to ask what is your pagination style?Now my pagination style is like five paging number,previous page,next page ,first page,last page (|< <1,2,3,4,5> >|)

Nur Zico Over a year ago

The Scroll/Scan/Search API can be use to get searched the data without any limitation. But your have to call for next chunk of result data within your defined time limit. But main problem is you can not think of it to paginate.. Just have to use it like scroll. I mean no way to back/forward in any page. You can use caching in frontend for backward page data.

Neilson3r Over a year ago

Sir,the thing is that only this module use ES as a DB,the others use Maria DB. Besides,all of the pages use pagination.So ,I have to make it looks like the others.I must have page number and size.Now I want to disable 'first page' and 'last page' button.The scroll API may not suit for this situation.

Collectives™ on Stack Overflow

how to do pagination with elasticsearch? from vs scroll API

2 Answers 2

1 Comment

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related