Fetch data from elasticsearch in pieces using Node js

Question

I am trying to query a large amount of data from elastic using node js. I have the following query which works perfectly -

client.search({
  "index": 'test',
  "size": 50,
  "type": consts.ReportType.VIOLATION,
  "body": searchQuery
}, callback);

My main goal is to fetch data in pieces, each time to get only 50 results as I'll probably have thousands of docs in elastic. I'm displaying the results in pages at client side (let's say like google search results) and wish to get more data from elastic only if needed.

So, is it possible to maintain some index which will tell elastic which is the last response I already got, and to fetch another 50 results from that point?

Thanks

fmdaboville · Accepted Answer · 2017-09-26 12:40:00Z

1

You can use pagination to perform this.

So, in your case, for example :

client.search({
  "index": 'test',
  "from": 2,
  "size": 50,
  "type": consts.ReportType.VIOLATION,
  "body": searchQuery
}, callback);

answered Sep 26, 2017 at 12:40

fmdaboville

1,9514 gold badges19 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Devaraj C · Accepted Answer · 2017-09-26 12:41:49Z

0

Change the from size for next 50 results

GET /_search { "from" : 0, "size" : 50, "query" : { "term" : { "user" : "kimchy" } } }

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html

Get bulk data Check scroll api https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/api-reference.html#api-scroll

answered Sep 26, 2017 at 12:41

Devaraj C

3371 gold badge3 silver badges11 bronze badges

Comments

Daniele Dellafiore · Accepted Answer · 2019-07-06 19:07:19Z

0

An even better way is use the Elastic Scroll API

answered Jul 6, 2019 at 19:07

Daniele Dellafiore

1,9171 gold badge15 silver badges19 bronze badges

Comments

Alex Nolasco · Accepted Answer · 2021-09-24 16:34:51Z

You can definitely use pagination to achieve this,

 /**
 * 
 * @param {object} elasticClient 
 * @param {{index: string, scroll: string, size: number: body: object}} searchQuery 
 */
async function* getRecords(elasticClient, searchQuery) {
  const response = await elasticClient.search(searchQuery);  
  const responseQueue = [];
  let counter = 0;
  
  responseQueue.push(response);
  while(responseQueue.length) {
    const { body } = responseQueue.shift();

    counter += body.hits.hits.length;
    for(const hit of body.hits.hits) {
      yield hit;
    }

    if (body.hits.total.value === counter) {
      break;
    }
    
    responseQueue.push(
      await elasticClient.scroll({
        scrollId: body._scroll_id,
        scroll: searchQuery.scroll
      })
    )
  }
}

Then your query

const body = { query: {"match_all": {}} } };
for await (const record of getRecords(elasticClient, {index: 'test', scroll: '30s', size: 100, body})) {
    console.log(record);
}

Collectives™ on Stack Overflow

Fetch data from elasticsearch in pieces using Node js

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related