How to use Scroll while passing raw json Query to ElasticSearch using NEST

Question

var query = @"
        {
          ""query"": {
            ""match_all"": { }                    
          }
        }";

    Func<SearchRequestParameters, SearchRequestParameters> requestParameters = a => 
                                                a.SearchType(SearchType.Scan).Scroll(TimeSpan.FromSeconds(60));

        var searchResult = await client.LowLevel.SearchAsync<SearchResponse<T>>(indexName, mappingName, query , requestParameters)

if (searchResult.Body.IsValid)
            {
                var scrollNo = 0;                
                var results = await client.ScrollAsync<T>("10s", searchResult.Body.ScrollId);

                while (results.Documents.Any())
                {
                    documents.AddRange(results.Documents);
                    scrollNo++;

                    results = await client.ScrollAsync<T>("10s", results.ScrollId);

                return new Customresponse<T>
                {
                    Documents = documents,
                    total = result.Body.Total
                };
            }

Would like to pull all data using scroll while passing raw json query. but scroll is not working properly while passing json raw query. Can anyone help on this ?.

Your query above is missing a scroll. What have you tried so far? — Russ Cam
– Russ Cam, Commented Jan 15, 2017 at 9:08
I tried as below. Func<SearchRequestParameters, SearchRequestParameters> requestParameters = a => a.SearchType(SearchType.Scan).Scroll(TimeSpan.FromSeconds(60)); — Sasi
– Sasi, Commented Jan 16, 2017 at 19:16
I tried for passing this request and while loop with scroll API but not working as expected. Func<SearchRequestParameters, SearchRequestParameters> requestParameters = a => a.SearchType(SearchType.Scan).Scroll(TimeSpan.FromSeconds(60)); — Sasi
– Sasi, Commented Jan 16, 2017 at 19:24
Please could you edit your question to add any additional details? — Russ Cam
– Russ Cam, Commented Jan 16, 2017 at 21:07
@RussCam, I updated my question with more details, can you suggest me if you have any?. — Sasi
– Sasi, Commented Jan 31, 2017 at 18:23

Russ Cam · Accepted Answer · 2017-01-31 22:01:54Z

1

Your example is nearly there but not quite; you're missing a closing brace for the while loop to collect all documents before returning the custom response.

Here's an example I just ran on the Stackoverflow data set, to return all questions tagged with nest

private IElasticClient _client;

void Main()
{
    var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
    var defaultIndex = "default-index";
    var connectionSettings = new ConnectionSettings(pool);

    _client = new ElasticClient(connectionSettings);

    var query = @"
    {
    ""query"": {
      ""term"": {
        ""tags"": {
          ""value"": ""nest""
          }
        }
      }
    }";

    var result = RunScrollAsync(query).Result.Dump();
}

private async Task<Customresponse<Question>> RunScrollAsync(string query)
{
    var scrollTime = "10s";

    // omit the .SearchType(Scan) which is deprecated. Not 
    // specifying means the first response contains the first set
    // of documents
    var esResponse = await _client.LowLevel.SearchAsync<SearchResponse<Question>>(
        "posts", 
        "question", 
        query, r => r.Scroll(TimeSpan.FromSeconds(10))).ConfigureAwait(false);

    if (esResponse.Body.IsValid && esResponse.Body.Documents.Any())
    {
        // assume you have less than 2,147,483,647 documents to return?
        var documents = new List<Question>((int)esResponse.Body.Total);
        documents.AddRange(esResponse.Body.Documents);

        var scrollNo = 0;

        var response = await _client.ScrollAsync<Question>(scrollTime, esResponse.Body.ScrollId).ConfigureAwait(false);;

        // keep scrolling until no more documents are returned
        while (response.Documents.Any())
        {
            documents.AddRange(response.Documents);
            scrollNo++;

            response = await _client.ScrollAsync<Question>(scrollTime, response.ScrollId).ConfigureAwait(false);;
        }

        return new Customresponse<Question>
        {
            Documents = documents,
            total = response.Total
        };
    }

    // return an empty result. 
    // Or throw an exception, or log - whatever you need to do
    return new Customresponse<Question>
    {
        Documents = Enumerable.Empty<Question>(),
        total = 0
    };
}

public class Customresponse<T>
{
    public IEnumerable<T> Documents { get; set; }

    public long total { get; set; }
}

This returns all 342 questions, with a total of 342 (Data set is from June 2016).

answered Jan 31, 2017 at 22:01

Russ Cam

126k34 gold badges211 silver badges275 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Sasi Over a year ago

can we get batch by batch using scroll like if I have 100k data I want to get 25k on each request using scroll on demand. like after we get 25k and when user clicks again to get next 25k data etc.?.

Russ Cam Over a year ago

the best place to start is the Scroll API documentation: elastic.co/guide/en/elasticsearch/reference/current/… Scroll keeps a cursor (term used loosely here) open on the cluster after which time it is closed. For doing pagination, used from and size, and for deep pagination, look at search_after

Sasi Over a year ago

Hi Russ, on the scrollsync I am getting 'Search Context missing exception', it was working fine but now am getting when I executed the above code. any idea ??.. # Response: {"error":{"root_cause":[{"type":"search_context_missing_exception","reason":"No search context found for id [56162426]"}

Collectives™ on Stack Overflow

How to use Scroll while passing raw json Query to ElasticSearch using NEST

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related