Random document in ElasticSearch

Question

Is there a way to get a truly random sample from an elasticsearch index? i.e. a query that retrieves any document from the index with probability 1/N (where N is the number of documents currently indexed)?

And as a follow-up question: if all documents have some numeric field s, is there a way to get a document through weighted random sampling, i.e. where the probability to get document i with value s_i is equal to s_i / sum(s_j for j in index)?

Koen. · Accepted Answer · 2017-02-14 19:30:37Z

78

I know it is an old question, but now it is possible to use random_score, with the following search query:

{
   "size": 1,
   "query": {
      "function_score": {
         "functions": [
            {
               "random_score": {
                  "seed": "1477072619038"
               }
            }
         ]
      }
   }
}

For me it is very fast with about 2 million documents.

I use current timestamp as seed, but you can use anything you like. The best is if you use the same seed, you will get the same results. So you can use your user's session id as seed and all users will have different order.

edited Feb 14, 2017 at 19:30

Koen.

27.3k7 gold badges87 silver badges80 bronze badges

answered Oct 31, 2016 at 23:50

Adam Wallner

2,22225 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Ian Kemp - SO dead by AI greed Over a year ago

For newer versions of ES: elastic.co/guide/en/elasticsearch/reference/5.4/…

Robin Zimmerman Over a year ago

NOTE: by default with the newer version of ES if you don't provide a seed the current timestamp is used. ALSO, I've found that if you let ES use it's own seed (current timestamp) the query is 20x faster (this is on a very large cluster, 6 seconds vs 150 seconds)

Alcanzar · Accepted Answer · 2014-09-17 13:46:33Z

7

The only way I know of to get random documents from an index (at least in versions <= 1.3.1) is to use a script:

sort: {
  _script: {
    script: "Math.random() * 200000",
    type: "number",
    params: {},
    order: "asc"
 }
}

You can use that script to make some weighting based on some field of the record.

It's possible that in the future they might add something more complicated, but you'd likely have to request that from the ES team.

answered Sep 17, 2014 at 13:46

Alcanzar

17.2k7 gold badges45 silver badges60 bronze badges

2 Comments

sudeepdino008 Over a year ago

Can't use seed with this. n documents will be grouped and having same score where n is the shard size.

Praneeth Kumar Over a year ago

does painless script Math.random() return a value between 0 and 1 inclusive?

Hassaan · Accepted Answer · 2017-06-21 11:24:58Z

6

You can use random_score with a function_score query.

{
    "size":1,
    "query": {
        "function_score": {
            "functions": [
                {
                    "random_score":  {
                        "seed": 11
                    }
                }
            ],
            "score_mode": "sum",
        }
    }
}

The bad part is that this will apply a random score to every document, sort the documents, and then return the first one. I don't know of anything that is smart enough to just pick a random document.

edited Jun 21, 2017 at 11:24

Hassaan

7,7407 gold badges34 silver badges53 bronze badges

answered Oct 23, 2015 at 20:35

JnBrymn

25.5k31 gold badges112 silver badges153 bronze badges

Comments

Eran Peled · Accepted Answer · 2020-10-13 07:36:50Z

5

NEST Way :

var result = _elastic.Search<dynamic>(s => s
        .Query(q => q
        .FunctionScore(fs => fs.Functions(f => f.RandomScore())
        .Query(fq => fq.MatchAll()))));

raw query way :

 GET index-name/_search
    "size": 1,
    "query": {
        "function_score": {
                "query" : { "match_all": {} },
               "random_score": {}
        }
    }
}

answered Oct 13, 2020 at 7:36

Eran Peled

9298 silver badges6 bronze badges

Comments

nycdatawrangler · Accepted Answer · 2015-06-11 21:51:15Z

2

You can use random_score to randomly order responses or retrieve a document with roughly 1/N probability.

Additional notes:

https://github.com/elastic/elasticsearch/issues/1170 https://github.com/elastic/elasticsearch/issues/7783

answered Jun 11, 2015 at 21:51

nycdatawrangler

7001 gold badge5 silver badges6 bronze badges

Collectives™ on Stack Overflow

Random document in ElasticSearch

5 Answers 5

2 Comments

2 Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

2 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related