Handle duplicate records in Elasticsearch

I think it's all about the document "_id".

If you had a unique "_id" per document, there would not be a problem, as you'll just "update" the document to the same value. You could even set the mapping to not allow an update, if needed.

Your problem is that you do not link the "_id" of the doc to the content of the document (which is fine, for some cases).

I guess a simple solution would be to create your own "my_id" field and set the path of "_id" to it, like here.

The problem then becomes how to create that "my_id" field. I'd use a hash on the document.

An example python snippet for it would be (i'm sure you could find an appropriate ruby plugin):

import hashlib
hash_object = hashlib.sha1(b"Product2,Language2,Date1,$12")
hex_dig = hash_object.hexdigest()
print(hex_dig)

I believe the first part of the solution will be to identify a set of values that if used together will be unique for a document. If not, then there is no way to separate duplicate documents from the genuine ones. For the sake of discussion, let say the four values (Product1,Language1,Date1,$1) define a document. If there is another document with the same set values, then it is a duplicate of the previous document and not a new document.

Consider you have (Product1,Language1,Date1,$1), you can first execute a query which search whether this document already exist in ElasticSearch or not. Something like:

{
"filter": {
    "bool": {
        "must": [
            {
                "term": {
                    "pdtField": "Product1"
                }
            },
            {
                "term": {
                    "langField": "Language1"
                }
            },
            {
                "term": {
                    "dateField": "Date1"
                }
            },
            {
                "term": {
                    "costField": "$1"
                }
            }
        ]
    }
}
}

Take care of the name of the fields used here, according to whatever you are actually using. If this filter result has doc_count != 0 then you need not create a new document for this. Else create a new document with the values in hand.

Alternatively, you can create a document ID using the hash created out of (Product1,Language1,Date1,$1), then use this hash as the _id of the document. First check if any document with this _id exists or not. If it does not exist then create a new document with the values in hand against the _id value generated by hash.

In case, you do not have control over the way an individual document is getting created, then maybe you can try preprocessing your CSV input using the strategy advised above, leave only the needed entries in the CSV and get rid of rest, then carry as usual with the resultant CSV.