MongoDb Java APi - Ignore errors in bulk import

Question

Hi i'm importing Documents into mongodb with this function

WriteResult com.mongodb.DBCollection.insert(List<DBObject> list)

some inserts fail because the data violates an index. Is it possible to ignore these errors and continue with the other documents?

Exception in thread "main" com.mongodb.WriteConcernException: { "serverUsed" : "localhost:27017" , "ok" : 1 , "n" : 0 , "err" : "insertDocument :: caused by :: 16755 Can't extract geo keys from object, malformed geometry?: XXXXX , "code" : 16755}
    at com.mongodb.CommandResult.getWriteException(CommandResult.java:90)
    at com.mongodb.CommandResult.getException(CommandResult.java:79)
    at com.mongodb.DBCollectionImpl.translateBulkWriteException(DBCollectionImpl.java:314)
    at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:189)

Christian P · Accepted Answer · 2014-06-24 19:59:52Z

1

If you're using MongoDB 2.6+ you can do an unordered bulk operation. If the error occurs when doing write operations, MongoDB will continue to process the remaining operations:

DBCollection coll = db.getCollection("test");
BulkWriteOperation bulk = coll.initializeUnorderedBulkOperation();
bulk.insert(new BasicDBObject("foo", 1));
bulk.insert(new BasicDBObject("bar", 2));
bulk.execute();

The downside to this approach is that you can't use it if your inserts needs to be executed in an order, but the upside is that the bulk inserts will be executed faster than by doing multiple inserts.

The additional benefit is that you can get the number of inserted documents from the BulkWriteResult object (returned from the execute method).

You can take a look at the Java documentation for bulk inserts here.

Edit: Just to be clear, I don't recommend that you ignore the errors, you should fix your data/inserts.

Edit 2 You can also execute a bulk operation with and set a write concern for the operation:

bulk.execute(new WriteConcern(0, 0, false, false, true));

edited Jun 24, 2014 at 19:59

answered Jun 24, 2014 at 19:34

Christian P

12.3k6 gold badges64 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

wutzebaer Over a year ago

thanks i'll try that... the order doesnt matter, could you say if my own answer is correct too?

wutzebaer Over a year ago

mh the BulkWriteOperation failed with the same exception, i'm trying it with the unacknowledged writes again, since i know how many objects it sould be in the end

sezanzeb Over a year ago

I can confirm that an UnorderedBulkOp ignores errors and continues to do the remaining operations in mongodb 4.4. .execute() will just an error in node that can be catched, while all other of my insert operations were completed. Using an OrderedBulkOp the operations were stopped after the first error.

wutzebaer · Accepted Answer · 2014-06-24 19:33:56Z

0

What worked for me:

wayCollection.insert(ways, new WriteConcern(0, 0, false, false, true));

answered Jun 24, 2014 at 19:33

wutzebaer

15k20 gold badges107 silver badges180 bronze badges

3 Comments

Christian P Over a year ago

This will "work". But by doing inserts this way you have no way of knowing if MongoDB actually inserted anything.

wutzebaer Over a year ago

will it be faster slower or equal execute than the bulk method?

Christian P Over a year ago

The unacknowledged writes are pretty fast, but I never compared the speed between bulk inserts and unacknowledged writes, so I can't really tell you for sure.

Collectives™ on Stack Overflow

MongoDb Java APi - Ignore errors in bulk import

2 Answers 2

3 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related