MongoDB Bulk Insert Operation instead of For-Loop

Question

Let's consider a social network to be built by NodeJS and MongoDB.
So, if a user creates a new post, it should be saved to his/her followers feed.

The straightforward implementation of this operation as follow (simplified):

var newPost="New Post";
//get list of followers of user 1
var listOfFollowers = followersCollection.find({u:"1"}); 

for(var i=0;i<listOfFollowers.length;i++){
    var followerId = listOfFollowers[i]._id;
    //insert new post of user 1 to every follower feed
    feedsCollection.insertOne(
       {ownerId:followerId,authorId:"1",content:newPost}
    ); 
}

This, of course, has very bad performance in case of big numbers in followers count. How can do this with a single fast performing MongoDB query?

A. Todkar · Accepted Answer · 2019-08-29 08:38:34Z

4

MongoDB provides bulk document insert functionality, check out this link - https://docs.mongodb.com/manual/reference/method/db.collection.initializeUnorderedBulkOp/

db.collection.initializeUnorderedBulkOp() It creates an unordered list of operations and mongodb executes this list in parallel, so it's fast and you don't have to take extra care of performance as mongo handles it.

For ordered insertions, you can use db.collection.initializeOrderedBulkOp().

e.g.

var newPost="New Post";
var bulk = db.followersCollection.initializeUnorderedBulkOp();
//get list of followers of user 1
var listOfFollowers = followersCollection.find({u:"1"}); 

for(var i=0;i<listOfFollowers.length;i++){
    var followerId = listOfFollowers[i]._id;
    //insert new post of user 1 to every follower feed
    bulk.insert( {ownerId:followerId,authorId:"1",content:newPost});
}
bulk.execute();

If you are using Mongoose then checkout Mongoose docs for the same. In the above example, I have just trying to explain how you can do it using plain MongoDB.

answered Aug 29, 2019 at 8:38

A. Todkar

4031 gold badge4 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

erfannf2 Over a year ago

Thank you. Although other answers are good to choose but this is the perfect answer. For the record, MongoDB NodeJS driver also supports UnorderedBulkOp which I use currently.

Aikansh Mann · Accepted Answer · 2019-08-29 07:46:42Z

2

Insert Many Read this document I think you will get the answer

answered Aug 29, 2019 at 7:46

Aikansh Mann

6666 silver badges12 bronze badges

Comments

Nilupul Sandeepa · Accepted Answer · 2019-08-29 09:15:21Z

1

Check this:

var newPost="New Post";

//Object Array
var collection = []

//get list of followers of user 1
var listOfFollowers = followersCollection.find({u:"1"}); 

for(var i=0;i<listOfFollowers.length;i++){
    var followerId = listOfFollowers[i]._id;
    collection.push({ownerId:followerId,authorId:"1",content:newPost})
}
feedsCollection.insert(collection); //Or use insertMany()

You can create your object array and insert it at once. Check documentation :- https://docs.mongodb.com/manual/reference/method/db.collection.insert/#insert-multiple-documents

Even though this is a simple answer for your question, If the collection array has a large number of elements, there still might be performance issues. So the best way to handle this is using triggers. https://docs.mongodb.com/stitch/triggers/

edited Aug 29, 2019 at 9:15

answered Aug 29, 2019 at 8:46

Nilupul Sandeepa

7687 silver badges20 bronze badges

1 Comment

erfannf2 Over a year ago

Thank you for your answer. As it is good enough but I think the UnorderedBulkOp thing is much better since it performs ops in parallel.

Collectives™ on Stack Overflow

MongoDB Bulk Insert Operation instead of For-Loop

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related