What is the efficient way of dynamically creating and reusing multiple mongodb connections in nodejs?

Question

I am using mongodb 2.2.3 and nodejs 0.10.5. I am creating an app where I am connecting to different mongodb instances on different hosts. What is the most efficient way of dynamically creating and reusing the different connections ?

For example I have several hosts in an array hostArray and I want to get all the collecions in it.

function getCollectionNames(hostsArray) {        
    async.map(hostsArray,function(item,callback){
        uri = "mongodb://" + item['user'] + ":" + item['passw'] + "@" + item['host'] + "/" + item['dbname'];
        var mongoClient = new MongoClient.connect(uri,function(err,db){
            if(!err) {
                 db.collectionNames(function(error,collections){
                    if(!err){
                        callback(null,collections);
                    }else{
                        callback(err,null);
                    }
                });
            } else {
                callback(error,null);       
            }
        });
    },function(err,results){
        res.send(results);
    });
}

There are too many open connections I can see when looked at the console of of of the mongodb server, and upon reaching higher numbers, it get crashed.

Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44311 #167 (87 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44312 #168 (88 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44313 #169 (89 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44314 #170 (90 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44315 #171 (91 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44316 #172 (92 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44317 #173 (93 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44318 #174 (94 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44319 #175 (95 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44320 #176 (96 connections now open)
Mon Oct 21 16:34:20 [conn167] end connection 127.0.0.1:44311 (95 connections now open)
Mon Oct 21 16:34:20 [conn169] end connection 127.0.0.1:44313 (94 connections now open)
Mon Oct 21 16:34:20 [conn171] end connection 127.0.0.1:44315 (94 connections now open)
Mon Oct 21 16:34:20 [conn173] end connection 127.0.0.1:44317 (92 connections now open)
Mon Oct 21 16:34:20 [conn175] end connection 127.0.0.1:44319 (92 connections now open)
Mon Oct 21 16:34:20 [conn168] end connection 127.0.0.1:44312 (90 connections now open)
Mon Oct 21 16:34:20 [conn170] end connection 127.0.0.1:44314 (89 connections now open)
Mon Oct 21 16:34:20 [conn172] end connection 127.0.0.1:44316 (88 connections now open)
Mon Oct 21 16:34:20 [conn174] end connection 127.0.0.1:44318 (87 connections now open)
Mon Oct 21 16:34:20 [conn176] end connection 127.0.0.1:44320 (86 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44321 #177 (87 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44322 #178 (88 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44323 #179 (89 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44324 #180 (90 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44325 #181 (91 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44326 #182 (92 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44327 #183 (93 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44328 #184 (94 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44329 #185 (95 connections now open)
Mon Oct 21 16:34:20 [initandlisten] connection accepted from 127.0.0.1:44330 #186 (96 connections now open)

robertklep · Accepted Answer · 2013-10-21 12:49:05Z

1

You can use async.mapLimit to limit the number of parallel requests/connections.

But you're also not closing your connections when you're done with them, which might (also) be the reason why you're running out of connections:

var mongoClient = new MongoClient.connect(uri,function(err,db){
  if (!err) { 
    db.collectionNames(function(error,collections){
      // done, close connection
      db.close();

      // call back with results or error
      if (!error){
        callback(null, collections);
      } else {
        callback(error, null);
      }
    });
  } else {
    callback(err, null);
  }
});

(the code above should also fix your mixing of the err and error variables)

answered Oct 21, 2013 at 12:49

robertklep

205k38 gold badges421 silver badges413 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

What is the efficient way of dynamically creating and reusing multiple mongodb connections in nodejs?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related