18

I am trying to profile the performance of my node.js app running mongodb currently configured to use 50 connection pools. Using Blazemeter I have been trying to do a test that sends 1000 simulated users to my endpoint. Running on a smaller amazon ec2 instance (4 CPUs and 7.5 GB of memory the performance seemed to be CPU bound). When I started moving up to a larger machine with at least 8 CPUs running in pm2 cluster mode, it seems that mongodb is running out of memory. When the the test gets up to about 300-500 simulated users the mongo process will fail:

I.E. I get an error from all db queries and I saw the following message when I try to launch the mongo shell:

2015-10-26T23:34:56.657+0000 warning: Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
2015-10-26T23:34:56.658+0000 Error: couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed at src/mongo/shell/mongo.js:146
exception: connect failed

The first time this happened, I also found the following error in the mongo log:

exception in initAndListen: 10309 Unable to create/open lock file: /var/lib/mongodb/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating

In following tests I only saw the above behavior but did not see any errors in the mongo log.

When running these tests mongo usually end up using about 80 percent of the systems memory before failing.

Here are the only mongo queries used by this endpoint:

    utility.getNextId(db, "projects", function(err, counter) {
    var pid = counter.seq;
    var newProject = {
        name: projectName,
        path: "/projects/"+user.name+"/"+projectName,
        created: utility.now(),
        modified: utility.now(),
        uid: user.uid,
        pid: pid,
        ip: ip
    }

    // Hierarchy of cloned projects
    if( parentPid )
        newProject.parent = parentPid;

    db.collection("projects").insert(newProject, function(err, inserted) {
        db.collection("users").update(
            {uid: user.uid},
            {$addToSet: { projects:pid }},
            function(err,_) {
                callback(err, newProject);
            }
        );
    });
});
};

exports.getNextId = function(db, name, callback) {
db.collection("counters").findAndModify(
    {_id:name},
    [["_id","asc"]],
    {$inc : {"seq":1}},
    {upsert:true, new:true},
    function(err, object) {
        callback(err, object);
    }
);
};

Most of this testing was done on an amazon ec2 m4.4xlarge (16 cpus and 64GB of ram).

Is a connection pool size of 50 to large for a machine with 64gb of RAM? I would think not. Is there a good way to calculate the amount of memory needed for n connection pools? Is my issue with the queries I am making?

EDIT: Here is a screenshot showing the mongostat right as mongo collapsed on the amazon ec2 m4.4xlarge with 16cpus and 64GB of ram

enter image description here

We create the mongo DB at the top with many other requires:

var mongo = require("mongodb");
var flash = require("connect-flash");
var session = require("express-session");
var auth = require("basic-auth");
var admin = require("./admin.js");

var mongoServer = new mongo.Server("localhost", 27017, {auto_recconnect:true, poolSize: 50});
var db = new mongo.Db("aqo", mongoServer, {safe:true});
var busboy = require('connect-busboy');

db.open(function(err,db) {
    if(err)
        console.warn("mongo-open err:",err);
});

EDIT: Here are my indexes for the users collection:

[
{
    "v" : 1,
    "key" : {
        "_id" : 1
    },
    "name" : "_id_",
    "ns" : "aqo.users"
},
{
    "v" : 1,
    "key" : {
        "uid" : 1
    },
    "name" : "uid_1",
    "ns" : "aqo.users"
}
]
Improve this question
7
  • connection pool of 50 is quite small/reasonable. I have a feeling, you may not be using your connection pool but re-creating a new client instance and 50 new connections with it for each request. Could you please post your app.js code or wherever you are creating your mongo client instance. Commented Nov 1, 2015 at 1:56
  • Before that, run the mongostat command line utility and under the conn tab, you should be able to see how many connections are open. It should NOT be more than 50. Also, provide a screenshot of that, in case, my initial hypothesis is incorrect. Commented Nov 1, 2015 at 2:03
  • @RahatMahbub thank you very much for your comments. I posted a screenshot of mongostat running on the live server without any load testing (if it would be helpful to get a mongostat on the the test server under extreme load let me know). What I notice from the mongo stat is that 201 connection are open, this could be because we are running this server in cluster mode using pm2 so each node instance fires up 50 connection pools, does this seem right (not sure about the extra 1). On my test server with 64GB of ram we were using cluster mode on 16 cpus. Commented Nov 1, 2015 at 17:27
  • I also posted the part of app.js where mongo is loaded. Commented Nov 1, 2015 at 17:27
  • the extra 1 connection is because of mongostat itself. Having 200 connections is also reasonable. However, over 1000 is not. It is also using very little resource as you can see. Now, can you provide another mongostat screenshot under load. It should quite clarify what's happening here. Commented Nov 1, 2015 at 21:53

3 Answers 3

Reset to default
9
+25

Although pool size of 50 isn't large for a machine with 64GB RAM, 800 certainly is. That's because you have 16 instances of your node process running with 50 each. The default number for max connections is 80% of the available file descriptors. If you are using Linux, the default is 1024 so you already have nearly max connections opened. Furthermore, each connection has an overhead of ~10MB, so you are using around 8GB for connections alone. This is obviously not ideal.

Ideally, You are supposed to reuse those connections in your connection pool as much as possible. So, start off your load testing with setting the poolSize to the default of 5. (i.e. 16*5=80 actually). You can trust pm2 to nicely handle the load in a round-robin fashion and that poolsize of 5 for each instance should be perfectly fine and give you an optimal performance. In case, 5 isn't enough, go up a little bit until you find something suitable.

Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

Thanks! Running the test with the default pool size of 5, mongo still runs out of memory. This time it happened at about 750 users. Is this just unavoidable at a certain point?
That's quite weird. 750 users is still almost negligible.
How much data do you have? How big are your indices?
For the two collections being used for this endpoing: projects has a totalIndexSize() of 2.5MB and users has .29MB. The total dataSize according to dbStats() is 15MB
The "each connection has an overhead of ~10MB" comment is very outdated. Looks like it was reduced to around 1MB around July 2011 for MongoDB 2.0
|
4

You've got a large number of queued reads and not many reads successfully occurring. My guess is that the uid in the users collection doesn't have an index (as you're querying by _id elsewhere and Mongo will automatically index that).

It also looks like you're running Mongo as a standalone server, it's designed to be run as replica sets, I'm not sure what problems this will cause but I'm guessing they haven't done much testing of a single Mongo instance working on its own so there could be memory problems.

Improve this answer

10 Comments

The users table is also indexed on uid (I added the index to my post). I see what you mean about the large number of queued reads I'm surprised that would be the case with 801 connection pools.
Those are queued on the mongo server so they've already been sent over the network (via one of the 801 pooled connections) rather than being queued to be picked up by a connection.
Wait a minute, you're querying in the counters collection for a document with _id "projects". I take it you've set the _id when creating the counter collection?
At a more fundamental level, there's a reason that Mongo uses ObjectIds instead of having incremental counters, do you really need to have a counter here or could you save an extra trip to mongo by using an ObjectId?
I agree that it doesn't necessarily follow from the information I've given but MongoDB does suck at scaling up. I haven't got any good sources that go into scaling Mongo up but everywhere that they mention performance they point out that Mongo was designed to run on commodity hardware and adding more CPUs or faster disks won't normally help. When 10gen (now MongoDB Inc) came into where I used to work they saw the hardware we were running it on and told us to run something like 5 instances on each to get the best performance. The comments I've made about testing are purely supposition.
|
0

To calculate this, you should know 2 things:

  1. Each connection consumes about 1MB of RAM.
  2. The connection pool is on a per-mongod/mongos basis, so when connecting to a 3-member replica there will be three connection pools (one per mongod), each with a maxPoolSize. Additionally, there is a required monitoring connection for each node as well, so you end up with (maxPoolSize+1)*number_of_nodes TCP connections.

Example

Let's say you have Atlas free cluster with 3 replica sets.

poolSize = N
numberOfNodes = 3

numberOfConnections = (N+1)*3

memoryRequired = (N+1)*3*1MB
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.