Python and mongoDB connection pool ( pymongo )

Question

I have a web application, and there are thousands of requests every minutes. the following is my python code for mongoDB connection:

Tool.py:

globalconnection = None

def getCollection(name,safe=False,readpref=ReadPreference.PRIMARY):

    global globalconnection
    while globalconnection is None:
            try:
                    if not globalconnection is None:
                            globalconnection.close()
                    globalconnection = Connection('mongodb://host:port',replicaSet='mysetname',safe=False,read_preference=ReadPreference.PRIMARY,network_timeout=30,max_pool_size=1024)
            except Exception as e:
                    globalconnection = None

    request_context.connection = globalconnection

    return request_context.connection["mydb"]["mycoll"]

web.py

@app.route("/test")
def test():
    request_collection = getCollection("user")
    results = request_collection.find()
    for result in results:
        #do something...
        request_collection.save(result)
    request_collection.end_request()

One http request gets the connection through this function,

and the http request calls end_request before the end of the request.

But I found that there are many AutoReconnect errors and over 20000 connections in mongoDB while increasing requests.

Do you have any suggestion?

This is unrelated, but it's more common to say if x is not None:. — Nathan Villaescusa
– Nathan Villaescusa, Commented Nov 22, 2012 at 3:27
not to mention that globalconnection cannot be not None on that line since we just got there past "while globalconnection is None:" — Asya Kamsky
– Asya Kamsky, Commented Nov 22, 2012 at 5:40
Because there would be several requests go into "while" in the same time, I add "if" to reduce the numbers of Connection object — Reiny Song
– Reiny Song, Commented Nov 22, 2012 at 6:08
As Asya said, the while loop will stop when the globalconnection is None, that means inside the while loop globalconnection will always be false. is not a do-while loop — Gianfranco P
– Gianfranco P, Commented Nov 23, 2012 at 12:43

Ilia Ross · Accepted Answer · 2012-11-25 09:08:13Z

2

For auto-reconnection you simply catch the exception, and try to get the connection again: http://api.mongodb.org/python/current/api/pymongo/errors.html
30 secs timeout sounds too long for, try shorter timeout instead?
Increase max number of connection from mongodb (default:20000) http://www.mongodb.org/display/DOCS/Connections

edited Nov 25, 2012 at 9:08

Ilia Ross

13.5k11 gold badges56 silver badges89 bronze badges

answered Nov 25, 2012 at 8:45

robbiecheng

211 bronze badge

Sign up to request clarification or add additional context in comments.

1 Comment

Reiny Song Over a year ago

1. Thank you for your suggestion, and I have implemented it. 2. do you have any suggestion? 3. I have no idea about this because there is a limit of 20,000 connections in mongoDB accroding to this website

Collectives™ on Stack Overflow

Python and mongoDB connection pool ( pymongo )

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related