Too many open connections with mongo 2.6 + pymongo 2.7.2

[Yahya Poonawala]

I was running Mongo 2.4 with a replicaset which contains 1 primary, 1 secondary and an arbiter. We were using pymongo 2.6 and mongoengine 0.8.2.

Recently, we performed an upgrade to Mongo 2.6, and also upgraded pymongo to 2.7.2 and mongoengine to 0.8.7. This setup worked fine for almost 12 hours, after which we started getting the below error :




[initandlisten] connection refused because too many open connections: 819

The ulimit is 1024 which worked perfectly with Mongo 2.4 and hence we have not increased it. Increasing it might solve the problem temporarily but we will hit it again in the future. Somehow the root cause seems to be the inability of Pymongo to close the connections. Any pointers why this happens?

[Bernie Hackett]

> Somehow the root cause seems to be the inability of Pymongo to close the connections.

PyMongo doesn't close connections. Instead it uses a connection pool, which you can read about here:

https://api.mongodb.com/python/2.7.2/faq.html#how-does-connection-pooling-work-in-pymongo

You need to determine how many instances of MongoClient are being created in your application, then multiply that by the number of threads running in your application. That will tell you how many client connections are being made to MongoDB.

[Yahya Poonawala]

Hi Bernie,

Thanks a lot for replying back. We read and understood the mechanism of connection pooling. However, our situation is peculiar.

With PyMongo 2.7.2, Mongoengine 0.8.7 and MongoDB 2.4.9, the maximum number of open connections observed at peak time is around 250. This has been working perfectly for us for the last few months.

With an upgrade to MongoDB 2.6, with the same amount of traffic and the same code, we were expecting the connections to be in the same range, however, they spiked to the maximum limit of 820 which has never ever happened before with MongoDB 2.4.9. Because of this spike, all further connections were refused and our website started malfunctioning.

So, a couple of questions :

1) With MongoDB 2.6, has the connection handling mechanism changed? Is it OK to see this spike in connections?

2) Will increasing the ulimit solve the problem?

3) Is there a compatibility issue between Pymongo 2.7.2, MongoEngine 0.8.7 and MongoDB 2.6? Would upgrading Pymongo to version 2.8.1 solve the issue?

Thanks a lot in advance.

[Bernie Hackett]

> 1) With MongoDB 2.6, has the connection handling mechanism changed? Is it OK to see this spike in connections?

I don't remember there being any connection handling changes in 2.6, but it's been a few years. Spikes in connections might be expected, depending on the load your server is currently experiencing. When the server can respond to requests quickly a given number of client side threads can share a smaller number of sockets. If the server is under load and slower to respond you may need a socket per thread on the client side. Remember, PyMongo doesn't close sockets once they are open (work to support that is ongoing for the next release), so you want to set max_pool_size to the maximum number of socket connections your server can tolerate from an individual app server.

> 2) Will increasing the ulimit solve the problem?

Maybe. It depends on how large the next load spike is. You probably want to set ulimit for the server to something greater than the maximum number of socket connections the server would need to support from your app servers. You should adjust MongoClient's max_pool_size option at the same time. 

> 3) Is there a compatibility issue between Pymongo 2.7.2, MongoEngine 0.8.7 and MongoDB 2.6? Would upgrading Pymongo to version 2.8.1 solve the issue?

There is no known incompatibility between PyMongo 2.7.2 and MongoDB 2.6. That combination is well tested. I have no idea about MongoEngine. I don't work on the project (it's not a MongoDB, Inc. product).

Please note that end of life was recently announced for MongoDB 2.6 - https://www.mongodb.com/blog/post/mongodb-2-6-end-of-life. PyMongo 2.7.2 is also almost 2 years old.

[Yahya Poonawala]

Hi Bernie,

Thanks for your response. We upgraded to Pymongo 2.8.1 and that seems to have solved the problem. After upgrading to 2.8.1, with the same spike in traffic the connections do not exceed 250-300, which was the case before. Thanks a lot for your help.