MongoDB Too Many Connections w/ PECL Driver

[George Milonas]

We have a 16 server API Front end, Connecting to Two, Single Master, Dual Read, MongoDB Clusters.

We are using PHP-FPM with NGINX in the API servers.

Mongo Driver is latest, 1.4.5.

The issue we are running into is, is that we are maxxing out every MongoDB server, with over 20K connections to each, causing the servers to crash.

Does the Mongo Driver really have pooled connections? And are they shared amonst PHP-FPM Child Processes?

What workaround can I use?

[Hannes Magnusson]

Due to the nature of the PHP language, we don't use connection pooling but persistent connections instead.

This means that each "PHP-FPM Child process" has its own connection to the ReplicaSet, and does not share it with its siblings. The connection is instead reused (as its persisted) in the subsequent requests.

If you are hitting the 20k connection limit then it seems like you have more then 1200 PHP-FPM children on all of the 16 frontend servers.

Currently, the only workarounds would be to

a) Limit your PHP-FPM workers

This could involve creating separate pools of PHP-FPM workers to isolate things. A dedicate pool of workers that hit MongoDB, another dedicate pool of workers that do xyz, ...

b) Connect to the ReplicaSet via multiple mongos'

For example run mongos on each frontend, and then connect to that mongos via localhost/UDS as mongos does connection pooling.

-Hannes

[George Milonas]

How would you go about setting up mongos for this type of situation.

George

[Hannes Magnusson]

You can follow the normal Sharding documentations like

http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/

With two exceptions:

- You don't actually want to shard your _data_, just add one shard to your "cluster" (e.g.

  do not run the sh.shardCollection() or sh.enableSharding("<database>")), you can

  short-circuit the tutorials at this point

- mongos needs to be started up with: --setParameter

  "releaseConnectionsAfterResponse=true"

You could then have as many mongos' as you want, say, 1 per frontend server.

To make it easier for you in the future when you do actually want to shard your

data I strongly recommend you follow the guidelines of having 3x config servers.

To quickly workaround your current problem you "could" add just one config server,

or even make your current ReplicaSet be the config server too...

I do however not recommend that as the system is simply not designed for that and

could cause you issues in the future.

Assuming your MongoDB Replicaset is called "RSNAME", where one of the ReplicaSet

members is running on mongodb1.example.com:27017 the correct procedure would be

something like:

  - Startup 3x config servers on 3 different servers

    configServer#1 (hostname cfg1server.example.com):

      $ mkdir /data/configdb

      $ mongod --configsvr --dbpath /data/configdb --port 27019

    configServer#2 (hostname cfg2server.example.com):

      $ mkdir /data/configdb

      $ mongod --configsvr --dbpath /data/configdb --port 27019

    configServer#3 (hostname cfg3server.example.com):

      $ mkdir /data/configdb

      $ mongod --configsvr --dbpath /data/configdb --port 27019

  - Startup ONE mongos

    mongosServer#1 (hostname frontend1.example.com):

      $ mongos --setParameter "releaseConnectionsAfterResponse=true" --configdb cfg1server.example.com:27019,cfg2server.example.com:27019,cfg3server.example.com:27019

  - Configure one shard, without any sharded data

    mongoServer#2 (hostname frontend2.example.com):

      $ mongo --host frontend1.example.com --port 27017

      > sh.addShard( "RSNAME/mongodb1.example.com:27017" )

      > exit

  - Startup mongos' wherever you choose (one per frontend server? dedicated servers?)

    mongoServer#2 (hostname frontend2.example.com):

        $ mongos --setParameter "releaseConnectionsAfterResponse=true" --configdb cfg1server.example.com:27019,cfg2server.example.com:27019,cfg3server.example.com:27019

    mongoServer#3 (hostname frontend3.example.com):

        $ mongos --setParameter "releaseConnectionsAfterResponse=true" --configdb cfg1server.example.com:27019,cfg2server.example.com:27019,cfg3server.example.com:27019

Now you have multiple mongos running, with one shard, and can start connecting to them

from your application. To do so you have to modify your connection string:

- Remove the "replicaSet=RSNAME" option from your connection string/arguments

- Change your seedlist from connecting to the ReplicaSet hosts, to connect to the mongos'

There should not be anything else you would have to change in your application,

unless you are somehow relying on connection counts, specific statistics, or

otherwise not-recommended strange procedures.

-Hannes

[Hannes Magnusson]

I should probably mention the following caveats:
http://docs.mongodb.org/manual/reference/parameters/#param.releaseConnectionsAfterResponse

So if you are running 2.2 or 2.4 and using the deprecated "Mongo"
object (rather then the MongoClient), or otherwise doing
"fire-and-forget" writes and manual "getLastError" calls.. Things may
go far south.

The driver makes this scenario very difficult to reach though, so you
have to do a fair amount of mean things intentionally for it to go bad
:)


Also, in MongoDB 2.6 (which is slated to be released "any day now"), I
believe this option is enabled by default so you do not need to pass
it in explicitly.
However, you do need to be running the latest driver for things to
work correctly (e.g. 1.5.0 or later release of the PHP MongoDB
driver), in which case we use a newer write API which does not use
"getLastError" at all, so this point becomes moot.


-Hannes

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "mongodb-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/mongodb-user/LPIHhPTOmJU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/4d5ed8a4-c73d-43e7-9ca1-2c20f88c78d3%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.