High connection count on configserver

[inte...@googlemail.com]

Hi,

we received an alert from the cloud manager "Connections has gone above 250" on one of our configs.

My first suspicion was that our little PHP tracking scripts doesn't do a clean shutdown of the MongoDB connection they open. But the connection count on our mongos is regularly at about six connections.

So I connected to the config and executed db.currentOp(true) and noticed that a high amount of threads are flagged with active: false.

I also executed the following command within the bash

sudo lsof | grep mongoDB | grep TCP

which returns tons of entries.

Is this a normal situation on a cluster with four nodes? Because a

db.serverStatus().connections

gave me the following result.

{ "current" : 253, "available" : 50947, "totalCreated" : NumberLong(818) }

253 from 50947 seems ok for me.

Just forgot to mention our used versions:

The complete cluster operates wih mongodb version 3.0.9. We exclusively use PHP to connect to the MongoDb.

PHP Version 5.5.9

PHP MongoDB Driver 1.6.11 with the following settings:

mongo

MongoDB Support => enabled

Version => 1.6.11

Streams Support => enabled

SSL Support => enabled

                   Supported Authentication Mechanisms

MONGODB-CR => enabled

SCRAM-SHA-1 => enabled

MONGODB-X509 => enabled

GSSAPI (Kerberos) => disabled

PLAIN => disabled

Directive => Local Value => Master Value

mongo.allow_empty_keys => 0 => 0

mongo.chunk_size => 261120 => 261120

mongo.cmd => $ => $

mongo.default_host => localhost => localhost

mongo.default_port => 27017 => 27017

mongo.is_master_interval => 15 => 15

mongo.long_as_object => 0 => 0

mongo.native_long => 1 => 1

mongo.ping_interval => 5 => 5

Thanks in advance!

Best regards - Marcus

[Kevin Adistambha]

Hi Marcus,

we received an alert from the cloud manager “Connections has gone above 250” on one of our configs.

The Connections has gone above 250 alert is not set by default in Cloud Manager, and therefore was manually set by a person with access to the group. You would be able to see this in your group’s Alert setting. Please see the Manage Alerts page for more detail.

Best regards,
Kevin

​

[inte...@googlemail.com]

Hi Kevin,

sorry for the late response, I was on vacation for the last two weeks ;)

The Connections has gone above 250 alert is not set by default in Cloud Manager, and therefore was manually set by a person with access to the group. You would be able to see this in your group’s Alert setting. Please see the Manage Alerts page for more detail.

Okay - I didn't know that this is no cloud manager default alert.

But the strange thing here is that the connections on the configserver are constantly increasing over the time. Could this be possible due to bad software (our) / driver (old php driver) design? The software / driver connects only to a MongoS and not directly to a configserver?

Best regards,

Marcus

[Kevin Adistambha]

Hi Marcus,

But the strange thing here is that the connections on the configserver are constantly increasing over the time. Could this be possible due to bad software (our) / driver (old php driver) design? The software / driver connects only to a MongoS and not directly to a configserver?

The drivers do not connect to the config servers directly. It is normal for the config servers to have relatively high connection count, since they act as a “map” of the data distribution of your sharded collections, and thus the whole cluster (mongos, mongod that make up the shards) would need the information contained in them. For more information, you can check the config server’s logs and look for a line similar to:

[initandlisten] connection accepted from X.X.X.X:65419 #19 (10 connections now open)

You may be able to determine if the IP address X.X.X.X is from expected sources (i.e. mongos or mongod, monitoring agents, etc.). There should be no connection from your application to the config servers, unless your application is specifically directed to connect to them.

Best regards,
Kevin

​

[inte...@googlemail.com]

Hi Kevin,

The drivers do not connect to the config servers directly.

I already thought so, but the probability that our software or the old php driver has some sort of connection related bug is a little bit higher than the mongoDB itself - I guess ;)

 

It is normal for the config servers to have relatively high connection count, since they act as a “map” of the data distribution of your sharded collections, and thus the whole cluster (mongos, mongod that make up the shards) would need the information contained in them.

Ahh okay - so the more distributed data we have, the more connections to the configserver we have?

Best regards,

Marcus

[Kevin Adistambha]

Hi Marcus,

Ahh okay - so the more distributed data we have, the more connections to the configserver we have?

Actually it’s the number of shards you have and the number of mongos you have, not the amount of data you have. For example, you can have a lot of data in a small number of shards, but you can also have a lot of shards with very little data.

It is perfectly normal for config servers to have many connections, since the operation of the cluster depends on them. For more information, please have a look at Config Servers and Production Cluster Architecture.

Best regards,
Kevin

​