Lots of ReplicaSetMonitorWatcher SocketException errors in logs

[Darshan Shah]

We are running mongodb 3.2.11 sharded 30 node cluster with each node being a 3 member replicaset.

Of late, I have noticed a lot of ReplicaSetMonitorWatcher SocketException errors in logs for both mongos and mongod.

Here is an excerpt of the kind of log messages:

2018-01-15T19:38:11.870-0500 I NETWORK  [ReplicaSetMonitorWatcher] Socket recv() timeout  192.168.12.34:29112

2018-01-15T19:38:11.870-0500 I NETWORK  [ReplicaSetMonitorWatcher] SocketException: remote: (NONE):0 error: 9001 socket exception [RECV_TIMEOUT] server [192.168.12.34:29112]

2018-01-15T19:38:11.870-0500 I NETWORK  [ReplicaSetMonitorWatcher] Detected bad connection created at 1515943956843337 microSec, clearing pool for mongod_server-16:29112 of 0 connections

2018-01-15T19:38:11.870-0500 I NETWORK  [ReplicaSetMonitorWatcher] Marking host mongod_server-16:29112 as failed :: caused by :: HostUnreachable: network error while attempting to run command 'ismaster' on host 'mongod_server-16:29112'

2018-01-15T19:38:27.305-0500 I COMMAND  [conn1616] command admin.$cmd command: listDatabases { listDatabases: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:1035 locks:{ Global: { acquireCount: { r: 26 } }, Database: { acquireCount: { r: 13 } } } protocol:op_command 101ms

2018-01-15T19:38:31.648-0500 I COMMAND  [conn1462] command admin.$cmd command: listDatabases { listDatabases: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:1035 locks:{ Global: { acquireCount: { r: 26 } }, Database: { acquireCount: { r: 13 } } } protocol:op_command 119ms

2018-01-15T19:39:27.135-0500 I NETWORK  [ReplicaSetMonitorWatcher] Socket recv() timeout  192.168.12.35:29112

2018-01-15T19:39:27.135-0500 I NETWORK  [ReplicaSetMonitorWatcher] SocketException: remote: (NONE):0 error: 9001 socket exception [RECV_TIMEOUT] server [192.168.12.35:29112]

2018-01-15T19:39:27.135-0500 I NETWORK  [ReplicaSetMonitorWatcher] Detected bad connection created at 1515943956721427 microSec, clearing pool for mongod_server-27:29112 of 0 connections

2018-01-15T19:39:27.135-0500 I NETWORK  [ReplicaSetMonitorWatcher] Marking host mongod_server-27:29112 as failed :: caused by :: HostUnreachable: network error while attempting to run command 'ismaster' on host 'mongod_server-27:29112'

2018-01-15T19:39:42.186-0500 I NETWORK  [ReplicaSetMonitorWatcher] Socket recv() timeout  192.168.12.35:29112

2018-01-15T19:39:42.186-0500 I NETWORK  [ReplicaSetMonitorWatcher] SocketException: remote: (NONE):0 error: 9001 socket exception [RECV_TIMEOUT] server [192.168.12.35:29112]

The servers referred to in the above log message are not a part of the replicaset of the primary where I see this message.

This means that Primary member of one replicaset is trying to connect to secondaries from another replicaset.

My question is - For a MongoD running as Primary in a replicaset, is this behavior normal and why?

Same question as well for a MongoS - is it normal for a MongoS to try to connect to Secondary members of replicasets and why?

Thanks!

[Darshan Shah]

I still see the same issue after upgrading to MongoDb 3.2.18.

I am guessing that I need a non-default setting for ShardingTaskExecutorPoolMaxConnecting - have posted a separate question on how to figure out a good value for that parameter.

However, the original questions still remain:

The servers referred to in the log message are not a part of the replicaset of the primary where I see this message.

This means that Primary member of one replicaset is trying to connect to secondaries from another replicaset.

My questions are:

1.  For a MongoD running as Primary in a replicaset, is this behavior normal and why?

2.  Same question as well for a MongoS - is it normal for a MongoS to try to connect to Secondary members of replicasets and why?

Can someone from the MongoDB team please reply with some info on the above two questions.

Thanks!

[Darshan Shah]

Bump - just in case this question got buried under other questions - please provide any links / documents that can help.

This is the link to my post requesting more details on the parameter settings.

Thanks!