ServerSelectionTimeoutError: No replica set members match selector "Primary()"

[David Miranda]

Hi,

This is the state of my replicaset:

{
"set" : "rs0",
"date" : ISODate("2018-02-28T15:58:37.228Z"),
"myState" : 7,
"term" : NumberLong(13),
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1519833700, 2),
"t" : NumberLong(13)
},
"appliedOpTime" : {
"ts" : Timestamp(1519833700, 2),
"t" : NumberLong(13)
},
"durableOpTime" : {
"ts" : Timestamp(0, 0),
"t" : NumberLong(-1)
}
},
"members" : [
{
"_id" : 1,
"name" : "mongo2:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 1106202,
"optime" : {
"ts" : Timestamp(1519833700, 2),
"t" : NumberLong(13)
},
"optimeDurable" : {
"ts" : Timestamp(1519833700, 2),
"t" : NumberLong(13)
},
"optimeDate" : ISODate("2018-02-28T16:01:40Z"),
"optimeDurableDate" : ISODate("2018-02-28T16:01:40Z"),
"lastHeartbeat" : ISODate("2018-02-28T15:58:34.259Z"),
"lastHeartbeatRecv" : ISODate("2018-02-28T15:58:35.285Z"),
"pingMs" : NumberLong(8),
"syncingTo" : "mongo1:27017",
"configVersion" : 23
},
{
"_id" : 5,
"name" : "arbiter:27017",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 1114291,
"configVersion" : 23,
"self" : true
},
{
"_id" : 6,
"name" : "mongo1:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 21350,
"optime" : {
"ts" : Timestamp(1519833700, 2),
"t" : NumberLong(13)
},
"optimeDurable" : {
"ts" : Timestamp(1519833700, 2),
"t" : NumberLong(13)
},
"optimeDate" : ISODate("2018-02-28T16:01:40Z"),
"optimeDurableDate" : ISODate("2018-02-28T16:01:40Z"),
"lastHeartbeat" : ISODate("2018-02-28T15:58:34.305Z"),
"lastHeartbeatRecv" : ISODate("2018-02-28T15:58:36.223Z"),
"pingMs" : NumberLong(7),
"electionTime" : Timestamp(1518724645, 1),
"electionDate" : ISODate("2018-02-15T19:57:25Z"),
"configVersion" : 23
}
],
"ok" : 1
}

I'm using the pymongo to connect to my replicaset and the MongoDB uri is:

mongodb://myuser:mypass@mongo1:27017,mongo2:27017/mydb?replicaSet=rs0

Sometimes I'm getting the following error in linking to my MongoDB replica set:

ServerSelectionTimeoutError: No replica set members match selector "Primary()"

NetworkTimeout: mongo1:27017: timed out 

This is intermittent issue since it does not always happen, because most of the time I can do write / read operations without getting these exceptions.  

 

Any tips?

Thanks

[Kevin Adistambha]

Hi

ServerSelectionTimeoutError: No replica set members match selector “Primary()”

This error saying that the client (Pymongo, in this case) is trying to connect to the primary node of the replica set, but couldn’t.

A replica set can have no primary if the primary node cannot see the majority of the replica set nodes. This is possible if:

• the majority of the secondary nodes are offline (e.g. 2 nodes are offline in a 3-node replica set)
• there is a network partition between the primary and the rest of the set.

Thus for example if you have a 3-node replica set and the two secondaries are offline, the primary node will immediately step down (becoming a secondary) and no writes are allowed in the set. This is to prevent a situation where a replica set was affected by a network partition, and thus could potentially have two primary nodes that cannot see each other (a “split brain” scenario). If a split brain occur and writes are occurring to two different primaries, it is very difficult to tell which primary has the most recent and the correct data.

What happened in your case could be such a scenario, where there is a temporary network partition and the primary node steps down. The intermittent nature of network partitions could be causing your intermittent connection issues.

Please see Automatic Failover for more details.

Best regards
Kevin

​

[David Miranda]

Hi Kevin,

Thanks for you reply. 

I realize what you mean, but I do not see this scenario happening, because my master is always the same (there is not a new election due to network problems). I also do not see in the logs any network communication error. Is something escaping me here? Any tips on how to make some debug to this situation?

Thanks.

[Kevin Adistambha]

Hi David

If your replica set is having connection issues between the nodes, you should see a message in the mongod logs that is similar to this:

2018-03-06T16:48:56.503+1100 I REPL_HB  [replexec-1] Error in heartbeat (requestId: 68) to localhost:27018, response status: HostUnreachable: Connection refused
2018-03-06T16:48:56.958+1100 I REPL     [replexec-2] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)

Note that the messages above were printed by MongoDB 3.6.3. It could be slightly different on older MongoDB versions.

If you’re certain that your replica set does not have an election or similar message during the connection timeout issue you’re seeing, then there could be intermittent network issues between the replica set and the client application.

Best regards
Kevin

​

[David Miranda]

Hi Kevin,

I searched the logs of my "mongos" and found no error related to what you said. Yes it may be an intermittent network issues, but so frequent? I just wanted to try to discard if it might be a configuration problem ...

Thank you.

David

[Kevin Adistambha]

Hi David

If your app can connect to the replica set and the replica set status is as you posted, I don’t believe that a MongoDB configuration issue is the source of the connection problems. If there is a replica set configuration issue, typically your replica set 1) would not be able to have a primary and a secondary, and 2) apps cannot connect to it all the time, not just some of the time. I suspect it’s a networking issue (reliability, config, etc.) instead.

Having said that, if you’re still having doubt that there could be a configuration issue, please post:

• Your MongoDB version and your OS version
• The output of rs.conf()
• Details of your deployment, e.g. is it a replica set or a sharded cluster, hardware spec, whether one machine has more than one mongod running on it, is it cloud hosted or physical machines, etc.
• What the mongod log shows during this connection issue
• Try to ping the server and see any packet loss from the application machine during this time.

Best regards
Kevin

​

[David Miranda]

Hi Kevin,

MongoDB version: 3.4.7

MongoDB rs.conf()

{
"_id" : "rs0",
"version" : 1,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 0,
"host" : "mongo1:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"catchUpTimeoutMillis" : 60000,
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
},
"replicaSetId" : ObjectId("59b0707a92de04d12999374f")
}
}

Deployment details:

- Running on cloud hosted (virtual machines)

- Single mongod process running on it

- Replica set

- Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz | 8GB RAM (All the 3 nodes)

- The mongod log files dont have nothing related to network issues or socket disconnections

Traceroute:

I have run a trace (in both directions) with mtr and no packet losses.

Thanks.