Automatic status change of the replica
Arrate
Hello,
we have a MongoDb (2.6) replica set setup on three nodes, and we have this in both errorlog. This is an example. All of them at same time in the diferent nodes.
Mar 27 10:31:49
NODO 1
Mar 27 10:31:49 ulpmon01 mongod.27017[1464]: [rsHealthPoll] replSet info ulpmon03.osasunet:27017 is down (or slow to respond):
Mar 27 10:31:49 ulpmon01 mongod.27017[1464]: [rsHealthPoll] replSet member ulpmon03.osasunet:27017 is now in state DOWN
Mar 27 10:31:53 ulpmon01 mongod.27017[1464]: [rsHealthPoll] replSet member ulpmon03.osasunet:27017 is up
Mar 27 10:31:53 ulpmon01 mongod.27017[1464]: [rsHealthPoll] replSet member ulpmon03.osasunet:27017 is now in state SECONDARY
NODO 2
Mar 27 10:31:43 ulpmon02 mongod.27017[1438]: [rsHealthPoll] DBClientCursor::init call() failed
Mar 27 10:31:43 ulpmon02 mongod.27017[1438]: [rsHealthPoll] replSet info ulpmon03.osasunet:27017 is down (or slow to respond):
Mar 27 10:31:43 ulpmon02 mongod.27017[1438]: [rsHealthPoll] replSet member ulpmon03.osasunet:27017 is now in state DOWN
Mar 27 10:31:50 ulpmon02 mongod.27017[1438]: [rsHealthPoll] replset info ulpmon03.osasunet:27017 heartbeat failed, retrying
Mar 27 10:31:53 ulpmon02 mongod.27017[1438]: [rsHealthPoll] replSet member ulpmon03.osasunet:27017 is up
Mar 27 10:31:53 ulpmon02 mongod.27017[1438]: [rsHealthPoll] replSet member ulpmon03.osasunet:27017 is now in state SECONDARY
NODO 3
Mar 27 10:31:53 ulpmon03 mongod.27017[1442]: [rsHealthPoll] replset info ulpmon01.osasunet:27017 thinks that we are down
Mar 27 10:31:53 ulpmon03 mongod.27017[1442]: [rsHealthPoll] replset info ulpmon02.osasunet:27017 thinks that we are down
Can anyone help me?
Thanks.
Arrate
Kevin Adistambha
Hi Arrate,
From the logs you posted, it appears that there is a network issue between Node 3 and the rest of the replica set. This is evident in the log from Node 1 and 2:
Mar 27 10:31:49 ulpmon01 mongod.27017[1464]: [rsHealthPoll] replSet info ulpmon03.osasunet:27017 is down (or slow to respond)
However, since it appears that Node 1 and 2 can still communicate with each other, the replica set should still have a Primary.
Does the network issue resolve itself after some time? If not, you might want to check the connection of Node 3.
Please note that MongoDB 2.6 reached its end of life on October 2016. If applicable, you may want to consider moving to a more recent version of MongoDB, which contains many bugfixes, improvements, and the WiredTiger storage engine. The latest MongoDB version is currently 3.4.4. Note that the only supported upgrade path from 2.6 to 3.4 involves upgrading to every major version (i.e. 2.6 -> 3.0 -> 3.2 -> 3.4).
You may find the following links helpful:
Best regards,
Kevin