MongoDB 3.6 replica stuck on STARTUP

[Mario Dagrada]

I am using mongodb with two servers physically located in respectively Italy and France.

The France server runs Ubuntu server is the Primary of the replica set, and works fine, i.e it accepts connections from mongo shell whether locally or remotely and is continuously updated by a python client.

The second server in Italy works fine standalone with CentOS, but get stuck to STARTUP status when starting as part of the replica set. The log indicates that it received the configuration from the primary.

I tried a several things to fix the issue but none of them works:

• - Try to use keyfile (or not) between members of the replica set.
• - Check the Firewall is open on both sides with nmap
• - Add iptables rules to assert the mongodb port accept in/out traffic.
• - Check mongod is up and listening on all IP on both sides.
• - Configure (or not) the same admin user on secondary before restarting with --replSet. This step can only be done when secondary is standalone, since started as member of the replica set, it is stuck in transient state and thus it does not accept user creation. Hence it follows that login to the secondary fails.

Also both members (France:primary and Italy:secondary) are started with --auth, --replSet rs0, --keyFile.

I researched the issue on the web before, and found the closest answer here: MonogoDB Replica Set Status Not changing from Startup to Secondary

But the author mention that the secondary did not received the configuration which makes the issue different than this one.

Thanks for your help.

[Kevin Adistambha]

Hi Mario

Could you please clarify a few points:

The second server in Italy works fine standalone with CentOS, but get stuck to STARTUP status when starting as part of the replica set

• Did you run rs.initiate() on both the French and the Italian server, or did you run the Italian server using the --replSet parameter and perform rs.initiate() and rs.add(<the Italian server's address>) on the French server only?
• What is the output of rs.conf() and rs.status() from both nodes?
• Could you post the mongod logs from both nodes?
• What is your MongoDB version?

Configure (or not) the same admin user on secondary before restarting with —replSet. This step can only be done when secondary is standalone

To clarify this point, you don’t need to create any user on the secondary, since all information will be replicated by the primary. In fact, you don’t need any data on the secondary since the secondary will automatically perform an initial sync when it joins a replica set for the first time.

It might be beneficial if you review and follow the procedure outlined in Deploy a Geographically Redundant Replica Set, and Deploy a Replica Set.

Please note, it is strongly recommended to deploy a replica set with three members as a minimum. This is because MongoDB uses a voting mechanism to elect a primary node (see Replica Set Elections for more information). A replica set only allow writes to the primary node, so in a 2-node replica set, if one of them is offline, the other node will become a secondary and the set will not accept writes (since the offline node’s vote is still counted).

For production environment, it is also strongly recommended to use SSL instead of keyfiles for better security. Please review the Production Notes for more details.

Best regards
Kevin

​