sharded cluster issue after changing IP addresses

[Anna Rybachkova]

.We have 3 nodes  sharded cluster version 3.6 with 1 shard, 3 members replica set, config server replica set and 2 mongos

We stopped the cluster, changed  IP addresses  on all 3 nodes 

  .Now we have a problem to start the cluster

: Start confServReplSet replica set

On ptkpl-mongo1 with user mongo/1q2w3e4r

mongod --configsvr --keyFile  /data/db/key_file/keyfile  --port 27019 --replSet confServReplSet  --logpath /data/db/cfg1/log.cfg_sh1_m1 --logappend  --dbpath /data/db/cfg1 --fork --bind_ip 0.0.0.0

On ptkpl-mongo2 with user mongo/1q2w3e4r

mongod --configsvr --keyFile  /data/db/key_file/keyfile   --port 27019 --replSet confServReplSet  --logpath /data/db/cfg2/log.cfg_sh1_m2 --logappend  --dbpath /data/db/cfg2 --fork --bind_ip 0.0.0.0

On  ptkpl-mongo3 with user mongo/1q2w3e4r

mongod --configsvr --keyFile  /data/db/key_file/keyfile --port 27019 --replSet confServReplSet  --logpath /data/db/cfg3/log.cfg_sh1_m3 --logappend  --dbpath /data/db/cfg3 --fork --bind_ip 0.0.0.0

 Now we want  to check replica set status:

mongo --port 27019

MongoDB shell version v3.6.2

connecting to: mongodb://127.0.0.1:27019/

MongoDB server version: 3.6.2

Error while trying to show server startup warnings: Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1517326691, 1) } with id: 0

Note: Cannot determine if automation is active

MongoDB server version: 3.6.2

Error while trying to show server startup warnings: timed out waiting for refresh

In log file we see a lot of errors like:

2018-02-19T17:35:52.891+0200 I REPL_HB  [replexec-2] Error in heartbeat (requestId: 370) to ptkpl-mongo2:27019, response status: InvalidReplicaSetConfig: Our replica set configuration is invalid or does not include us

2018-02-19T17:35:52.892+0200 I REPL_HB  [replexec-1] Error in heartbeat (requestId: 372) to ptkpl-mongo3:27019, response status: InvalidReplicaSetConfig: Our replica set configuration is invalid or does not include us

2018-02-19T17:35:57.858+0200 I REPL     [rsBackgroundSync] waiting for 4 pings from other members before syncing

Please advise

[Kevin Adistambha]

Hi

InvalidReplicaSetConfig: Our replica set configuration is invalid or does not include us

I believe this error is caused by your replica set configuration still using the old hostnames. This is what it means when it says it “does not include us”.

If you’re planning to change the hostname of your replica set, it’s recommended to follow the procedure outlined in Change Hostnames in a Replica Set. For your particular situation, I believe the section titled Change All Hostnames at the Same Time is relevant. Note that the steps include changing the replica set configuration to reflect the new hostnames, and reconfiguring the replica set.

Best regards
Kevin

​