Recovering from IP change on Primary

[Geoff]

Hello.

I have a three server replica cluster working fine for 3 years.  Mongo 2.6.10.  Replica were designated using IP instead of hostname :(

AWS rebooted all three servers and all received new IP addresses.  Leaving replication broken.

To continue service immediately, I commented out `replSet = testReplSet` on the known primary, restarted, and pointed the web servers to this single server.  Effectively putting it in stand-alone mode.

Now it is time to re-establish replication.

Can I fix the IP address of the primary in the replica set, re-enable `replSet = testReplSet`, restart mongo, and will it resume the role of primary? ( After which I will add new secondaries)

---------------------------------------------

1) UPDATE THE IP ADDRESS (JUST FOR THE PRIMARY _id=0)

Suppose we have a configuration such as the one shown here:

> rs.config() { "_id" : "testReplSet", "version" : 2, "members" : [ { "_id" : 0, "host" : "10.35.1.10: 27017" }, { "_id" : 1, "host" : "10.35.1.11: 27017" }, { "_id" : 2, "host" : "10.35.1.12: 27017" } ] }

Can I load the current configuration in the shell and then change the relevant IP:

> var config = rs.config()
> config.members[ 1]. host = "10.35.2.10: 27017"

Then send it to the database using the rs.reconfig helper:

> rs.reconfig( config)

2) MODIFY /etc/mongodb.conf

Enable `replSet = testReplSet`

3) RESTART MONGO

---------------------------------------------

Will this trigger mongodb and this server into resuming the role of primary?


If not, what are other options?

Thanks
Geoff

[Geoff]

Ok. This approach did not work.

The original PRIMARY came back up in "SECONDARY" mode.  It did not permit further `rs` commands stating this procedure needs to be done on the primary.  This server was the primary.  I could not find a way to tell this secondary server to assume the PRIMARY role. 

Any suggestions ?

[Stephen Steneker]

On Tuesday, 9 January 2018 12:22:24 UTC+11, Geoff wrote:

Ok. This approach did not work.

The original PRIMARY came back up in "SECONDARY" mode.  It did not permit further `rs` commands stating this procedure needs to be done on the primary.  This server was the primary.  I could not find a way to tell this secondary server to assume the PRIMARY role. 

Hi Geoff,

Based on your original description it sounds possible that you may have restarted your (former) primary in standalone mode and then continued with writes:

To continue service immediately, I commented out `replSet = testReplSet` on the known primary, restarted, and pointed the web servers to this single server.  Effectively putting it in stand-alone mode.

Any writes in standalone mode will not be written to the replication oplog, so if this is the case you will need to drop the local database on your former primary (in standalone mode) and re-sync the other two replica set members to ensure consistency. After dropping the local database in standalone mode, the procedure will be the same as the Convert a Standalone to a Replica Set tutorial.

If there have definitely been no writes in standalone mode and only the IP addresses of the replica set members have changed, you can force reconfigure with current IPs as per the Reconfigure a Replica Set with Unavailable Members tutorial. The force reconfigure procedure should be used with caution: do not use this if there is already a primary in your replica set or potentially divergent oplogs.

Mongo 2.6.10

You are probably already aware, but MongoDB 2.6 reached end of life in October, 2016. After you get your deployment back online I would recommend planning to upgrade to MongoDB 3.2 or newer. MongoDB 3.0 will reach end of life next month (February, 2018).

Regards,

Stennie

[Geoff]

Thanks.   I will upgrade mongo.

This did the trick.

> use admin

> var cfg = rs.conf()

> cfg.members = [cfg.members[1]]
[ { "_id" : 6, "host" : "10.27.207.54:27017" } ]

> cfg.members[0].host='10.167.37.180:27017'
10.117.37.180:27017

> cfg
{
    "_id" : "xxx",
    "version" : 101520,
    "members" : [
        {
            "_id" : 6,
            "host" : "10.117.37.180:27017"
        }
    ]
}

> rs.reconfig(cfg, {force : true})
{
    "msg" : "will try this config momentarily, try running rs.conf() again in a few seconds",
    "ok" : 1
}
>
>
xxx:PRIMARY>