Changing IP's of replica servers

[raylu]

I have some replica servers with two IP addresses and I'd like to
change the replica set config from one to the other.

Related: http://groups.google.com/group/mongodb-user/browse_thread/thread/d0dde3ac8ea5aea4/440fae63594e5058

> You can use --fastsync to change the ip quickly.

What exactly is the procedure for this? I'm thinking it's a bad idea
to add the second IP to the config while the first one is still there,
so I should:
1. remove the first IP
2. take down the mongod
3. bring it back up with --fastsync
4. add the second IP

I'm wondering if the local config that the mongod stores will cause
problems.

[k]

You should just be able to change the "host" field in the member's
config.

On Jun 30, 4:43 pm, raylu <lur...@gmail.com> wrote:
> I have some replica servers with two IP addresses and I'd like to
> change the replica set config from one to the other.
>

> Related:http://groups.google.com/group/mongodb-user/browse_thread/thread/d0dd...

[raylu]

While all the mongod's in the replica set are running?

[k]

Yes. Do config = rs.conf(), change the host, and do an rs.reconfig.
I'd recommend doing one host at a time.

[raylu]

That didn't really work. I set up a test replica set using 1.8.2 with
2 servers and made it a part of a autosharded cluster. Using

{
"_id" : "testr",
"version" : 1,
"members" : [
{
"_id" : 0,
"host" : "box1ip1:27017"
},
{
"_id" : 1,
"host" : "box2ip1:27017"
}
]
}

When mongos connects to the configdb, it logs:
Mon Jun 27 23:01:20 [Balancer] updated set (testr) to: testr/
box1ip1:27017,box2ip1:27017

box2 is the primary.
testr:PRIMARY> cfg = rs.conf()
testr:PRIMARY> cfg.members[1].host = "box2ip2:27017"
testr:PRIMARY> rs.reconfig(cfg)

On box2, rs.status() and rs.conf() look fine. However, on box1, which
remained secondary, rs.conf() looks fine but rs.status() still lists
box2ip1. The mongos logged
Thu Jun 30 21:33:01 [ReplicaSetMonitorWatcher] updated set (testr) to:
testr/box1ip1:27017,box2ip1:27017,box2ip2:27017

After restarting the mongos,
Thu Jun 30 21:34:43 [Balancer] updated set (testr) to: testr/
box1ip1,box2ip1:27017

Thinking that I should only change the secondary, I switched the IP
back to the original configuration, reconfiged. Then, I changed the
secondary in the same way:
testr:PRIMARY> cfg = rs.conf()
testr:PRIMARY> cfg.members[0].host = "box1ip2:27017"
testr:PRIMARY> rs.reconfig(cfg)

Again, rs.conf() on both boxes look fine but rs.status() on the
primary, which I ran the reconfig from, still shows box1ip1.
rs.status() from the secondary looks fine. This time, mongos didn't
log anything while running. Restarting the mongos was scary:
Thu Jun 30 21:58:26 [Balancer] updated set (testr) to: testr/
box1ip1:27017,box1ip2:27017
Thu Jun 30 21:58:26 [Balancer] updated set (testr) to: testr/
box1ip1:27017,box1ip2:27017,box2ip1:27017

I'm thinking it would be better to perform the procedure I originally
suggested.

-raylu

[k]

You should just restart the mongod that doesn't pick up on the
reconfig. There was a bug in 1.8 where, under certain circumstances,
reconfigures were only half-"noticed" by some members until they were
restarted.

[raylu]

OK. Restarting the mongod fixed the nodes in the replica set. However,
the mongos still says

testr/box1ip1:27017,box1ip2:27017,box2ip1:27017

for db.shards.find() and db.runCommand({listShards: 1}). Querying the
configsvr itself with db.shards.find() gives the same result.

[k]

Cool!

You'll probably have to bounce mongos to update its server list. It
wasn't very smart about updating that stuff in 1.8.

[raylu]

But the problem isn't with the mongos. Restarting it gave the same set
of 3 IPs. Querying the config db directly got those 3 also.

-raylu

[raylu]

I just ran into the same problem with the remove, fastsync, add
procedure.

The first secondary I took down went fine. Restarting the mongos
revealed a good configuration with the ip1 removed and ip2 added.

I had the master step down then and removed the second node. After
bringing it back up and adding it, both ip2s were there but the ip1 of
the original master was also being returned by the mongos, even after
a restart. It also appeared on the configdb itself.

-raylu

[k]

Hmm, I think this is bug, this really should be updated automatically
by mongos. I've created a case for it: https://jira.mongodb.org/browse/SERVER-3355.

However, you can hack around it by updating the shards collection:

> use config
> db.shards.update({_id: "testr"}, {$set : {host : newHostString}})

Then restart the mongos.

[Mike K]

I've seen this issue happen with the config servers not picking up the changes; in some cases, adding members updates the config servers while removing them does not. In one case, the config servers appended all the new members while leaving the old members, but after an hour or so seemed to notice the changes and 'cleared' themselves out.