Config server maintenance

[Erez Zarum]

I have 3 config servers in a sharded environment hostA,hostB,hostC, i have to take down hostA for maintenance.

I will first shutdown the config server on hostA (it will automatically failover to hostB), then i will backup the data on hostA.

I assume the maintenance will take around 1 hour, after i am done with it, i will copy the data back to hostA (after maintenance) and start it again.

I couldn't find any information as of how much time can i let hostA be down without it going out of sync and not able to "resync"? is there any rule of thumb for that?

[Asya Kamsky]

Hi Erez,

The three config servers are not a replica set, they are three independent mongod servers.  All three are used by mongos processes.

When you take one of them down, the other config servers will become read-only - your sharded cluster will continue to operate, but no chunk splitting or balancing will happen (in other words, the sharding configuration becomes read-only, not your data).

This means that when you restore the data to hostA and start it again, it will not be "behind" the other two config servers (since those were read-only during the maintenance window) and it will not need to "catch up".   

Once there are three config servers again, the sharded cluster will be able to resume splitting and balancing as needed.

Asya

[Asya Kamsky]

I left off a link I meant to include with my reply:

http://www.mongodb.org/display/DOCS/Changing+Config+Servers

The last example seems very close to your scenario, except your hostA won't be dead and you'll have a back-up from it so you won't need to get it from hostC.

[Erez Zarum]

So in case even one config server going down (for any reason, maintenance, etc..) all of the other config server are going into read-only mode?

What about inserting data to the cluster? isn't it creating metadata on the config server?

Thanks.

[Asya Kamsky]

If any of the three config servers are down, the other config servers are going to become read-only.

Data can still be inserted into the sharded cluster.  What can't change is the chunk and shard information which is what is in the config servers.

Just inserting new data into the cluster doesn't change config server data.

Another short write-up of this is here: http://wiki.mongodb.org/display/DOCS/Sharding+and+Failover in section 4.

Asya