Down shard server

[Ted]

Hi,

I have a test cluster running 1.6.5 with 3 servers. One of the shard
servers died with no chance of recovery.

Is this post (http://groups.google.com/group/mongodb-user/
browse_thread/thread/f76f1d8504101243/59633b2d68f6db1d?lnk=gst&q=failed
+shard+server#59633b2d68f6db1d) the best way to handle this situation?

I just want to get the cluster going again and ignore the lost data.

Thanks.

[Alvin Richards]

Yes, this is the best way to deal with this.

-Alvin

[Ted]

I read through the thread a little too quick.  What should I do if I don't have a replacement server?

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Eliot Horowitz]

What state do you want to end up?
You can do some hackery on the config db if you need?

i.e. shutdown all shards + mongos
remove that shard from the shards collection
change the owner of all chunks that were on that shard somewhere else

obviously on a live system that would cause havoc, but should work

[Ted]

I would like to have a working cluster with 2 servers.  No worries about the lost chunks from the down server since this is a test system (I was bummed because the db was almost at a 1B rows).

Do I change owners or just delete the chunk's existence on the config server?

[Eliot Horowitz]

Change owners, deleting the chunks would be bad.

[Ted]

I assume by owner you mean the shard tag (vice the ns tag) from the shards collection.  Do I just make up a dummy shard?

Curious to know why deleting the chunk's existence is bad.  The chunk is gone.

[Eliot Horowitz]

Right, the shard field.
You HAVE to set to a real shard, not a dummy.

The reason is you have to have chunks that take up the entire key space.
If not, when you try to insert something there it will fail badly.
The system also makes sure the key space is complete and sane.

[Ted]

So what will happen when mongo tries to access a chunk that is suppose to exist on a (running) shard server but isn't there?

And what happens if there is an insert into that chunk?

[Eliot Horowitz]

A chunk is just a logical key range.
If you try to access a chunk that was put there by this process, it
will act normally as if it had no data.
Inserts to that chunk will work normally.

[Ted]

We installed a replacement server at the remote data center, fired up mongod, and restarted the cluster.
The missing chunks are "draining" from the newly installed server to the cluster without errors.  Mongo is even
decreasing the total row count as the chunks are found empty.

This is pretty cool that mongo can handle this situation.  Eliot - I'm impressed.

Thanks again for your time and help.