How to remove a replica set from MongoDB Cluster when all the nodes of the replicset are down

[Himanshu Joshi]

Hi,

I have a mongodb cluster with 2 replica sets each with 3 nodes (1 primary, 1 secondary and 1 arbiter). Currently all nodes (primary as well as secondary) of one of these replica set are down. So I am neither able to recover it nor able to delete it. I have the backup of data, so I want to delete this replica set and recreate it with the backup data. I don't want to delete the complete cluster as there is another healthy replica set present in it.

Is it possible ? I am using mongodb version 2.6.7.

I have also posted my query yesterday at  http://stackoverflow.com/questions/34308484/mongodb-replica-set-failed

But didn't dot any response yet.

Plz help.

Thanks & Regards

Himanshu Joshi

[Kevin Adistambha]

Hi Himanshu,

If you have a complete backup of the cluster, it is strongly recommended to restore a sharded cluster as a whole to guard against inconsistencies, missing data, etc. If you only restore a single shard from a backup, there may be missing or duplicate documents due to subsequent migrations as part of normal sharded cluster balancing activity.

Having said that, yes you can restore a single shard. However, it potentially involves a lot of manual work with no guarantee of success:

1. You should check the config server changelog to see if there have been any migrations to or from this shard subsequent to the backup you restored.

2. You should restore the new replica set using the same settings as the old ones (i.e. the same hostnames, the same ports, etc.).

3. If there were any migrations involving this shard after the backup:

   3a) For documents in chunk ranges that migrated to this shard, you will need to restore those documents from an alternative backup source.

   3b) You may wish to run cleanupOrphaned to remove documents in chunk ranges that migrated from this shard.

On a side note, a RAID-0 configuration is not recommended to be used with MongoDB.

Good luck with your restore. Let us know if you have any further questions.

Best regards,
Kevin