MongoDB 3.2.10 Index rebuilding did not complete: 11000 E11000 duplicate key error collection when creating replica set wiredtiger storage engine

[Sanjay Gupta]

MongoDB 3.2.10

Getting following error when adding replica set as WiredTiger. This is sharded cluster ( 3 shards nodes ). 2 Shards replica sets created using wiredTiger storage so only 1 shard is having problem when creating replica set.

This is our QA/DEV setup:

3 Shards on 3 Nodes ( Also each node runs Arbiter for replica set)

4th Node ( has replica set for all 3 shards, different DB data paths)

2016-11-11T00:51:55.005+0000 I -        [initandlisten]   Index Build: 39291000/                                                                        39372845 99%
2016-11-11T00:51:58.032+0000 I -        [initandlisten]   Index Build: 39345000/                                                                        39372845 99%
2016-11-11T00:52:04.696+0000 E INDEX    [initandlisten] Index rebuilding did not                                                                         complete: 11000 E11000 duplicate key error collection: XYZ.ALBUM index: _id_                                                                         dup key: { : 59000012923579 }
2016-11-11T00:52:04.696+0000 I INDEX    [initandlisten] note: restart the server                                                                         with --noIndexBuildRetry to skip index rebuilds
2016-11-11T00:52:04.696+0000 I -        [initandlisten] Fatal Assertion 26100
2016-11-11T00:52:04.696+0000 I -        [initandlisten]

2016-11-11T00:52:04.696+0000 E INDEX    [initandlisten] Index rebuilding did not complete: 11000 E11000 duplicate key error collection: XYZ.My_Collection index: _id_ dup key: { : 59000012923579 }
2016-11-11T00:52:04.696+0000 I INDEX    [initandlisten] note: restart the server with --noIndexBuildRetry to skip index rebuilds




I did not have issue when I was using mmapv1 replica set but now I have dropped old replica set and was converting to WiredTiger storage. 2 Replica sets were created without any issues but only 1 is having issue. When this issue happens, Mongod process also shutdowns. 

I checked count for _id= 59000012923579  and it shows 1.

mongos> db.My_Collection.find({_id:59000012923579}).count()

1

I am not sure why it is giving dup key error ?

How to fix this issue ?

[Sanjay Gupta]

I have deleted all data from dbpath and again restarted replica set sync and it looks it is going fine now. I was trying to restart a failed replica set sync and it looks like MongoDB was not able to handle it, once I cleaned up failed replica set data and restart mongo and looks like it is passed point where it was failing before.

Basically,I have removed host from relica set configuration, cleanup dbpath, restarted mongo and again added host in replica set configuration.