Mongo replica set *secondary* duplicate key issue...

[Henry]

Hi,

We've been having regular issues in our production environment where we see duplicate key errors on our secondaries like this (from rs.status()):
        {
                        "_id" : 14,
                        "name" : "db35:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 20419,
                        "optime" : {
                                "t" : 1353085117000,
                                "i" : 39
                        },
                        "optimeDate" : ISODate("2012-11-16T16:58:37Z"),
                        "lastHeartbeat" : ISODate("2012-11-16T22:13:55Z"),
                        "pingMs" : 1,
                        "errmsg" : "syncTail: 11000 E11000 duplicate key error index: data.aa_b098d94b:_Installation.$deviceToken_1 dup key: { : \"1581a95a\"}, syncing: { ts: Timestamp 1353085117000|40"
                }

We suspect that when we occasionally kill operations on the primary, the operations are still trickling to the secondaries, which results in the above situation.  This state prevents the secondaries from continuing to sync to the primary.


The situation we suspect that leads to this situation looks something like:
Assume that there's a collection with a uniquely indexed field.  This collection exists on a replica set with a primary and a secondary.

Phase 1:
- An op inserting a new row into the collection (containing the uniquely indexed field with some value) hits the replica set primary.
- The op gets killed at the primary using killOp.
- The op somehow continues to secondary and is accepted.  (<-- possible bug?)
- Collection on secondary now has the row and the primary does not.

Phase 2
- After the above, a new op comes in that inserts a row that's identical to the row above -- except for a different object id.
- Op is accepted and indexed by the primary since uniqueness constraint is met.
- Op passes on to secondary.
- Op on the secondary results in a duplicate key error since it already has that value for the uniquely indexed field.
- This results in the above situation.

Would you please pass this theory on to someone at 10gen that is well versed with the replication operations above?  Clearing up these inconsistencies results in a fairly large operational overhead.

Thanks in advance.

Henry

[DaveC]

This sounds oddly similar to the issue I'm seeing: https://groups.google.com/forum/m/?fromgroups#!topic/mongodb-user/eDPGj9nS6_A

What version of MongoDB are you running?