Secondary not syncing / building indexes

[David K Storrs]

We added a new secondary (dbs1a) to a replica set, planning to let it
sync and then step down / remove the original primary (dbs1d1). The
new machine (dbs1a) sync'd all data and changed from STARTUP to
SECONDARY status, which made us think that it was up and good to go.
As soon as we stepped down the existing primary (dbs1d1), we
discovered that the new machine had neither synced the indexes nor
built them and was not in fact ready for production.

I see a lot of messages that look like this in the logs on the dbs1a;
all of these happened right about when we did the step down:

Thu Sep 27 19:05:38 [conn2270] stale version detected during query
over cm_prod.video_latest_stats : { $err: "version changed during
initial query ( ns : cm_prod.video_latest_stats, received : 160|1||
000000000000000000000000, wanted : 0|0||00000000000000000000...",
code: 13388, ns: "cm_prod.video_latest_stats", vReceived: Timestamp
160000|1, vReceivedEpoch: ObjectId('000000000000000000000000'),
vWanted: Timestamp 0|0, vWantedEpoch:
ObjectId('000000000000000000000000') }

These stopped as soon as we put the original machine back in as
primary.

These are both v2.2 instances. One is CentOS 6.2, one is Ubuntu
10.04. Not sure what else to add. Add suggestions?

Dave

[Kristina Chodorow]

There's a known bug (https://jira.mongodb.org/browse/SERVER-7199) that should be fixed for 2.2.1 that can prevent indexes from being built during initial sync (recloning ops can produce duplicate keys, so unique index builds fail).  If you're missing non-unique indexes, there is no known bug for that, definitely let us know if that's the case.

You can build the required indexes on dbs1a by restarting it on a different port without the --replSet option, building the indexes, and then restarting it as a member of the set again.  Unique indexes may fail to build if there are duplicate keys in your collection, make sure you run db.foo.getIndexes() and verify the correct indexes are there before restarting.

[David K Storrs]

Replying well after the fact, sorry:

Actually, it turned out to be a separate problem -- probably a
hardware bug of some sort -- but what was happening was that the
machine got partway through building indices, then crashed. We
swapped everything except the RAM, RAID controller, and drives into a
new chassis and the issue stopped.

Thanks for your help!

Dave