We added a new secondary (dbs1a) to a replica set, planning to let it
sync and then step down / remove the original primary (dbs1d1).  The
new machine (dbs1a) sync'd all data and changed from STARTUP to
SECONDARY status, which made us think that it was up and good to go.
As soon as we stepped down the existing primary (dbs1d1), we
discovered that the new machine had neither synced the indexes nor
built them and was not in fact ready for production.

I see a lot of messages that look like this in the logs on the dbs1a;
all of these happened right about when we did the step down:

Thu Sep 27 19:05:38 [conn2270] stale version detected during query
over cm_prod.video_latest_stats : { $err: "version changed during
initial query ( ns : cm_prod.video_latest_stats, received : 160|1||
000000000000000000000000, wanted : 0|0||00000000000000000000...",
code: 13388, ns: "cm_prod.video_latest_stats", vReceived: Timestamp
160000|1, vReceivedEpoch: ObjectId('000000000000000000000000'),
vWanted: Timestamp 0|0, vWantedEpoch:
ObjectId('000000000000000000000000') }

These stopped as soon as we put the original machine back in as
primary.

These are both v2.2 instances.  One is CentOS 6.2, one is Ubuntu
10.04.  Not sure what else to add.  Add suggestions?

Dave