Replication funky state

[Kyle VanderBeek]

Some pathologically long queries were running and not noticed on my 2-node+arbiter configuration.  MongoDB (1.8.2) became largely unresponsive and today (after normal restarts) I'm getting this in the logs on the non-PRIMARY.  Any ideas how to get past this?

Thu Sep 22 11:13:24 [replica set sync] replset rollback error resyncing collection metrics_production.tmp.mr.tracked_events_tracked_events_backfill_29

Thu Sep 22 11:13:24 [replica set sync] replSet unexpected exception in syncThread()

Thu Sep 22 11:13:24 [dur] lsn set 252454

Thu Sep 22 11:14:21 [dur] lsn set 309714

Thu Sep 22 11:14:25 [replica set sync] replSet our last op time written: Sep 21 17:41:29:898

Thu Sep 22 11:14:25 [replica set sync] replset source's GTE: Sep 22 10:50:31:1

Thu Sep 22 11:14:25 [replica set sync] replSet rollback 0

Thu Sep 22 11:14:25 [replica set sync] replSet rollback 1

Thu Sep 22 11:14:25 [replica set sync] replSet rollback 2 FindCommonPoint

Thu Sep 22 11:14:25 [replica set sync] replSet info rollback our last optime:   Sep 21 17:41:29:898

Thu Sep 22 11:14:25 [replica set sync] replSet info rollback their last optime: Sep 22 11:00:27:1

Thu Sep 22 11:14:25 [replica set sync] replSet info rollback diff in end of log times: -62338 seconds

Thu Sep 22 11:14:25 [replica set sync] replSet info rollback of renameCollection is slow in this version of mongod

Thu Sep 22 11:14:25 [replica set sync] replSet info rollback of renameCollection is slow in this version of mongod

Thu Sep 22 11:14:25 [replica set sync] replSet info rollback of renameCollection is slow in this version of mongod

Thu Sep 22 11:14:25 [replica set sync] replSet rollback found matching events at Sep 21 17:41:27:1

Thu Sep 22 11:14:25 [replica set sync] replSet rollback findcommonpoint scanned : 4339

Thu Sep 22 11:14:25 [replica set sync] replSet replSet rollback 3 fixup

Thu Sep 22 11:14:26 [replica set sync] replSet rollback 3.5

Thu Sep 22 11:14:26 [replica set sync] replSet rollback 4 n:4311

Thu Sep 22 11:14:26 [replica set sync] replSet minvalid=Sep 22 11:00:27 4e7b77bb:1

Thu Sep 22 11:14:26 [replica set sync] replSet rollback 4.1 coll resync metrics_production.tmp.mr.tracked_events_tracked_events_backfill_29

Thu Sep 22 11:14:26 [replica set sync] building new index on { _id: 1 } for metrics_production.tmp.mr.tracked_events_tracked_events_backfill_29

Thu Sep 22 11:14:26 [replica set sync] done for 0 records 0secs

Thu Sep 22 11:14:26 [replica set sync] Assertion: 13312:replSet error : logOp() but not primary?

0x471fec 0x617651 0x614582 0x688103 0x62c428 0x62cc35 0x600591 0x60583e 0x606939 0x60dc5e 0x60ef28 0x60ef7d 0x60f5c2 0x822157 0x821d5f 0x977c6d 0x9b9a99 

 [0x471fec]

 [0x617651]

 [0x614582]

 [0x688103]

 [0x62c428]

 [0x62cc35]

 [0x600591]

 [0x60583e]

 [0x606939]

 [0x60dc5e]

 [0x60ef28]

 [0x60ef7d]

 [0x60f5c2]

 [0x822157]

 [0x821d5f]

 [0x977c6d]

 [0x9b9a99]

Thu Sep 22 11:14:26 [replica set sync] replset rollback error resyncing collection metrics_production.tmp.mr.tracked_events_tracked_events_backfill_29

Thu Sep 22 11:14:26 [replica set sync] replSet unexpected exception in syncThread()

Thu Sep 22 11:14:26 [dur] lsn set 314404

Thu Sep 22 11:15:21 [dur] lsn set 369714

--
Kyle.

Change.org

[Kristina Chodorow]

This looks like a bug in the rollback code. You should have a valid
rollback dump, but you're probably going to have to resync this node
(or start it from a backup).

I'll look into fixing the underlying problem...

[Kyle VanderBeek]

Is there a JIRA ticket that I can follow?

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

--
Kyle.

Change.org

[Kristina Chodorow]

Now there is: https://jira.mongodb.org/browse/SERVER-3939 :)

On Sep 23, 2:57 pm, Kyle VanderBeek <k...@change.org> wrote:
> Is there a JIRA ticket that I can follow?
>