[rsBackgroundSync] Fatal Assertion 18752 and abort mongodb process

[tao hiko]

My MongoDB version 3.2.10.

Mongod process has been terminated and then I checked in log file and found this content.

2017-05-24T06:26:14.824+0000 I REPL     [ReplicationExecutor] This node is <node 1> in the config
2017-05-24T06:26:14.824+0000 I REPL     [ReplicationExecutor] transition to STARTUP2
2017-05-24T06:26:14.824+0000 I REPL     [ReplicationExecutor] Starting replication applier threads
2017-05-24T06:26:14.825+0000 I REPL     [ReplicationExecutor] transition to RECOVERING
2017-05-24T06:26:14.827+0000 I REPL     [ReplicationExecutor] transition to SECONDARY
2017-05-24T06:26:14.827+0000 I REPL     [ReplicationExecutor] Member <node 2> is now in state PRIMARY
2017-05-24T06:26:15.045+0000 I REPL     [ReplicationExecutor] Member <node 1> is now in state ARBITER
2017-05-24T06:26:16.358+0000 I NETWORK  [initandlisten] connection accepted from 10.52.202.233:35445 #2 (2 connections now open)
2017-05-24T06:26:17.825+0000 I REPL     [ReplicationExecutor] syncing from: <node 2>
2017-05-24T06:26:17.829+0000 I REPL     [SyncSourceFeedback] replset setting syncSourceFeedback to <node 2>
2017-05-24T06:26:17.829+0000 I REPL     [rsBackgroundSync] replSet our last op time fetched: May 24 01:02:13:2
2017-05-24T06:26:17.830+0000 I REPL     [rsBackgroundSync] replset source's GTE: May 24 01:02:56:1
2017-05-24T06:26:17.830+0000 I REPL     [rsBackgroundSync] beginning rollback
2017-05-24T06:26:17.830+0000 I REPL     [rsBackgroundSync] rollback 0
2017-05-24T06:26:17.830+0000 I REPL     [ReplicationExecutor] transition to ROLLBACK
2017-05-24T06:26:17.830+0000 I REPL     [rsBackgroundSync] rollback 1
2017-05-24T06:26:17.830+0000 I REPL     [rsBackgroundSync] rollback 2 FindCommonPoint
2017-05-24T06:26:17.831+0000 I REPL     [rsBackgroundSync] replSet info rollback our last optime:   May 24 01:02:13:2
2017-05-24T06:26:17.831+0000 I REPL     [rsBackgroundSync] replSet info rollback their last optime: May 24 06:25:57:3
2017-05-24T06:26:17.831+0000 I REPL     [rsBackgroundSync] replSet info rollback diff in end of log times: -19424 seconds
2017-05-24T06:26:19.583+0000 F REPL     [rsBackgroundSync] warning: log line attempted (8477k) over max size (10k), printing beginning and end ... replSet error can't rollback this command yet: {... 2017-05-24T06:26:19.583+0000 I REPL     [rsBackgroundSync] replSet cmdname=applyOps
2017-05-24T06:26:19.583+0000 E REPL     [rsBackgroundSync] replica set fatal exception
2017-05-24T06:26:19.583+0000 I -        [rsBackgroundSync] Fatal Assertion 18752
2017-05-24T06:26:19.583+0000 I -        [rsBackgroundSync]

How can I bring it back and what is "log line attempted (8477k) over max size (10k)" and "Fatal Assertion 18752"?

Currently, node2 became to Primary.

Thanks,

Hiko

[tao hiko]

I'm running "--repair" when start mongod process and I will keep update when it's done.

[tao hiko]

This method cannot help after repaired and then start it, mongod still shows the same error msg.

How can I skip rollback process to avoid this error? Do I have only solution to bring it back by copying offline from another node? :(

[tao hiko]

Finally, I have deleted all data files and re-synced from another node.  It's done.

[tao hiko]

The root cause of this problem as my opinion is oplog replay process had been terminated by something and server was restarted and then the transaction was broken and couldn't rollback. After re-synced, I applied oplog again and has no error.