Help! Replica set nodes won't start

[jdmerth]

our production servers are going down. every time they come back up we get:

Wed Sep 05 23:01:02 [conn51] end connection 10.28.218.27:52083 (11 connections now open)

Wed Sep 05 23:01:02 [repl writer worker 1] unindex failed (key too big?) tracking.packages.$ps_1_ts_1_ltd_1 key: { : 4, : 7, : new Date(1346884488824) } _id: "374_001111143007811742"

Wed Sep 05 23:01:02 [conn30] run command admin.$cmd { replSetHeartbeat: "rs", v: 72243, pv: 1, checkEmpty: false, from: "10.28.186.59:27017", $auth: {} }

Wed Sep 05 23:01:02 [conn30] command admin.$cmd command: { replSetHeartbeat: "rs", v: 72243, pv: 1, checkEmpty: false, from: "10.28.186.59:27017", $auth: {} } ntoreturn:1 keyUpdates:0  reslen:148 0ms

Wed Sep 05 23:01:02 [repl writer worker 1] ERROR: writer worker caught exception: btree: key+recloc already in index on: { ts: Timestamp 1346885686000|2, h: 5252254742187484922, op: "u", ns: "tracking.packages", o2: { _id: "374_0" }, o: { $addToSet: { acts: { $each: [ {-----snip-----}...

Wed Sep 05 23:01:02 [repl writer worker 1]   Fatal Assertion 16360

Wed Sep 05 23:01:03 [repl writer worker 1] mongod.exe    ...\src\mongo\util\stacktrace.cpp(161)                                      mongo::printStackTrace+0x3e

Wed Sep 05 23:01:03 [repl writer worker 1] mongod.exe    ...\src\mongo\util\assert_util.cpp(126)                                     mongo::fassertFailed+0x43

Wed Sep 05 23:01:03 [repl writer worker 1] mongod.exe    ...\src\mongo\db\repl\rs_sync.cpp(120)                                      `mongo::replset::multiSyncApply'::`1'::catch$0+0xb5

Wed Sep 05 23:01:03 [repl writer worker 1] mongod.exe    f:\dd\vctools\crt_bld\SELF_64_amd64\crt\prebuild\eh\amd64\handlers.asm(44)  _CallSettingFrame+0x20

Wed Sep 05 23:01:03 [repl writer worker 1] mongod.exe    f:\dd\vctools\crt_bld\self_64_amd64\crt\prebuild\eh\frame.cpp(1337)         __CxxCallCatchBlock+0xeb

Wed Sep 05 23:01:03 [repl writer worker 1] ntdll.dll                                                                                 RtlRestoreContext+0x2e3

Wed Sep 05 23:01:03 [repl writer worker 1] mongod.exe    ...\src\mongo\db\repl\rs_sync.cpp(116)                                      mongo::replset::multiSyncApply+0x50

Wed Sep 05 23:01:03 [repl writer worker 1] mongod.exe    ...\src\mongo\util\concurrency\thread_pool.cpp(66)                          mongo::threadpool::Worker::loop+0x94

Wed Sep 05 23:01:03 [repl writer worker 1] mongod.exe    ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(180)             boost::`anonymous namespace'::thread_start_function+0x21

Wed Sep 05 23:01:03 [repl writer worker 1] mongod.exe    f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314)                 _callthreadstartex+0x17

Wed Sep 05 23:01:03 [repl writer worker 1] mongod.exe    f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(292)                 _threadstartex+0x7f

Wed Sep 05 23:01:03 [repl writer worker 1] kernel32.dll                                                                              BaseThreadInitThunk+0xd

Wed Sep 05 23:01:03 [repl writer worker 1] 

can anyone help??

[thomasr]

Hi, in order to help, can you provide some more information? 

1. What version of MongoDB are you running?
2. Can you provide information about the collection that raises the error? Is it capped? Unique index?
3. Looking at the log messages, it seems you're trying to insert an index key that's too big. The current maximum (as of version 2.0) is 1024 bytes (prior to that it was 819 bytes). Is it possible you use a larger index key?
4. It would be helpful if you could set the log level verbosity to 5 and keep the server running until the error occurs. You increase verbosity with the shell command:  db.adminCommand({setParameter:1, logLevel:5}) and afterwards turn it back to 0. Can you provide the logfile for that period? (If you are concerned about sharing sensitive information, we can figure out some other way to securely submit the log files).

Regards,

Thomas

--

Thomas Rueckstiess

10gen - Technical Support

[jdmerth]

Sorry. I was a bit frantic...

1. version 2.2.0 running on windows server 2008 R2 Ent

2. the collection is not capped, not unique
3. from what I can tell the key does not look to big: 

key: { : 4, : 7, : new Date(1346884488824) }

[James Devine]

did you ever fix this?  I am running into this same error after one of my machines crashed.

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb