2016-05-03T01:04:55.211-0400 [conn28306] authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" }
2016-05-03T01:04:55.273-0400 [journal] mongod.exe ...\src\mongo\util\stacktrace.cpp(169) mongo::printStackTrace+0x43
2016-05-03T01:04:55.273-0400 [journal] mongod.exe ...\src\mongo\util\signal_handlers.cpp(107) mongo::`anonymous namespace'::abruptQuit+0xf2
2016-05-03T01:04:55.273-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\src\winsig.c(593) raise+0x1ed
2016-05-03T01:04:55.273-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\src\abort.c(81) abort+0x18
2016-05-03T01:04:55.273-0400 [journal] mongod.exe ...\src\mongo\db\instance.cpp(342) mongo::mongoAbort+0x6e
2016-05-03T01:04:55.273-0400 [journal] mongod.exe ...\src\mongo\db\dur.cpp(647) `mongo::dur::groupCommitWithLimitedLocks'::`1'::catch$0+0x97
2016-05-03T01:04:55.273-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\SELF_64_amd64\crt\prebuild\eh\amd64\handlers.asm(44) _CallSettingFrame+0x20
2016-05-03T01:04:55.273-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\prebuild\eh\frame.cpp(1337) __CxxCallCatchBlock+0xeb
2016-05-03T01:04:55.273-0400 [journal] ntdll.dll RtlCaptureContext+0x3c3
2016-05-03T01:04:55.273-0400 [journal] mongod.exe ...\src\mongo\db\dur.cpp(643) mongo::dur::groupCommitWithLimitedLocks+0x27
2016-05-03T01:04:55.273-0400 [journal] mongod.exe ...\src\mongo\db\dur.cpp(783) mongo::dur::durThreadGroupCommit+0x6d
2016-05-03T01:04:55.273-0400 [journal] mongod.exe ...\src\mongo\db\dur.cpp(855) mongo::dur::durThread+0x27a
2016-05-03T01:04:55.273-0400 [journal] mongod.exe ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185) boost::`anonymous namespace'::thread_start_function+0x21
2016-05-03T01:04:55.273-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314) _callthreadstartex+0x17
2016-05-03T01:04:55.273-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(292) _threadstartex+0x7f
2016-05-03T01:04:55.273-0400 [journal] KERNEL32.DLL BaseThreadInitThunk+0xd
2016-05-03T01:04:55.273-0400 [journal] SEVERE: Got signal: 22 (SIGABRT).
Backtrace:
2016-04-17T01:14:22.408-0400 [conn3032850] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [#.#.#.#:53821]
2016-04-17T01:14:22.892-0400 [journal] error exception in dur::journal error appending to file D:\MongoDB\data\journal\j._390 8192 8192 errno:112 There is not enough space on the disk.
2016-04-17T01:14:22.908-0400 [journal] dbexception in groupCommitLL causing immediate shutdown: 13517 error appending to file D:\MongoDB\data\journal\j._390 8192 8192 errno:112 There is not enough space on the disk.
2016-04-17T01:14:22.924-0400 [journal] SEVERE: dur1
2016-04-17T01:14:25.689-0400 [conn3032205] authenticate db: admin { authenticate: 1, nonce: "xxx", user: "mmsagentuser", key: "xxx" }
2016-04-17T01:14:28.127-0400 [conn3032682] end connection #.#.#.#:52202 (82 connections now open)
2016-04-17T01:14:28.127-0400 [initandlisten] connection accepted from #.#.#.#:52682 #3032928 (83 connections now open)
2016-04-17T01:14:28.127-0400 [conn3032928] authenticate db: MyDB { authenticate: 1, user: "MyUser", nonce: "xxx", key: "xxx" }
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\util\stacktrace.cpp(169) mongo::printStackTrace+0x43
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\util\signal_handlers.cpp(107) mongo::`anonymous namespace'::abruptQuit+0xf2
2016-04-17T01:14:32.517-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\src\winsig.c(593) raise+0x1ed
2016-04-17T01:14:32.517-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\src\abort.c(81) abort+0x18
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\db\instance.cpp(342) mongo::mongoAbort+0x6e
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\db\dur.cpp(647) `mongo::dur::groupCommitWithLimitedLocks'::`1'::catch$0+0x97
2016-04-17T01:14:32.517-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\SELF_64_amd64\crt\prebuild\eh\amd64\handlers.asm(44) _CallSettingFrame+0x20
2016-04-17T01:14:32.517-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\prebuild\eh\frame.cpp(1337) __CxxCallCatchBlock+0xeb
2016-04-17T01:14:32.517-0400 [journal] ntdll.dll RtlCaptureContext+0x3c3
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\db\dur.cpp(643) mongo::dur::groupCommitWithLimitedLocks+0x27
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\db\dur.cpp(783) mongo::dur::durThreadGroupCommit+0x6d
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\db\dur.cpp(855) mongo::dur::durThread+0x27a
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185) boost::`anonymous namespace'::thread_start_function+0x21
2016-04-17T01:14:32.517-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314) _callthreadstartex+0x17
2016-04-17T01:14:32.517-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(292) _threadstartex+0x7f
2016-04-17T01:14:32.517-0400 [journal] KERNEL32.DLL BaseThreadInitThunk+0xd
2016-04-17T01:14:32.517-0400 [journal] SEVERE: Got signal: 22 (SIGABRT).
Backtrace:
Enter code here...
I found the first occurrence of this issue happening in our environment. It looks like it started after one of the replica set members ran out of disk space. The disk space issue has since been fixed but the crashes continue to intermittently happen with the same error messages. Does this mean my database is corrupt? Would a db.repairDatabase() fix it? Is there any way to determine this for certain?Here's the disk space message we got on the 17th of April along with the first crash message:
2016-04-17T01:14:22.408-0400 [conn3032850] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [#.#.#.#:53821]
2016-04-17T01:14:22.892-0400 [journal] error exception in dur::journal error appending to file D:\MongoDB\data\journal\j._390 8192 8192 errno:112 There is not enough space on the disk.
2016-04-17T01:14:22.908-0400 [journal] dbexception in groupCommitLL causing immediate shutdown: 13517 error appending to file D:\MongoDB\data\journal\j._390 8192 8192 errno:112 There is not enough space on the disk.
2016-04-17T01:14:22.924-0400 [journal] SEVERE: dur1
2016-04-17T01:14:25.689-0400 [conn3032205] authenticate db: admin { authenticate: 1, nonce: "xxx", user: "mmsagentuser", key: "xxx" }
2016-04-17T01:14:28.127-0400 [conn3032682] end connection #.#.#.#:52202 (82 connections now open)
2016-04-17T01:14:28.127-0400 [initandlisten] connection accepted from #.#.#.#:52682 #3032928 (83 connections now open)
2016-04-17T01:14:28.127-0400 [conn3032928] authenticate db: MyDB { authenticate: 1, user: "MyUser", nonce: "xxx", key: "xxx" }
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\util\stacktrace.cpp(169) mongo::printStackTrace+0x43
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\util\signal_handlers.cpp(107) mongo::`anonymous namespace'::abruptQuit+0xf2
2016-04-17T01:14:32.517-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\src\winsig.c(593) raise+0x1ed
2016-04-17T01:14:32.517-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\src\abort.c(81) abort+0x18
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\db\instance.cpp(342) mongo::mongoAbort+0x6e
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\db\dur.cpp(647) `mongo::dur::groupCommitWithLimitedLocks'::`1'::catch$0+0x97
2016-04-17T01:14:32.517-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\SELF_64_amd64\crt\prebuild\eh\amd64\handlers.asm(44) _CallSettingFrame+0x20
2016-04-17T01:14:32.517-0400 [journal] mongod.exe f:\dd\vctools\crt_bld\self_64_amd64\crt\prebuild\eh\frame.cpp(1337) __CxxCallCatchBlock+0xeb
2016-04-17T01:14:32.517-0400 [journal] ntdll.dll RtlCaptureContext+0x3c3
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\db\dur.cpp(643) mongo::dur::groupCommitWithLimitedLocks+0x27
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\db\dur.cpp(783) mongo::dur::durThreadGroupCommit+0x6d
2016-04-17T01:14:32.517-0400 [journal] mongod.exe ...\src\mongo\db\dur.cpp(855) mongo::dur::durThread+0x27a
2016-04-17T01:14:32.517-0400 [
...