Mongodb failed to start after a disk 100% full problem [WiredTiger.wt corrupted]

Paddy Zhu

unread,

Sep 22, 2016, 4:47:50 PM9/22/16

to mongodb-user

My server had a 'No space left on device' problem because our disk was running out of inode early today.

But after we fixed the disk problem, our mongodb was still unable to start or repair.

I desperately need your help : (

And the log information as follow:

2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] MongoDB starting : pid=10992 port=27017 dbpath=/ick/data2 64-bit host=localhost.localdomain
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] db version v3.2.3
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] git version: b326ba837cf6f49d65c2f85e1b70f6f31ece7937
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] modules: none
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] build environment:
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] distmod: rhel62
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] distarch: x86_64
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] target_arch: x86_64
2016-07-25T17:29:53.558+0800 I CONTROL  [initandlisten] options: { repair: true, storage:

{ dbPath: "/ick/data2" }

}
2016-07-25T17:29:53.847+0800 I -  [initandlisten] Detected data files in /ick/data2 created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-07-25T17:29:53.847+0800 I STORAGE  [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2016-07-25T17:29:53.847+0800 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-07-25T17:29:53.898+0800 E STORAGE  [initandlisten] WiredTiger (0) [1469438993:898124][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 196608: block header checksum of 675114611 doesn't match expected checksum of 1687733155
2016-07-25T17:29:53.898+0800 E STORAGE  [initandlisten] WiredTiger (0) [1469438993:898190][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2016-07-25T17:29:53.898+0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1469438993:898209][10992:0x7f69edc18d40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-07-25T17:29:53.898+0800 I -  [initandlisten] Fatal Assertion 28558
2016-07-25T17:29:53.898+0800 I -  [initandlisten]

***aborting after fassert() failure

2016-07-25T17:29:53.946+0800 F - [initandlisten] Got signal: 6 (Aborted).

0x130e1e2 0x130d119 0x130d922 0x3ee060f7e0 0x3ee0232625 0x3ee0233e05 0x1295b32 0x108b3e3 0x1a57e3c 0x1a57fdd 0x1a583c4 0x199067c 0x1990bf3 0x198dbd7 0x19919de 0x19ae1a8 0x19e2140 0x1a56bfb 0x1a57117 0x1a5726b 0x19f02f8 0x1a54415 0x1a20b7f 0x1a20c5e 0x19df078 0x107322e 0x106f6a3 0xf99268 0x974701 0x97815d 0x3ee021ed5d 0x972279
----- BEGIN BACKTRACE -----
{"backtrace":[

{"b":"400000","o":"F0E1E2","s":"_ZN5mongo15printStackTraceERSo"}

,

{"b":"400000","o":"F0D119"}

,

{"b":"400000","o":"F0D922"}

,

{"b":"3EE0600000","o":"F7E0"}

,

{"b":"3EE0200000","o":"32625","s":"gsignal"}

,

{"b":"3EE0200000","o":"33E05","s":"abort"}

,

{"b":"400000","o":"E95B32","s":"_ZN5mongo13fassertFailedEi"}

,

{"b":"400000","o":"C8B3E3"}

,

{"b":"400000","o":"1657E3C","s":"__wt_eventv"}

,

{"b":"400000","o":"1657FDD","s":"__wt_err"}

,

{"b":"400000","o":"16583C4","s":"__wt_panic"}

,

{"b":"400000","o":"159067C","s":"__wt_block_extlist_read"}

,

{"b":"400000","o":"1590BF3","s":"__wt_block_extlist_read_avail"}

,

{"b":"400000","o":"158DBD7","s":"__wt_block_checkpoint_load"}

,

{"b":"400000","o":"15919DE"}

,

{"b":"400000","o":"15AE1A8","s":"__wt_btree_open"}

,

{"b":"400000","o":"15E2140","s":"__wt_conn_btree_open"}

,

{"b":"400000","o":"1656BFB","s":"__wt_session_get_btree"}

,

{"b":"400000","o":"1657117","s":"__wt_session_get_btree"}

,

{"b":"400000","o":"165726B","s":"__wt_session_get_btree_ckpt"}

,

{"b":"400000","o":"15F02F8","s":"__wt_curfile_open"}

,

{"b":"400000","o":"1654415"}

,

{"b":"400000","o":"1620B7F","s":"__wt_metadata_cursor_open"}

,

{"b":"400000","o":"1620C5E","s":"__wt_metadata_cursor"}

,

{"b":"400000","o":"15DF078","s":"wiredtiger_open"}

,

{"b":"400000","o":"C7322E","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"}

,

{"b":"400000","o":"C6F6A3"}

,

{"b":"400000","o":"B99268","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"}

,

{"b":"400000","o":"574701","s":"_ZN5mongo13initAndListenEi"}

,

{"b":"400000","o":"57815D","s":"main"}

,

{"b":"3EE0200000","o":"1ED5D","s":"__libc_start_main"}

,

{"b":"400000","o":"572279"}

],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "2.6.32-573.22.1.el6.x86_64", "version" : "#1 SMP Wed Mar 23 03:35:39 UTC 2016", "machine" : "x86_64" }

, "somap" : [

{ "elfType" : 2, "b" : "400000", "buildId" : "B389239A9108B604FCA797D0F968E253E7BF9F5E" }

,

{ "b" : "7FFEC8C76000", "elfType" : 3, "buildId" : "2F78F7B8A7307DD9C340F3CA735BE2CAA9C157D1" }

,

{ "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B84C31B86733DE212F6886FE6F55630FE56180A9" }

,

{ "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "A30A68D2F579614CBEA988BDAAC20CD56D8C48FC" }

,

{ "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "95159178F1A4A3DBDC7819FBEA2C80E5FCDD6BAC" }

,

{ "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "29B61382141595ECBA6576232E44F2310C3AAB72" }

,

{ "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "5D42D8781F5FF4C06B4F839B85B21786AA44488D" }

,

{ "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }

,

{ "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4BDFA184364D81F7DFD789474C3FB8F98A00" }

,

{ "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C56DD1B811FC0D9263248EBB308C73FCBCD80FC1" }

,

{ "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8E6FA4C4B0594C355C1B90C1D49990368C81A040" }

,

{ "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "959C5E10A47EE8A633E7681B64B4B9F74E242ED5" }

,

{ "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "441FA45097A11508E50D55A3D1FF169BF2BE7C62" }

,

{ "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "F62622218875795666E08B92D176A50791183EEC" }

,

{ "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }

,

{ "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B8DEDADC140347276164C729418C7A37B7224135" }

,

{ "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }

,

{ "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4BDFC7A19C1F328EB4FCFBCE7A1E27606928610D" }

,

{ "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }

,

{ "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C39D7FFB49DFB1B55AD09D1D711AD802123F6623" }

,

{ "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" }

] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130e1e2]
mongod(+0xF0D119) [0x130d119]
mongod(+0xF0D922) [0x130d922]
libpthread.so.0(+0xF7E0) [0x3ee060f7e0]
libc.so.6(gsignal+0x35) [0x3ee0232625]
libc.so.6(abort+0x175) [0x3ee0233e05]
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x1295b32]
mongod(+0xC8B3E3) [0x108b3e3]
mongod(__wt_eventv+0x40C) [0x1a57e3c]
mongod(__wt_err+0x8D) [0x1a57fdd]
mongod(__wt_panic+0x24) [0x1a583c4]
mongod(__wt_block_extlist_read+0x6C) [0x199067c]
mongod(__wt_block_extlist_read_avail+0x33) [0x1990bf3]
mongod(__wt_block_checkpoint_load+0x3B7) [0x198dbd7]
mongod(+0x15919DE) [0x19919de]
mongod(__wt_btree_open+0xC68) [0x19ae1a8]
mongod(__wt_conn_btree_open+0x140) [0x19e2140]
mongod(__wt_session_get_btree+0xEB) [0x1a56bfb]
mongod(__wt_session_get_btree+0x607) [0x1a57117]
mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a5726b]
mongod(__wt_curfile_open+0x1E8) [0x19f02f8]
mongod(+0x1654415) [0x1a54415]
mongod(__wt_metadata_cursor_open+0x5F) [0x1a20b7f]
mongod(__wt_metadata_cursor+0x7E) [0x1a20c5e]
mongod(wiredtiger_open+0x15F8) [0x19df078]
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x45E) [0x107322e]
mongod(+0xC6F6A3) [0x106f6a3]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf99268]
mongod(_ZN5mongo13initAndListenEi+0x321) [0x974701]
mongod(main+0x15D) [0x97815d]
libc.so.6(__libc_start_main+0xFD) [0x3ee021ed5d]
mongod(+0x572279) [0x972279]
----- END BACKTRACE -----

Pierre ARLAUD

unread,

Sep 23, 2016, 5:57:56 AM9/23/16

to mongodb-user

Just FYI I may have a similar issue here https://groups.google.com/forum/#!topic/mongodb-user/PEeTIWC190k

Kevin Adistambha

unread,

Oct 4, 2016, 8:21:11 PM10/4/16

to mongodb-user

Hi,

My server had a ‘No space left on device’ problem because our disk was running out of inode early today.
But after we fixed the disk problem, our mongodb was still unable to start or repair.

I noticed you have opened a SERVER ticket (https://jira.mongodb.org/browse/SERVER-26237) and the issue appears to be fixed.

Best regards,
Kevin

Reply all

Reply to author

Forward