Mongodb crashed with "File too large"

[al...@seoclarity.net]

Hi There

we have a 8TB db. and Yesterday db crashed without any reasons. after dig into log and we are seeing the below errors.

i tried to restart that node,it won't start.also we had replication on another server but that server e crashed too. please take a look

2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (27) [1486569324:661608][9029:0x7f8bef870700], file:collection-9-2207001049530720609.wt, eviction-worker: collection-9-2207001049530720609.wt: handle-write: pwrite: failed to write 4096 bytes at offset 2194719883264: File too large
2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (27) [1486569324:661933][9029:0x7f8bef870700], eviction-worker: cache eviction worker error: File too large
2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (-31804) [1486569324:661979][9029:0x7f8bef870700], eviction-worker: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-02-08T09:55:24.662-0600 I - [thread1] Fatal Assertion 28558
2017-02-08T09:55:24.662-0600 E STORAGE [initandlisten] WiredTiger (-31804) [1486569324:661994][9029:0x7f8bf3298cc0], file:sizeStorer.wt, txn-recover: Operation failed during recovery: WT_PANIC: WiredTiger library panic
2017-02-08T09:55:24.662-0600 I - [thread1]

[al...@seoclarity.net]

2017-02-08T10:58:35.004-0600 I -        [initandlisten] Detected data files in /opt/R1/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.

2017-02-08T10:58:35.004-0600 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=15G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),

2017-02-08T10:58:38.297-0600 E STORAGE  [thread1] WiredTiger (27) [1486573118:296987][9532:0x7f4f6a07e700], file:collection-9-2207001049530720609.wt, eviction-worker: collection-9-2207001049530720609.wt: handle-write: pwrite: failed to write 4096 bytes at offset 2194719883264: File too large

2017-02-08T10:58:38.297-0600 E STORAGE  [thread1] WiredTiger (27) [1486573118:297392][9532:0x7f4f6a07e700], eviction-worker: cache eviction worker error: File too large

2017-02-08T10:58:38.297-0600 E STORAGE  [thread1] WiredTiger (-31804) [1486573118:297433][9532:0x7f4f6a07e700], eviction-worker: the process must exit and restart: WT_PANIC: WiredTiger library panic

2017-02-08T10:58:38.297-0600 I -        [thread1] Fatal Assertion 28558

2017-02-08T10:58:38.297-0600 I -        [thread1]

***aborting after fassert() failure

2017-02-08T10:58:38.297-0600 E STORAGE  [initandlisten] WiredTiger (-31804) [1486573118:297567][9532:0x7f4f6caa4cc0], file:sizeStorer.wt, txn-recover: Operation failed during recovery: WT_PANIC: WiredTiger library panic

2017-02-08T10:58:38.297-0600 I -        [initandlisten] Fatal Assertion 28558

2017-02-08T10:58:38.297-0600 I -        [initandlisten]

***aborting after fassert() failure

2017-02-08T10:58:38.304-0600 F -        [thread1] Got signal: 6 (Aborted).

0x1315982 0x1314ad9 0x13152e2 0x7f4f6b71f340 0x7f4f6b380cc9 0x7f4f6b3840d8 0x129f992 0x109ad23 0x1a7676c 0x1a76c2d 0x1a77014 0x1a22473 0x7f4f6b717182 0x7f4f6b44447d

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"400000","o":"F15982","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F14AD9"},{"b":"400000","o":"F152E2"},{"b":"7F4F6B70F000","o":"10340"},{"b":"7F4F6B34A000","o":"36CC9","s":"gsignal"},{"b":"7F4F6B34A000","o":"3A0D8","s":"abort"},{"b":"400000","o":"E9F992","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C9AD23"},{"b":"400000","o":"167676C","s":"__wt_eventv"},{"b":"400000","o":"1676C2D","s":"__wt_err"},{"b":"400000","o":"1677014","s":"__wt_panic"},{"b":"400000","o":"1622473"},{"b":"7F4F6B70F000","o":"8182"},{"b":"7F4F6B34A000","o":"FA47D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.6", "gitVersion" : "05552b562c7a0b3143a729aaa0838e558dc49b25", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.2.0-36-generic", "version" : "#41~14.04.1-Ubuntu SMP Tue Apr 19 17:03:32 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B35B0D2B9CB934315BA086AB5C88C919B9B5BA50" }, { "b" : "7FFFFD9CD000", "elfType" : 3, "buildId" : "8B2559B12BD6E6C7212A35F980FB4E639F6DAB3B" }, { "b" : "7F4F6C631000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "74864DB9D5F69D39A67E4755012FB6573C469B3D" }, { "b" : "7F4F6C255000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "AAE7CFF8351B730830BDBCE0DCABBE06574B7144" }, { "b" : "7F4F6C04D000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "B376100CAB1EAC4E5DE066EACFC282BF7C0B54F3" }, { "b" : "7F4F6BE49000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "67699FFDA9FD2A552032E0652A242E82D65AA10D" }, { "b" : "7F4F6BB43000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "EF3F6DFFA1FBE48436EC6F45CD3AABA157064BB4" }, { "b" : "7F4F6B92D000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F4F6B70F000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "AF06068681750736E0524DF17D5A86CB2C3F765C" }, { "b" : "7F4F6B34A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "5382058B69031CAA9B9996C11061CD164C9398FF" }, { "b" : "7F4F6C890000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "2A816C3EBBA4E12813FBD34B06FBD25BC892A67F" } ] }}

mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1315982]

mongod(+0xF14AD9) [0x1314ad9]

mongod(+0xF152E2) [0x13152e2]

libpthread.so.0(+0x10340) [0x7f4f6b71f340]

libc.so.6(gsignal+0x39) [0x7f4f6b380cc9]

libc.so.6(abort+0x148) [0x7f4f6b3840d8]

mongod(_ZN5mongo13fassertFailedEi+0x82) [0x129f992]

mongod(+0xC9AD23) [0x109ad23]

mongod(__wt_eventv+0x42C) [0x1a7676c]

mongod(__wt_err+0x8D) [0x1a76c2d]

mongod(__wt_panic+0x24) [0x1a77014]

mongod(+0x1622473) [0x1a22473]

libpthread.so.0(+0x8182) [0x7f4f6b717182]

libc.so.6(clone+0x6D) [0x7f4f6b44447d]

-----  END BACKTRACE  -----

[Weishan Ang]

Hello!

What filesystem are you using?

[al...@seoclarity.net]

HI 

we are using EXT3 

On Thursday, February 9, 2017 at 9:28:41 AM UTC-6, al...@seoclarity.net wrote:

[Stephen Steneker]

On Friday, 10 February 2017 03:32:06 UTC+11, alps wrote:

we are using EXT3 

Hi,

As you noted on SERVER-27948 (which you also reported for this issue), the maximum file size for EXT3 with 4KiB blocks is 2TiB so the "File too large" error is expected for a 2 TiB data file. The solution for this issue is to upgrade your filesystem; more modern filesystems like XFS and EXT4 support significantly larger limits.

Per the MongoDB Production Notes, use of XFS is strongly recommended with WiredTiger. It would also be worth reviewing ulimit settings to ensure adequate resource limits for your use case.

Regards,

Stennie

[al...@seoclarity.net]

thank you