2020-05-21T10:11:01.770+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none' 2020-05-21T10:11:01.776+0000 W ASIO [main] No TransportLayer configured during NetworkInterface startup 2020-05-21T10:11:01.779+0000 I CONTROL [initandlisten] MongoDB starting : pid=139 port=27017 dbpath=/data/db 64-bit host=5071f576509d 2020-05-21T10:11:01.780+0000 I CONTROL [initandlisten] db version v4.2.6 2020-05-21T10:11:01.782+0000 I CONTROL [initandlisten] git version: 20364840b8f1af16917e4c23c1b5f5efd8b352f8 2020-05-21T10:11:01.783+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1 11 Sep 2018 2020-05-21T10:11:01.785+0000 I CONTROL [initandlisten] allocator: tcmalloc 2020-05-21T10:11:01.785+0000 I CONTROL [initandlisten] modules: none 2020-05-21T10:11:01.786+0000 I CONTROL [initandlisten] build environment: 2020-05-21T10:11:01.787+0000 I CONTROL [initandlisten] distmod: ubuntu1804 2020-05-21T10:11:01.788+0000 I CONTROL [initandlisten] distarch: x86_64 2020-05-21T10:11:01.789+0000 I CONTROL [initandlisten] target_arch: x86_64 2020-05-21T10:11:01.790+0000 I CONTROL [initandlisten] options: { config: "/tmp/docker-entrypoint-temp-config.json", net: { bindIp: "127.0.0.1", port: 27017, tls: { mode: "disabled" } }, processManagement: { fork: true, pidFilePath: "/tmp/docker-entrypoint-temp-mongod.pid" }, systemLog: { destination: "file", logAppend: true, path: "/data/db/docker-initdb.log" } } 2020-05-21T10:11:01.849+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3394M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress], 2020-05-21T10:11:02.611+0000 E STORAGE [initandlisten] WiredTiger error (95) [1590055862:611142][139:0x7fe5fe559b00], file:WiredTiger.wt, connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not supported Raw: [1590055862:611142][139:0x7fe5fe559b00], file:WiredTiger.wt, connection: __posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not supported 2020-05-21T10:11:02.625+0000 E STORAGE [initandlisten] WiredTiger error (95) [1590055862:625022][139:0x7fe5fe559b00], wiredtiger_open: __posix_open_file, 667: /data/db/WiredTiger.lock: handle-open: open: Operation not supported Raw: [1590055862:625022][139:0x7fe5fe559b00], wiredtiger_open: __posix_open_file, 667: /data/db/WiredTiger.lock: handle-open: open: Operation not supported 2020-05-21T10:11:02.628+0000 E STORAGE [initandlisten] WiredTiger error (95) [1590055862:628694][139:0x7fe5fe559b00], wiredtiger_open: __posix_open_file, 667: /data/db/WiredTiger.lock: handle-open: open: Operation not supported Raw: [1590055862:628694][139:0x7fe5fe559b00], wiredtiger_open: __posix_open_file, 667: /data/db/WiredTiger.lock: handle-open: open: Operation not supported 2020-05-21T10:11:02.632+0000 E STORAGE [initandlisten] WiredTiger error (95) [1590055862:632835][139:0x7fe5fe559b00], wiredtiger_open: __posix_open_file, 667: /data/db/WiredTiger.lock: handle-open: open: Operation not supported Raw: [1590055862:632835][139:0x7fe5fe559b00], wiredtiger_open: __posix_open_file, 667: /data/db/WiredTiger.lock: handle-open: open: Operation not supported 2020-05-21T10:11:02.636+0000 E STORAGE [initandlisten] WiredTiger error (95) [1590055862:636325][139:0x7fe5fe559b00], wiredtiger_open: __posix_open_file, 667: /data/db/WiredTiger.lock: handle-open: open: Operation not supported Raw: [1590055862:636325][139:0x7fe5fe559b00], wiredtiger_open: __posix_open_file, 667: /data/db/WiredTiger.lock: handle-open: open: Operation not supported 2020-05-21T10:11:02.637+0000 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version. 2020-05-21T10:11:02.638+0000 F STORAGE [initandlisten] Reason: 95: Operation not supported 2020-05-21T10:11:02.639+0000 F - [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 915 2020-05-21T10:11:02.639+0000 F - [initandlisten] ***aborting after fassert() failure
which I understand as referring to code lines https://github.com/wiredtiger/wiredtiger/blob/8de74488f2bb2b5cba0404c345f568a2f72478d3/src/os_posix/os_fs.c#L661-L667WT_SYSCALL_RETRY(((pfh->fd = open(name, f, mode)) == -1 ? -1 : 0), ret); if (ret != 0) WT_ERR_MSG(session, ret, pfh->direct_io ? "%s: handle-open: open: failed with direct I/O configured, " "some filesystem types do not support direct I/O" : "%s: handle-open: open", name);
and I assume that gvfs does not fulfil certain posix standards expected by WiredTiger, but what exactly is missing here for compatibility?
With smbnetfs (in connection with bindfs), I had more success initially, and the conceptually working setup can be found at https://github.com/IMTEK-Simulation/mongod-on-smb/tree/smbnetfs (with
the particular mount options here https://github.com/IMTEK-Simulation/mongod-on-smb/blob/4ec17068bdadd33a0c3db9f1c10f8b156866c9bd/compose/local/mongodb/docker-entrypoint.sh#L51-L55 and here https://github.com/IMTEK-Simulation/mongod-on-smb/blob/4ec17068bdadd33a0c3db9f1c10f8b156866c9bd/compose/local/mongodb/docker-entrypoint.sh#L75-L78),
but apparently that configuration causes other difficulties:
2020-05-25T12:26:28.174+0000 E STORAGE [WTCheckpointThread] WiredTiger error (16) [1590409588:174508][38:0x7f06b7748700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_turtle_update, 395: WiredTiger.turtle: fatal turtle file update error: Device or resource busy Raw: [1590409588:174508][38:0x7f06b7748700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_turtle_update, 395: WiredTiger.turtle: fatal turtle file update error: Device or resource busy |
2020-05-25T12:26:28.174+0000 E STORAGE [WTCheckpointThread] WiredTiger error (-31804) [1590409588:174545][38:0x7f06b7748700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_panic, 490: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1590409588:174545][38:0x7f06b7748700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_panic, 490: the process must exit and restart: WT_PANIC: WiredTiger library panic |
2020-05-25T12:26:28.174+0000 F - [WTCheckpointThread] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 414 |
2020-05-25T12:26:28.174+0000 F - [WTCheckpointThread] |
|
***aborting after fassert() failure |
|
|
2020-05-25T12:26:28.210+0000 F - [WTCheckpointThread] Got signal: 6 (Aborted). |
0x5558e47eb5b1 0x5558e47eabcc 0x5558e47eac56 0x7f06c3064890 0x7f06c2c9fe97 0x7f06c2ca1801 0x5558e2c46c19 0x5558e2983466 0x5558e2de96ab 0x5558e2993cd7 0x5558e299413b 0x5558e2db4a4f 0x5558e2db279c 0x5558e2db03fa 0x5558e2db0e9e 0x5558e2dfcb2f 0x5558e2dfda81 0x5558e2dff3ee 0x5558e2dff8b6 0x5558e29932ce 0x5558e2d31dee 0x5558e46c094f 0x5558e46c0c6e 0x5558e491127f 0x7f06c30596db 0x7f06c2d8288f |
----- BEGIN BACKTRACE ----- |
{"backtrace":[{"b":"5558E1F35000","o":"28B65B1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5558E1F35000","o":"28B5BCC"},{"b":"5558E1F35000","o":"28B5C56"},{"b":"7F06C3052000","o":"12890"},{"b":"7F06C2C61000","o":"3EE97","s":"gsignal"},{"b":"7F06C2C61000","o":"40801","s":"abort"},{"b":"5558E1F35000","o":"D11C19","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5558E1F35000","o":"A4E466"},{"b":"5558E1F35000","o":"EB46AB"},{"b":"5558E1F35000","o":"A5ECD7","s":"__wt_err_func"},{"b":"5558E1F35000","o":"A5F13B","s":"__wt_panic"},{"b":"5558E1F35000","o":"E7FA4F","s":"__wt_turtle_update"},{"b":"5558E1F35000","o":"E7D79C","s":"__wt_metadata_update"},{"b":"5558E1F35000","o":"E7B3FA"},{"b":"5558E1F35000","o":"E7BE9E","s":"__wt_meta_ckptlist_set"},{"b":"5558E1F35000","o":"EC7B2F"},{"b":"5558E1F35000","o":"EC8A81","s":"__wt_checkpoint"},{"b":"5558E1F35000","o":"ECA3EE"},{"b":"5558E1F35000","o":"ECA8B6","s":"__wt_txn_checkpoint"},{"b":"5558E1F35000","o":"A5E2CE"},{"b":"5558E1F35000","o":"DFCDEE","s":"_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv"},{"b":"5558E1F35000","o":"278B94F","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"5558E1F35000","o":"278BC6E"},{"b":"5558E1F35000","o":"29DC27F"},{"b":"7F06C3052000","o":"76DB"},{"b":"7F06C2C61000","o":"12188F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.6", "gitVersion" : "20364840b8f1af16917e4c23c1b5f5efd8b352f8", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.18.0-147.8.1.el8_1.x86_64", "version" : "#1 SMP Thu Apr 9 13:49:54 UTC 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "5558E1F35000", "elfType" : 3, "buildId" : "66B48B81895E2FEAAB0D26C828C647755EB81440" }, { "b" : "7FFD63158000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "AB7BC9738B0D294EFA3164C39D64FA9FBBAC368A" }, { "b" : "7F06C45A6000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "1C6BC2C0699CE0F7E848CA0B267E0CF07553F6AB" }, { "b" : "7F06C438B000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F06C3EC0000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "68CECD8742C0C91CDC82709C3E52A8C9F9451FC5" }, { "b" : "7F06C3C33000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "38AE5B4499DA422AA6D86BAED1902662DED5F730" }, { "b" : "7F06C3A2F000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F06C3827000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F06C3489000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F06C3271000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "679F3AE11120EC7C483BC9295345D836F5C104F7" }, { "b" : "7F06C3052000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F06C2C61000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F06C4825000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F06C2A3C000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7F06C281F000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "EE6E9462BA2491F4EE8C4E52C3323274A9366614" }, { "b" : "7F06C2603000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7F06C23F5000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7F06C21AA000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "00F419F64B0E70D8C5EEF7050369AA40B2A6E090" }, { "b" : "7F06C1F58000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "70EEF126558D1559A0A4E334FB68E4E9AABE90CB" }, { "b" : "7F06C1D4A000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "C14042EC7BD22B9A07D2C16563FE3C2606F52AB7" }, { "b" : "7F06C1B2D000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F06C17AF000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "0E2784298E7D3F4D894FE130ACEFA77C3E624F72" }, { "b" : "7F06C1449000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "1954013F9BBF9FF3A328A8E516481A2BBD8658EB" }, { "b" : "7F06C1215000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, { "b" : "7F06C0FDF000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, { "b" : "7F06C0D5E000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7F06C0A88000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "69FBCF425EE6DF03DE93B82FBC2FC33790E68A96" }, { "b" : "7F06C0856000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "F400D5D643A7F9696DF0E6148FA99BEE6C1BDDF7" }, { "b" : "7F06C0652000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "17107881DF65C66B4C6D38CAB37C285FA44663BD" }, { "b" : "7F06C0447000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "D78D71E8E016A534281B25B97CD7E5E9DB5FE00A" }, { "b" : "7F06C022C000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "6E98533B96F674F77C1BD83AA3565D974D9C4372" }, { "b" : "7F06BFFEB000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7F06BFCBC000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, { "b" : "7F06BFAA9000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7F06BF8A5000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, { "b" : "7F06BF69C000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7F06BF40F000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7F06BF16D000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7F06BEF37000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7F06BED21000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7F06BEB19000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7F06BE8F0000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfTy |
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5558e47eb5b1] |
mongod(+0x28B5BCC) [0x5558e47eabcc] |
mongod(+0x28B5C56) [0x5558e47eac56] |
libpthread.so.0(+0x12890) [0x7f06c3064890] |
libc.so.6(gsignal+0xC7) [0x7f06c2c9fe97] |
libc.so.6(abort+0x141) [0x7f06c2ca1801] |
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5558e2c46c19] |
mongod(+0xA4E466) [0x5558e2983466] |
mongod(+0xEB46AB) [0x5558e2de96ab] |
mongod(__wt_err_func+0x90) [0x5558e2993cd7] |
mongod(__wt_panic+0x39) [0x5558e299413b] |
mongod(__wt_turtle_update+0x1AF) [0x5558e2db4a4f] |
mongod(__wt_metadata_update+0x26C) [0x5558e2db279c] |
mongod(+0xE7B3FA) [0x5558e2db03fa] |
mongod(__wt_meta_ckptlist_set+0x26E) [0x5558e2db0e9e] |
mongod(+0xEC7B2F) [0x5558e2dfcb2f] |
mongod(__wt_checkpoint+0xA1) [0x5558e2dfda81] |
mongod(+0xECA3EE) [0x5558e2dff3ee] |
mongod(__wt_txn_checkpoint+0x256) [0x5558e2dff8b6] |
mongod(+0xA5E2CE) [0x5558e29932ce] |
mongod(_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv+0x8DE) [0x5558e2d31dee] |
mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x9F) [0x5558e46c094f] |
mongod(+0x278BC6E) [0x5558e46c0c6e] |
mongod(+0x29DC27F) [0x5558e491127f] |
libpthread.so.0(+0x76DB) [0x7f06c30596db] |
libc.so.6(clone+0x3F) [0x7f06c2d8288f] |
----- END BACKTRACE ----- |
This happened twice in the middle of restoring a ~100 GB database with the "mongorestore" utility. And it apparently destroys the database, at the next launch mongod would rename the previous files and start from scratch again.
I assume some underlying issues within smbnetfs or bindfs (geographically, storage platform and actual server machine probably stand within the same room, thus real connectivity issues are pretty unlikely and we never encountered any with the cifs setup).
Also here I would be happy for any ideas and interpretations, and I might point the smbnetfs (or bindfs) developers towards that issue.
Now, we resort to mounting the smb shares on the host system with cifs (as before) and forwarding them to the rootless container as volumes (master branch of https://github.com/IMTEK-Simulation/mongod-on-smb),
but since a reliable "mongodb on smb" setup is possible with cifs, why not an entirely rootless setup?
I am aware that the described issues lie within the underlying file system drivers and not within WiredTiger, but I would like to understand a bit more about why I get a working, reliable setup with cifs and two different issues with gvfs and smbnetfs.
Looking forward to any input and discussion.
Thanks!