MongoDB can't startup after a mongodb 4.0 upgrade failure

[Andy Fan]

we can reproduce this error like this:

1.  have a running mongodb 3.6 replication set.

2.  set the rs protocolVersion=0 and rs.reconfig()  # to make the 4.0 startup failure

3.stop one mongo instance and start it with mongodb 4.0  # mongodb 4.0 will failed since it doesn't support protocolVersion=0

4.start the instance with mongodb 3.6, it still failed with error: 

2018-12-25T15:36:57.413+0800 E STORAGE  [initandlisten] WiredTiger error (-31802) [1545723350:852671][102919:0x7ffff22eca80], connection: Version incompatibility detected: unsupported WiredTiger file version: this build requires a maximum version of 2, and the file is version 3: WT_ERROR: non-specific WiredTiger error

now we have to resync the data. 

looks the issue is caused by mongodb 4.0 create a journal file and crashed, then 3.6 can't start the newer version wiredTiger file. 

(gdb) bt

#0  __wt_wiredtiger_error (error=-31802) at src/third_party/wiredtiger/src/conn/api_strerror.c:30

#1  0x00007ffff4b434e0 in __wt_strerror (session=0x7ffff9015000, error=-31802, errbuf=0x0, errlen=0)

    at src/third_party/wiredtiger/src/os_common/os_errno.c:37

#2  0x00007ffff4b86046 in __eventv (session=0x7ffff9015000, msg_event=false, error=-31802, file_name=0x0, line_number=0,

    fmt=0x7ffff6ba5a40 "Version incompatibility detected: unsupported WiredTiger file version: this build requires a maximum version of %u, and the file is version %u", ap=0x7fffffffa350) at src/third_party/wiredtiger/src/support/err.c:250

#3  0x00007ffff4b8657e in __wt_err (session=0x7ffff9015000, error=-31802,

    fmt=0x7ffff6ba5a40 "Version incompatibility detected: unsupported WiredTiger file version: this build requires a maximum version of %u, and the file is version %u") at src/third_party/wiredtiger/src/support/err.c:328

#4  0x00007ffff4c412a9 in __log_open_verify (session=0x7ffff9015000, id=12, fhp=0x0, lsnp=0x0, versionp=0x7fffffffa4ee)

    at src/third_party/wiredtiger/src/log/log.c:983

#5  0x00007ffff4c42a99 in __wt_log_open (session=0x7ffff9015000) at src/third_party/wiredtiger/src/log/log.c:1613

#6  0x00007ffff4c09863 in __wt_logmgr_create (session=0x7ffff9015000, cfg=0x7fffffffa720)

    at src/third_party/wiredtiger/src/conn/conn_log.c:1067

#7  0x00007ffff4b158af in __wt_connection_workers (session=0x7ffff9015000, cfg=0x7fffffffa720)

    at src/third_party/wiredtiger/src/conn/conn_open.c:216

#8  0x00007ffff4b11778 in wiredtiger_open (home=0x7ffff8ddd000 "/data/u11/zhifan/scripts/data/demo2/node3",

    event_handler=0x7ffff8fde6a0,

    config=0x7ffff8ff9000 "create,cache_size=1024M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,compatibility=(release=\"3.0\",require_max=\"3.0\"),log=(enabled=tr"..., connectionp=0x7ffff8fde698)

    at src/third_party/wiredtiger/src/conn/conn_api.c:2718

#9  0x00007ffff4ac3bdf in mongo::WiredTigerKVEngine::WiredTigerKVEngine (this=0x7ffff8fde680, canonicalName=..., path=..., cs=

    0x7ffff8feb8c0, extraOpenOptions=..., cacheSizeMB=1024, durable=true, ephemeral=false, repair=false, readOnly=false)

    at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:414

#10 0x00007ffff4aab44f in mongo::(anonymous namespace)::WiredTigerFactory::create (this=0x7ffff8b98380, params=..., lockFile=

    0x7ffff8dbdd20) at src/mongo/db/storage/wiredtiger/wiredtiger_init.cpp:113

#11 0x00007ffff4d18d33 in mongo::ServiceContextMongoD::initializeGlobalStorageEngine (this=0x7ffff8cc0300)

    at src/mongo/db/service_context_d.cpp:202

#12 0x00007ffff4a62fe7 in mongo::(anonymous namespace)::_initAndListen (listenPort=27022) at src/mongo/db/db.cpp:749

#13 0x00007ffff4a652d1 in mongo::(anonymous namespace)::initAndListen (listenPort=27022) at src/mongo/db/db.cpp:1072

#14 0x00007ffff4a672e9 in mongo::mongoDbMain (argc=3, argv=0x7fffffffc218, envp=0x7fffffffc238) at src/mongo/db/db.cpp:1424

#15 0x00007ffff4a4a998 in main (argc=3, argv=0x7fffffffc218, envp=0x7fffffffc238) at src/mongo/db/dbmain.cpp:49

(gdb) up 4

#4  0x00007ffff4c412a9 in __log_open_verify (session=0x7ffff9015000, id=12, fhp=0x0, lsnp=0x0, versionp=0x7fffffffa4ee)

    at src/third_party/wiredtiger/src/log/log.c:983

983 WT_ERR_MSG(session, WT_ERROR,

(gdb) p *fh

$1 = {name = 0x7ffff8ddbba0 "journal/WiredTigerLog.0000000012", name_hash = 12158138220901498757, q = {tqe_next = 0x7ffff8ed6cc0,

    tqe_prev = 0x7ffff8fff8b8}, hashq = {tqe_next = 0x0, tqe_prev = 0x7ffff8fff108}, ref = 1, handle = 0x7ffff8febbe0}

[Pierre-Charles BERTINEAU]

Hello Andy,

I get the same error with a standalone upgrade. How did you solve your issue ?

Thanks,

[Kevin Adistambha]

Hi Pierre,

Please note that you’re replying to an old thread from 2018, and the original situation might be quite different although superficially they look similar. For example ,protocolVersion=0 is a replica set only setting, and you mentioned that you’re trying to upgrade a standalone instance.

Could you please open a new thread with some additional details:

• The exact error you are seeing
• From what exact version to what exact version are you trying to upgrade
• Step by step description on your upgrade effort

Best regards,
Kevin

​