Cannot Start MongoDB - WT_ERROR: non-specific WiredTiger error

[Victor Inocentes]

Hello everyone, 

Hope there is someone who can help me in this situation.

I am running a MongoDB EC2 Instance and ran out of space so the service has shutdown. I have taken it up with the Bitnami Community but we've failed to reach a resolution to this problem. We tried restoring the permissions of the files affected to the original state of the instance prior to shutdown. 

The log below is the last message so I believe that we have narrowed down the error to its true source but I still cannot restart the service

mongodb@ip-10-10-222-193:/home/bitnami$ /opt/bitnami/mongodb/bin/mongod --repair --dbpath /opt/bitnami/mongodb/data/db
2018-11-21T13:10:00.866+0000 I CONTROL  [initandlisten] MongoDB starting : pid=24158 port=27017 dbpath=/opt/bitnami/mongodb/data/db 64-bit host=ip-10-10-222-193
2018-11-21T13:10:00.866+0000 I CONTROL  [initandlisten] db version v3.4.10
2018-11-21T13:10:00.866+0000 I CONTROL  [initandlisten] git version: 078f28920cb24de0dd479b5ea6c66c644f6326e9
2018-11-21T13:10:00.866+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-11-21T13:10:00.866+0000 I CONTROL  [initandlisten] modules: none
2018-11-21T13:10:00.866+0000 I CONTROL  [initandlisten] build environment:
2018-11-21T13:10:00.866+0000 I CONTROL  [initandlisten]     distarch: x86_64
2018-11-21T13:10:00.866+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2018-11-21T13:10:00.866+0000 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/opt/bitnami/mongodb/data/db" } }
2018-11-21T13:10:00.888+0000 I -        [initandlisten] Detected data files in /opt/bitnami/mongodb/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-11-21T13:10:00.888+0000 I STORAGE  [initandlisten]
2018-11-21T13:10:00.888+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-11-21T13:10:00.888+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-11-21T13:10:00.888+0000 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2018-11-21T13:10:00.888+0000 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=3479M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-11-21T13:10:00.896+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1542805800:896663][24158:0x7f3fc3466d40], file:WiredTiger.wt, connection: /opt/bitnami/mongodb/data/db/WiredTiger.wt: handle-read: pread: failed to read 4096 bytes at offset 53248: WT_ERROR: non-specific WiredTiger error
2018-11-21T13:10:00.897+0000 I -        [initandlisten] Assertion: 28718:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 253
2018-11-21T13:10:00.897+0000 I STORAGE  [initandlisten] exception in initAndListen: 28718 -31802: WT_ERROR: non-specific WiredTiger error, terminating
2018-11-21T13:10:00.897+0000 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2018-11-21T13:10:00.897+0000 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-11-21T13:10:00.897+0000 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2018-11-21T13:10:00.897+0000 I CONTROL  [initandlisten] now exiting
2018-11-21T13:10:00.897+0000 I CONTROL  [initandlisten] shutting down with code:100

Sincerely appreciate the assistance and thank you in advance for the help!! 

 

[Kevin Adistambha]

Hi Victor,

The error message you posted seems to indicate that the WiredTiger.wt file is corrupted.

Having said that, MongoDB 4.0 has improved --repair behaviour. If you are able to, could you run --repair on a copy of the data files using the 4.0 binary and see if it allows MongoDB to start:

1. Download the MongoDB 4.0 binary
2. Take a backup of the dbpath
3. Run mongod --repair on the dbpath using the MongoDB 4.0 binary
4. Try starting with MongoDB 3.4.10

Note that this is not guaranteed to be successful, and should be done as a last resort if you don’t have a recent backup.

Best regards,
Kevin

​