cannot repair database after WiredTiger.turtle got erased because disk got full

[Timothee Cour]

I'm running mongod with dbpath mounted on NFS (which may not be a good idea in hindsight after seeing this happened)

the NFS disk got full, and I couldn't restart mongod, looking at the logs (see [1]) it wasn't super clear.

but the issue seems that mongodb /WiredTiger.turtle is empty (which i'm guessing is due to mongod attempting to write to it and then ending up clearing this file after the NFS mounted disk was full).

How can I recover my data from a missing WiredTiger.turtle ?

I tried:

mongod --dbpath mongodb --repair --storageEngine wiredTiger

which gave [2] and didn't help because it didn't seem to know how to handle the existing files, eg: unexpected file _mdb_catalog.wt found, renamed to _mdb_catalog.wt.1; unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.1

Now when I restart mongod and run mongo client almost all my data is gone (but, curiously, I'm getting about 32 documents in 3 collections out of millions of documents in ~ 10 collections)

Please advise.

NOTE: no backup in this case (yes I know...)

[1]

2017-04-30T19:08:11.653-0700 I CONTROL  [main] ***** SERVER RESTARTED *****

2017-04-30T19:08:11.661-0700 I CONTROL  [initandlisten] MongoDB starting : pid=44543 port=27017 dbpath=mongodb 64-bit host=face1

2017-04-30T19:08:11.661-0700 I CONTROL  [initandlisten] db version v3.4.1

2017-04-30T19:08:11.661-0700 I CONTROL  [initandlisten] git version: 5e103c4f5583e2566a45d740225dc250baacfbd7

2017-04-30T19:08:11.661-0700 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2k  26 Jan 2017

2017-04-30T19:08:11.661-0700 I CONTROL  [initandlisten] allocator: tcmalloc

2017-04-30T19:08:11.661-0700 I CONTROL  [initandlisten] modules: none

2017-04-30T19:08:11.661-0700 I CONTROL  [initandlisten] build environment:

2017-04-30T19:08:11.661-0700 I CONTROL  [initandlisten]     distarch: x86_64

2017-04-30T19:08:11.661-0700 I CONTROL  [initandlisten]     target_arch: x86_64

2017-04-30T19:08:11.661-0700 I CONTROL  [initandlisten] options: { config: "mongodb.conf", net: { bindIp: "..." }, storage: { dbPath: "mongodb" }, systemLog: {destination: "file", logAppend: true, path: "mongodb.log" } }

2017-04-30T19:08:11.661-0700 W -        [initandlisten] Detected unclean shutdown -mongodb/mongod.lock is not empty.

2017-04-30T19:08:11.705-0700 I -        [initandlisten] Detected data files in mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.

2017-04-30T19:08:11.705-0700 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.

2017-04-30T19:08:11.705-0700 I STORAGE  [initandlisten]

2017-04-30T19:08:11.705-0700 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine

2017-04-30T19:08:11.705-0700 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem

2017-04-30T19:08:11.705-0700 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=112286M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),

2017-04-30T19:08:11.715-0700 I -        [initandlisten] Assertion: 28595:2: No such file or directory src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 267

2017-04-30T19:08:11.715-0700 I STORAGE  [initandlisten] exception in initAndListen: 28595 2: No such file or directory, terminating

2017-04-30T19:08:11.715-0700 I NETWORK  [initandlisten] shutdown: going to close listening sockets...

2017-04-30T19:08:11.715-0700 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock

2017-04-30T19:08:11.715-0700 I NETWORK  [initandlisten] shutdown: going to flush diaglog...

2017-04-30T19:08:11.715-0700 I CONTROL  [initandlisten] now exiting

2017-04-30T19:08:11.715-0700 I CONTROL  [initandlisten] shutting down with code:100

[2]

2017-04-30T21:11:55.161-0700 I CONTROL  [initandlisten] MongoDB starting : pid=17179 port=27017 dbpath=mongodb 64-bit host=face1

2017-04-30T21:11:55.161-0700 I CONTROL  [initandlisten] db version v3.4.1

2017-04-30T21:11:55.161-0700 I CONTROL  [initandlisten] git version: 5e103c4f5583e2566a45d740225dc250baacfbd7

2017-04-30T21:11:55.161-0700 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2k  26 Jan 2017

2017-04-30T21:11:55.161-0700 I CONTROL  [initandlisten] allocator: tcmalloc

2017-04-30T21:11:55.161-0700 I CONTROL  [initandlisten] modules: none

2017-04-30T21:11:55.161-0700 I CONTROL  [initandlisten] build environment:

2017-04-30T21:11:55.161-0700 I CONTROL  [initandlisten]     distarch: x86_64

2017-04-30T21:11:55.161-0700 I CONTROL  [initandlisten]     target_arch: x86_64

2017-04-30T21:11:55.161-0700 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "mongodb", engine: "wiredTiger" } }

2017-04-30T21:11:55.208-0700 I STORAGE  [initandlisten]

2017-04-30T21:11:55.208-0700 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine

2017-04-30T21:11:55.208-0700 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem

2017-04-30T21:11:55.208-0700 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.

2017-04-30T21:11:55.208-0700 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=112286M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),

2017-04-30T21:11:55.310-0700 E STORAGE  [initandlisten] WiredTiger error (17) [1493611915:310444][17179:0x7fc182dd2cc0], connection: mongodb/WiredTiger.wt: handle-open: open: File exists

2017-04-30T21:11:55.310-0700 I STORAGE  [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.1

2017-04-30T21:11:56.094-0700 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=112286M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),

2017-04-30T21:11:56.177-0700 E STORAGE  [initandlisten] WiredTiger error (17) [1493611916:177387][17179:0x7fc182dd2cc0], WT_SESSION.create: mongodb/_mdb_catalog.wt: handle-open: open: File exists

2017-04-30T21:11:56.177-0700 I STORAGE  [initandlisten] WiredTiger message unexpected file _mdb_catalog.wt found, renamed to _mdb_catalog.wt.1

2017-04-30T21:11:56.218-0700 I CONTROL  [initandlisten]

2017-04-30T21:11:56.218-0700 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.

2017-04-30T21:11:56.218-0700 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.

2017-04-30T21:11:56.218-0700 I CONTROL  [initandlisten]

2017-04-30T21:11:56.220-0700 I CONTROL  [initandlisten]

2017-04-30T21:11:56.220-0700 I CONTROL  [initandlisten] ** WARNING: You are running on a NUMA machine.

2017-04-30T21:11:56.220-0700 I CONTROL  [initandlisten] **          We suggest launching mongod like this to avoid performance problems:

2017-04-30T21:11:56.220-0700 I CONTROL  [initandlisten] **              numactl --interleave=all mongod [other options]

2017-04-30T21:11:56.220-0700 I CONTROL  [initandlisten]

2017-04-30T21:11:56.220-0700 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.

2017-04-30T21:11:56.220-0700 I CONTROL  [initandlisten] **        We suggest setting it to 'never'

2017-04-30T21:11:56.220-0700 I CONTROL  [initandlisten]

2017-04-30T21:11:56.220-0700 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.

2017-04-30T21:11:56.220-0700 I CONTROL  [initandlisten] **        We suggest setting it to 'never'

2017-04-30T21:11:56.220-0700 I CONTROL  [initandlisten]

2017-04-30T21:11:56.220-0700 I STORAGE  [initandlisten] finished checking dbs

2017-04-30T21:11:56.220-0700 I NETWORK  [initandlisten] shutdown: going to close listening sockets...

2017-04-30T21:11:56.221-0700 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock

2017-04-30T21:11:56.221-0700 I NETWORK  [initandlisten] shutdown: going to flush diaglog...

2017-04-30T21:11:56.221-0700 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down

2017-04-30T21:11:56.264-0700 I STORAGE  [initandlisten] shutdown: removing fs lock...

2017-04-30T21:11:56.264-0700 I CONTROL  [initandlisten] now exiting

2017-04-30T21:11:56.264-0700 I CONTROL  [initandlisten] shutting down with code:0

[Kevin Adistambha]

Hi Timothee,

How can I recover my data from a missing WiredTiger.turtle ?

The WiredTiger.turtle file contains information required by the WiredTiger storage engine to understand the other data files. Unfortunately, there is no automated method currently available to recreate this file. Restoring from a known good backup is probably the best avenue at this point.

There is some information on using NFS in the Production Notes, but generally it is not recommended since it involves performance degradation at best, and data loss at worst.

I also noticed there are some warnings printed to the mongod logs regarding your environment (access control, NUMA hardware, and transparent huge page). I would recommend reviewing the Production Notes and the Security Checklist for best practices.

Best regards,
Kevin

​