...encountered an illegal file format or internal value

[ctasb...@gmail.com]

Hello,

We are using MongoDB 3.4.7 with wiredTiger on numerous remote stand alone systems where the client & server reside on the same box.  Unfortunately these systems are subject to being forcibly shut down.  Occasionally on some systems after a forced shutdown, our MongoDB service does not start and the MongoDB log files contain something resembling the following error: 

2018-04-19T16:29:37.425-0500 W -                   [initandlisten] Detected unclean shutdown - c:\Data\MongoDB\data\db\mongod.lock is not empty.

2018-04-19T16:29:37.426-0500 I -                     [initandlisten] Detected data files in c:\Data\MongoDB\data\db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.

2018-04-19T16:29:37.426-0500 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.

2018-04-19T16:29:37.426-0500 I STORAGE    [initandlisten] wiredtiger_open config: create,cache_size=3527M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),

2018-04-19T16:29:37.431-0500 E STORAGE  [initandlisten] WiredTiger error (0) [1524173377:431385][7268:1993686752], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value

2018-04-19T16:29:37.431-0500 E STORAGE  [initandlisten] WiredTiger error (-31804) [1524173377:431385][7268:1993686752], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic

2018-04-19T16:29:37.431-0500 I -                    [initandlisten] Fatal Assertion 28558 at src\mongo\db\storage\wiredtiger\wiredtiger_util.cpp 361

2018-04-19T16:29:37.431-0500 I -                    [initandlisten] 

***aborting after fassert() failure

From what I have learned, this error indicates corruption on the disk. Is it likely that these issues are being caused by a forced shutdown?  Does anybody know of any steps to help prevent this issue?  (Aside from the obvious of preventing the systems from being forcibly shut down.)  Or does anybody know an effective way of recovering from this issue with minimal data loss?

Thank You,

Chris

[Kevin Adistambha]

Hi Chris

From what I have learned, this error indicates corruption on the disk. Is it likely that these issues are being caused by a forced shutdown?

Corruption appears to be the main issue from the log you posted. Forced shutdown is definitely one reason, since in a forced shutdown, there is no way WiredTiger can confirm that all data are persisted correctly. Unfortunately in the face of corrupt data, the best way forward is to restore from a known good backup.

Having said that, forced shutdowns are not good for any system. It is best if you can design your system around the need to pull the plug as part of a normal operation.

Another possibility is that you have a hardware defect. Do you see this corruption more often on a certain node compared to others?

Best regards
Kevin

​

[ctasb...@gmail.com]

Thanks Kevin,

I've seen 1 instance where it looked like a hardware issue, otherwise based on our logging, it appears the system was forcibly shut down. 

And unfortunately our system has to be hard to shut down, with only a small set of administrators in theory knowing how to do so.  Unfortunately these administrators often don't know how to shut the system down or don't want to take the time to shut it down properly.  So they just pull the plug.  And, unfortunately, administrator training and supervision is outside my company's control.

Thanks Again,

Chris