Mongo database critical issue

[Roman Claudiu]

Hello guys,

We've got an emergency situation regarding a production website which had just been released. We just got hit by a heavy DDoS-attack and servers got overloaded by requests. Once the attack had stopped, we've restarted the server with the database and we noticed that the 'WiredTiger.wt' file had just disappeared and that Mongo no longer works. I'm wondering what could we do to restore the database and have Mongo running back, taking into account that the 'WiredTiger.wt' file is missing ?
We've got an instance which has it's MONGO_URL set to another instance where we have the database (here's where the DDoS-attack happened).

Here are some logs that you could check:
----------------------------------------
- When I try on the database server to connect "mongo NAME_OF_DATABASE":

MongoDB shell version: 3.2.11
connecting to: NAME_OF_DATABASE
Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:229:14
@(connect):1:6

exception: connect failed



- When I check the logs "mongo log /var/log/mongodb/mongod.log":

2017-04-04T11:05:18.252+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2017-04-04T11:05:18.255+0000 I CONTROL  [initandlisten] MongoDB starting : pid=2716 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-xxx-xxx-xxx-xxx
2017-04-04T11:05:18.256+0000 I CONTROL  [initandlisten] db version v3.2.11
2017-04-04T11:05:18.256+0000 I CONTROL  [initandlisten] git version: 009580ad490190ba33d1c6253ebd8d91808923e4
2017-04-04T11:05:18.256+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2017-04-04T11:05:18.256+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-04-04T11:05:18.256+0000 I CONTROL  [initandlisten] modules: none
2017-04-04T11:05:18.256+0000 I CONTROL  [initandlisten] build environment:
   2017-04-04T11:05:18.256+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2017-04-04T11:05:18.256+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-04-04T11:05:18.256+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-04-04T11:05:18.256+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
2017-04-04T11:05:18.279+0000 E NETWORK  [initandlisten] Failed to unlink socket file /tmp/mongodb-27017.sock errno:1 Operation not permitted
2017-04-04T11:05:18.279+0000 I -        [initandlisten] Fatal Assertion 28578
2017-04-04T11:05:18.279+0000 I -        [initandlisten]



- When I run "sudo service mongodb status":

mongodb.service - High-performance, schema-free document-oriented database
  Loaded: loaded (/etc/systemd/system/mongodb.service; enabled; vendor preset: enabled)
  Active: failed (Result: exit-code) since Tue 2017-04-04 11:51:06 UTC; 8s ago
 Process: 1384 ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf (code=exited, status=14)
Main PID: 1384 (code=exited, status=14)

Apr 04 11:51:06 ip-172-31-42-103 systemd[1]: Started High-performance, schema-free document-oriented database.
Apr 04 11:51:06 ip-172-31-42-103 systemd[1]: mongodb.service: Main process exited, code=exited, status=14/n/a
Apr 04 11:51:06 ip-172-31-42-103 systemd[1]: mongodb.service: Unit entered failed state.
Apr 04 11:51:06 ip-172-31-42-103 systemd[1]: mongodb.service: Failed with result 'exit-code'.


Thank you!

[Rhys Campbell]

Have you got backups? Thread  here...

http://stackoverflow.com/questions/40891411/recovering-a-missing-wiredtiger-wt

Look like you also have a permissions issue on /tmp/mongodb-27017.sock

Check with...

ls -lh /tmp/mongodb-27017.sock