I installed MongoDB on Ubuntu 14.04 as per the documentation. It runs and works fine, except that it mysteriously stops every few hours and requires a manual restart with service mongod restart
to get going again. There's nothing in the mongod.log when this happens.
In addition, when I restart and then check status with service mongod status
, it gives me mongod stop/waiting
event hough the application is able to connect to it. I've been searching around on how to debug this, but am stuck at this point.
Hi James,
I installed MongoDB on Ubuntu 14.04 as per the documentation. It runs and works fine, except that it mysteriously stops every few hours and requires a manual restart with
service mongod restart
to get going again. There’s nothing in the mongod.log when this happens.
Could you please provide the following relevant information?
mongod
process was killed by the OOMkiller. Could you check the system log for any hint if this is the case?mongod
log path from /etc/mongod.conf
and then check the mongo.log for any notable event, i.e. “Detected unclean shutdown”.In addition, when I restart and then check status with
service mongod status
, it gives memongod stop/waiting
event hough the application is able to connect to it.
If your mongod service is stopped you should not be able to connect to it. Another possibility is there is a mongod
instance already running on the default port, so the service cannot be started. Do you usually have other mongod
processes running on this machine? It may be the case that if your application is connecting to another mongod
instance not started by the service process. Please check the relevant logs to discover if this is the case.
Regards,
Tom
Hi James,
To verify if your mongod was killed by OOMkiller, you can run the following command: dmesg | egrep -i 'killed process'
to see if you have the following outputs:
Jan 13 02:13:49 ip-xxx-xxx-xxx-xxx kernel: [xxxxxx.xxxxxx] Out of memory: Kill process 'PID' (Process name) score xxx or sacrifice child
Jan 13 02:13:49 ip-xxx-xxx-xxx-xxx kernel: [xxxxxx.xxxxxx] Killed process PID (Process name) total-vm:937032kB, anon-rss:887396kB, file-rss:4kB
[initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
This message typically appears if the mongod
process was killed with SIGKILL (kill -9), please refer to Unix_signal for more details about SIGKILL. OOMkiller sends SIGKILL to any process it kills. Please note that this signal is very disruptive and could result in data loss (see https://docs.mongodb.com/v3.2/tutorial/manage-mongodb-processes/#use-kill).
If you discover the messages above, then the OOMkiller is the reason why your MongoDB was terminated. You may want to review the Production Notes, specifically the section titled Disk and Storage Systems - Swap
If you’re still having issues with your deployment, could you please provide more information:
free -m
to get this information.db.stats()
command? Regards,
Tom