mongod shutting down unexpectedly

[Matt Walters]

Hi guys, 

I'm having difficulty troubleshooting the following issue. When deploying our application, after doing so N number of times we see mongod die with the following logs: 

Thu Nov 14 14:39:30.787 [initandlisten] connection accepted from 10.68.102.64:9950 #91 (30 connections now open)

Thu Nov 14 14:39:30.788 [conn91] CMD: drop lcm.LocationInfo

Thu Nov 14 14:39:30.796 [conn91] build index lcm.LocationInfo { _id: 1 }

Thu Nov 14 14:39:30.797 [conn91] build index done.  scanned 0 total records. 0 secs

Thu Nov 14 14:39:30.815 [journal] exception in dur::groupCommitLL causing immediate shutdown: boost::filesystem::current_path: No such file or directory

Thu Nov 14 14:39:30.815 dur4

Thu Nov 14 14:39:30.815 Got signal: 6 (Aborted).

 

Thu Nov 14 14:39:30.818 Backtrace:

0xdddd81 0x6d0d29 0x3931c32920 0x3931c328a5 0x3931c34085 0x9eded7 0x91f44e 0x921439 0x92180b 0xe26609 0x3932407851 0x3931ce811d

mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdddd81]

mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0d29]

/lib64/libc.so.6() [0x3931c32920]

/lib64/libc.so.6(gsignal+0x35) [0x3931c328a5]

/lib64/libc.so.6(abort+0x175) [0x3931c34085]

mongod(_ZN5mongo10mongoAbortEPKc+0x47) [0x9eded7]

mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x42e) [0x91f44e]

mongod() [0x921439]

mongod(_ZN5mongo3dur9durThreadEv+0x2fb) [0x92180b]

mongod() [0xe26609]

/lib64/libpthread.so.0() [0x3932407851]

/lib64/libc.so.6(clone+0x6d) [0x3931ce811d]

Has anyone seen this exception before or have insight into what may be happening?

Thanks,

Matt

[Asya Kamsky]

Haven't seen this before but the error suggests some issue with file system/directory MongoDB is using - what's the disk set-up?  Anything unusual about it?

--
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb
 
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[William Zola]

What's happening is that the directory where the journal files are supposed to go is vanishing, at least from the perspective of the 'mongod' process. 

You wouldn't happen to be running a networked filesystem, would you?  Other possibilities include that you're running in a virtualized environment which hasn't been configured properly.  For no good reason, my intuition is telling me that NFS is part of the picture.

 -William

[Matt Walters]

Thanks for the responses, guys. The file system is 100GB san storage presented to the VM via VMFS.

Any ideas?

Thx. 

Matt

[William Zola]

The most likely problem is an error in communicating with the SAN.  How are you mounting the SAN?  Are you using NFS, iSCSI, or some other protocol?  If you examine the SAN for errors, do you find them?  Any networking errors?  Any errors logged in the VM Hypervisor?

 -William 

  -William

-- 

William Zola

MongoDB Inc, Member of Technical Staff

"Bene diagnosticur, bene curator"

--

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb
 
---

You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/LJbmAv_tKzg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.