Cannot start Mongo after new installation v2.4.8 - "Got Signal 6"

[bassrehæb dæmōn]

Hi guys,
I installed Mongo 2.4.8 using the standard instructions given on my Centos 6.4 - Dedicated server.
But i am getting the following error when i try to start the mongod

root@ks4005334 [/]# service mongod start
Starting mongod: Fri Nov 29 11:26:57.863 terminate() called, printing stack (if implemented for platform):
0xde05e1 0x6cf3de 0x35b41644be6 0x35b41644c13 0x35b41644d0e 0xdf7bb1 0xdd6a51 0xdd7309 0xdd59a5 0xdd5b09 0x70088c 0x700d2c 0x7010d2 0x701111 0x701563 0x6dd695 0x6df459 0x35b40d79cdd 0x6cf1c9
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo11myterminateEv+0x3e) [0x6cf3de]
 /usr/lib64/libstdc++.so.6(+0xbcbe6) [0x35b41644be6]
 /usr/lib64/libstdc++.so.6(+0xbcc13) [0x35b41644c13]
 /usr/lib64/libstdc++.so.6(+0xbcd0e) [0x35b41644d0e]
 /usr/bin/mongod(_ZN5boost11filesystem36detail6statusERKNS0_4pathEPNS_6system10error_codeE+0x231) [0xdf7bb1]
 /usr/bin/mongod(_ZN5mongo11ProcessInfo16checkNumaEnabledEv+0x41) [0xdd6a51]
 /usr/bin/mongod(_ZN5mongo11ProcessInfo10SystemInfo17collectSystemInfoEv+0x3a9) [0xdd7309]
 /usr/bin/mongod(_ZN5mongo11ProcessInfo20initializeSystemInfoEv+0xa5) [0xdd59a5]
 /usr/bin/mongod(_ZN5mongo36_mongoInitializerFunction_SystemInfoEPNS_18InitializerContextE+0x9) [0xdd5b09]
 /usr/bin/mongod(_ZN5boost6detail8function17function_invoker1IPFN5mongo6StatusEPNS3_18InitializerContextEES4_S6_E6invokeERNS1_15function_bufferES6_+0xc) [0x70088c]
 /usr/bin/mongod(_ZNK5mongo11Initializer7executeERKSt6vectorISsSaISsEERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x13c) [0x700d2c]
 /usr/bin/mongod(_ZN5mongo21runGlobalInitializersERKSt6vectorISsSaISsEERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x32) [0x7010d2]
 /usr/bin/mongod(_ZN5mongo26runGlobalInitializersOrDieERKSt6vectorISsSaISsEERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x21) [0x701111]
 /usr/bin/mongod(_ZN5mongo26runGlobalInitializersOrDieEiPKPKcS3_+0x3b3) [0x701563]
 /usr/bin/mongod() [0x6dd695]
 /usr/bin/mongod(main+0x9) [0x6df459]
 /lib64/libc.so.6(__libc_start_main+0xfd) [0x35b40d79cdd]
 /usr/bin/mongod(__gxx_personality_v0+0x499) [0x6cf1c9]
Fri Nov 29 11:26:57.867 Got signal: 6 (Aborted).

Fri Nov 29 11:26:57.872 Backtrace:
0xde05e1 0x6d0559 0x35b40d8d960 0x35b40d8d8e5 0x35b40d8f0c5 0x6cf3e3 0x35b41644be6 0x35b41644c13 0x35b41644d0e 0xdf7bb1 0xdd6a51 0xdd7309 0xdd59a5 0xdd5b09 0x70088c 0x700d2c 0x7010d2 0x701111 0x701563 0x6dd695
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0559]
 /lib64/libc.so.6(+0x32960) [0x35b40d8d960]
 /lib64/libc.so.6(gsignal+0x35) [0x35b40d8d8e5]
 /lib64/libc.so.6(abort+0x175) [0x35b40d8f0c5]
 /usr/bin/mongod(_ZN5mongo11myterminateEv+0x43) [0x6cf3e3]
 /usr/lib64/libstdc++.so.6(+0xbcbe6) [0x35b41644be6]
 /usr/lib64/libstdc++.so.6(+0xbcc13) [0x35b41644c13]
 /usr/lib64/libstdc++.so.6(+0xbcd0e) [0x35b41644d0e]
 /usr/bin/mongod(_ZN5boost11filesystem36detail6statusERKNS0_4pathEPNS_6system10error_codeE+0x231) [0xdf7bb1]
 /usr/bin/mongod(_ZN5mongo11ProcessInfo16checkNumaEnabledEv+0x41) [0xdd6a51]
 /usr/bin/mongod(_ZN5mongo11ProcessInfo10SystemInfo17collectSystemInfoEv+0x3a9) [0xdd7309]
 /usr/bin/mongod(_ZN5mongo11ProcessInfo20initializeSystemInfoEv+0xa5) [0xdd59a5]
 /usr/bin/mongod(_ZN5mongo36_mongoInitializerFunction_SystemInfoEPNS_18InitializerContextE+0x9) [0xdd5b09]
 /usr/bin/mongod(_ZN5boost6detail8function17function_invoker1IPFN5mongo6StatusEPNS3_18InitializerContextEES4_S6_E6invokeERNS1_15function_bufferES6_+0xc) [0x70088c]
 /usr/bin/mongod(_ZNK5mongo11Initializer7executeERKSt6vectorISsSaISsEERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x13c) [0x700d2c]
 /usr/bin/mongod(_ZN5mongo21runGlobalInitializersERKSt6vectorISsSaISsEERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x32) [0x7010d2]
 /usr/bin/mongod(_ZN5mongo26runGlobalInitializersOrDieERKSt6vectorISsSaISsEERKSt3mapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x21) [0x701111]
 /usr/bin/mongod(_ZN5mongo26runGlobalInitializersOrDieEiPKPKcS3_+0x3b3) [0x701563]
 /usr/bin/mongod() [0x6dd695]

Same error is reported when i explicitly try to invoke the init script : /etc/rc.d/init.d/mongod start
but interestingly, the mongod instance runs when i just issue "mongod" at the command line, and mongo accepts connections on 27017 & 18.


I have read around the web for possible solutions, and implemented them but it hasnt solved the problem..
eg: export LC_ALL=C
   export LC_CTYPE="en_US.UTF-8"

i have included the above even in my .bashrc and .bashrc_profile but none have solved the issue.


Can you guys suggest a solution or give a heads up?

Thank you.


[Adam C]

This looks like what happens if the user you are running MongoDB as (usually mongod or mongodb) cannot access the required folders to actually start because of permissions issues.  You should take a look at your log, data, folder permissions (and the files therein) to make sure they are all owned by the mongodb user.  This usually happens when you have run MongoDB as root and hence have root ownership of the folders/files in question, but there are other reasons too.

Adam

[bassrehæb dæmōn]

Hi Adam,

Thanks for the reply. 

I checked all the folders (and files) i know of, all are owned by mongod user.

/var/lib/mongo 

/var/log/mongo

/data/db

but it still gives the sme error when trying to launch using 'service mongod start'. what else can I try?

[Adam C]

Can you post the relevant lines from the mongod log when this happens?  I would like to see the preceding lines in the log, look for more clues there - it looks like what happens with a permissions problem, but perhaps there is another way to trigger the problem.  Oh, and can you double check the init script itself and make sure the user is being set correctly?

Adam.

[bassrehæb dæmōn]

Hi Adam,

I checked the mongo log at /var/log/mongo/mongod.log is empty

options from the init script

CONFIGFILE="/etc/mongod.conf"

OPTIONS=" -f $CONFIGFILE"

SYSCONFIG="/etc/sysconfig/mongod"

mongod=${MONGOD-/usr/bin/mongod}

MONGO_USER=mongod

MONGO_GROUP=mongod

relevant Contents of /etc/mongod.conf

logpath=/var/log/mongo/mongod.log

logappend=true

fork = true

#port = 27017

dbpath=/var/lib/mongo

pidfilepath = /var/run/mongodb/mongod.pid

I see the dbpath at mongod.conf set at /var/lib/mongo but i also had created "/data/db" earlier where i can see the journals,etc.

the init script/daemon "service mongod start" doesnt run and throws the mentioned errors, but the fork "mongo" instance runs

root@ks4005334 [~]# mongod

mongod --help for help and startup options

Fri Nov 29 21:38:07.103 [initandlisten] MongoDB starting : pid=9696 port=27017 dbpath=/data/db/ 64-bit host=ks4005334.ip-192-99-10.net

Fri Nov 29 21:38:07.103 [initandlisten] db version v2.4.8

Fri Nov 29 21:38:07.104 [initandlisten] git version: a350fc38922fbda2cec8d5dd842237b904eafc14

Fri Nov 29 21:38:07.104 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49

Fri Nov 29 21:38:07.104 [initandlisten] allocator: tcmalloc

Fri Nov 29 21:38:07.104 [initandlisten] options: {}

Fri Nov 29 21:38:07.179 [initandlisten] journal dir=/data/db/journal

Fri Nov 29 21:38:07.179 [initandlisten] recover : no journal files present, no recovery needed

Fri Nov 29 21:38:07.266 [websvr] admin web console waiting for connections on port 28017

Fri Nov 29 21:38:07.266 [initandlisten] waiting for connections on port 27017

[bassrehæb dæmōn]

Hi Adam, I am attaching some of the files and folder/file permissions so that its easy to correlate.

 

[Adam C]

The fact that the log is empty might provide a clue here - perhaps it is /var/log that is the problem.  Although the file permissions look correct, if the user is unable to CWD to the directory you could see this issue too.  What about the permissions of /var/log and/or the other parent directories?

Another way to debug this would be to switch the log to a different folder, try a different path for dbpath as a test etc.  If you work your way through them you should be able to narrow it down.  Or alternatively use su to try accessing the various paths with the mongod user.

Adam

[bassrehæb dæmōn]

Hi Adam,

I created a logfile at " /" mongolog, chomod it -rwxrwxrwx   1 mongod mongod     0 Nov 29 22:43 mongolog*

db path is now set at /data/db with ownership for mongod:mongod

it still gives the same error- wont start.

I noticed in mongod.conf

pidfilepath = /var/run/mongodb/mongod.pid

/var/run/mongodb - doesnot exist..

could this be the issue?

[Ernie Hershey]

Can you check permissions under /sys/devices/system? You might be hitting the issue mentioned here - https://jira.mongodb.org/browse/SERVER-10623. 

Ernie

--
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb
 
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[bassrehæb dæmōn]

Hi Ernie,

I have tried most of the work arounds of shifting and properly setting up permissions for log/db folders earlier  to no avail. I will go through the link you mentioned.

Here's the file permissions break ups. 

root@ks4005334 [/sys/devices]# ls -l

total 0

drwxr-xr-x 15 root root 0 Nov 29 23:27 ./

dr-xr-xr-x 11 root root 0 Nov 29 23:27 ../

drwx------  2 root root 0 Nov 29 23:27 breakpoint/

drwx------  4 root root 0 Nov 29 23:27 cpu/

drwx------  3 root root 0 Nov 29 23:27 ide0/

drwx------  3 root root 0 Nov 29 23:27 ide1/

drwx------ 21 root root 0 Nov 29 23:27 LNXSYSTM:00/

drwx------ 33 root root 0 Nov 29 23:27 pci0000:00/

drwx------ 22 root root 0 Nov 29 23:27 pci0000:ff/

drwx------ 12 root root 0 Nov 29 23:27 platform/

drwx------ 17 root root 0 Nov 29 23:27 pnp0/

drwx------  2 root root 0 Nov 29 23:27 software/

drwxr-xr-x  8 root root 0 Nov 29 23:27 system/

drwx------  4 root root 0 Nov 29 23:27 uncore/

drwx------ 21 root root 0 Nov 29 23:27 virtual/

root@ks4005334 [/sys/devices/system]# ls -l

total 0

drwxr-xr-x   8 root root 0 Nov 29 23:27 ./

drwxr-xr-x  15 root root 0 Nov 29 23:27 ../

drwx------   3 root root 0 Nov 29 23:27 clocksource/

drwxr-xr-x  13 root root 0 Nov 29 23:27 cpu/

drwx------   4 root root 0 Nov 29 23:27 edac/

drwx------  10 root root 0 Nov 29 23:27 machinecheck/

drwx------ 194 root root 0 Nov 29 23:27 memory/

drwx------   3 root root 0 Nov 29 23:27 node/

root@ks4005334 [/sys/devices/system/node]# ls -l

total 0

drwx------ 3 root root    0 Nov 29 23:27 ./

drwxr-xr-x 8 root root    0 Nov 29 23:27 ../

-r--r--r-- 1 root root 4096 Nov 30 03:49 has_cpu

-r--r--r-- 1 root root 4096 Nov 30 03:49 has_normal_memory

drwx------ 2 root root    0 Nov 29 23:27 node0/

-r--r--r-- 1 root root 4096 Nov 30 03:49 online

-r--r--r-- 1 root root 4096 Nov 30 03:49 possible

-rw-r--r-- 1 root root 4096 Nov 30 03:49 uevent

root@ks4005334 [/proc/self]# ls -l

-r--r--r--   1 root root 0 Nov 30 03:50 numa_maps

[Ernie Hershey]

It looks like only the root user can read or list files under /sys/devices/system/node, which mongod uses at startup to examine information about NUMA on the system.

One way to work around the problem would be to open up permissions in that directory: chmod -R go+rx /sys/devices/system/. 

Ernie

--

[bassrehæb dæmōn]

Ernie,

You were spot on! That worked perfectly.

i had to chmod -R go+rx /sys too (as mongod log showed subsequent errors ; error no.100)

Sat Nov 30 06:34:17.361 [initandlisten] exception in initAndListen std::exception: boost::filesystem::status: Permission denied: "/sys/dev/block/8:1/queue/read_ahead_k$

Sat Nov 30 06:34:17.361 dbexit:

Steps taken to resolve (for anyone who might face the same issue)

1)  chmod -R go+rx /sys/devices/system/

and/or, 2)  chmod -R go+rx /sys

Thanks a lot Ernie, i could not have figured this.

With best regards,

Subhadip M.

[Ernie Hershey]

That's great! I'm glad it worked. 

Ernie