MongoDB 3.2.5 service fails to start in CentOS 7

[Mikko Tommila]

My server was running MongoDB 3.2.4 fine but after MongoDB 3.2.5 was released, now it won't start. The operating system is CentOS 7. When I start the mongod service, it just immediately fails and there's even nothing written in the log. If I uninstall the mongodb 3.2.5 packages and install mongodb-org-3.2.4 then the service starts just fine.

The error messages are:

[root@ip-192-168-41-156 ~]# service mongod start

Starting mongod (via systemctl):  Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.

                                                           [FAILED]

[root@ip-192-168-41-156 ~]# systemctl status mongod.service

● mongod.service - SYSV: Mongo is a scalable, document-oriented database.

   Loaded: loaded (/etc/rc.d/init.d/mongod)

   Active: failed (Result: exit-code) since Fri 2016-04-15 10:45:46 UTC; 43s ago

     Docs: man:systemd-sysv-generator(8)

  Process: 24605 ExecStart=/etc/rc.d/init.d/mongod start (code=exited, status=1/FAILURE)

 Main PID: 13547 (code=exited, status=0/SUCCESS)

Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: Starting SYSV: Mongo is a scalable, document-oriented database....

Apr 15 10:45:46 ip-192-168-41-156 runuser[24612]: pam_unix(runuser:session): session opened for user mongod by (uid=0)

Apr 15 10:45:46 ip-192-168-41-156 runuser[24612]: pam_unix(runuser:session): session closed for user mongod

Apr 15 10:45:46 ip-192-168-41-156 mongod[24605]: Starting mongod: [FAILED]

Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: mongod.service: control process exited, code=exited status=1

Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..

Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: Unit mongod.service entered failed state.

Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: mongod.service failed.

[root@ip-192-168-41-156 ~]# journalctl -xe

Apr 15 10:45:14 ip-192-168-41-156 yum[24479]: Installed: mongodb-org-3.2.5-1.el7.x86_64

Apr 15 10:45:46 ip-192-168-41-156 polkitd[13528]: Registered Authentication Agent for unix-process:24600:1421059 (system bus name :1.95 [/usr/bin/pkttyagent --n

Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: Starting SYSV: Mongo is a scalable, document-oriented database....

-- Subject: Unit mongod.service has begun start-up

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit mongod.service has begun starting up.

Apr 15 10:45:46 ip-192-168-41-156 runuser[24612]: pam_unix(runuser:session): session opened for user mongod by (uid=0)

Apr 15 10:45:46 ip-192-168-41-156 runuser[24612]: pam_unix(runuser:session): session closed for user mongod

Apr 15 10:45:46 ip-192-168-41-156 mongod[24605]: Starting mongod: [FAILED]

Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: mongod.service: control process exited, code=exited status=1

Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..

-- Subject: Unit mongod.service has failed

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit mongod.service has failed.

--

-- The result is failed.

Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: Unit mongod.service entered failed state.

Apr 15 10:45:46 ip-192-168-41-156 systemd[1]: mongod.service failed.

Apr 15 10:45:46 ip-192-168-41-156 polkitd[13528]: Unregistered Authentication Agent for unix-process:24600:1421059 (system bus name :1.95, object path /org/free

The access rights to the data directory, log directory and tmp directory seem to be correct.

What could be wrong?

Mikko

[Weishan Ang]

Hi,

What's the SELinux status?

[Mikko Tommila]

In /etc/selinux/config I have:

SELINUX=enforcing

SELINUXTYPE=targeted

I have also run:

semanage port -a -t mongod_port_t -p tcp 27017

(but whether I run the above command or not does not seem to make a difference, nor does it with MongoDB 3.2.4)

Mikko

[etter...@gmail.com]

Same here, when I set setenforce 0 it actually starts with 3.2.5 as well. So I downgraded to 3.2.4 to leave setenforce to 1

[Everyharu]

me too.

same problem

2016년 4월 15일 금요일 오후 8시 2분 59초 UTC+9, Mikko Tommila 님의 말:

[Wan Bachtiar]

Hi Mikko,

I tested this with two CentOS instances:

• /etc/redhat-release : CentOS Linux release 7.2.1511 (Core) and CentOS Linux release 7.1.1503 (Core)
• uname: 3.10.0-229.14.1.el7.x86_64 x86_64 x86_64

Using MongoDB v3.2.5:

> mongod --version 
db version v3.2.5
git version: 34e65e5383f7ea1726332cb175b73077ec4a1b02
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
allocator: tcmalloc
modules: none
build environment:
    distmod: rhel70
    distarch: x86_64
    target_arch: x86_64

Both tests ran as expected using sudo service mongod start.

In my test I used the values of SELINUX=enforcing and SELINUX=targeted to try to replicate your environment. However for a guide of SELinux setup please refer to the manual Configure SELinux with MongoDB.

Did you change any configurations between the two versions ? i.e. storage engine, data path, etc
If you change the data path of your mongod, the default SELinux policies will prevent mongod from having write access on the new data path if you don’t change the security context. See also data directories and permissions.

If you are still having difficulties with MongoDB v3.2.5 on CentOS 7, could you:

• Check the few last lines of mongod output log in /var/log/mongodb/mongod.log
• Try running mongod directly without going through service. If the process failed, please post the output log of mongod
• Post the output of mongod --version
• Post the content of /etc/redhat-release

Kind regards,

Wan.

​

[Everyharu]

Have you tested CentOS7 minimal version?

I installed and tested this with CentOS Linux release 7.2.1511 (Core).
Do not change the other settings.

But this can not be executed with systemctl(service).

So I installed the 3.0.11 version.

2016년 4월 22일 금요일 오후 12시 16분 26초 UTC+9, Wan Bachtiar 님의 말:

[etter...@gmail.com]

Hi Wan

thanks for the reply.

• Check the few last lines of mongod output log in /var/log/mongodb/mongod.log

sudo service mongod start

Starting mongod (via systemctl):  Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.
                                                           [FAILED]

And nothing happens to the log with

tail -f /var/log/mongodb/mongod.log

• Try running mongod directly without going through service. If the process failed, please post the output log of mongod

running mongod directly fails because it wants `/data/db` After creating `/data/db` the process succeeds 

However running with the configuration file like the service would do:

sudo mongod --config /etc/mongod.conf

about to fork child process, waiting until server is ready for connections.

forked process: 19162

child process started successfully, parent exiting

gives the following output to the log, which apparently works as well, actually after issuing the command twice:

2016-04-07T05:52:04.468-0400 I CONTROL  [main] ***** SERVER RESTARTED *****

2016-04-07T05:52:04.474-0400 I CONTROL  [initandlisten] MongoDB starting : pid=19162 port=27017 dbpath=/var/lib/mongo 64-bit host=localhost.localdomain

2016-04-07T05:52:04.474-0400 I CONTROL  [initandlisten] db version v3.2.5

2016-04-07T05:52:04.474-0400 I CONTROL  [initandlisten] git version: 34e65e5383f7ea1726332cb175b73077ec4a1b02

2016-04-07T05:52:04.474-0400 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013

2016-04-07T05:52:04.474-0400 I CONTROL  [initandlisten] allocator: tcmalloc

2016-04-07T05:52:04.474-0400 I CONTROL  [initandlisten] modules: none

2016-04-07T05:52:04.474-0400 I CONTROL  [initandlisten] build environment:

2016-04-07T05:52:04.474-0400 I CONTROL  [initandlisten]     distmod: rhel70

2016-04-07T05:52:04.474-0400 I CONTROL  [initandlisten]     distarch: x86_64

2016-04-07T05:52:04.474-0400 I CONTROL  [initandlisten]     target_arch: x86_64

2016-04-07T05:52:04.474-0400 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }

2016-04-07T05:52:04.493-0400 I -        [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.

2016-04-07T05:52:04.493-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),

2016-04-07T05:52:04.668-0400 I CONTROL  [initandlisten]

2016-04-07T05:52:04.668-0400 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.

2016-04-07T05:52:04.668-0400 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.

2016-04-07T05:52:04.668-0400 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.

2016-04-07T05:52:04.668-0400 I CONTROL  [initandlisten]

2016-04-07T05:52:04.672-0400 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/diagnostic.data'

2016-04-07T05:52:04.672-0400 I NETWORK  [initandlisten] waiting for connections on port 27017

2016-04-07T05:52:04.672-0400 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker

• Post the output of mongod --version

mongod --version

db version v3.2.5

git version: 34e65e5383f7ea1726332cb175b73077ec4a1b02

OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013

allocator: tcmalloc

modules: none

build environment:

    distmod: rhel70

    distarch: x86_64

    target_arch: x86_64

 

• Post the content of /etc/redhat-release

CentOS Linux release 7.2.1511 (Core)

I'll keep on investigating and post a solution if I find one

Best

Adrian

[Wan Bachtiar]

Hi Adrian,

I’ve managed to reproduced the problem with CentOS 7 Minimal, and have opened SERVER-23863 ticket on MongoDB JIRA issue tracker.

Please feel free to watch or upvote for updates.

Regards,

Wan.

​

​

[etter...@gmail.com]

Cool thanks

I think I found a work around:

When starting the mongod service while tailing the audit.log it produces a few lines:

tail -f /var/log/audit/audit.log
service mongod start

So I created a type enforcing file with:

grep mongod /var/log/audit/audit.log | audit2allow -m mongod > mongod.te

produced the following file

module mongod 1.0;

require {

type locale_t;

type mongod_t;

type ld_so_cache_t;

class file execute;

}

#============= mongod_t ==============

allow mongod_t ld_so_cache_t:file execute;

allow mongod_t locale_t:file execute;

Added the module:

grep mongod /var/log/audit/audit.log | audit2allow -M mongod

And loaded the module:

semodule -i mongod.pp

et voila

sudo service mongod start

Starting mongod (via systemctl):                           [  OK  ]

so it must have to do something with a policy ...

[Mikko Tommila]

Looking at /var/log/audit/audit.log indeed reveals selinux denying access to various files e.g. /usr/lib/locale/locale-archive and if you allow the output by modifying /etc/init.d/mongod then you can see that it outputs this error:

mongod[2069]: Starting mongod: 2016-05-18T10:06:52.967+0000 F CONTROL  [main] Failed global initialization: BadValue: Invalid or no user locale set. Please ensure LANG and/or LC_* environment variables are set correctly.

I guess this is caused by a bug that will be fixed in 3.2.7:

https://jira.mongodb.org/browse/SERVER-24101

Mikko

[boncalo mihai]

Have you tried

chcon -Rv --type=mongod_var_lib_t $dbpath
chcon -Rv --type=mongod_log_t $logpath

setmanage permissive -a mongod_t

http://linux.die.net/man/8/mongod_selinux

[leck...@gmail.com]

Have the same issue. Have tried some recipes mentione here, like the ones form Mikko with semodule and boncalo but the error persist. Hopefully will be fixed in 3.2.7

[Wan Bachtiar]

Have the same issue. Have tried some recipes mentione here, like the ones form Mikko with semodule and boncalo but the error persist

Hi leckozol,

The problem described in this thread is related to setting SELINUX=enforcing on MongoDB v3.2.5 and v3.2.6.

If you have tried setting SELINUX to ‘permissive’ and is still having an issue to start mongod, it may be a different problem causing the issue you are seeing. 
Please start a new thread discussion with relevant details of:

• MongoDB version.
• Specific OS version. i.e. CentOS 7 minimal distribution.
• The content of /etc/selinux/config
• The error message that you are getting. i.e. log messages.
• Any steps to replicate the issue.

Hopefully will be fixed in 3.2.7

There is a patch committed into git master for the issue described in this thread, and will be targeted for release for v3.2.7. Please see JIRA ticket SERVER-24117 for more information.

Kind regards,

Wan.

​

[Balazs Varga]

I still facing with this error in centos 7 minimal install and mongodb 3.2.8

Can you give me advice ? 

[Wan Bachtiar]

I still facing with this error in centos 7 minimal install and mongodb 3.2.8

Hi Balazs,

I ran a test to install and start MongoDB 3.2.8 on CentOS Minimal 7.2 without any issues. As mentioned in ticket SERVER-24117, the issue has been resolved since MongoDB v3.2.7.

For installation steps see MongoDB install on RHEL.

If you are still having issues installing and/or starting MongoDB 3.2.8 in CentOS 7 Minimal, please open a new thread discussion with the following relevant information:

• Specific CentOS version number.
• The command that you executed, and the error message that you are getting.
• The content of /etc/selinux/config
• The content of /etc/yum.repos.d/mongodb-org-3.2.repo

Regards,

Wan.

​

​

[Balazs Varga]

Thanks Wan!

Found was on my side :) 

[davidcaughill]

I know this is a very old thread, but for those who arrive here via Google, we hit the same issue on an older version of mongo (3.2.4) at one point, and tweaking the `SELinux` perms did pull things back online in that case as well: We went from: 

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

#     enforcing - SELinux security policy is enforced.

#     permissive - SELinux prints warnings instead of enforcing.

#     disabled - No SELinux policy is loaded.

SELINUX=enforcing

# SELINUXTYPE= can take one of three values:

#     targeted - Targeted processes are protected,

#     minimum - Modification of targeted policy. Only selected processes are protected.

#     mls - Multi Level Security protection.

SELINUXTYPE=targeted

to: 

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

#     enforcing - SELinux security policy is enforced.

#     permissive - SELinux prints warnings instead of enforcing.

#     disabled - No SELinux policy is loaded.

SELINUX=permissive

# SELINUXTYPE= can take one of three values:

#     targeted - Targeted processes are protected,

#     minimum - Modification of targeted policy. Only selected processes are protected.

#     mls - Multi Level Security protection.

SELINUXTYPE=targeted

...which was enough to allow the mongo service to come back up. 

Supposedly that bug does not impact 3.2.4 (only 3.2.5 on) but that did not seem to be the case in this situation.