mongodb service error: IllegalOperation: Attempted to create a lock file on a read-only directory: /data, terminating

[DH]

Hello,

I'm running into an issue with the storage engine (WiredTiger) not able to create the lock file in the data directory.

I have checked many times and the permission looks correct.

It ONLY fails if mongod is run as a service but if run from command line it works fine (see sudo command below).

I had two machines with same setup and their services are running fine, but for some reason, the mongod service cannot run in this third machine.

I've searched the web but all suggestions have to do with typical permission issue, which I feel certain that the permission is ok.  For testing purpose only, I even tried chmod 777 to the /data directory without luck.

What am I missing?

Thanks!

_____________________________

$ whoami

root

$ mkdir /data

$ chown mongod:mongod /data

$ ls -l / | grep data

drwxr-xr-x.   7 mongod mongod   6 Aug 29 13:19 data

$ cat /etc/mongod.conf

# mongod.conf

# for documentation of all options, see:

#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.

systemLog:

  destination: file

  logAppend: false

  path: /var/log/mongodb/mongod.log

  verbosity: 0

# Where and how to store data.

storage:

  dbPath: /data

  directoryPerDB: true

#  engine:

#  mmapv1:

#  wiredTiger:

# how the process runs

processManagement:

  pidFilePath: /var/run/mongodb/mongod.pid

  timeZoneInfo: /usr/share/zoneinfo

  fork: true

# network interfaces

net:

  port: 27017

  bindIpAll: true

  #bindIp: localhost

# authentication

#security:

#operationProfiling:

# Note that for 3-member PRIMARY-SECONDARY-ARBITER configuration,

# set enableMajorityReadConcern to false.

# See https://docs.mongodb.com/manual/reference/read-concern-majority

#replication:

#sharding:

## Enterprise-Only Options

#auditLog:

#snmp:

$ cat /lib/systemd/system/mongod.service

[Unit]

Description=MongoDB Database Server

Documentation=https://docs.mongodb.org/manual

After=network.target

[Service]

User=mongod

Group=mongod

Environment="OPTIONS=-f /etc/mongod.conf"

EnvironmentFile=-/etc/sysconfig/mongod

ExecStart=/usr/bin/mongod $OPTIONS

ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb

ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb

ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb

PermissionsStartOnly=true

PIDFile=/var/run/mongodb/mongod.pid

Type=forking

# file size

LimitFSIZE=infinity

# cpu time

LimitCPU=infinity

# virtual memory size

LimitAS=infinity

# open files

LimitNOFILE=64000

# processes/threads

LimitNPROC=64000

# locked memory

LimitMEMLOCK=infinity

# total threads (user+kernel)

TasksMax=infinity

TasksAccounting=false

# Recommended limits for for mongod as specified in

# http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings

[Install]

WantedBy=multi-user.target

$ systemctl start mongod

Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.

$ systemctl status mongod.service

● mongod.service - MongoDB Database Server

   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Thu 2019-08-29 13:37:23 PDT; 5s ago

     Docs: https://docs.mongodb.org/manual

  Process: 6947 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=100)

  Process: 6944 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)

  Process: 6941 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)

  Process: 6937 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)

Aug 29 13:37:23 msvdb01 systemd[1]: Starting MongoDB Database Server...

Aug 29 13:37:23 msvdb01 mongod[6947]: about to fork child process, waiting until server is ready for connections.

Aug 29 13:37:23 msvdb01 mongod[6947]: forked process: 6950

Aug 29 13:37:23 msvdb01 mongod[6947]: ERROR: child process failed, exited with error number 100

Aug 29 13:37:23 msvdb01 mongod[6947]: To see additional information in this output, start without the "--fork" option.

Aug 29 13:37:23 msvdb01 systemd[1]: mongod.service: control process exited, code=exited status=100

Aug 29 13:37:23 msvdb01 systemd[1]: Failed to start MongoDB Database Server.

Aug 29 13:37:23 msvdb01 systemd[1]: Unit mongod.service entered failed state.

Aug 29 13:37:23 msvdb01 systemd[1]: mongod.service failed.

$ cat /var/log/mongdb/mongod.log

2019-08-29T13:24:32.398-0700 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'

2019-08-29T13:24:32.408-0700 I  CONTROL  [initandlisten] MongoDB starting : pid=6500 port=27017 dbpath=/data 64-bit host=msvdb01

2019-08-29T13:24:32.408-0700 I  CONTROL  [initandlisten] db version v4.2.0

2019-08-29T13:24:32.408-0700 I  CONTROL  [initandlisten] git version: a4b751dcf51dd249c5865812b390cfd1c0129c30

2019-08-29T13:24:32.408-0700 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013

2019-08-29T13:24:32.408-0700 I  CONTROL  [initandlisten] allocator: tcmalloc

2019-08-29T13:24:32.408-0700 I  CONTROL  [initandlisten] modules: none

2019-08-29T13:24:32.408-0700 I  CONTROL  [initandlisten] build environment:

2019-08-29T13:24:32.408-0700 I  CONTROL  [initandlisten]     distmod: rhel70

2019-08-29T13:24:32.408-0700 I  CONTROL  [initandlisten]     distarch: x86_64

2019-08-29T13:24:32.408-0700 I  CONTROL  [initandlisten]     target_arch: x86_64

2019-08-29T13:24:32.408-0700 I  CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "*", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid", timeZoneInfo: "/usr/share/zoneinfo" }, storage: { dbPath: "/data", directoryPerDB: true }, systemLog: { destination: "file", logAppend: false, path: "/var/log/mongodb/mongod.log", verbosity: 0 } }

2019-08-29T13:24:32.409-0700 I  STORAGE  [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data, terminating

2019-08-29T13:24:32.409-0700 I  NETWORK  [initandlisten] shutdown: going to close listening sockets...

2019-08-29T13:24:32.409-0700 I  NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock

2019-08-29T13:24:32.409-0700 I  -        [initandlisten] Stopping further Flow Control ticket acquisitions.

2019-08-29T13:24:32.409-0700 I  CONTROL  [initandlisten] now exiting

2019-08-29T13:24:32.409-0700 I  CONTROL  [initandlisten] shutting down with code:100

$ sudo -u mongod mongod -f /etc/mongod.conf

about to fork child process, waiting until server is ready for connections.

forked process: 6835

child process started successfully, parent exiting

_____________________________

[Mathew Allen]

Hi,

The issue you describe may be related to SElinux being enabled on the host that you are installing mongodb. You can check to see if SElinux is enabled by reviewing the settings in the /etc/selinux/config file:

SELINUX=enforcing
SELINUXTYPE=targeted

If SELinux is in enforcing mode, you must configure SELinux for MongoDB if you are not using the default MongoDB directories, which is documented here. Otherwise you can disable SElinux and reboot the server.

Best regards,

Matt

​

[DH]

Thanks for the tip.  I have worked around it but your pointer was what I needed.