Primary Mongo Replica set crashed in Production env

51 views

Skip to first unread message

Sai Kiran

unread,

May 2, 2018, 7:48:59 PM5/2/18

to mongodb-user

We have one primary and two secondary replica sets for our production env and the primary replica set is crashed every minute when I bring it to PRIMARY state and when I check the status of Mongod service it shows as failed:

mongod.service - SYSV: Mongo is a scalable, document-oriented database.

Loaded: loaded (/etc/rc.d/init.d/mongod)

Active: failed (Result: exit-code) since Wed 2018-05-02 21:56:03 UTC; 9min ago

Process: 27285 ExecStart=/etc/rc.d/init.d/mongod start (code=exited, status=1/FAILURE)

production mongod[27285]: Error starting mongod. /var/run/mongodb/mongod.pid exists.

production systemd[1]: mongod.service: control process exited, code=exited status=1

production systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..

production systemd[1]: Unit mongod.service entered failed state.

Tried restarting the service several times and still unable to get the Primary replica set to primary state. As followed from other blogs, I've tried deleting the /tmp/mongodb-27017.sock file several times and restarted the service with our .congf file.

I have the Logs generated in /var/log/mongodb/mongod.log

*2017-11-25T09:01:33.834+0000 I CONTROL [main] ***** SERVER RESTARTED ******

2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] MongoDB starting : pid=16025 port=27017 dbpath=/var/lib/mongo 64-bit host=new-mittens-production

2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] db version v3.2.12

2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14

2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013

2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] allocator: tcmalloc

2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] modules: none

2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] build environment:

2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] distmod: rhel70

2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] distarch: x86_64

2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] target_arch: x86_64

*2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:

{ bindIp: "0.0.0.0", port: 27017 }

, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal:

{ enabled: true }

}, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }*

2017-11-25T09:01:33.877+0000 E NETWORK [initandlisten] Failed to unlink socket file /tmp/mongodb-27017.sock errno:1 Operation not permitted

2017-11-25T09:01:33.877+0000 I - [initandlisten] Fatal Assertion 28578

2017-11-25T09:01:33.877+0000 I - [initandlisten]

***aborting after fassert() failure

Any suggestions or help is appreciated. Thanks in advance.

Wan Bachtiar

unread,

Jul 19, 2018, 8:33:37 PM7/19/18

to mongodb-user

Any suggestions or help is appreciated.

Hi Sai,

The log entry that is of interest is:

production mongod[27285]: Error starting mongod. /var/run/mongodb/mongod.pid exists.

Check the existence of /var/run/mongodb/mongod.pid file before you’re starting the primary.
The service stop should have remove this file, but in your case due to a crash (unknown cause), it may have not had the chance to remove it.