Is mongodb causing file system to go into read-only mode?

[Raidon]

Hi,

Mongo shard is not getting started. It has a dedicated 10 TB storage out of which it is currently occupying 3.5 TB.

Following is error given in logs: 

Mon Oct 2 22:08:29 [initandlisten] User Assertion: 10309:Unable to create/open lock file: /u01/shard1/mongod.lock errno:30 Read-only file system Is a mongod instance already running?

Mon Oct 2 22:08:29 [initandlisten] exception in initAndListen: 10309 Unable to create/open lock file: /u01/shard1/mongod.lock errno:30 Read-only file system Is a mongod instance already running?, terminating

The mongo shard is started in repair mode with the options --repair --nojournal.

I have tried restarting the machine also. Mongo starts in repair mode. But after some time file system goes into read-only mode and then mongo shard fails to start.

Is Mongodb doing some activities because of which OS makes filesystem read-only? Or it is some hardware issue?

How can we approach towards this issue?

 

 

[Stephen Steneker]

On Wednesday, 4 October 2017 00:39:20 UTC+11, Raidon wrote:

Mongo shard is not getting started. It has a dedicated 10 TB storage out of which it is currently occupying 3.5 TB.

Following is error given in logs: 

Mon Oct 2 22:08:29 [initandlisten] User Assertion: 10309:Unable to create/open lock file: /u01/shard1/mongod.lock errno:30 Read-only file system Is a mongod instance already running?

Mon Oct 2 22:08:29 [initandlisten] exception in initAndListen: 10309 Unable to create/open lock file: /u01/shard1/mongod.lock errno:30 Read-only file system Is a mongod instance already running?, terminating

I have tried restarting the machine also. Mongo starts in repair mode. But after some time file system goes into read-only mode and then mongo shard fails to start.

Hi Raidon,

I believe the error messages here are referring to read-only files (or files that cannot be open for write access) rather than a read-only filesystem. MongoDB will not change filesystem mount options.

I suspect you may have run a repair operation as the root user and be running your mongod service with a different user account which does not have write access to some files in your dbPath.

Can you confirm the specific version of MongoDB server and O/S you are using?

Can you also check that the permissions in your dbPath are set correctly? If you aren't sure, please provide the output of ls -la /u01/shard1/mongod.lock and confirm how you normally start your mongod server.

The mongo shard is started in repair mode with the options --repair --nojournal.

Two obvious caveats to be aware of:

• If this shard is part of a replica set you should be re-syncing from a healthy member of the replica set rather than running repair. If there are any issues with the data integrity on a replica set member, a repair operation may introduce data inconsistencies.
  
  
• Journaling should be left enabled for a production environment.

Regards, Stennie