MongoDB fails to allocate data files on an NFS share

[Kevin Tran]

Background:

I use Vagrant (http://vagrantup.com/) to create a virtual machine for development. I use a Python build system called Buildout to manage and build my project. I use a Buildout recipe that installs and configures MongoDB. It puts the MongoDB data into a directory in the project. This all works fine when not using virtual machines. Vagrant is setup so that the source code repository exists on the host machine and is accessed by the virtual machine through NFS.

What I want to do:

I want to run mongod as the vagrant user (in the guest OS) and have the data directory in the NFS share. Trying to start the server like this causes an error:

    vagrant@natty:/nfs_share$ bin/mongod --dbpath /nfs_share/mongodb/data --master --port 27001 --directoryperdb -vvvvv

    Thu Oct 13 18:30:38 isInRangeTest passed

    Thu Oct 13 18:30:38 shardKeyTest passed

    Thu Oct 13 18:30:38 shardObjTest passed

    Thu Oct 13 18:30:38 must specify both start and end of balancing window: { start: 1 }

    Thu Oct 13 18:30:38 must specify both start and end of balancing window: { stop: 1 }

    Thu Oct 13 18:30:38 cannot parse active window (use hh:mm 24hs format): { start: "21:30", stop: "28:35" }

    Thu Oct 13 18:30:38 BalancingWidowObjTest passed

    Thu Oct 13 18:30:38 BackgroundJob starting: DataFileSync

    Thu Oct 13 18:30:38 versionCmpTest passed

    Thu Oct 13 18:30:38 [initandlisten] MongoDB starting : pid=3240 port=27001 dbpath=/nfs_share/mongodb/data master=1 64-bit 

    Thu Oct 13 18:30:38 [initandlisten] db version v1.8.1, pdfile version 4.5

    Thu Oct 13 18:30:38 [initandlisten] git version: a429cd4f535b2499cc4130b06ff7c26f41c00f04

    Thu Oct 13 18:30:38 [initandlisten] build sys info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41

    Thu Oct 13 18:30:39 [initandlisten] query: local.system.namespaces{ name: /^local.temp./ }

    Thu Oct 13 18:30:39 [initandlisten] Accessing: local for the first time

    Thu Oct 13 18:30:39 [initandlisten]    used cursor: 0x182e180

    Thu Oct 13 18:30:39 [initandlisten] query local.system.namespaces reslen:36 nreturned:0 1ms

    Thu Oct 13 18:30:39 [initandlisten] enter repairDatabases (to check pdfile version #)

    Thu Oct 13 18:30:39 [initandlisten] done repairDatabases

    Thu Oct 13 18:30:39 [initandlisten] waiting for connections on port 27001

    Thu Oct 13 18:30:39 [initandlisten] master=true

    Thu Oct 13 18:30:39 [initandlisten] ******

    Thu Oct 13 18:30:39 [initandlisten] creating replication oplog of size: 944MB...

    Thu Oct 13 18:30:39 [initandlisten] create collection local.oplog.$main { size: 990000000.0, capped: true, autoIndexId: false }

    Thu Oct 13 18:30:39 [FileAllocator] allocating new datafile /nfs_share/mongodb/data/local/local.ns, filling with zeroes...

    Thu Oct 13 18:30:39 [FileAllocator] warning: posix_fadvise fails /nfs_share/mongodb/data/local/local.ns errno:1 Operation not permitted

    Thu Oct 13 18:30:39 [FileAllocator] FileAllocator: posix_fallocate failed: errno:9 Bad file descriptor falling back

    Thu Oct 13 18:30:39 [FileAllocator] error failed to allocate new file: /nfs_share/mongodb/data/local/local.ns size: 16777216 errno:9 Bad file descriptor

    Thu Oct 13 18:30:39 [initandlisten] New namespace: local.oplog.$main

    New namespace: local.system.namespaces

    Assertion: 12520:new file allocation failure

    Thu Oct 13 18:30:39 [initandlisten] exception in initAndListen std::exception: new file allocation failure, terminating

    Thu Oct 13 18:30:39 dbexit: 

    Thu Oct 13 18:30:39 [initandlisten] shutdown: going to close listening sockets...

    Thu Oct 13 18:30:39 [initandlisten] shutdown: going to flush diaglog...

    Thu Oct 13 18:30:39 [initandlisten] shutdown: going to close sockets...

    Thu Oct 13 18:30:39 [initandlisten] shutdown: waiting for fs preallocator...

    Thu Oct 13 18:30:39 [initandlisten] shutdown: closing all files...

    Thu Oct 13 18:30:39 closeAllFiles() finished

    Thu Oct 13 18:30:39 [initandlisten] shutdown: removing fs lock...

    Thu Oct 13 18:30:39 BackgroundJob starting: ClientCursorMonitor

    Thu Oct 13 18:30:39 BackgroundJob starting: snapshot

    Thu Oct 13 18:30:39 dbexit: really exiting now

mongod cannot create the data files. The error message is "warning: posix_fadvise fails /nfs_share/mongodb/data/local/local.ns errno:1 Operation not permitted". This initially suggests that the file permissions are wrong. I don't think they are because the data directory is group writeable and vagrant is in the group "dialout" which is the group owner of the NFS share. My second guess was that NFS was to blame. When I run this command on a file system that is not NFS, all is well. I am not sure what is wrong. A search on the MongoDB mailing list found a message (http://groups.google.com/group/mongodb-user/browse_thread/thread/4129325d3a9a3237) from someone saying that NFS wouldn't work with MongoDB if the data directory doesn't support exclusive locking.

Possible Solution:

Put a symlink in the repository that points to a data directory in a non-NFS location

    This is what I'm currently doing. It's not very nice because it doesn't work with the Buildout recipe and must be done manually.

    There's no easy way to determine if Buildout is running in a virtual machine. Maybe it can detect if the file system is NFS and be scripted to act accordingly?

    It's a little annoying to have components of the project in different places. On the contrary, the Postgres data is elsewhere so why not put the MongoDB data elsewhere too?

Questions:

    Is the error I'm seeing actually related to NFS? Or is there another possible cause?

    Is exclusive locking a feature of NFS? Can it be enabled/disabled?

    I read reports on the MongoDB mailing list that people were running MongoDB on NFS. Is there any special consideration needed to do this?

[Kevin Tran]

I solved this by placing the MongoDB data in /usr/local/var/mongodb/. Now that I think about it, it makes more sense to store the data outside the NFS share and use the NFS share for storing code only.