Mongodb and NAS

[del]

Hi guys,

I've been running mongodb for some time but now the data has grown
quite a lot and I need to hold the data in a NAS. Our IT dept has
attached one and I believe it's via CIFS to my linux box. Now when I
try to start mongodb, I get the following message :


Startup log:
Mon Feb 6 09:54:17 [initandlisten] MongoDB starting : pid=2535
port=27017 dbpath=/export/dset/HKDATAP03.daiwaasia.local/mongodb/data
64-bit host=HKDATAP03.daiwaasia.local
Mon Feb 6 09:54:17 [initandlisten] db version v2.0.1, pdfile version
4.5
Mon Feb 6 09:54:17 [initandlisten] git version:
3a5cf0e2134a830d38d2d1aae7e88cac31bdd684
Mon Feb 6 09:54:17 [initandlisten] build info: Linux bs-
linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28
EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Mon Feb 6 09:54:17 [initandlisten] options: { dbpath: "/export/dset/
HKDATAP03.daiwaasia.local/mongodb/data", quota: true, quotaFiles: 60 }
Mon Feb 6 09:54:17 [initandlisten] Assertion: 13651:Couldn't fsync
directory '/export/dset/HKDATAP03.daiwaasia.local/mongodb/data': errno:
22 Invalid argument
0x588cb2 0x58748f 0x88ae2a 0xa8d932 0xa8e32d 0xa9805a 0x3f2641d994
0x500689
/opt/loghome/mongodb/latest/bin/mongod(_ZN5mongo11msgassertedEiPKc
+0x112) [0x588cb2]
/opt/loghome/mongodb/latest/bin/
mongod(_ZN5mongo16flushMyDirectoryERKN5boost10filesystem10basic_pathISsNS1_11path_traitsEEE
+0x33f) [0x58748f]
/opt/loghome/mongodb/latest/bin/mongod(_ZN5mongo15acquirePathLockEb
+0x1da) [0x88ae2a]
/opt/loghome/mongodb/latest/bin/mongod(_ZN5mongo14_initAndListenEi
+0x342) [0xa8d932]
/opt/loghome/mongodb/latest/bin/mongod(_ZN5mongo13initAndListenEi
+0x1d) [0xa8e32d]
/opt/loghome/mongodb/latest/bin/mongod(main+0x9aaa) [0xa9805a]
/lib64/libc.so.6(__libc_start_main+0xf4) [0x3f2641d994]
/opt/loghome/mongodb/latest/bin/mongod(__gxx_personality_v0+0x3e9)
[0x500689]
Mon Feb 6 09:54:17 [initandlisten] exception in initAndListen: 13651
Couldn't fsync directory '/export/dset/HKDATAP03.daiwaasia.local/
mongodb/data': errno:22 Invalid argument, terminating
Mon Feb 6 09:54:17 dbexit:
Mon Feb 6 09:54:17 [initandlisten] shutdown: going to close listening
sockets...
Mon Feb 6 09:54:17 [initandlisten] shutdown: going to flush
diaglog...
Mon Feb 6 09:54:17 [initandlisten] shutdown: going to close
sockets...
Mon Feb 6 09:54:17 [initandlisten] shutdown: waiting for fs
preallocator...
Mon Feb 6 09:54:17 [initandlisten] shutdown: lock for final commit...
Mon Feb 6 09:54:17 [initandlisten] shutdown: final commit...
Mon Feb 6 09:54:17 [initandlisten] shutdown: closing all files...
Mon Feb 6 09:54:17 [initandlisten] closeAllFiles() finished
Mon Feb 6 09:54:17 [initandlisten] shutdown: removing fs lock...
Mon Feb 6 09:54:17 dbexit: really exiting now

Need some guidance on this.

Is it the NAS itself or that it's using CIFS? I've read a few posts
that NFS works.

Any help appreciated.

Thanks
del.

[Sam Millman]

Is: /export/dset/HKDATAP03.

daiwaasia.local/mongodb/data

The folder on your NAS?

Is this connected at startup through your fstab file? What does the line in your fstab look like?

This is 90% a cifs problem (good old cifs...not). I have to be honest when I say that I find sharing (cifs or nfs) mongo directories they are notoriously fragile and prone to corruption.

del.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[del]

Hi Sam,

Correct. That is the folder in the NAS. The NAS is connected at
startup.

The /etc/fstab line looks like :
//hkdatnas01/PROD /export/dset cifs credentials=/etc/
naslogin_dats,uid=514,gid=1000,file_mode=0644,dir_mode=0755 1 2

Just so you know, the NAS is a Dell PowerVault NX3100
http://www.dell.com/us/enterprise/p/powervault-nx3100/pd

I'll see what else I can dig up regarding CIFS and mongo. Maybe NFS
will work? I'll see what I can do to get the sysadmins to change this
but let me know if you need more information.

Thanks
del.

On Feb 6, 7:48 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> Is: /export/dset/HKDATAP03.
> daiwaasia.local/mongodb/data
>
> The folder on your NAS?
>
> Is this connected at startup through your fstab file? What does the line in
> your fstab look like?
>
> This is 90% a cifs problem (good old cifs...not). I have to be honest when
> I say that I find sharing (cifs or nfs) mongo directories they are
> notoriously fragile and prone to corruption.
>

[Scott Hernandez]

I would not suggest NFS as there are all kinds of issues there
depending on the version and def. around performance even if you can
start the server with NFS mounted volumes.

I would make sure you have the latest CIFS/SAMBA client on your server
as this is the pertinent error : "Couldn't fsync directory" coming
from the OS/Driver.

If you can't fsync/msync to that mount then nothing is going to work.

Reading some bugs against redhat/debian/other-distros/sqlite/postgres
it seems like this is generally a problem with cifs and servers/client
not implementing flush correctly or with any level of performance. I
am sure there are people running on windows with remote drives but
that is whole different cifs/smb client/server implementation.

[del]

Thanks Scott.  I'm looking at my options now with the CIFS driver.

As my dev env and production env are in two different countries, the
prod setup of the NAS is mounted via NFS.  (Totally different IT
depts)
The /etc/fstab looks like this :
10.173.240.66:/vol/vas_datlc002/nas/dat /home/dset      nfs    
rw,vers=3,rsize=8192,wsize=8192,bg,hard,intr    0       0

Now we tried to start mongo on the NAS and it didn't start there
either but the error message is different.  We tried to change the --
pidfilepath to something on the local machine as it couldn't get a
lock but that didn't help.

Tue Feb  7 20:11:06 [initandlisten] MongoDB starting : pid=16583
port=28020 dbpath=/home/dset/mongotemp 64-bit host=dat-apsv0060
Tue Feb  7 20:11:06 [initandlisten]
Tue Feb  7 20:11:06 [initandlisten] ** WARNING: You are running on a
NUMA machine.
Tue Feb  7 20:11:06 [initandlisten] **          We suggest launching
mongod like this to avoid performance problems:
Tue Feb  7 20:11:06 [initandlisten] **              numactl --
interleave=all mongod [other options]
Tue Feb  7 20:11:06 [initandlisten]
Tue Feb  7 20:11:06 [initandlisten] ** WARNING: /proc/sys/vm/
zone_reclaim_mode is 1
Tue Feb  7 20:11:06 [initandlisten] **          We suggest setting it
to 0
Tue Feb  7 20:11:06 [initandlisten] **          http://www.kernel.org/
doc/Documentation/sysctl/vm.txt
Tue Feb  7 20:11:06 [initandlisten]
Tue Feb  7 20:11:06 [initandlisten] db version v2.0.1, pdfile version
4.5
Tue Feb  7 20:11:06 [initandlisten] git version:
3a5cf0e2134a830d38d2d1aae7e88cac31bdd684
Tue Feb  7 20:11:06 [initandlisten] build info: Linux bs-

linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28
EST 2009 x86_64 BOOST_LIB_VERSION=1_41

Tue Feb  7 20:11:06 [initandlisten] options: { dbpath: "/home/dset/
mongotemp", pidfilepath: "/home/dats/temp", port: 28020, quota: true,
quotaFiles: 4 }
Tue Feb  7 20:11:36 [initandlisten] exception in initAndListen: 10310
Unable to acquire lock for lockfilepath: /home/dset/mongotemp/
mongod.lock, terminating
Tue Feb  7 20:11:36 dbexit:
Tue Feb  7 20:11:36 [initandlisten] shutdown: going to close listening
sockets...
Tue Feb  7 20:11:36 [initandlisten] shutdown: going to flush
diaglog...
Tue Feb  7 20:11:36 [initandlisten] shutdown: going to close
sockets...
Tue Feb  7 20:11:36 [initandlisten] shutdown: waiting for fs
preallocator...
Tue Feb  7 20:11:36 [initandlisten] shutdown: lock for final commit...
Tue Feb  7 20:11:36 [initandlisten] shutdown: final commit...
Tue Feb  7 20:11:36 [initandlisten] shutdown: closing all files...
Tue Feb  7 20:11:36 [initandlisten] closeAllFiles() finished
Tue Feb  7 20:11:36 dbexit: really exiting now

I'll keep searching around but if anyone as any ideas, please let me
know.

Thanks
del.

On Feb 7, 2:22 pm, Scott Hernandez <scotthernan...@gmail.com> wrote:
> I would not suggest NFS as there are all kinds of issues there
> depending on the version and def. around performance even if you can
> start the server with NFS mounted volumes.
>
> I would make sure you have the latest CIFS/SAMBA client on your server
> as this is the pertinent error : "Couldn't fsync directory" coming
> from the OS/Driver.
>
> If you can't fsync/msync to that mount then nothing is going to work.
>
> Reading some bugs against redhat/debian/other-distros/sqlite/postgres
> it seems like this is generally a problem with cifs and servers/client
> not implementing flush correctly or with any level of performance. I
> am sure there are people running on windows with remote drives but
> that is whole different cifs/smb client/server implementation.
>
>
>
>
>
>
>
> On Mon, Feb 6, 2012 at 10:23 PM, del <del.abdul...@gmail.com> wrote:
> > Hi Sam,
>

> > Correct.  That is the folder in theNAS.  TheNASis connected at

> > startup.
>
> > The /etc/fstab line looks like :
> > //hkdatnas01/PROD       /export/dset    cifs    credentials=/etc/
> > naslogin_dats,uid=514,gid=1000,file_mode=0644,dir_mode=0755    1 2
>

> > Just so you know, theNASis a Dell PowerVault NX3100
> >http://www.dell.com/us/enterprise/p/powervault-nx3100/pd
>
> > I'll see what else I can dig up regarding CIFS andmongo.  Maybe NFS

> > will work?  I'll see what I can do to get the sysadmins to change this
> > but let me know if you need more information.
>
> > Thanks
> > del.
>
> > On Feb 6, 7:48 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> >> Is: /export/dset/HKDATAP03.
> >> daiwaasia.local/mongodb/data
>
> >> The folder on yourNAS?
>
> >> Is this connected at startup through your fstab file? What does the line in
> >> your fstab look like?
>
> >> This is 90% a cifs problem (good old cifs...not). I have to be honest when

> >> I say that I find sharing (cifs or nfs)mongodirectories they are

> >> notoriously fragile and prone to corruption.
>
> >> On 6 February 2012 11:43, del <del.abdul...@gmail.com> wrote:
>
> >> > Hi guys,
>
> >> > I've been running mongodb for some time but now the data has grown

> >> > quite a lot and I need to hold the data in aNAS.  Our IT dept has

> >> > Is it theNASitself or that it's using CIFS?  I've read a few posts

[Barrie]

Hey del,

The lock problem is because you can't get an exclusive lock for
mongod.lock.
You should look into upgrading your NFS server version to something
that supports exclusive locks.
And you might want to turn off NUMA control, as mentioned in the log.

Hope this helps.

Barrie

[John Arrowwood]

I got around the problem by making mongod.lock a symlink to a file in another directory on a local filesystem.  The process was able to get the exclusive lock, and all was well.

It shouldn't prove to be a problem, because the database files are not actually shared by multiple servers, the disk is on a NAS for convenience purposes.  Wish that wasn't the case, but that's beside the point.

John Arrowwood,
Consulting Hypnotist / Hypnotherapist
http://www.JohnArrowwood.com
503.863.4823

[del]

For my NAS in Tokyo using NFS, that symlink trick worked!!!   Many
thanks for your help.

Just need to get this CIFS issue fixed. I have the Dell engineers
looking at the NAS in their lab.

> ...
>
> read more »

[Scott Hernandez]

Using a symlink like this could cause serious (data) problems if you
aren't using journaling and can't guarantee that no other instance
will try to use that dbpath. It is not a suggested setup because of
these possibilities.

> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.