MongoDB mmap() failed on VxFS

[Chimeng Wong]

Hi all,

OS:           CentOS 6.4

VxFS:       5.1

MongoDB: 2.4.8/2.4.9

I got an unexpected error during i install the MongoDB instance on the VxFS disk. i have done plenty research on this case but there is no result for this case. Also, i have try below settings or procedures but still can not solve the problem. 

However, VxFS supposed can be installed MongoDB on SAS disk because i have installed it on RHEL5 with veritas 4 before, and it is running well so far, i wanna upgrade my OS to CentOS 6.4 but i only can 0install it on localdisk(/opt) successfully. set the (ulimit -n) to over 64000 is not work for me as well, the error message can be occurred again.

Also, for this case, i think the error message is not enough for DBAs to solve the problem.

Do you guys have any solution or recommendation for me? Big thanks !

Error message:

START ============================================

Tue Feb 18 11:39:32.454 [initandlisten] db version v2.4.8

Tue Feb 18 11:39:32.454 [initandlisten] git version: a350fc38922fbda2cec8d5dd842237b904eafc14

Tue Feb 18 11:39:32.454 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49

Tue Feb 18 11:39:32.454 [initandlisten] allocator: tcmalloc

Tue Feb 18 11:39:32.454 [initandlisten] options:

{ config: "/mounted_disk/mongodb-vdb01/conf/shard0rs0.conf", dbpath: "/mounted_disk/mongodb-vdb01/data/shard0rs0", fork: "true", journal: "true", logappend: "true", logpath: "/mounted_disk/mongodb-vdb01/log/shard0rs0.log", noMoveParanoia: "true", pidfilepath: "/mounted_disk/mongodb-vdb01/tmp/shard0rs0.pid", port: 27020, rest: "true", shardsvr: "true" }

Tue Feb 18 11:39:32.455 [initandlisten] journal dir=/mounted_disk/mongodb-vdb01/data/shard0rs0/journal

Tue Feb 18 11:39:32.455 [initandlisten] recover : no journal files present, no recovery needed

Tue Feb 18 11:39:32.481 [FileAllocator] allocating new datafile /mounted_disk/mongodb-vdb01/data/shard0rs0/local.ns, filling with zeroes...

Tue Feb 18 11:39:32.481 [FileAllocator] creating directory /mounted_disk/mongodb-vdb01/data/shard0rs0/_tmp

Tue Feb 18 11:39:32.483 [FileAllocator] done allocating datafile /mounted_disk/mongodb-vdb01/data/shard0rs0/local.ns, size: 16MB, took 0 secs

Tue Feb 18 11:39:32.483 [initandlisten] local.startup_log Assertion failure _filename.empty() src/mongo/util/mmap.cpp 200

0xde05e1 0xda15bd 0xdc115d 0xdc3231 0xdbeda3 0xa5b063 0xa6070e 0x8d5419 0x8d56c8 0x8d57af 0x8d5a4e 0xab76bc 0xac59a7 0xac6b7f 0x9f4de9 0x9f5859 0x9fc0e8 0x9fc973 0x731de1 0x6d54f1

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xda15bd]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo9MongoFile11setFilenameERKSs+0x22d) [0xdc115d]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo16MemoryMappedFile3mapEPKcRyi+0x51) [0xdc3231]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo16MemoryMappedFile14mapWithOptionsEPKci+0x63) [0xdbeda3]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo8MongoMMF4openERKSsb+0x43) [0xa5b063]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo14NamespaceIndex5_initEv+0x7e) [0xa6070e]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo8Database7getFileEiib+0x319) [0x8d5419]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo8Database8addAFileEib+0x38) [0x8d56c8]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo8Database12suitableFileEPKcibb+0xaf) [0x8d57af]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo8Database11allocExtentEPKcibb+0x9e) [0x8d5a4e]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo19insert_newNamespaceEPKcib+0xcc) [0xab76bc]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbbPb+0x1367) [0xac59a7]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEbb+0x4f)[0xac6b7f]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE+0x119) [0x9f4de9]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x929) [0x9f5859]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xab8)[0x9fc0e8]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo14DBDirectClient3sayERNS_7MessageEbPSs+0x83) [0x9fc973]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo12DBClientBase6insertERKSsNS_7BSONObjEi+0x291) [0x731de1]

/opt/third-party/mongodb/product/current/bin/mongod(_ZN5mongo10logStartupEv+0x8d1) [0x6d54f1]

Tue Feb 18 11:39:32.490 [initandlisten] insert local.startup_log keyUpdates:0 exception: assertion src/mongo/util/mmap.cpp:200 6ms

END ====================================================================================

vxtunefs settings ===================================================================

read_pref_io = 65536

read_nstream = 1

read_unit_io = 65536

write_pref_io = 65536

write_nstream = 1

write_unit_io = 65536

pref_strength = 10

buf_breakup_size = 1048576

discovered_direct_iosz = 262144

max_direct_iosz = 1048576

default_indir_size = 8192

odm_cache_enable = 0

write_throttle = 0

max_diskq = 1048576

initial_extent_size = 8

max_seqio_extent_size = 2048

max_buf_data_size = 8192

hsm_write_prealloc = 0

read_ahead = 1

inode_aging_size = 0

inode_aging_count = 0

fcl_maxalloc = 973264896

fcl_keeptime = 0

fcl_winterval = 3600

fcl_ointerval = 600

pdir_enable = 0

pdir_threshold = 32768

lazy_copyonwrite = 0

shared_pg_enabled = 0

oltp_load = 0

delicache_enable = 1

thin_friendly_alloc = 0

dalloc_enable = 0

dalloc_limit = 90

lazy_isize_enable = 0

END==================================================================================

From the error message, i think the root caused is the mmap function did not work on VxFS(version 5.1 on CentOS 6.4), but i have no idea how can i fix it well. 

BTW, since CentOS 6.4 only support Veritas File System version 5.1(version 4 is out of service on CentOS 6),  can anyone use HA with MongoDB on CentOS 6 successfully? Any recommendation for me? Thanks a lot !

[Stephen Steneker]

Hi Chimeng,

We do not currently test with VxFS and it doesn't look like anyone has reported similar issues in Jira yet.

Are you using VxFS clustered file system features? If so, can you describe how you are sharing the VxFS mount points?

Would you be able to test with a dbpath on another partition on this server which is ext4 or XFS? That would help narrow the scope of the problem to VxFS versus something amiss in your Centos 6 configuration.

Can you also post an `ls -laR`  of the dbpath (/mounted_disk/mongodb-vdb01/data/shard0rs0)?

Thanks,

Stephen