errno:487 Attempt to access invalid address

[René M. Andersen]

We are from time to time on our build servers experiencing an error like shown below:

Thu Jun 26 12:21:20.008 [conn3579] MapViewOfFileEx for d:\mongodb\data\Test_635393820466449562\Test_635393820466449562.ns failed with errno:487 Attempt to access invalid address. (file size is 536870912) in MemoryMappedFile::map

Thu Jun 26 12:21:20.008 [conn3579]  Test_635393820466449562.system.indexes Fatal Assertion 16166

Result is that the mongod process terminates.

I have no idea why the error occurs, anyone can help?

Log file attached.

Environment is:

Virtual Windows 2008 R2 64bit 4GB RAM

MongoDB 2.4.8

Regards René

[Asya Kamsky]

Hi,

This happens because the mongod process fails to successfully read the memory mapped file - it got an error from the OS.

MapViewOfFileEx for d:\mongodb\data\Test_635393820466449562\Test_635393820466449562.ns failed with errno:487 Attempt to access invalid address. (file size is 536870912) in MemoryMappedFile::map

This error is from this file: https://github.com/mongodb/mongo/blob/master/src/mongo/util/mmap_win.cpp#L323

I did notice that you previously successfully allocate the same size files, so it's not clear what the issue may be.   What version is this?  How often does this happen?

Is it always on the same file/test_db?   What can you tell me about the file system that the data directory is on (d:\mongodb\data)

Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/ace1f76b-604d-4d08-99e6-f39587412828%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[William Zola]

Also, you mentioned that this was "Virtual Windows".  What virtualization software are you using?  How is the D: drive being implemented in this environment?  And would you happen to be using Azure?

 -William 

[René M. Andersen]

Hi,

Thanks for the input.

The MongoDB version is 2.4.8.

The virtual environment is VMWare hosted locally at our company. All disk drives in this setup maps to a SAN.

The error occurs in our auto tests against MongoDB. The Test_xxx databases is dynamically generated as part of the individual test setup and dropped when the test completes. This means we create many different Test_xxx databases during an entire test run. I have noticed any pattern in when the error occurs. It is not everytime the tests run.

Regards

René

[William Zola]

Hi Rene!

Given what you've said, the most likely cause is problems with the SAN or the VMWare environment.  MongoDB is reporting that the file isn't visible to the process within the VM.  That could mean one of many things:

 - A (possibly transient) error on the SAN making that LUN unavailable

 - A (possibly transient) error on the SAN making that file unavailable

 - A (possibly transient) networking error so that an attempt to access that particular file fails

 - A (possibly transient) OS error if the OS mistakenly thinks that the file does not exist

 - A combination of any or all of the above

We generally recommend the following when using VMWare ESX:

 - Do not over-allocate RAM

 - Reserve all RAM that has been allocated to the VM running MongoDB

 - If using a SAN, mount the SAN to the ESX host and expose the network storage to the guest as a local device

We generally recommend the following when using a SAN:

 - Do not use NFS or CIFS as the network protocol.  Use iSCSI, FiberChannel, or FiberChannel over Ethernet

 - For best performance, do not share spindles between disks used by MongoDB with other disks; the LUNs that are used for MongoDB data and journal files should be dedicated solely for use by MongoDB nodes

 - For high availability, each node in a replica set should use a LUN that maps to different physical disks; ideally, each node would be connected to a different SAN

I hope you find this useful in diagnosing your problem.

 -William Z

[René M. Andersen]

Hi Wiiliam,

Thanks a lot. I will investigate your suggestions/findings.

Regards

René