mmap() failed, errno: 12 Cannot allocated memory after datafile allocation

[MJ]

Hi,

We experienced an issue today as the mongod started generating the
following errors after a datafile creation:

Wed May 2 11:21:09 [FileAllocator] done allocating datafile /{path}/
fix.3, size: 512MB, took 44.306 secs
Wed May 2 11:21:09 [conn114] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn114] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn114] insert fix.messages exception: can't map
file memory code:10085 47267ms
Wed May 2 11:21:09 [conn126] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn126] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn133] end connection 142.205.5.254:18898
Wed May 2 11:21:09 [conn117] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn117] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn117] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn117] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn126] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn126] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn114] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn114] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn123] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn123] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn120] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn120] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn120] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn120] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn120] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn120] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn120] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn120] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn111] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn111] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn129] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn129] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn129] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn129] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn123] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn123] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn111] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory
Wed May 2 11:21:09 [conn111] ERROR: mmap failed with out of memory.
(64 bit build)
Wed May 2 11:21:09 [conn111] ERROR: mmap() failed for /{path}/fix.3
len:536870912 errno:12 Cannot allocate memory

I shutdown the mongod and all replicasets, and restarted them but got
the same error. In the end I had to db.dropDatabase() the db and start
again which isn't ideal. I was also left with a FATAL replicaset so
had to take it offline, remove the data files and start it back again
to allow it to take a fresh copy.

Basic Setup:
2 x machines (replicasets)
1 x machine (arb)

Machine Details:

SUSE Linux 11 Service Pack 2
2Gb Ram
30Gb Disk

uname -a
Linux {host} 3.0.13-0.27-default 1 SMP Wed Feb 15 13:33:49 UTC 2012
(d73692b) x86_64 x86_64 x86_64 GNU/Linux

If anyone could shed some light on this it would great!!!

Cheers.

[Kyle Banker]

This is sometime due to a lack of swap space. Can you check your swap size? (swapon -s)

[MJ]

Thanks,

I get:

Filename: /dev/mapper/system-swap

Type: partition

Size: 1986556

Used: 9260

Priority: -1

If this is due to swap size, then any idea why I would have run out?

Cheers.

[Kyle Banker]

Can you also post ulimit -a.

[MJ]

As requested:

core file size          (blocks, -c) 1

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 15452

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) 1691160

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 15452

virtual memory          (kbytes, -v) 3180960

file locks                      (-x) unlimited

Thanks again.

[Kyle Banker]

If I'm reading that correctly, you have a max virtual memory size of around 3 GB. You need to raise it.

This is usually done by editing the values in /etc/security/limits.conf. You'll probably just want to set that limit to 'unlimited'.

[MJ]

Thanks Kyle.

I ran through some tests today and set the limit to 1gb and managed to reproduce the issue. I'm now looking at setting this to be unlimited.

Thanks again.