Bazel Native memory allocation (mmap) failed to map bytes for committing reserved memory.

[lo...@xperiel.com]

Summary:
Bazel test runner running out of memory while running parallel Spock tests.

Error:
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000756580000, 365953024, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 365953024 bytes for committing reserved memory.

Temporary fix:
we bumped up our machines from c4.4xlarge to c4.8xlarge and it seems to be working but we don't want to continue with the added cost.

MORE INFO

jenkins: Jenkins ver. 2.73.3
Bazel Version: 0.13.0
docker:
Client:
Version: 18.04.0-ce
API version: 1.24 (downgraded from 1.29)
Go version: go1.9.4
Git commit: 3d479c0
Built: Tue Apr 10 18:22:52 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 1.12.6
API version: 1.24 (minimum version )
Go version: go1.6.2
Git commit: 78d1802
Built: Tue Jan 31 23:35:14 2017
OS/Arch: linux/amd64
Experimental: false

os:
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

AWS instance:
c4.4xlarge

[László Csomor]

Hi,

The error seems to be that Bazel failed to allocate enough memory, i.e. the machine ran out of RAM.

As a workaround, you could call "bazel shutdown" at the end of each CI run. This means you lose caching between builds on the CI machines, and so your builds and tests will be slower, but I think it'll help with the memory pressure.

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/85e81068-155b-45b4-b859-8f9a7a4ab7c5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[lo...@xperiel.com]

That seems a little problematic since our goal is to keep build node cost while also not extending the lifetime of each build exponentially is there a setting we can apply to bazel testrunner that would manage memory more efficiently?

[Laurent Le Brun]

Eric, Tom, any idea about this issue? Do you know who can help?

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/9bf8f697-cfbd-4370-95e4-cda06eff3e16%40googlegroups.com.

[lo...@xperiel.com]

Im looking here at our build configuration and it seems as though we're doing a bazel run on custom bazel rule that defines test runners for our tests. It's a simple way to consolidate build within jenkins but I'm wondering if this could somehow cause memory allocations to fall outside the scope of the bazel server target.

[Eric Fellheimer]

What is the difference between c4.4xlarge and c4.8xlarge? Do these have swapping enabled? If not, you'll have to examine what is actually consuming the memory. You can consider decreasing the Xmx setting of blaze itself or the tests it's running to fit into a smaller memory footprint, if possible. Also, make sure you don't have a virtual memory limit here ("ulimit -v")

Eric Fellheimer
fe...@google.com

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/8e5a617f-3655-4969-b73f-84885b5ee91c%40googlegroups.com.

[lo...@xperiel.com]

Model vCPU Mem (GiB) Storage
Dedicated EBS Bandwidth (Mbps)
c4.large 2 3.75 EBS-Only 500
c4.xlarge 4 7.5 EBS-Only 750
c4.2xlarge 8 15 EBS-Only 1,000
c4.4xlarge 16 30 EBS-Only 2,000
c4.8xlarge 36 60 EBS-Only 4,000

https://aws.amazon.com/ec2/instance-types/

[lo...@xperiel.com]

no virtual memory limit either

[lo...@xperiel.com]

[Eric Fellheimer]

can you monitor your memory with "top" before you hit the memory failure condition? what processes are consuming the memory at what levels?

Eric Fellheimer
fe...@google.com

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/a9f6b336-fc1c-42b7-be25-a59029718cd0%40googlegroups.com.

[lo...@xperiel.com]

It's the java processes, I assume these are the hazelcast nodes?
top - 15:57:17 up 3:57, 0 users, load average: 22.11, 6.99, 5.10
Tasks: 98 total, 8 running, 90 sleeping, 0 stopped, 0 zombie
%Cpu(s): 64.8 us, 5.1 sy, 0.0 ni, 30.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 61825552 total, 4189528 free, 53803360 used, 3832664 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 7372068 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
51424 root 20 0 19.657g 412712 35760 S 287.0 0.7 0:08.64 java
50721 root 20 0 19.869g 940608 37084 S 265.1 1.5 0:26.60 java
50361 root 20 0 19.873g 1.049g 36684 S 256.8 1.8 0:36.39 java
46778 root 20 0 19.881g 4.113g 37044 S 174.1 7.0 1:35.97 java
47236 root 20 0 19.879g 3.969g 37140 S 167.8 6.7 1:32.60 java
46849 root 20 0 19.944g 4.526g 36840 S 157.1 7.7 1:34.99 java
47164 root 20 0 19.884g 4.545g 37120 S 150.8 7.7 1:43.72 java
47754 root 20 0 19.881g 3.584g 37060 S 150.2 6.1 1:34.76 java
48726 root 20 0 19.938g 2.985g 36632 S 135.2 5.1 1:05.67 java
47200 root 20 0 19.883g 3.961g 37312 S 134.6 6.7 1:48.87 java
47287 root 20 0 19.883g 3.305g 37256 S 129.6 5.6 1:43.36 java
47090 root 20 0 19.883g 4.201g 37064 S 125.6 7.1 1:45.34 java
46966 root 20 0 19.946g 4.065g 36812 S 124.6 6.9 1:47.52 java
47391 root 20 0 19.887g 2.642g 37248 S 116.6 4.5 1:46.38 java
47448 root 20 0 19.878g 2.679g 36932 S 106.3 4.5 1:51.56 java
45977 root 20 0 34.788g 1.858g 19340 S 13.0 3.2 1:49.71 java

[Eric Fellheimer]

Would be good to confirm what those java processes are. Can you show with their full command line?

Eric Fellheimer
fe...@google.com

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/33bfefa0-d579-4cda-9a20-225d22b0674e%40googlegroups.com.