Native Linux memory leak in Jenkins master running in Docker?

[John Hovell]

For the last few weeks we've been experiencing container restarts due to Jenkins running out of memory in Docker. We allocated 3GB for the container and heap usage is quite low (a few hundred MB, limited to 512 MB) but after 12-24 hours overall container usage grows to 3GB.

We're using the jenkins/jenkins:lts image from here https://hub.docker.com/r/jenkins/jenkins/ (version 2.60.2)

Jstat shows low memory usage and top shows that the Java process overall is taking up 1.3 GB (uptime is misleading because this is for the host not container). The other 1.7 GB of memory seems to be unaccounted for and thinking it is a memory leak in the Docker container, either from Jenkins or some forked process.

Has anyone seen this behavior or know what other tools I can use to try to figure out where the extra memory is getting eaten up? 

Thanks!

$ jstat -gc 7

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT   

2048.0 2048.0  0.0   2047.0 166400.0 44755.3   341504.0   186831.1  157516.0 135388.1 18816.0 14700.4   3841   31.287   8      1.827   33.114

$java -version

openjdk version "1.8.0_141"

OpenJDK Runtime Environment (build 1.8.0_141-8u141-b15-1~deb9u1-b15)

OpenJDK 64-Bit Server VM (build 25.141-b15, mixed mode)

$ docker --version

Docker version 17.03.1-ce, build 7392c3b/17.03.1-ce

top - 19:09:38 up 8 days, 12:29,  0 users,  load average: 0.49, 0.57, 0.74

Tasks:   5 total,   1 running,   4 sleeping,   0 stopped,   0 zombie

%Cpu(s):  1.0 us,  0.5 sy,  0.0 ni, 98.4 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem : 15402128 total,  2567768 free,  8970084 used,  3864276 buff/cache

KiB Swap:        0 total,        0 free,        0 used.  5957332 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                             

    7 jenkins   20   0 6350548 1.320g  35608 S   0.7  9.0  41:18.58 java                                                                

    1 jenkins   20   0    1104      4      0 S   0.0  0.0   0:01.28 tini                                                                

  560 jenkins   20   0   19956   3600   3036 S   0.0  0.0   0:00.03 bash                                                                

15235 jenkins   20   0   19960   3532   3080 S   0.0  0.0   0:00.02 bash                                                                

15240 jenkins   20   0   42792   3580   3092 R   0.0  0.0   0:00.00 top