RunDeck locks up all Java - JVM processing until restarted

[Aaron Sheppard]

Good Afternoon All,

This weekend we had RunDeck locked up from Saturday around noon till Monday at 9:00 am when we had to restart the server to get it to start processing jobs again.  

This has happened in the past before, but usually due to outside RunDeck factors like backups failing or restarts not completing.  In this case nothing on the OS side was out of the ordinary.  

The CPU tends to stay at or near 70% and just hangs with no other jobs being processed until the tomcat server is rebooted.

In trying to find out what was happening I found the following from the catalina logs.  Does anyone know what we need to be checking?  How can we prevent this from happening especially on overnights and weekends? I couldn't get a thread dump because it wasn't when we could observe it as the webgui won't respond at all.

Catalina Logs: 

Saturday-

11-Jun-2022 12:54:22.653 WARNING [pool-11-thread-2] com.google.common.cache.LocalCache$Segment$1.run Exception thrown during refresh
    java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

...

...

    Caused by: java.lang.OutOfMemoryError: Java heap space

And again today-

14-Jun-2022 12:51:44.775 WARNING [Thread-495] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [rundeck] appears to have started a thread named [Thread-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.nio.fs.WindowsNativeDispatcher.GetQueuedCompletionStatus0(Native Method)
 sun.nio.fs.WindowsNativeDispatcher.GetQueuedCompletionStatus(WindowsNativeDispatcher.java:1007)
 sun.nio.fs.WindowsWatchService$Poller.run(WindowsWatchService.java:586)
 java.lang.Thread.run(Thread.java:748)

14-Jun-2022 12:34:16.974 WARNING [pool-11-thread-1] com.google.common.cache.LocalCache$Segment$1.run Exception thrown during refresh
    java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded

SETUP:

Server: Windows 2016 (8 core / 35 GB ram)

Tomcat: 9.0.55

JAVA_OPTS=-XX:MaxPermSize=256m -Xmx1024m -Xms512m

RunDeck

APIVERSION : 40

BUILD : 3.4.10-20220118
BUILDGIT : v3.4.10-0-g01b8e84
VERSION : 3.4.10-20220118

ThreadPoolSize:10

RDECK_CLI_OPTS=-Xms1028m -Xmx4096m

JVM

IMPLEMENTATIONVERSION : 25.312-b07
NAME : OpenJDK 64-Bit Server VM
VENDOR : Amazon.com Inc.
VERSION : 1.8.0_312

DB

MySQL MariaDB: 10.6.3

[Aaron Sheppard]

I have some of the thread dumps and metrics from today as well as the GCP screenshot growth

[rac...@rundeck.com]

Hi Aaron,

A good approach is to assign more memory to Rundeck via -Xmx/-Xms parameters, take a look at this (it’s a specific section for java.lang.OutOfMemoryError: Java heap space error).

Regards.

​

[Aaron Sheppard]

I have seen that link before; my apologies for not listing the profile file

rundeck\etc\profile ---------------

RDECK_BASE=/Apache/webapps/rundeck/rundeck
export RDECK_BASE

JAVA_HOME=/Amazon Corretto/jre8
export JAVA_HOME

PATH=$JAVA_HOME/bin:$RDECK_BASE/tools/bin:$PATH
export PATH

export JAVA_CMD=$JAVA_HOME/bin/java

if test -n "$JRE_HOME"
then
   unset JRE_HOME
fi

#
# Set min/max heap size
#
export RDECK_JVM="$RDECK_JVM -Xmx5g -Xms1g -XX:MaxMetaspaceSize=256m -server"

export RDECK_SSL_OPTS="-Djavax.net.ssl.trustStore=$RDECK_BASE/etc/truststore -Djavax.net.ssl.trustStoreType=jks -Djava.protocol.handler.pkgs=com.sun.net.ssl.internal.www.protocol"

umask 002

rundeck\etc\profile.bat ---------------

set RDECK_BASE=\Apache\webapps\rundeck\rundeck

set JAVA_HOME=\Amazon Corretto\jdk11.0.15_9

:: Unsetting JRE_HOME to ensure there is no conflict with JAVA_HOME
(set JRE_HOME=)

set Path=%JAVA_HOME%\bin;%RDECK_BASE%\tools\bin;%Path%

set RDECK_SSL_OPTS="-Djavax.net.ssl.trustStore=%RDECK_BASE%\etc\truststore -Djavax.net.ssl.trustStoreType=jks -Djava.protocol.handler.pkgs=com.sun.net.ssl.internal.www.protocol"
set RDECK_CLI_OPTS=-Xms1028m -Xmx4096m
set RD_LIBDIR=%RDECK_BASE%\tools\lib
:: set RDECK_JVM="$RDECK_JVM -Xmx6144m -Xms512m -XX:MaxPermSize=256m -server"

Apache\bin\setenv.bat ---------------

set "JRE_HOME=\Amazon Corretto\jdk11.0.15_9"
set "JAVA_OPTS=-XX:MaxPermSize=256m -Xmx1024m -Xms512m -server -Drdeck.base=\Apache\webapps\rundeck\rundeck -Drundeck.config.location=\Apache\webapps\rundeck\rundeck\server\config\rundeck-config.properties -Drundeck.server.logDir=\Apache\webapps\rundeck\rundeck\server\logs --add-opens=java.base/java.lang.invoke=ALL-UNNAMED"

----------------------

Total physical memory: 
36,699,156 kbytes
Free physical memory: 
31,009,600 kbytes
Total swap space: 
42,204,180 kbytes
Free swap space: 
20,826,488 kbytes

[rac...@rundeck.com]

Hi Aaron,

Could you increase those values on the profile.bat file? like this.

Greetings.

​

[rac...@rundeck.com]

I mean, on the setenv.bat file! my mistake.

Regards

[Aaron Sheppard]

I can increase the settings but I'm already at a steady 20Gb reserved between RunDeck, MariaDB and Tomcat; which only leaves 15Gb un reserved for the processes to flex up to out of the total 35Gb of the entire server.  How much is necessary for jobs that barely take 30 Mb to process on average with only upto 4 going at one time?

export RDECK_JVM="$RDECK_JVM -Xmx5g -Xms1g -XX:MaxMetaspaceSize=256m -server"

set RDECK_CLI_OPTS=-Xms1028m -Xmx4096m

The rest is with the mariadb buffer size:
innodb_buffer_pool_size=15G
innodb_log_file_size=4G

Any increase may not leave enough resources for jobs to process, or is that not correct?


Regards,



Aaron Sheppard

[rac...@rundeck.com]

Right Aaron, but under Tomcat, Rundeck takes the values from the setenv.bat file which is defined as: -XX:MaxPermSize=256m -Xmx1024m -Xms512m, basically, Tomcat manages the Rundeck memory parameters. Check the complete guide here.

Regards.

​