Extremely Slow Gui - High cpu usage with no/few jobs running

[Aaron Sheppard]

My Rundeck detail

• Rundeck version: 3.4.10

• install type: war on tomcat 9.0.55

• OS Name/version: Windows Server 2016 Standard

• DB Type/version: default h2

Expected Behavior
Anytime we click on the UI the cpu spikes to 100% and stays for several seconds. Each click can take between 30 seconds and 4 minutes to clear depending on the web page. The Activity page and the Jobs page are the worst offenders.

How to Reproduce
Click on projects or Jobs with few or no jobs running; or click on the Activity page(s).

We have 5 projects each with 4-5 jobs and at any given time there may be 1 or 2 jobs running but the longest we have only runs for 10 minutes with the average job taking under a minute. We were using RunDeck 3.1.1 before but after the log4j vulnerability fix was made available we have upgraded. Both versions have been slow, but 3.4.1 seems to have been the slowest.

Our server is a VM with:

Windows Server 2016 Standard
Ram: 8Gb
CPU: Quad 2.1 Ghz
Disk: 249 GB (184 GB free)

The health api always says 'healthy':
{"dataSource.connection.time":{"healthy":true,"message":"Datasource connection healthy with timeout 5 seconds"},"quartz.scheduler.threadPool":{"healthy":true}}

I have set our setenv.bat in tomcat to the following:

set "JRE_HOME=C:\Program Files\Amazon Corretto\jre8" set "JAVA_OPTS=-XX:MaxPermSize=1g -Xmx2g -Xms1g -server -Drdeck.base=D:\Apache\webapps\rundeck\rundeck -Drundeck.config.location=D:\Apache\webapps\rundeck\rundeck\server\config\rundeck-config.properties -Drundeck.server.logDir=D:\Apache\webapps\rundeck\rundeck\server\logs"

Additionally I set our profile.bat in rundeck to:

set RDECK_BASE=D:\Apache\webapps\rundeck\rundeck set JAVA_HOME=C:\Program Files\Amazon Corretto\jre8 :: Unsetting JRE_HOME to ensure there is no conflict with JAVA_HOME (set JRE_HOME=) set Path=%JAVA_HOME%\bin;%RDECK_BASE%\tools\bin;%Path% set RDECK_SSL_OPTS="-Djavax.net.ssl.trustStore=%RDECK_BASE%\etc\truststore -Djavax.net.ssl.trustStoreType=jks -Djava.protocol.handler.pkgs=com.sun.net.ssl.internal.www.protocol" set RDECK_CLI_OPTS=-Xms256m -Xmx4096m set RD_LIBDIR=%RDECK_BASE%\tools\lib

[rac...@rundeck.com]

Hi Aaron,

Checking your system info I noticed that the instance is using only 247 MB as Xmx memory.

Could you try using this setenv.bat (in a non-prod env) and check the system info value?

set CATALINA_OPTS=-XX:MaxPermSize=256m -Xmx2048m -Xms1024m -server -Drdeck.base=C:\your\rundeck\path

More tips to configure:

1. Please move to a “real database”, H2 isn’t designed for production and usually is a root cause of slowness, take a look.
2. Keep your executions “low”, you can use the Execution History Clean feature.

Regards.

​

[Sheppard, Aaron]

Thank you,

 rac...@rundeck.com

Our current setenv.bat is using the following settings from before the upgrade:

set "JAVA_OPTS=-XX:MaxPermSize=1g -Xmx2g -Xms1g -server -Drdeck.base=D:\Apache\webapps\rundeck\rundeck -Drundeck.config.location=D:\Apache\webapps\rundeck\rundeck\server\config\rundeck-config.properties -Drundeck.server.logDir=D:\Apache\webapps\rundeck\rundeck\server\logs"

Should we be using CATALINA_OPTS instead?  Will this work with  tomcat 9.0.55?

Regards,

Aaron Sheppard

Programmer

GPSO Centralized Team

6929 N Lakewood Ave

Tulsa, OK 74117

O 918.590.9937 | M 918.344.1791

aaron.s...@verizon.com

https://verizonone.webex.com/join/aaron.sheppard

Digital Finance support/questions: GPSOCen...@verizon.com

Click to get a Ticket started!

--
You received this message because you are subscribed to a topic in the Google Groups "rundeck-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rundeck-discuss/vj65ADl9Pk4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/f3661f47-b647-4c12-b871-4447751683dcn%40googlegroups.com.

[rac...@rundeck.com]

Hi Aaron,

It works in that way over Tomcat 9.0.55.

Greetings.

[Sheppard, Aaron]

Thanks rac...@rundeck.com,

I stopped tomcat, then I made the CATALINA_OPTS change in setenv.bat and changed the settings in rundeck\etc\profile.bat but it looks like the memory isn't being recognized, is there somewhere else that needs this change?

setenv.bat:

set "JRE_HOME=D:\Amazon Corretto\jre8"
set "CATALINA_OPTS=-XX:MaxPermSize=256m -Xmx1024m -Xms512m -server -Drdeck.base=D:\Apache\webapps\rundeck\rundeck -Drundeck.config.location=D:\Apache\webapps\rundeck\rundeck\server\config\rundeck-config.properties -Drundeck.server.logDir=D:\Apache\webapps\rundeck\rundeck\server\logs"

profile.bat:

set RDECK_BASE=D:\Apache\webapps\rundeck\rundeck

set JAVA_HOME=D:\Amazon Corretto\jre8

:: Unsetting JRE_HOME to ensure there is no conflict with JAVA_HOME
(set JRE_HOME=)

set Path=%JAVA_HOME%\bin;%RDECK_BASE%\tools\bin;%Path%

set RDECK_SSL_OPTS="-Djavax.net.ssl.trustStore=%RDECK_BASE%\etc\truststore -Djavax.net.ssl.trustStoreType=jks -Djava.protocol.handler.pkgs=com.sun.net.ssl.internal.www.protocol"

set RDECK_CLI_OPTS=-Xms1028m -Xmx4096m
set RD_LIBDIR=%RDECK_BASE%\tools\lib

thanks,

Aaron Sheppard

Programmer

GPSO Centralized Team

6929 N Lakewood Ave

Tulsa, OK 74117

O 918.590.9937 | M 918.344.1791

aaron.s...@verizon.com

https://verizonone.webex.com/join/aaron.sheppard

Digital Finance support/questions: GPSOCen...@verizon.com

Click to get a Ticket started!

To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/95577d87-d6d2-4473-8d50-183e9c54e7b0n%40googlegroups.com.

[Sheppard, Aaron]

This is the current java monitor report for memory:

VM Summary Thursday, March 3, 2022 at 10:15:58 AM Eastern Standard Time
Current heap size:     41,149 kbytes Maximum heap size:  2,097,152 kbytes		Committed memory:  51,200 kbytes Pending finalization:  0 objects
Garbage collector:  Name = 'G1 Young Generation', Collections = 79, Total time spent = 0.183 seconds Garbage collector:  Name = 'G1 Old Generation', Collections = 0, Total time spent = 0.000 seconds
Operating System:  Windows Server 2016 10.0 Architecture:  amd64 Number of processors:  4 Committed virtual memory:  192,616 kbytes		Total physical memory:  8,387,604 kbytes Free physical memory:  3,198,208 kbytes Total swap space:  9,698,324 kbytes Free swap space:  3,921,168 kbytes

Thanks again,

Aaron Sheppard

Programmer

GPSO Centralized Team

6929 N Lakewood Ave

Tulsa, OK 74117

O 918.590.9937 | M 918.344.1791

aaron.s...@verizon.com

https://verizonone.webex.com/join/aaron.sheppard

Digital Finance support/questions: GPSOCen...@verizon.com

Click to get a Ticket started!

To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/95577d87-d6d2-4473-8d50-183e9c54e7b0n%40googlegroups.com.

[rac...@rundeck.com]

Hi Aaron,

Could you try using OpenJDK instead of Amazon Corretto? My test environment is based on this OpenJDK distribution.

Regards.

[Aaron Sheppard]

Thank you very much for the suggestion, we have to use Amazon Corretto per our security department; so unfortunately I cannot use OpenJDK.

Is there any Corretto alternative techniques?

[rac...@rundeck.com]

Hi Aaron,

I tested Rundeck 3.4.10 on Tomcat 9.0.55 under Amazon Corretto JRE 1.8.0_312 and 11 (on Windows Server 2019 virtual machine) from scratch and it works as expected, let me share with you my config files:

The setenv.bat file (at tomcat bin directory) I added the first line just to test your configuration, but it works without that line:

set "JRE_HOME=C:\jre8
set "CATALINA_OPTS=-XX:MaxPermSize=256m -Xmx1024m -Xms512m -server -Drdeck.base=C:\rdeck"

The profile.bat file (at C:/rdeck/etc directory), this file keeps untouched.

set RDECK_BASE=C:\rdeck

set JAVA_HOME=C:\jre8

:: Unsetting JRE_HOME to ensure there is no conflict with JAVA_HOME
(set JRE_HOME=)

set Path=%JAVA_HOME%\bin;%RDECK_BASE%\tools\bin;%Path%

set RDECK_SSL_OPTS="-Djavax.net.ssl.trustStore=%RDECK_BASE%\etc\truststore -Djavax.net.ssl.trustStoreType=jks -Djava.protocol.handler.pkgs=com.sun.net.ssl.internal.www.protocol"
set RDECK_CLI_OPTS=-Xms256m -Xmx4096m
set RD_LIBDIR=%RDECK_BASE%\tools\lib

The rundeck-config.properties file (at C:\rdeck\server\config directory, my test instance is outside Tomcat base directory):

#loglevel.default is the default log level for jobs: ERROR,WARN,INFO,VERBOSE,DEBUG
loglevel.default=INFO
rdeck.base=C:/rdeck

#rss.enabled if set to true enables RSS feeds that are public (non-authenticated)
rss.enabled=false
server.address=0.0.0.0
server.port=8080
grails.serverURL=http://localhost:8080/rdeck
server.servlet.context-path=/rdeck
dataSource.dbCreate = none
dataSource.url = jdbc:h2:file:C:/rdeck/server/data/grailsdb;DB_CLOSE_ON_EXIT=FALSE
grails.plugin.databasemigration.updateOnStart=true
# Pre Auth mode settings
rundeck.security.authorization.preauthenticated.enabled=false
rundeck.security.authorization.preauthenticated.attributeName=REMOTE_USER_GROUPS
rundeck.security.authorization.preauthenticated.delimiter=,
# Header from which to obtain user name
rundeck.security.authorization.preauthenticated.userNameHeader=X-Forwarded-Uuid
# Header from which to obtain list of roles
rundeck.security.authorization.preauthenticated.userRolesHeader=X-Forwarded-Roles
# Redirect to upstream logout url
rundeck.security.authorization.preauthenticated.redirectLogout=false
rundeck.security.authorization.preauthenticated.redirectUrl=/oauth2/sign_in

rundeck.feature.repository.enabled=true

The framework.properties file (at C:\rdeck\etc path)

# framework.properties -
#

# ----------------------------------------------------------------
# Server connection information
# ----------------------------------------------------------------

framework.server.name = localhost
framework.server.hostname = localhost
framework.server.port = 8080
framework.server.url = http://localhost:8080/rdeck

# ----------------------------------------------------------------
# Installation locations
# ----------------------------------------------------------------

rdeck.base=C:/rdeck

framework.projects.dir=C:/rdeck/projects
framework.etc.dir=C:/rdeck/etc
framework.var.dir=C:/rdeck/var
framework.tmp.dir=C:/rdeck/var/tmp
framework.logs.dir=C:/rdeck/var/logs
framework.libext.dir=C:/rdeck/libext

# ----------------------------------------------------------------
# SSH defaults for node executor and file copier
# ----------------------------------------------------------------

framework.ssh.keypath = C:/Users/Administrator/.ssh/id_rsa
framework.ssh.user = Administrator

# ssh connection timeout after a specified number of milliseconds.
# "0" value means wait forever.
framework.ssh.timeout = 0

# ----------------------------------------------------------------
# Auto generated server UUID: 9ca61cca-d4af-4c76-bd2c-60b01fec427f
# ----------------------------------------------------------------
rundeck.server.uuid = 9ca61cca-d4af-4c76-bd2c-60b01fec427f

# ----------------------------------------------------------------
# System-wide global variables.
# ----------------------------------------------------------------

# Expands to ${globals.var1}
#framework.globals.var1 = value1

# Expands to ${globals.var2}
#framework.globals.var2 = value2

In this research I found that the Allocated Memory always reach the 100% on Java 8 (Corretto and AdoptJDK/OpenJDK), switching to Java 11 (Corretto and AdopJDK/OpenJDK) this value keeps around 50%, so, I think Java 11 is more efficient to manage Rundeck memory allocation under Tomcat.

So, as a recommendation, switch to Java 11, configure your instances against a “real” database and make sure that you’re editing the right files, the Tomcat+Rundeck config don’t need extra stuff (at least basically), all is defined on the setenv.bat file (also take a look on the file permissions).

Regards!

​

[Aaron Sheppard]

Thanks again  rac...@rundeck.com for your very thorough answers.  I will definitely review our setup and report back if I have any successes. Something we learned today is that there is a backup software running on our server, which is a vm, that was failing.  As a result, we uninstalled the backup software and since then the cpu has been consistently under 10% except when accessing the gui; but even then the spike to 80%+ is only temporary and goes back down with in a few moments.  While it's still slower than expected (14-seconds avg screen change) it is a huge improvement.

Again, I appreciate your help in this matter.

[Aaron Sheppard]

 rac...@rundeck.com thanks again for your help.  I went back and got some things updated. but still running into bottlenecks if more than one process is running at a time.  Granted the cpu is only 'maxing' at 56% for a few minutes then going back down to 2% which is a HUGE improvement; but if I have two processes running and a user trying to update a record it can take as much as 30 seconds to 2 minutes for the screen to update.
 
I have upgraded to 35 Gb of RAM,
added 4 CPUs to make it a total of 8 CPUs.  
I updated to Amazon Corretto 11.
Installed MySQL (MariaDB) for the DB and gave it 20 GB ram in the settings.
Updated the files you listed but the 'system report' screen still shows jvm of 1.8.0_312.  

The following are the entries from the configs.  Is there anything I am missing?  

I need to be able to run up to 300 processes on this machine and at any given time 24-50 will be happening at the same time; but at the moment more than 2 and the users can't use the gui at all.

The setenv.bat file (at tomcat bin directory):

```

set "JRE_HOME=[dir]\Amazon Corretto\jdk11.0.14_9"
set "JAVA_OPTS=-XX:MaxPermSize=256m -Xmx1024m -Xms512m -server -Drdeck.base=[dir]\Apache\webapps\rundeck\rundeck -Drundeck.config.location=[dir]\Apache\webapps\rundeck\rundeck\server\config\rundeck-config.properties -Drundeck.server.logDir=[dir]\Apache\webapps\rundeck\rundeck\server\logs""

```
The profile.bat file

```

set RDECK_BASE=[dir]\Apache\webapps\rundeck\rundeck

set JAVA_HOME=[dir]\Amazon Corretto\jdk11.0.14_9

:: Unsetting JRE_HOME to ensure there is no conflict with JAVA_HOME
(set JRE_HOME=)

set Path=%JAVA_HOME%\bin;%RDECK_BASE%\tools\bin;%Path%

set RDECK_SSL_OPTS="-Djavax.net.ssl.trustStore=%RDECK_BASE%\etc\truststore -Djavax.net.ssl.trustStoreType=jks -Djava.protocol.handler.pkgs=com.sun.net.ssl.internal.www.protocol"

set RDECK_CLI_OPTS=-Xms1028m -Xmx4096m
set RD_LIBDIR=%RDECK_BASE%\tools\lib
:: set RDECK_JVM="$RDECK_JVM -Xmx6144m -Xms512m -XX:MaxPermSize=256m -server"
```
The rundeck-config.properties file:

```

#loglevel.default is the default log level for jobs: ERROR,WARN,INFO,VERBOSE,DEBUG
loglevel.default=INFO

rdeck.base=[dir]/Apache/webapps/rundeck/rundeck

#rss.enabled if set to true enables RSS feeds that are public (non-authenticated)
rss.enabled=false

server.address=[server ip]
server.port=8080
grails.serverURL=http:// [server ip] :8080/rundeck
#dataSource.dbCreate = none
dataSource.dbCreate = update
#dataSource.url = jdbc:h2:file:[dir]/Apache/webapps/rundeck/rundeck/server/data/grailsdb;DB_CLOSE_ON_EXIT=FALSE
dataSource.url = jdbc:mysql://localhost/rundeck?autoReconnect=true&useSSL=false
dataSource.username = [user]
dataSource.password = [password]
dataSource.driverClassName = org.mariadb.jdbc.Driver
grails.plugin.databasemigration.updateOnStart=true
# email settings
grails.mail.host=host
grails.mail.port=25

grails.plugin.databasemigration.updateOnStart=true
# Pre Auth mode settings
rundeck.security.authorization.preauthenticated.enabled=false
rundeck.security.authorization.preauthenticated.attributeName=REMOTE_USER_GROUPS
rundeck.security.authorization.preauthenticated.delimiter=,
# Header from which to obtain user name
rundeck.security.authorization.preauthenticated.userNameHeader=X-Forwarded-Uuid
# Header from which to obtain list of roles
rundeck.security.authorization.preauthenticated.userRolesHeader=X-Forwarded-Roles
# Redirect to upstream logout url
rundeck.security.authorization.preauthenticated.redirectLogout=false
rundeck.security.authorization.preauthenticated.redirectUrl=/oauth2/sign_in

rundeck.feature.repository.enabled=true
```
The framework.properties file

```

# framework.properties -
#

# ----------------------------------------------------------------
# Server connection information
# ----------------------------------------------------------------

framework.server.name = localhost
framework.server.hostname = localhost
framework.server.port = 8080

framework.server.url = http://localhost:8080/rundeck

# ----------------------------------------------------------------
# Installation locations
# ----------------------------------------------------------------

rdeck.base=[dir]/Apache/webapps/rundeck/rundeck

framework.projects.dir=[dir]/Apache/webapps/rundeck/rundeck/projects
framework.etc.dir=[dir]/Apache/webapps/rundeck/rundeck/etc
framework.var.dir=[dir]/Apache/webapps/rundeck/rundeck/var
framework.tmp.dir=[dir]/Apache/webapps/rundeck/rundeck/var/tmp
framework.logs.dir=[dir]/Apache/webapps/rundeck/rundeck/var/logs
framework.libext.dir=[dir]/Apache/webapps/rundeck/rundeck/libext

# ----------------------------------------------------------------
# SSH defaults for node executor and file copier
# ----------------------------------------------------------------

framework.ssh.keypath = .../.ssh/id_rsa
framework.ssh.user = [user]

# ssh connection timeout after a specified number of milliseconds.
# "0" value means wait forever.
framework.ssh.timeout = 0

# ----------------------------------------------------------------

# Auto generated server UUI[dir] ...
# ----------------------------------------------------------------
rundeck.server.uuid = uuid

# ----------------------------------------------------------------
# System-wide global variables.
# ----------------------------------------------------------------

# Expands to ${globals.var1}
#framework.globals.var1 = value1

# Expands to ${globals.var2}
#framework.globals.var2 = value2

```

[Aaron Sheppard]

I made all the suggested changes; we added 8 CPUs and 35 GB ram; and yet we still get very poor performance, this photo is when there were no jobs running at all. It took seven minutes to login, open a project, edit a job, add three emails to the notifications, save the job and navigate back to the 'all projects' screen.  At one point the CPU maxed out at 80%.  Earlier in the day we were trying to update a job while other jobs were running and it completely locked up the server, causing 3 jobs to fail and not even write out the log file to determine the issue.  It wasn't until I turned off the tomcat service, let it set for five minutes then cleared the dns cache and did a gpupdate and a reboot that it came back to normal.  This is becoming all too common.  I'm not sure what else to throw at it.

[rac...@rundeck.com]

Hi Aaron,

Could you check if your Tomcat environment has some global memory limitation or special config? (probably in the catalina.bat init script) Could you test in another tomcat environment with the latest Rundeck version?

Regards.