Jenkins Performance loading slow

[Sverre Moe]

Many of our users are reporting that Jenkins is very slow to load most of the times.

We have a legacy build system, built inhouse and which is much faster. It checks regularly git and builds all changed repositories in one operation. Thus Jenkins which is comprised of several distinct projects takes a little longer time. However this is not the main concern regarding performance. Usage of Jenkins web page is reported very slow very often.

I'm not quite sure what we can do to remedy this.

I who have spear-pointed us using Jenkins and am concerned for the future of Jenkins in our organisation.

Software:

Jenkins 2.168

Over 300 projects

Multibranch pipelines with Git

Most of the software is C/C++, and only a handful of Java. Some projects contains only configuration.

Hardware: Jenkins server running on a VM

OS: SUSE Linux Enterprise Server 12 SP3

CPU: 6 Cores Intel Xeon CPU E5-4620 2.20GHz

RAM: 8GB (Jenkins max-heap-size 6GB)

HDD: 400G 157G 244G 40% /var/lib/jenkins

[Service]
Environment=MALLOC_ARENA_MAX=1
ExecStart=/usr/bin/java -Xmx6g -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=300 -Djava.awt.headless=true -Dhudso
n.tasks.MailSender.SEND_TO_UNKNOWN_USERS=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins
.log --httpsPort=8443 --httpsKeyStore=/var/lib/jenkins/.keystore --httpsKeyStorePassword=changeit
User=jenkins

Jenkins is configured with 32 build agents. Each on their own VM.

List of installed plugins:

Active Directory plugin 2.12
Agiletestware Pangolin Connector for TestRail 2.6
Analysis Model API Plug-in 2.1.2
AnsiColor 0.6.2
Ant Plugin 1.9
Apache HttpComponents Client 4.x API Plugin 4.5.5-3.0
Audit Trail 2.4
Authentication Tokens API Plugin 1.3
Autofavorite for Blue Ocean 1.2.3
Badge 1.7
Basic Branch Build Strategies Plugin 1.1.1
Bitbucket Branch Source Plugin 2.4.2
Bitbucket Pipeline for Blue Ocean 1.13.2
Blue Ocean 1.13.2
Blue Ocean Core JS 1.13.2
Blue Ocean Pipeline Editor 1.13.2
Branch API Plugin 2.1.2
Build Rotator 1.2
Build Trigger Badge Plugin 2.10
CMake plugin 2.6.0
CVS Plug-in 2.14
Checkstyle Plug-in 4.0.0
Chroot Plugin 0.1.4
CloudBees Docker Custom Build Environment Plugin 1.7.3
Cobertura Plugin 1.13
Code Coverage API Plugin 1.0.7
Command Agent Launcher Plugin 1.3
Common API for Blue Ocean 1.13.2
Conditional BuildStep 1.3.6
Config API for Blue Ocean 1.13.2
Config File Provider Plugin 3.5
Configuration as Code Plugin 1.7
Coverity plugin 1.11.3
Cppcheck Plug-in 1.24
Credentials Binding Plugin 1.18
Credentials Plugin 2.1.18
Cron Column Plugin 1.4
Dashboard View 2.10
Dashboard for Blue Ocean 1.13.2
Dependency Graph Viewer Plugin 0.13
Design Language 1.13.2
Display URL API 2.3.0
Display URL for Blue Ocean 2.2.0
Docker API Plugin 3.0.14
Docker Commons Plugin 1.13
Docker Pipeline 1.17
Docker Slaves Plugin 1.0.7
Docker plugin 1.1.6
Doxygen Plug-in 0.18
Durable Task Plugin 1.29
Dynamic Search View Plugin 0.3.0
Email Extension Plugin 2.65
Embeddable Build Status Plugin 2.0
EnvInject API Plugin 1.5
Events API for Blue Ocean 1.13.2
External Monitor Job Type Plugin 1.7
Favorite 2.3.2
FindBugs Plug-in 5.0.0
Folders Plugin 6.7
GIT server Plugin 1.7
Git Parameter Plug-In 0.9.10
Git Pipeline for Blue Ocean 1.13.2
Git Tag Message Plugin 1.6.1
Git client plugin 2.7.6
Git plugin 3.9.3
GitHub API Plugin 1.95
GitHub Branch Source Plugin 2.4.2
GitHub Pipeline for Blue Ocean 1.13.2
GitHub plugin 1.29.4
GitLab Plugin 1.5.11
Gradle Plugin 1.30
Green Balls 1.15
Groovy 2.2
Groovy Postbuild 2.4.3
HTML Publisher plugin 1.18
Handy Uri Templates 2.x API Plugin 2.1.7-1.0
Infrastructure plugin for Publish Over X 0.22
JDK Tool Plugin 1.2
JIRA Integration for Blue Ocean 1.13.2
JIRA Pipeline Steps 1.4.5
JIRA Trigger Plugin 0.6.3
JIRA plugin 3.0.5
JSch dependency plugin 0.1.55
JUnit Attachments Plugin 1.5
JUnit Plugin 1.27
JWT for Blue Ocean 1.13.2
JaCoCo plugin 3.0.4
Jackson 2 API Plugin 2.9.8
JavaScript GUI Lib: ACE Editor bundle plugin 1.1
JavaScript GUI Lib: Handlebars bundle plugin 1.1.1
JavaScript GUI Lib: Moment.js bundle plugin 1.1.1
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin 1.2.1
Javadoc Plugin 1.5
JiraTestResultReporter plugin 2.0.7
Job DSL 1.72
LDAP Plugin 1.20
Lockable Resources plugin 2.4
Log Parser Plugin 2.0
Mailer Plugin 1.23
Managed Scripts 1.4
MapDB API Plugin 1.0.9.0
Matrix Authorization Strategy Plugin 2.3
Matrix Project Plugin 1.14
Maven Integration plugin 3.2
Maven Release Plug-in Plug-in 0.14.0
Mercurial plugin 2.5
Monitoring 1.76.0
Multi slave config plugin 1.2.0
Multiple SCMs plugin 0.6
Nexus Artifact Uploader 2.10
Nexus Task Runner Plugin 0.9.2
Node and Label parameter plugin 1.7.2
OWASP Markup Formatter Plugin 1.5
PAM Authentication plugin 1.4
Parameterized Scheduler 0.6.3
Parameterized Trigger plugin 2.35.2
Personalization for Blue Ocean 1.13.2
Pipeline 2.6
Pipeline Graph Analysis Plugin 1.9
Pipeline Maven Integration Plugin 3.6.7
Pipeline SCM API for Blue Ocean 1.13.2
Pipeline Utility Steps 2.3.0
Pipeline implementation for Blue Ocean 1.13.2
Pipeline: API 2.33
Pipeline: Basic Steps 2.14
Pipeline: Build Step 2.7
Pipeline: Declarative 1.3.6
Pipeline: Declarative Agent API 1.1.1
Pipeline: Declarative Extension Points API 1.3.6
Pipeline: Groovy 2.64
Pipeline: Input Step 2.9
Pipeline: Job 2.32
Pipeline: Milestone Step 1.3.1
Pipeline: Model API 1.3.6
Pipeline: Multibranch 2.21
Pipeline: Nodes and Processes 2.29
Pipeline: REST API Plugin 2.10
Pipeline: SCM Step 2.7
Pipeline: Shared Groovy Libraries 2.13
Pipeline: Stage Step 2.3
Pipeline: Stage Tags Metadata 1.3.6
Pipeline: Stage View Plugin 2.10
Pipeline: Step API 2.19
Pipeline: Supporting APIs 3.2
Plain Credentials Plugin 1.5
Pub-Sub "light" Bus 1.12
Publish Over SSH 1.20.1
REST API for Blue Ocean 1.13.2
REST Implementation for Blue Ocean 1.13.2
Rake plugin 1.8.0
RubyMetrics plugin for Jenkins 1.6.5
Run Condition Plugin 1.2
SCM API Plugin 2.3.0
SSH Agent Plugin 1.17
SSH Credentials Plugin 1.15
SSH Slaves plugin 1.29.4
SSH plugin 2.6.1
Script Security Plugin 1.54
Scriptler 2.9
Selenium HTML report 1.0
Server Sent Events (SSE) Gateway Plugin 1.17
Shelve Project Plugin 2.3
Simple Theme Plugin 0.5.1
SonarQube Scanner for Jenkins 2.8.1
Static Analysis Collector Plug-in 2.0.0
Static Analysis Utilities 1.95
Structs Plugin 1.17
Subversion Plug-in 2.12.1
TAP Plugin 2.2.2
Task Scanner Plug-in 4.53
Template Project plugin 1.5.2
TestLink Plugin 3.16
TestNG Results Plugin 1.15
Token Macro Plugin 2.7
Trilead API Plugin 1.0.1
Unleash Maven Plugin 2.3.0
Valgrind Plug-in 0.28
Variant Plugin 1.2
View Job Filters 2.1.1
WMI Windows Agents Plugin 1.4
Warnings Next Generation Plugin 3.0.3
Warnings Plug-in 5.0.1
Web for Blue Ocean 1.13.2
artifact-promotion 0.5.1
bouncycastle API Plugin 2.17
cron-shelve 1.3
database 1.5
database-postgresql 1.0
i18n for Blue Ocean 1.13.2
jQuery UI plugin 1.0.2
jQuery plugin 1.12.4-0
promoted builds plugin 3.2
ruby-runtime 0.13
user build vars plugin 1.5
xUnit plugin 2.3.3

[denis mone]

Hi there.

We had similar issues with our Jenkins installation. Mainly regarding the time it took to load the build page. We have 100+ jobs and around 20 build running every hour on average.

Initially our Jenkins master setup was:

 * An EC2 t2.large instance (2vCPU, 8GB RAM)
 * A 300GB EBS volume on SSD disk.
 * Jenkins JVM was deployed with Xmx=4GB

We had serious complains for the performance of Jenkins and after installing the monitoring plugin we discovered that Jenkins would reach it's max memory limit
too often and spent a lot of time doing GC thus we decided to upgrade the Jenkins master machine.

Currently our setup is

 * Jenkins master on an EC2 t3.xlarge AWS machine with 4 core vCPU (Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz) and 16GB
 * A 400GB EBS volume of SSD storage.
 * Jenkins JVM is deployed with Xmx=8GB

We have this setup for 4 months now and we haven't experienced the same performance problems. On days with higher number of builds, Jenkins master responds more slowly but never with 504

We have also upgrades our monitoring system to using node-exporter + Prometheus + Grafana in order to get more accurate metrics from our Jenkins master and slaves.

From the specs you have posted i'd suggest that you upgrade your system to one with more RAM and give half of it on Jenkins and if possible mount JENKINS_HOME on an SSD disk.

I'd also recommend installing monitoring plugin in order to be able to see what's going on when your users complain about Jenkins performance.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ab410cf3-2bd7-4fd8-98ef-863296681b32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Victor Martinez]

There are a couple of interesting articles/presentations to configure the JVM accordingly, see the below reply:
- https://groups.google.com/d/msg/jenkinsci-users/T7F9FHzSYtY/xq5k9JdjBAAJ

Cheers

[Stuart Rowe]

Since you have the Active Directory plugin installed, can you confirm that you have caching enabled? We were bit by this when upgrading a test instance to LTS 2.150.2.
See: https://issues.jenkins-ci.org/browse/JENKINS-56243

[Sverre Moe]

We have enabled cache for LDAP authentication

Cache size 20

Cache TTL 5 min 0 sec

Is there a workaround? Disabling cache perhaps?

[Sverre Moe]

The LDAP caching is supposed to increase performance

> Some LDAP servers may be slow, or rate limit client requests. In such cases enabling caching may improve performance of Jenkins with the risk of delayed propagation of user changes from LDAP and increased memory usage on the master. 

[Sverre Moe]

I tried increasing the cache

Size 50

TTL 1 hour

Still loading is really slow for some.

Refresh main page, fast. Then I try to enter a project. and I click and click and click, wait and wait, but nothing. Then it just loads after several seconds.

[Sverre Moe]

I did try to delete the cookies from Jenkins, and it became much faster at loading.

[Sverre Moe]

Seems there has been some improvements in a new version of Jenkins. Much better, but not quite out of the woods yet.

Upgrading to 2.181 seems to solve the slowness loading problem.

Let me repeat some of my comments from the Jenkins issue:

https://issues.jenkins-ci.org/browse/JENKINS-56243

I have been trying Jenkins 2.181 on our Jenkins Test server. Really fast loading of UI, not seeing any slowness.

Though the Jenkins Test server does not have as much build history, the real test would be to upgrade our Jenkins Production server.

Opening a multibranch project branch, the page seems to have loaded completely, but loading is still ongoing for some more ~30 seconds. Looked at what was loading and seems to be prototype.js. There was an fix for this in JENKINS-49319, but it was reverted.

Firefox is having much more problems than browsers with the Chromium engine. So it seems to be a javascript problem that last loading.

What I find very odd. If I log in to Jenkins (using LDAP) all loading problems goes away. Why should logging in solve the loading slowness?

The last remaining loading on Multibranch Pipeline branch project. Developer Tools in my browser showed me what was remaining trying to load.

ajax 200 xhr prototype.js:1585 310 B 15 ms
runs?since=%232&fullStages=true&_=1561103938220 200 xhr jquery2.js:998 48.8 KB 18 ms

These two repeated several times for about 30+ seconds until the page was actually finished loading.