[JIRA] (JENKINS-56243) Jenkins GUI is slow -removing cookie fixes it (temporarily)

247 views
Skip to first unread message

henjovr@gmail.com (JIRA)

unread,
Feb 22, 2019, 2:46:01 AM2/22/19
to jenkinsc...@googlegroups.com
Henjo van Rees created an issue
 
Jenkins / Bug JENKINS-56243
Jenkins GUI is slow -removing cookie fixes it (temporarily)
Issue Type: Bug Bug
Assignee: Unassigned
Components: core
Created: 2019-02-22 07:45
Priority: Minor Minor
Reporter: Henjo van Rees

The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

Situation:

  • In a clean browser (no cache, cookies) Jenkins is very fast
  • After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
    Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
    Jenkins just isn't workable for users at that time.
  • Logging out + in again does not fix it for that user.
  • Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again.

 

So, what happens with the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE? 
Why does it cause the slowness after hours of use?
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

henjovr@gmail.com (JIRA)

unread,
Feb 22, 2019, 2:50:02 AM2/22/19
to jenkinsc...@googlegroups.com
Henjo van Rees updated an issue
Change By: Henjo van Rees
The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

Situation:
* In a clean browser (no cache, cookies) Jenkins is very fast
* After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:

Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
Jenkins just isn't workable for users at that time.
* Logging out + in again does not fix it for that user.
* _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

 

So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 

Why does it cause the slowness after hours of use?

driver@hotmail.co.uk (JIRA)

unread,
Feb 25, 2019, 2:44:02 PM2/25/19
to jenkinsc...@googlegroups.com
Ian Driver commented on Bug JENKINS-56243
 
Re: Jenkins GUI is slow -removing cookie fixes it (temporarily)

We are also experiencing this problem on the latest LTS release (2.150.3)

lig@hms.se (JIRA)

unread,
Feb 26, 2019, 1:20:02 AM2/26/19
to jenkinsc...@googlegroups.com

Its also present in 2.150.2. We are experiencing it after upgrade from 2.73.3 to 2.150.2.

jenkins@anarres.org (JIRA)

unread,
Mar 1, 2019, 11:15:03 AM3/1/19
to jenkinsc...@googlegroups.com
Shevek . commented on Bug JENKINS-56243

This appears to be affecting us after the update to 2.150.3 and has made jenkins unusable

jenkins@anarres.org (JIRA)

unread,
Mar 1, 2019, 11:41:02 AM3/1/19
to jenkinsc...@googlegroups.com
Shevek . commented on Bug JENKINS-56243

Downgrade to 2.150.1 appears to solve the issue.

jenkins@anarres.org (JIRA)

unread,
Mar 1, 2019, 11:59:03 AM3/1/19
to jenkinsc...@googlegroups.com
Shevek . commented on Bug JENKINS-56243

Tempted to say this is a major or blocker as it kills our usage of Jenkins on the first non-login request. We do NOT get an hour of usability, we get NO usability on 2.150.3

henjovr@gmail.com (JIRA)

unread,
Mar 3, 2019, 8:31:03 AM3/3/19
to jenkinsc...@googlegroups.com

So, 2.150.1 doesn't seem to have the problem.
2.150.2 and higher have the problem. 

When I look at the 2.150.2 changelog I immediately see this fix:

"Deleting a user in an external security realm did not invalidate their session or 'Remember me' cookie
SECURITY-901 / CVE-2019-1003004
When using an external security realm such as LDAP or Active Directory, deleting a user from the security realm does not result in the user losing access to Jenkins.

While deleting the user record from Jenkins did invalidate the 'Remember me' cookie, there was no way to invalidate active sessions besides restarting Jenkins or terminating sessions through other means, such as Monitoring Plugin.

Jenkins now encodes a per-user seed value in sessions, 'Remember me' cookies, and cached authentications of the remoting-based CLI, that can manually be reset by a user themselves, or an administrator, on the user’s configuration page. Doing so will invalidate all current sessions, 'Remember me' cookies, and cached CLI authentications, requiring credentials to be entered again to authenticate. Deleting a user record in Jenkins will now also invalidate existing sessions, as the current seed value is deleted as well."

 

So, concluding: This security fix introduces are very nasty slowdown when using Remember Me and LDAP/AD.

How can we escalate this issue further? 

henjovr@gmail.com (JIRA)

unread,
Mar 3, 2019, 8:31:04 AM3/3/19
to jenkinsc...@googlegroups.com
Henjo van Rees edited a comment on Bug JENKINS-56243
So, 2.150.1 doesn't seem to have the problem.
2.150.2 and higher have the problem. 

When I look at the 2.150.2 changelog I immediately see this fix:
_ "Deleting a user in an external security realm did not invalidate their session or 'Remember me' cookie cookie_
SECURITY _SECURITY -901 / CVE-2019- 1003004 1003004_
When _When using an external security realm such as LDAP or Active Directory, deleting a user from the security realm does not result in the user losing access to Jenkins. _

While _While deleting the user record from Jenkins did invalidate the 'Remember me' cookie, there was no way to invalidate active sessions besides restarting Jenkins or terminating sessions through other means, such as Monitoring Plugin. _

Jenkins _Jenkins now encodes a per-user seed value in sessions, 'Remember me' cookies, and cached authentications of the remoting-based CLI, that can manually be reset by a user themselves, or an administrator, on the user’s configuration page. Doing so will invalidate all current sessions, 'Remember me' cookies, and cached CLI authentications, requiring credentials to be entered again to authenticate. Deleting a user record in Jenkins will now also invalidate existing sessions, as the current seed value is deleted as well." _

 

So, concluding: This security fix introduces are very nasty slowdown when using Remember Me and LDAP/AD.

How can we escalate this issue further? 

henjovr@gmail.com (JIRA)

unread,
Mar 3, 2019, 8:31:06 AM3/3/19
to jenkinsc...@googlegroups.com
Henjo van Rees updated an issue
Change By: Henjo van Rees
Priority: Minor Major

henjovr@gmail.com (JIRA)

unread,
Mar 3, 2019, 8:32:02 AM3/3/19
to jenkinsc...@googlegroups.com
Henjo van Rees updated an issue
The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

Situation:
* In a clean browser (no cache, cookies) Jenkins is very fast
* After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
Jenkins just isn't workable for users at that time.
* Logging out + in again does not fix it for that user.
* _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

 So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 
Why does it cause the slowness after hours of use?


 

[update]

 

henjovr@gmail.com (JIRA)

unread,
Mar 3, 2019, 8:33:03 AM3/3/19
to jenkinsc...@googlegroups.com
Henjo van Rees updated an issue
The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

Situation:
* In a clean browser (no cache, cookies) Jenkins is very fast
* After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
Jenkins just isn't workable for users at that time.
* Logging out + in again does not fix it for that user.
* _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

 So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 
Why does it cause the slowness after hours of use?

 

_* [update] *_

  _*SECURITY-901 / CVE-2019-1003004 in Jenkins 2.150.2 introduced a security fix, but with a side effect that after some time (hours) the Jenkins GUI for that user starts to slow down to a crawl.*_

dbeck@cloudbees.com (JIRA)

unread,
Mar 15, 2019, 12:57:03 PM3/15/19
to jenkinsc...@googlegroups.com
Daniel Beck commented on Bug JENKINS-56243
 
Re: Jenkins GUI is slow -removing cookie fixes it (temporarily)

Could we get thread dumps for this while Jenkins is busy? https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump

Note that this might simply be a side effect of a bad performing security realm (AD server) that's getting hit more frequently.

dbeck@cloudbees.com (JIRA)

unread,
Mar 16, 2019, 11:34:04 AM3/16/19
to jenkinsc...@googlegroups.com

CC Wadeck Follonier just in case.

wfollonier@cloudbees.com (JIRA)

unread,
Mar 18, 2019, 10:02:03 AM3/18/19
to jenkinsc...@googlegroups.com

Henjo van Rees, Shevek ., Linus Geson, Ian Driver, could you provide additional information about your configuration? Especially the security realm that is used, in which version (if from plugin), with specific configuration as well (like AD/LDAP cache configuration).

In addition, when you are seeing such performance problem, could you check the cookies of one of your request? (especially the number of cookie that is sent)

From my PoV there is no huge performance impact on the REMEMBER_ME cookie as the only addition there is a User.getById call, that is doing nothing with external security realm.

lig@hms.se (JIRA)

unread,
Mar 18, 2019, 10:25:01 AM3/18/19
to jenkinsc...@googlegroups.com

We are using Active Directory security realm using Active Directory Plugin version 2.12. The only additional configuration enabled is "Enable StartTLS".

I'll try to get back to you with the other information as well.

stuartr@ea.com (JIRA)

unread,
Mar 18, 2019, 12:04:02 PM3/18/19
to jenkinsc...@googlegroups.com

We experienced this same issue after upgrading to LTS 2.150.2 also using Active Directory. The problem was resolved after enabling cache under the security realm configuration. We thought this setting had been configured already - perhaps it was disabled by the upgrade to LTS 2.150.2 or just misconfiguration on our end.

wfollonier@cloudbees.com (JIRA)

unread,
Mar 18, 2019, 12:27:02 PM3/18/19
to jenkinsc...@googlegroups.com

perhaps it was disabled by the upgrade to LTS 2.150.2

Nothing done during the security around that, sorry

Linus Geson Try enabling the cache feature of the plugin, that will help you a lot I imagine.

jenkins@anarres.org (JIRA)

unread,
Mar 18, 2019, 1:07:02 PM3/18/19
to jenkinsc...@googlegroups.com
Shevek . commented on Bug JENKINS-56243

Wadeck Follonier We are using built in user authentication with the built in access matrix. Nothing special whatsoever. Stock install via apt.

 

dbeck@cloudbees.com (JIRA)

unread,
Mar 18, 2019, 1:27:03 PM3/18/19
to jenkinsc...@googlegroups.com

Could we get thread dumps for this while Jenkins is busy? https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump
Add Comment Add Comment
 

lig@hms.se (JIRA)

unread,
Mar 19, 2019, 2:24:04 AM3/19/19
to jenkinsc...@googlegroups.com
Linus Geson updated an issue
 
Change By: Linus Geson
Attachment: Thread dump [Jenkins].html

lig@hms.se (JIRA)

unread,
Mar 19, 2019, 2:27:02 AM3/19/19
to jenkinsc...@googlegroups.com
Linus Geson commented on Bug JENKINS-56243
 
Re: Jenkins GUI is slow -removing cookie fixes it (temporarily)

Thread dump [Jenkins].html

Here comes a thread dump while the GUI is slow. I logged in yesterday when leaving work (checked keep me signed in) and shut down my computer. Arriving now this morning and booting up my laptop and opening the Jenkins web it is slow, not unusable but extremely slow.

dbeck@cloudbees.com (JIRA)

unread,
Mar 19, 2019, 2:32:04 AM3/19/19
to jenkinsc...@googlegroups.com

At the time the thread dump was recorded, that was the only request being handled (and it doesn't show problems). I suppose it's best in this case to take the thread dump another way (e.g. signal), while a slow request is being handled by Jenkins.

lig@hms.se (JIRA)

unread,
Mar 19, 2019, 2:46:03 AM3/19/19
to jenkinsc...@googlegroups.com
Linus Geson updated an issue
Change By: Linus Geson
Attachment: Thread dump2 [Jenkins].html

lig@hms.se (JIRA)

unread,
Mar 19, 2019, 2:51:04 AM3/19/19
to jenkinsc...@googlegroups.com
Linus Geson commented on Bug JENKINS-56243
 
Re: Jenkins GUI is slow -removing cookie fixes it (temporarily)

This thread dump is taken in one chrome tab while another tab is busy loading another Jenkins page. The threadDump tab finished while the other tab was still busy. I could try the signalling method as well if this dump is not good enough, I just hurried to get the dump before reading your comment entirely and realizing you asked for a different method

 

Thread dump2 [Jenkins].html

dbeck@cloudbees.com (JIRA)

unread,
Mar 19, 2019, 3:22:06 AM3/19/19
to jenkinsc...@googlegroups.com

Interesting.

https://github.com/jenkinsci/jenkins/blob/1c9eb43283e7321ee4d3a0e1e9995453493ff04a/core/src/main/java/hudson/security/TokenBasedRememberMeServices2.java#L240 is new in the security fix.

With a slow security realm, this will affect not just existing user lookups e.g. from changelogs ( GitChangeSet.findOrCreateUser shows up 4 times in the thread dump), but also any other request. Four separate requests go through TokenBasedRememberMeServices2.retrieveAuthFromCookie – caching means only one request to the security realm, but if that's slow, all of them are handled slowly.

The thread dump also indicates AD goes through referrals, which slows everything down further. I wonder whether the security realm config is just terrible. Make sure you're using caching, if available, and that you're contacting the global catalog (which is all I remember from working in an AD environment).

lig@hms.se (JIRA)

unread,
Mar 19, 2019, 3:30:04 AM3/19/19
to jenkinsc...@googlegroups.com

I don't know anything at all about AD but we do have some performance issues related to it that has affected our initial login to Jenkins (the regular log in takes 20 seconds) even before the upgrade that brought the current problems in this issue. Our IT department have some kind of plan for handling those issue down the road, but yes we do have a slow security realm.

Thanks for the tips I'll look into it on my end.

lig@hms.se (JIRA)

unread,
Mar 19, 2019, 3:35:03 AM3/19/19
to jenkinsc...@googlegroups.com
Linus Geson edited a comment on Bug JENKINS-56243
I don't know anything at all about AD but we do have some performance issues related to it that has affected our initial login to Jenkins (the regular log in takes 20 seconds) even before the upgrade that brought the current problems in this issue. Our IT department have some kind of plan for handling those issue down the road, but yes we do have a slow security realm.

Thanks for the tips I'll look into it on my end. ...

...I enable the cache and the GUI went from slow to normal fast without having to delete any cookie.

wfollonier@cloudbees.com (JIRA)

unread,
Mar 19, 2019, 3:52:06 AM3/19/19
to jenkinsc...@googlegroups.com

is new in the security fix.

Not really, it was just "moved" from super class to that class, in order to have the time-independant equal method, nothing else changed in the method.

frederic.seneque@inova-software.com (JIRA)

unread,
Apr 8, 2019, 4:25:04 AM4/8/19
to jenkinsc...@googlegroups.com

Hi all,

We are currently running Jenkins 2.164.1 LTS with the AD Plugin for authentication and we are facing the same issues. Enabling cache has improved a lot the response time of all HTTP requests (static resources/xhr ..) to jenkins. Is it considered as a work around or a mandatory configuration while using this AD plugin?

sverre.moe@gmail.com (JIRA)

unread,
Apr 10, 2019, 11:51:05 AM4/10/19
to jenkinsc...@googlegroups.com

I cannot find a 2.150.2, mentioned above but 2.150-1.2
https://pkg.jenkins.io/opensuse/jenkins-2.150-1.2.noarch.rpm

I Just tried to downgrade to 2.149 and it did not go so well, as several plugins do not work now.
Our users are having serous trouble with slow Jenkins user experience. I hope this can be fixed soon.

dbeck@cloudbees.com (JIRA)

unread,
Apr 10, 2019, 11:56:03 AM4/10/19
to jenkinsc...@googlegroups.com

sverre.moe@gmail.com (JIRA)

unread,
Apr 10, 2019, 12:09:02 PM4/10/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Bug JENKINS-56243
I cannot find a 2.150.2, mentioned above but 2.150-1.2
https://pkg.jenkins.io/opensuse/jenkins-2.150-1.2.noarch.rpm

I Just tried to downgrade to 2.149 and it did not go so well, as several plugins do not work now.
Our users are having serous serious trouble with slow Jenkins user experience. I hope this can be fixed soon.

sverre.moe@gmail.com (JIRA)

unread,
Apr 10, 2019, 12:16:04 PM4/10/19
to jenkinsc...@googlegroups.com

I was unaware of the opensuse-stable builds. So this is the Jenkins LTS releases.
I tried to downgrade to the 2.150.1 LTS and it worked with our installed plugins.

sverre.moe@gmail.com (JIRA)

unread,
Apr 10, 2019, 12:26:02 PM4/10/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Bug JENKINS-56243
I was unaware of the opensuse-stable builds. So this is the Jenkins LTS releases.
I tried to downgrade to the 2.150.1 LTS and it worked with our installed plugins.


The LTS upgrade guide seems to have a workaround for this bug
https://jenkins.io/doc/upgrade-guide/2.150/

dpeschman@godaddy.com (JIRA)

unread,
Apr 10, 2019, 12:27:01 PM4/10/19
to jenkinsc...@googlegroups.com

Enabling AD caching worked for us too.

dbeck@cloudbees.com (JIRA)

unread,
Apr 10, 2019, 12:28:03 PM4/10/19
to jenkinsc...@googlegroups.com

The LTS upgrade guide seems to have a workaround for this bug

Different issue. You're not even logged in with that one.

 

sverre.moe@gmail.com (JIRA)

unread,
Apr 10, 2019, 12:33:02 PM4/10/19
to jenkinsc...@googlegroups.com

So the workaround -Djenkins.security.seed.UserSeedProperty.disableUserSeed=true will not fix this issue?
It says its a workaround for SECURITY-901 which mentioned in the description caused the regression.

sverre.moe@gmail.com (JIRA)

unread,
Apr 10, 2019, 12:34:03 PM4/10/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Bug JENKINS-56243
So the workaround -Djenkins.security.seed.UserSeedProperty.disableUserSeed=true will not fix this issue?
It says its a workaround for SECURITY-901 which mentioned in the description here caused the regression.

dbeck@cloudbees.com (JIRA)

unread,
Apr 10, 2019, 12:36:03 PM4/10/19
to jenkinsc...@googlegroups.com

Unsure, may or may not help.

sverre.moe@gmail.com (JIRA)

unread,
Apr 10, 2019, 12:43:03 PM4/10/19
to jenkinsc...@googlegroups.com

Well if it doesn't downgrading to Jenkins LTS 2.150.1 worked on our Test Jenkins, so we have that to fall back to if it doesn't work.

sverre.moe@gmail.com (JIRA)

unread,
Apr 10, 2019, 12:44:02 PM4/10/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Bug JENKINS-56243
Well if it doesn't , the downgrading to Jenkins LTS 2.150.1 worked on our Test Jenkins, so we have that to fall back to if it  doesn't work.

jenkins@anarres.org (JIRA)

unread,
Apr 10, 2019, 2:10:09 PM4/10/19
to jenkinsc...@googlegroups.com
Shevek . commented on Bug JENKINS-56243

The active-directory thing is a total red herring. We are affected by this, and we do not and never have used AD. The downgrade to 2.150.1 does work.

gianvittorio.mauri@gmail.com (JIRA)

unread,
Apr 12, 2019, 10:36:02 AM4/12/19
to jenkinsc...@googlegroups.com

We had the same issue, the workaround https://jenkins.io/doc/upgrade-guide/2.150/ worked for us.

andrew@enospc.com (JIRA)

unread,
Apr 22, 2019, 3:10:03 PM4/22/19
to jenkinsc...@googlegroups.com

I upgraded our jenkins masters from 2.150.1 to 2.164.2 a few days ago.  The next day, we were intermittently seeing long page loads.  Today, I have consistently seen pages take 15-20 seconds to load.  This is from an instance of chrome 71 on Linux; this browser has been open for a very long time (weeks, at least).  I did a service jenkins restart on one of the masters, and page loads became fast again from that same client browser, even after logging back into jenkins.  If I open a new browser (I opened an instance of Firefox), it sees fast page loads on the server that was just restarted, and on the servers where the jenkins process was not restarted, even after logging into jenkins.  If I turn on the AD cache on the jenkins masters, page loads become fast on all machines, even from the long-running client browser.  I am unconvinced that AD server performance was the root cause, since page loads are fast from a newly-opened web browser to a jenkins server with the AD cache still turned off.

nelson.work@gmail.com (JIRA)

unread,
May 21, 2019, 1:49:03 PM5/21/19
to jenkinsc...@googlegroups.com

So far, the only thing that has worked for us on Jenkins 2.150+ is disabling the Remember Me option in Global Security.

henjovr@gmail.com (JIRA)

unread,
May 22, 2019, 4:48:06 AM5/22/19
to jenkinsc...@googlegroups.com

So, summary from the comments:

  • With AD post 2.150.1 become slow over time with Remember Me on
  • Without AD post 2.150.1 become slow over time with Remember Me on
  • Disabling Remember Me in Global Security fixes the problem, no slowdown after several hours
  • _SECURITY-901 / CVE-2019-1003004 ++_is de cause of the issue

 

@Jenkins developers:

Please look into the problem! Since people reported the same problem WITHOUT Active Directoy, please do not blame AD or directory services for the issue.

Disabling Remember Me is a good workaround, but nothing more than a workaround!

 

henjovr@gmail.com (JIRA)

unread,
May 22, 2019, 4:49:08 AM5/22/19
to jenkinsc...@googlegroups.com
Henjo van Rees edited a comment on Bug JENKINS-56243
So, summary from the comments:
* *With AD* post 2.150.1 become slow over time with Remember Me on
* *Without* *AD* post 2.150.1 become slow over time with Remember Me on
* Disabling *Remember Me* in Global Security fixes the problem, no slowdown after several hours
* _ *SECURITY-901 / CVE-2019-1003004*  ++_is is de cause of the issue

 

@Jenkins developers:

*Please look into the problem!* Since people reported the same problem WITHOUT Active Directoy, please do not blame AD or directory services for the issue.


Disabling Remember Me is a good workaround, but nothing more than a workaround!

 

jladan@fortinet.com (JIRA)

unread,
May 22, 2019, 1:29:04 PM5/22/19
to jenkinsc...@googlegroups.com
James Ladan edited a comment on Bug JENKINS-56243
[~henjovr] - did you try enabling the cache in Jenkins' LDAP plugin config? That helped for us. See Stuart Rowe's comment, above: [#comment-363002 ] .

jladan@fortinet.com (JIRA)

unread,
May 22, 2019, 1:29:05 PM5/22/19
to jenkinsc...@googlegroups.com

Henjo van Rees - did you try enabling the cache in Jenkins' LDAP plugin config? That helped for us. See Stuart Rowe's comment, above: [#comment-363002.

jladan@fortinet.com (JIRA)

unread,
May 22, 2019, 1:31:03 PM5/22/19
to jenkinsc...@googlegroups.com
James Ladan edited a comment on Bug JENKINS-56243
[~henjovr] - did you try enabling the cache in Jenkins' LDAP plugin config? That helped for us. See Stuart Rowe's comment, above: [ comment-363002|https://issues.jenkins-ci.org/browse/JENKINS-56243?focusedCommentId=363002&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel #comment-363002] .

henjovr@gmail.com (JIRA)

unread,
May 23, 2019, 9:22:08 AM5/23/19
to jenkinsc...@googlegroups.com

The LDAP was already enabled before the upgrade, still enabled after the upgrade from 2.150.1.

So no, that did not fix it unfortunately.

sverre.moe@gmail.com (JIRA)

unread,
May 31, 2019, 7:33:03 AM5/31/19
to jenkinsc...@googlegroups.com

We have tried several workarounds. None of them works.

  • Disable security
  • LDAP (Logged-in users can do anything)
  • LDAP (Anyone can do anything)
  • Disable remember me
  • Downgrade to Jenkins LTS 2.150.1

Our users still experience that Jenkins UI is slow and loading takes time, sometimes hang.

bshahadat@gmail.com (JIRA)

unread,
Jun 14, 2019, 6:32:02 PM6/14/19
to jenkinsc...@googlegroups.com

Is there any update on this? We are using 2.174, and it is painfully slow. Clearing cookies can help for short span of time.

gwittel@proofpoint.com (JIRA)

unread,
Jun 14, 2019, 6:49:03 PM6/14/19
to jenkinsc...@googlegroups.com

Looking at the code added by SECURITY-901, the code for UserSeedProperty concerns me:

 https://github.com/jenkinsci/jenkins/commit/8c490d14c4ffe6162f6e97d25a66612330fe2ace#diff-3ae1a1f58f660098ff61a9afeb8d3a10

The RNG call RANDOM.generateSeed is effectively single threaded based on how SecureRandom works.  I'm not a Jenkins expert, but it looks like the HttpSessionContextIntegrationFilter2 or AuthenticationProcessingFilter2 can cause creation of a User object with properties (via AllUsers). The UserSeedProperty instance then gets generated via the User properties constructor (causing a new RNG call). If that gets called a lot, that can be a severe bottleneck.

A test may be to disable the user seed property (note this exposes the issue that SECURITY-901 tries to fix) per this link

Set jenkins.security.seed.UserSeedProperty.disableUserSeed to true

gwittel@proofpoint.com (JIRA)

unread,
Jun 14, 2019, 7:00:06 PM6/14/19
to jenkinsc...@googlegroups.com
Greg Wittel edited a comment on Bug JENKINS-56243
Looking at the code added by SECURITY-901, the code for UserSeedProperty concerns me:

 [https://github.com/jenkinsci/jenkins/commit/8c490d14c4ffe6162f6e97d25a66612330fe2ace#diff-3ae1a1f58f660098ff61a9afeb8d3a10]

The RNG call {{RANDOM.generateSeed}} is effectively single threaded based on how SecureRandom works.   I'm not a Jenkins expert, but it looks like the {{HttpSessionContextIntegrationFilter2}} or {{AuthenticationProcessingFilter2}} can cause creation of a {{User}} object with properties (via AllUsers). The {{UserSeedProperty}} instance then gets generated via the {{User}} properties constructor (causing a new RNG call).  If that gets called a lot, that can be a severe bottleneck.
Similarly {{HudsonPrivateSecurityRealm}} also triggers the seed re-generation.

 A test may be to disable the user seed property (note this exposes the issue that SECURITY-901 tries to fix) per [this link|https://wiki.jenkins.io/display/JENKINS/Plugins+affected+by+the+SECURITY-901+fix]

Set {{jenkins.security.seed.UserSeedProperty.disableUserSeed}} to true

mpandey@elementcorp.com (JIRA)

unread,
Jun 18, 2019, 12:35:03 PM6/18/19
to jenkinsc...@googlegroups.com

Hi,

I am using 2.176.1. And it is painfully slow. I changed the security settings to 'Jenkin's Own User Database'. Speed was good. Due to abrupt reboot of Jenkins yesterday, its not allowing any kind of authentication now. But can someone tell me the exact version which do not have GUI slowness issues please. I am trying to install Jenkins from fresh on a new box so I would rather install a version which has no known issues. 

mpandey@elementcorp.com (JIRA)

unread,
Jun 18, 2019, 1:08:04 PM6/18/19
to jenkinsc...@googlegroups.com
Meghna Pandey updated an issue
 
Jenkins / Bug JENKINS-56243
Change By: Meghna Pandey
Priority: Major Critical

mpandey@elementcorp.com (JIRA)

unread,
Jun 18, 2019, 6:32:06 PM6/18/19
to jenkinsc...@googlegroups.com
Meghna Pandey edited a comment on Bug JENKINS-56243
 
Re: Jenkins GUI is slow -removing cookie fixes it (temporarily)
* This issue is resolved after upgrading the Jenkins from 2.176 to 2.179. Jenkins is working perfectly fine now. *  

mpandey@elementcorp.com (JIRA)

unread,
Jun 18, 2019, 6:32:06 PM6/18/19
to jenkinsc...@googlegroups.com

This issue is resolved after upgrading the Jenkins from 2.176 to 2.179. Jenkins is working perfectly fine now. 
Add Comment Add Comment
 

sverre.moe@gmail.com (JIRA)

unread,
Jun 20, 2019, 4:33:03 PM6/20/19
to jenkinsc...@googlegroups.com

I have been trying Jenkins 2.181 on our Jenkins Test server. Really fast loading of UI, not seeing any slowness.
Though the Jenkins Test server does not have as much build history, the real test would be to upgrade our Jenkins Production server.

sverre.moe@gmail.com (JIRA)

unread,
Jun 20, 2019, 4:50:02 PM6/20/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Bug JENKINS-56243
I have been trying Jenkins 2.181 on our Jenkins Test server. Really fast loading of UI, not seeing any slowness.
Though the Jenkins Test server does not have as much build history, the real test would be to upgrade our Jenkins Production server.


Edit: Until I went to Manage Jenkins, then loading the icons just hanging page loading.

sverre.moe@gmail.com (JIRA)

unread,
Jun 21, 2019, 3:31:04 AM6/21/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Bug JENKINS-56243
I have been trying Jenkins 2.181 on our Jenkins Test server. Really fast loading of UI, not seeing any slowness.
Though the Jenkins Test server does not have as much build history, the real test would be to upgrade our Jenkins Production server.

Edit: Until I went to Manage Jenkins, then loading the icons just hanging page loading.


Edit: Opening a multibranch project branch, the page seems to have loaded completely, but loading is still ongoing for some more ~30 seconds. Looked at what was loading and seems to be prototype.js. There was an fix for this in JENKINS-49319, but it was reverted.
Firefox is having much more problems than browsers with the Chromium engine. So it seems to be a javascript problem that last loading.

sverre.moe@gmail.com (JIRA)

unread,
Jun 21, 2019, 3:33:04 AM6/21/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Bug JENKINS-56243
I have been trying Jenkins 2.181 on our Jenkins Test server. Really fast loading of UI, not seeing any slowness.
Though the Jenkins Test server does not have as much build history, the real test would be to upgrade our Jenkins Production server.

Edit: Until I went to Manage Jenkins, then loading the icons just hanging page loading.

Edit: Opening a multibranch project branch, the page seems to have loaded completely, but loading is still ongoing for some more ~30 seconds. Looked at what was loading and seems to be prototype.js. There was an fix for this in JENKINS-49319, but it was reverted.
Firefox is having much more problems than browsers with the Chromium engine. So it seems to be a javascript problem that last loading.


Edit:
All good things must come to an end: Now I am getting the same problem in Chrome. Several resources are not loading.

sverre.moe@gmail.com (JIRA)

unread,
Jun 21, 2019, 3:36:03 AM6/21/19
to jenkinsc...@googlegroups.com

What I find very odd. If I log in to Jenkins (using LDAP) all loading problems goes away. Why should logging in solve the loading slowness?

sverre.moe@gmail.com (JIRA)

unread,
Jun 21, 2019, 4:12:02 AM6/21/19
to jenkinsc...@googlegroups.com

The last remaining loading on Multibranch Pipeline branch project. Developer Tools in my browser showed me what was remaining trying to load.
ajax 200 xhr prototype.js:1585 310 B 15 ms
runs?since=%232&fullStages=true&_=1561103938220 200 xhr jquery2.js:998 48.8 KB 18 m
These two repeated several times for about 30+ seconds until the page was actually finished loading.

sverre.moe@gmail.com (JIRA)

unread,
Jun 21, 2019, 4:12:07 AM6/21/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Bug JENKINS-56243
The last remaining loading on Multibranch Pipeline branch project. Developer Tools in my browser showed me what was remaining trying to load.
{code}
ajax 200 xhr prototype.js:1585 310 B 15 ms
runs?since=%232&fullStages=true&_=1561103938220 200 xhr jquery2.js:998 48.8 KB 18 m
{code}
These two repeated several times for about 30+ seconds until the page was actually finished loading.

sverre.moe@gmail.com (JIRA)

unread,
Jun 21, 2019, 4:13:08 AM6/21/19
to jenkinsc...@googlegroups.com
Sverre Moe edited a comment on Bug JENKINS-56243
The last remaining loading on Multibranch Pipeline branch project. Developer Tools in my browser showed me what was remaining trying to load.
{code}
ajax 200 xhr prototype.js:1585 310 B 15 ms
runs?since=%232&fullStages=true&_=1561103938220 200 xhr jquery2.js:998 48.8 KB 18 m ms
{code}
These two repeated several times for about 30+ seconds until the page was actually finished loading.

sverre.moe@gmail.com (JIRA)

unread,
Jun 21, 2019, 4:40:05 AM6/21/19
to jenkinsc...@googlegroups.com
Sverre Moe updated an issue
 
Change By: Sverre Moe
Labels: user-experience

boards@gmail.com (JIRA)

unread,
Jun 26, 2019, 3:44:02 PM6/26/19
to jenkinsc...@googlegroups.com
Matt Sicker commented on Bug JENKINS-56243
 
Re: Jenkins GUI is slow -removing cookie fixes it (temporarily)

If there's contention in the SecureRandom instance, that could be causing issues. Let me see if I can reproduce any slowdowns with a benchmark test.

Thanks for verifying that this isn't AD-specific at least. I might be able to help figure this out.

boards@gmail.com (JIRA)

unread,
Jun 28, 2019, 12:42:03 PM6/28/19
to jenkinsc...@googlegroups.com

So far from my testing, I'm not finding any slow code in seed renewal. Some basic JMH tests in this branch: https://github.com/jenkinsci/jenkins/compare/master...jvz:user-seed-perf-JENKINS-56243?expand=1

Right now, my hypothesis is that if a SecurityRealm is having any performance issues, multiple requests to load the same user's details could be piling up due to the remember me cookie validation check. The same happens in the session cookie itself. Basically, the reason why it was performing better before was because it wasn't validating authentication properly in the first place.

I'm working on some basic load tests to compare 2.150.1 and 2.150.2 to see if I can reproduce this idea. Based on the comments so far, it sounds like this should even be potentially reproducible using just the built-in user database. The JMH tests above only use an in-memory user database, so introducing lag in the calls to loadUserDetails() could be an interesting way to potentially test this as well.

boards@gmail.com (JIRA)

unread,
Jun 28, 2019, 1:18:03 PM6/28/19
to jenkinsc...@googlegroups.com

I discovered that the remember me service bypasses the user details cache entirely. I've made a draft PR with this fixed: https://github.com/jenkinsci/jenkins/pull/4093

sverre.moe@gmail.com (JIRA)

unread,
Jun 29, 2019, 7:10:03 AM6/29/19
to jenkinsc...@googlegroups.com

We have disabled the Remember Me option and still experience the slowness. The slowness only happens when users are not logged in.

boards@gmail.com (JIRA)

unread,
Jul 1, 2019, 2:34:12 PM7/1/19
to jenkinsc...@googlegroups.com

I chatted with Wadeck Follonier earlier today, and we've found that the most likely culprit is that TokenBasedRememberMeServices2 does not cache the user seed property in their session. I'll submit a PR later to address this.

boards@gmail.com (JIRA)

unread,
Jul 1, 2019, 2:34:17 PM7/1/19
to jenkinsc...@googlegroups.com
Matt Sicker assigned an issue to Matt Sicker
 
Change By: Matt Sicker
Assignee: Matt Sicker

boards@gmail.com (JIRA)

unread,
Jul 1, 2019, 2:34:17 PM7/1/19
to jenkinsc...@googlegroups.com
Matt Sicker started work on Bug JENKINS-56243
 
Change By: Matt Sicker
Status: Open In Progress

boards@gmail.com (JIRA)

unread,
Jul 1, 2019, 5:30:10 PM7/1/19
to jenkinsc...@googlegroups.com
Change By: Matt Sicker
Status: In Progress Review

boards@gmail.com (JIRA)

unread,
Jul 1, 2019, 5:30:10 PM7/1/19
to jenkinsc...@googlegroups.com

boards@gmail.com (JIRA)

unread,
Jul 2, 2019, 10:46:09 AM7/2/19
to jenkinsc...@googlegroups.com
Change By: Matt Sicker
Status: In Review Progress

boards@gmail.com (JIRA)

unread,
Jul 2, 2019, 10:46:17 AM7/2/19
to jenkinsc...@googlegroups.com
 
Re: Jenkins GUI is slow -removing cookie fixes it (temporarily)

Need to add another test, but this looks to be about fixed.

boards@gmail.com (JIRA)

unread,
Jul 2, 2019, 12:00:14 PM7/2/19
to jenkinsc...@googlegroups.com
Change By: Matt Sicker
Status: In Progress Review

sverre.moe@gmail.com (JIRA)

unread,
Jul 2, 2019, 1:37:03 PM7/2/19
to jenkinsc...@googlegroups.com
Sverre Moe commented on Bug JENKINS-56243
 
Re: Jenkins GUI is slow -removing cookie fixes it (temporarily)

Looking forward to testing it out. Our developers are getting frustrated.

boards@gmail.com (JIRA)

unread,
Jul 2, 2019, 3:58:03 PM7/2/19
to jenkinsc...@googlegroups.com

Incremental release available: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/cli/2.184-rc28433.92d6063c40c3/

Still waiting for reviews before someone can merge it for the next weekly.

sverre.moe@gmail.com (JIRA)

unread,
Jul 2, 2019, 4:33:02 PM7/2/19
to jenkinsc...@googlegroups.com

I can test the incremental release on our Test Jenkins instance. I dare not install it in production.

wfollonier@cloudbees.com (JIRA)

unread,
Jul 3, 2019, 4:18:05 AM7/3/19
to jenkinsc...@googlegroups.com

Sverre Moe as you said before, if the "Disable remember me" workaround was not working for you, do not expect this change to work either. It's "just" the correction of the root cause of this issue. From my PoV, wiht all the information you gave, you have another (unknown?) problem that is different from this one.

sverre.moe@gmail.com (JIRA)

unread,
Jul 3, 2019, 5:42:08 AM7/3/19
to jenkinsc...@googlegroups.com

There is another issue I have been tracking I think can be related to our problem of slowness. JENKINS-49319

dbeck@cloudbees.com (JIRA)

unread,
Jul 4, 2019, 7:14:09 AM7/4/19
to jenkinsc...@googlegroups.com
Daniel Beck updated an issue
 
Change By: Daniel Beck
Labels: lts-candidate user-experience

o.v.nenashev@gmail.com (JIRA)

unread,
Jul 8, 2019, 8:52:06 AM7/8/19
to jenkinsc...@googlegroups.com
Oleg Nenashev updated Bug JENKINS-56243
 

The fix was released in Jenkins 2.184
Change By: Oleg Nenashev
Status: In Review Resolved
Resolution: Fixed
Released As: Jenkins 2.184

dar.amit@gmail.com (JIRA)

unread,
Jul 8, 2019, 1:22:04 PM7/8/19
to jenkinsc...@googlegroups.com
Amit Dar commented on Bug JENKINS-56243
 
Re: Jenkins GUI is slow -removing cookie fixes it (temporarily)

will this be included in the next LTS release?

dbeck@cloudbees.com (JIRA)

unread,
Jul 8, 2019, 2:11:02 PM7/8/19
to jenkinsc...@googlegroups.com

Next baseline for sure. 2.176.2 certainly not. 2.176.3 possibly.

ogondza@gmail.com (JIRA)

unread,
Aug 1, 2019, 7:01:03 AM8/1/19
to jenkinsc...@googlegroups.com
Oliver Gondža updated an issue
 
Change By: Oliver Gondža
Labels: lts 2.176.3 - candidate fixed user-experience
Reply all
Reply to author
Forward
0 new messages