[JIRA] (JENKINS-61211) QueueItemAuthenticator causes Build Queue contention

16 views
Skip to first unread message

aburdajewicz@cloudbees.com (JIRA)

unread,
Feb 24, 2020, 9:17:03 PM2/24/20
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ created an issue
 
Jenkins / Improvement JENKINS-61211
QueueItemAuthenticator causes Build Queue contention
Issue Type: Improvement Improvement
Assignee: Unassigned
Components: core
Created: 2020-02-25 02:16
Environment: core:2.222
Labels: performance
Priority: Major Major
Reporter: Allan BURDAJEWICZ

When using the Authorize Project plugin to configure authorization for build execution, this has severe consequences on the overall performances of a Jenkins master because it request information repeatedly "trying to get user details for impersonation" from the Security Realm. This can happen during in a thread that hold the Queue.Lock

An instance will periodically become unavailable or take a very long time in builds especially when the build queue becomes large or when a lot of builds are happening all at the same time.

If start a fresh instance with 2 master executors, configure a "Project default Build Authorization" under *Manage Jenkins > configure Global Security*, then create a single pipeline and execute it, you will see through FINE logging of hudson.model.User that the {{User#getUserDetailsForImpersonation} is requested 8 times:

eb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
      1. Workaround

Caching can be configured at the Security Realm level depending on the solution (for example for LDAP / Active directory).

      1. Improvement Proposal

I do however think that something could be improved and that since impersonation is happening so frequently, maybe the UserDetailsCache can be used to improve performance.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

aburdajewicz@cloudbees.com (JIRA)

unread,
Feb 24, 2020, 10:06:03 PM2/24/20
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ started work on Improvement JENKINS-61211
 
Change By: Allan BURDAJEWICZ
Status: Open In Progress

aburdajewicz@cloudbees.com (JIRA)

unread,
Feb 24, 2020, 10:06:03 PM2/24/20
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ assigned an issue to Allan BURDAJEWICZ
Change By: Allan BURDAJEWICZ
Assignee: Allan BURDAJEWICZ

aburdajewicz@cloudbees.com (JIRA)

unread,
Feb 24, 2020, 10:07:02 PM2/24/20
to jenkinsc...@googlegroups.com
Change By: Allan BURDAJEWICZ
Status: In Progress Review

aburdajewicz@cloudbees.com (JIRA)

unread,
Feb 24, 2020, 10:07:02 PM2/24/20
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ updated an issue
When using the Authorize Project plugin to configure authorization for build execution, this has severe consequences on the overall performances of a Jenkins master because it request information repeatedly "trying to get user details for impersonation" from the Security Realm. This can happen during in a thread that hold the {{Queue.Lock}}

An instance will periodically become unavailable or take a very long time in builds especially when the build queue becomes large or when a lot of builds are happening all at the same time.

If start a fresh instance with 2 master executors, configure a "Project default Build Authorization" under *Manage Jenkins > configure Global Security*, then create a single pipeline and execute it, you will see through FINE logging of {{hudson.model.User}} that the {{User#getUserDetailsForImpersonation} is requested 8 times:

{code}
 eb Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation

Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
{code}


### Workaround


Caching can be configured at the Security Realm level depending on the solution (for example for LDAP / Active directory).

### Improvement Proposal

I do however think that something could be improved and that since impersonation is happening so frequently, maybe the [UserDetailsCache|https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/security/UserDetailsCache.java] can be used to improve performance.

aburdajewicz@cloudbees.com (JIRA)

unread,
Feb 24, 2020, 10:07:03 PM2/24/20
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ updated an issue
When using the Authorize Project plugin to configure authorization for build execution, this has severe consequences on the overall performances of a Jenkins master because it request information repeatedly "trying to get user details for impersonation" from the Security Realm. This can happen during in a thread that hold the {{Queue.Lock}}

An instance will periodically become unavailable or take a very long time in builds especially when the build queue becomes large or when a lot of builds are happening all at the same time.

If start a fresh instance with 2 master executors, configure a "Project default Build Authorization" under *Manage Jenkins > configure Global Security*, then create a single pipeline and execute it, you will see through FINE logging of {{hudson.model.User}} that the {{User#getUserDetailsForImpersonation} is requested 8 times:

{code}
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
{code}


### h3. Workaround


Caching can be configured at the Security Realm level depending on the solution (for example for LDAP / Active directory).

### h3. Improvement Proposal


I do however think that something could be improved and that since impersonation is happening so frequently, maybe the [UserDetailsCache|https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/security/UserDetailsCache.java] can be used to improve performance.

aburdajewicz@cloudbees.com (JIRA)

unread,
Feb 24, 2020, 10:07:03 PM2/24/20
to jenkinsc...@googlegroups.com
Allan BURDAJEWICZ updated an issue
When using the Authorize Project plugin to configure authorization for build execution, this has severe consequences on the overall performances of a Jenkins master because it request information repeatedly "trying to get user details for impersonation" from the Security Realm. This can happen during in a thread that hold the {{Queue.Lock}}

An instance will periodically become unavailable or take a very long time in builds especially when the build queue becomes large or when a lot of builds are happening all at the same time.

If start a fresh instance with 2 master executors, configure a "Project default Build Authorization" under * * Manage Jenkins > configure Global Security* * , then create a single pipeline and execute it, you will see through FINE logging of {{hudson.model.User}} that the {{User#getUserDetailsForImpersonation} is requested 8 times:

{code}

eb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
Feb 24, 2020 2:42:42 PM FINE hudson.model.User getUserDetailsForImpersonation
Impersonation of the user admin was a success
{code}


### Workaround


Caching can be configured at the Security Realm level depending on the solution (for example for LDAP / Active directory).

### Improvement Proposal


I do however think that something could be improved and that since impersonation is happening so frequently, maybe the [UserDetailsCache|https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/security/UserDetailsCache.java] can be used to improve performance.
Reply all
Reply to author
Forward
0 new messages