[JIRA] (JENKINS-58879) EC2 plugin queue locks have resurfaced

[joyce.z.yee@gmail.com (JIRA)]

Joyce Yee created an issue

Jenkins / JENKINS-58879 EC2 plugin queue locks have resurfaced Issue Type: Bug Assignee: FABRIZIO MANFREDI Components: ec2-plugin Created: 2019-08-09 18:57 Environment: Jenkins 2.186 (from the jenkins/jenkins:2.186 docker image) Amazon EC2 plugin (1.44.1) Priority: Blocker Reporter: Joyce Yee After upgrading to 1.44.1, we are observing the return of long-standing queue locking as described in this [PR|https://github.com/jenkinsci/ec2-plugin/pull/346]. It looks like the issue is reintroduced within the scope of this commit. I think it's because the `clock` is now instantiated within the queue lock, and therefore halts the ability for the `EC2RetentionStrategy` to skip the `NodeProvisioner.update()` cycle triggered with `internalCheck`, if the last check happened < 1 minute ago.  Downgrading back down to version `1.42` where we patch in the code introduced in the original commit (https://github.com/jenkinsci/ec2-plugin/pull/346) fixes the issue and we no longer observe these frequently occurring queue locks.   Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[joyce.z.yee@gmail.com (JIRA)]

Joyce Yee updated an issue

Jenkins / JENKINS-58879 EC2 plugin queue locks have resurfaced

Change By: Joyce Yee After upgrading to 1.44.1, we are observing the return of long-standing queue locking as described in this [ PR | ( [https://github.com/jenkinsci/ec2-plugin/pull/346] ] ) . It looks like the issue is reintroduced within the scope of this [commit| [https://github.com/jenkinsci/ec2-plugin/commit/4973e9f4371b25abdcf78605380e12e57be336aa #diff-e577be8c21e85e6a1f2ff095f583f5d2R237]].

I think it's because the `clock` is now instantiated within the queue lock, and therefore halts the ability for the `EC2RetentionStrategy` to skip the `NodeProvisioner.update()` cycle triggered with `internalCheck`, if the last check happened < 1 minute ago.

Downgrading back down to version `1.42` where we patch in the code introduced in the original commit ([https://github.com/jenkinsci/ec2-plugin/pull/346]) fixes the issue and we no longer observe these frequently occurring queue locks.

Add Comment

[joyce.z.yee@gmail.com (JIRA)]

Joyce Yee updated an issue

Jenkins / JENKINS-58879 EC2 plugin queue locks have resurfaced Change By: Joyce Yee

After upgrading to 1.44.1, we are observing the return of long-standing queue locking as described in this PR ([https://github.com/jenkinsci/ec2-plugin/pull/346]). It looks like the issue is reintroduced within the scope of this [commit|#diff-e577be8c21e85e6a1f2ff095f583f5d2R237] ] .

I think it's because the `clock` is now instantiated within the queue lock, and therefore halts the ability for the `EC2RetentionStrategy` to skip the `NodeProvisioner.update()` cycle triggered with `internalCheck`, if the last check happened < 1 minute ago.  Downgrading back down to version `1.42` where we patch in the code introduced in the original commit ([https://github.com/jenkinsci/ec2-plugin/pull/346]) fixes the issue and we no longer observe these frequently occurring queue locks.

Add Comment

[joyce.z.yee@gmail.com (JIRA)]

Joyce Yee updated an issue

Jenkins / JENKINS-58879 EC2 plugin queue locks have resurfaced Change By: Joyce Yee

After upgrading to 1.44.1, we are observing the return of long-standing queue locking as described in this PR ([https://github.com/jenkinsci/ec2-plugin/pull/346]).

It looks like the issue is reintroduced within the scope of this commit ( [ https://github.com/jenkinsci/ec2-plugin/ commit | /4973e9f4371b25abdcf78605380e12e57be336aa #diff-e577be8c21e85e6a1f2ff095f583f5d2R237] ) .

I think it's because the `clock` is now instantiated within the queue lock, and therefore halts the ability for the `EC2RetentionStrategy` to skip the `NodeProvisioner.update()` cycle triggered with `internalCheck`, if the last check happened < 1 minute ago.  Downgrading back down to version `1.42` where we patch in the code introduced in the original commit ([https://github.com/jenkinsci/ec2-plugin/pull/346]) fixes the issue and we no longer observe these frequently occurring queue locks.

Add Comment

[joyce.z.yee@gmail.com (JIRA)]

Joyce Yee updated an issue

Jenkins / JENKINS-58879 EC2 plugin queue locks have resurfaced Change By: Joyce Yee

After upgrading to 1.44.1, we are observing the return of long-standing queue locking as described in this PR ([https://github.com/jenkinsci/ec2-plugin/pull/346]). It looks like the issue is reintroduced within the scope of this commit ( [ https://github.com/jenkinsci/ec2-plugin/ commit /4973e9f4371b25abdcf78605380e12e57be336aa | #diff-e577be8c21e85e6a1f2ff095f583f5d2R237] )

. I think it's because since the `clock` is now only instantiated within  the queue lock `readResolve` , and therefore halts not in the ability constructor of `EC2RetentionStrategy`, the problem is re-introduced for workers that are being newly created (not for workers that were persisted from XML). The clock may not exist when the ` EC2RetentionStrategy check ` method is invoked, which prevents the strategy's ability to to skip the `NodeProvisioner.update()` cycle triggered with `internalCheck`, if the last check happened < 1 minute ago.   The solution is probably to instantiate the clock in both the constructor and `readResolve`.

Downgrading back down to version `1.42` where we patch in the code introduced in the original commit ([https://github.com/jenkinsci/ec2-plugin/pull/346]) fixes the issue and we no longer observe these frequently occurring queue locks.

Add Comment

[fabrizio.manfredi@gmail.com (JIRA)]

FABRIZIO MANFREDI started work on JENKINS-58879

Change By: FABRIZIO MANFREDI Status: Open In Progress

Add Comment

[fabrizio.manfredi@gmail.com (JIRA)]

FABRIZIO MANFREDI commented on JENKINS-58879

Re: EC2 plugin queue locks have resurfaced Some  improvement has been released in the 1.45 to reduce the query and the locking of the queue, can you test ?

Add Comment

[fabrizio.manfredi@gmail.com (JIRA)]

FABRIZIO MANFREDI updated JENKINS-58879

Jenkins / JENKINS-58879

EC2 plugin queue locks have resurfaced

Change By: FABRIZIO MANFREDI Status: In Progress Review

Add Comment

[mrjunzehe@gmail.com (JIRA)]

Junze He commented on JENKINS-58879

Re: EC2 plugin queue locks have resurfaced We tested it with 1.45 and didn't notice any locks. Thanks!

Add Comment