[JIRA] (JENKINS-60536) Jenkins job hangs in Jira call after git rev-list command

0 views
Skip to first unread message

mark.earl.waite@gmail.com (JIRA)

unread,
Jan 7, 2020, 8:46:02 AM1/7/20
to jenkinsc...@googlegroups.com
Mark Waite updated an issue
 
Jenkins / Bug JENKINS-60536
Jenkins job hangs in Jira call after git rev-list command
Change By: Mark Waite
Summary: Jenkins job hangs in Jira call after git rev-list command
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

d.france@crossflight.com (JIRA)

unread,
Jan 7, 2020, 9:05:05 AM1/7/20
to jenkinsc...@googlegroups.com
David France edited a comment on Bug JENKINS-60536
 
Re: Jenkins job hangs in Jira call after git rev-list command
We've being seeing much the same problem stack that Kieran reported, trying to progress issues by a workflow action in Jira.  We first became aware of the problem on 20/12/2019:
{noformat}
"Executor #1 for master : executing deployToUAT #1619" #647 daemon prio=5 os_prio=0 cpu=436.80ms elapsed=236.46s tid=0x000000006
2b92000 nid=0x2590 waiting on condition  [0x000000006644d000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java...@11.0.5/Native Method)
        - parking to wait for  <0x0000000083d46f58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(java...@11.0.5/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java...@11.0.5/AbstractQueuedSynchronizer.j
ava:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java...@11.0.5/AbstractQueuedSynchronizer.java:917)        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java...@11.0.5/AbstractQueuedSynchronizer.java:1240)
        at java.util.concurrent.locks.ReentrantLock.lock(java...@11.0.5/ReentrantLock.java:267)
        at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.j
ava:295)
        at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
        at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseH
ttpPromiseAsyncClient.java:36)
        at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417)
        at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364)
        at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297)
        at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianH
ttpClientDecorator.java:83)
        at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253)
        at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.j
ava:69)
        at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient.
java:109)
        at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94
)
        at hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:196)
        at hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136)
        at hudson.plugins.jira.JiraSite.progressMatchingIssues(JiraSite.java:1043)
        at hudson.plugins.jira.JiraIssueUpdateBuilder.perform(JiraIssueUpdateBuilder.java:105)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
        at hudson.model.Build$BuildExecution.build(Build.java:206)
        at hudson.model.Build$BuildExecution.doRun(Build.java:163)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
        at hudson.model.Run.execute(Run.java:1853)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:427) {noformat}
 

 The problem looks to be a problem with an earlier connection in the connection pool in the _{{HttpAsyncClient}}_. The earlier open connection is failing to be closed before processing the new request.

We did a couple of experiments in the jira-plugin source, and managed to find a way get the connection to complete. (This is hardly a submission quality change, but perhaps it offers a clue or two).

The first hope was that changing the _{{HttpClientOptions.maxConnectionsPerHost}}_ value would stop the connection pool holding the connections, but forcing this (in the code) to either 1 or 2000 had no effect.

Instead, as a rough workaround we tried bypassing the connection pooling, by creating a new _{{HttpAsyncClientBuilder}}_ against a new _{{PoolingNHttpClientConnectionManager}}_ for each request. This is will be leaky - there is nothing to close the new connection pools. We don't know how soon to expect a problem though, Jenkins has stayed up so far.

The change is [here|https://github.com/xft-devs/jira-plugin/commit/ab0613b85e721364b7bd7d9fded68fae7a2a48e2]. (Essentially it extracts a method in the constructor, to create the _{{HttpAsyncClientBuilder}}_, then also calls it from _{{doExecute()}}_.)

 As mentioned, this was just a minimal/dirty change to get us up and running for now. For example even an _{{HttpAsyncClient}}_ without connection pooling would be better and more stable.

For reference this is working against JIRA Cloud. Not sure why the connections started failing to close, but this was enough to force our builds through for the time being.

d.france@crossflight.com (JIRA)

unread,
Jan 7, 2020, 9:06:06 AM1/7/20
to jenkinsc...@googlegroups.com

d.france@crossflight.com (JIRA)

unread,
Jan 7, 2020, 9:07:02 AM1/7/20
to jenkinsc...@googlegroups.com
As mentioned, this was just a minimal/dirty change to get us up and running for now. For example even an _{{HttpAsyncClient}}_ without connection pooling would be better cleaner, and more stable.


For reference this is working against JIRA Cloud.

d.france@crossflight.com (JIRA)

unread,
Jan 7, 2020, 9:11:04 AM1/7/20
to jenkinsc...@googlegroups.com
As mentioned, this was just a minimal/dirty change to get us up and running for now. For example even an _{{HttpAsyncClient}}_ without connection pooling would be cleaner, and more stable.

For reference this is working against JIRA Cloud.   We're not sure why the connections started failing to close, but this was enough to force our builds through for the time being.

d.france@crossflight.com (JIRA)

unread,
Jan 7, 2020, 10:02:12 AM1/7/20
to jenkinsc...@googlegroups.com
David France edited a comment on Bug JENKINS-60536
We've being seeing much the same problem stack that Kieran reported, trying to progress issues by a workflow action in Jira.  We first became aware of the problem on 20/12/2019 .

The locking thread :
{noformat}
" Handling GET /view/Pipelines/job/.../457/ from a.b.c.d : qtp64133603-15 AbstractBuild/index.jelly SubversionChangeLogS
et/digest.jelly" #15 prio=5 os_prio=0 cpu=8985.66ms elapsed=5244.95s tid=0x0000000064d07000 nid=0x1fdc waiting for monitor entry  [0
x00000000665b7000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605)
        - waiting to lock <0x000000008613f660> (a org.apache.http.nio.reactor.ssl.SSLIOSession)
        at org.apache.http.impl.nio.NHttpConnectionBase.close(NHttpConnectionBase.java:511)
        at org.apache.http.impl.nio.conn.CPoolEntry.closeConnection(CPoolEntry.java:75)
        at org.apache.http.impl.nio.conn.CPoolEntry.close(CPoolEntry.java:101)
        at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:423)
        at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)

        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.j
ava:295)
        at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
        at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseH
ttpPromiseAsyncClient.java:36)
        at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417)
        at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364)
        at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297)
        at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianH
ttpClientDecorator.java:83)
        at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253)
        at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.j
ava:69)
        at com.atlassian.jira.rest.client.internal.async.AsynchronousIssueRestClient.getIssue(AsynchronousIssueRestClient.java:186)
        at com.atlassian.jira.rest.client.internal.async.AsynchronousIssueRestClient.getIssue(AsynchronousIssueRestClient.java:177)
        at hudson.plugins.jira.JiraRestService.getIssue(JiraRestService.java:154)
        at hudson.plugins.jira.JiraSession.getIssue(JiraSession.java:126)
        at hudson.plugins.jira.JiraSite.lambda$getIssue$0(JiraSite.java:878)
        at hudson.plugins.jira.JiraSite$$Lambda$303/0x000000002262d840.call(Unknown Source)
        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
        - locked <0x000000008644f218> (a com.google.common.cache.LocalCache$StrongAccessEntry)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
        at hudson.plugins.jira.JiraSite.getIssue(JiraSite.java:873)
        at hudson.plugins.jira.JiraChangeLogAnnotator.annotate(JiraChangeLogAnnotator.java:107)
        at hudson.scm.ChangeLogSet$Entry.getMsgAnnotated(ChangeLogSet.java:252)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java...@11.0.5/Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java...@11.0.5/NativeMethodAccessorImpl.java:62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java...@11.0.5/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java...@11.0.5/Method.java:566)
        at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
        at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
        at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
        at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
        at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
        at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
        at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
        at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
...
{noformat}
 
And a waiting thread:
{noformat}
" Executor #1 for master : executing deployToUAT #1619" #647 daemon prio=5 os_prio=0 cpu=436.80ms elapsed=236.46s tid=0x000000006
 The problem looks to be a problem with an earlier connection in the connection pool in the _{{HttpAsyncClient}}_. The earlier open connection is failing to be closed before processing the new request (in the first thread) .


We did a couple of experiments in the jira-plugin source, and managed to find a way get the connection to complete.

The first hope was that changing the _{{HttpClientOptions.maxConnectionsPerHost}}_ value would stop the connection pool holding the connections, but forcing this (in the code) to either 1 or 2000 had no effect.

Instead, as a rough workaround we tried bypassing the connection pooling, by creating a new _{{HttpAsyncClientBuilder}}_ against a new _{{PoolingNHttpClientConnectionManager}}_ for each request. This is will be leaky - there is nothing to close the new connection pools. We don't know how soon to expect a problem though, Jenkins has stayed up so far.

The change is [here|https://github.com/xft-devs/jira-plugin/commit/ab0613b85e721364b7bd7d9fded68fae7a2a48e2]. (Essentially it extracts a method in the constructor, to create the _{{HttpAsyncClientBuilder}}_, then also calls it from _{{doExecute()}}_.)

As mentioned, this was just a minimal/dirty change to get us up and running for now. For example even an _{{HttpAsyncClient}}_ without connection pooling would be cleaner, and more stable.

For reference this is working against JIRA Cloud.  We're not sure why the connections started failing to close, but this was enough to force our builds through for the time being.

d.france@crossflight.com (JIRA)

unread,
Jan 8, 2020, 6:51:07 AM1/8/20
to jenkinsc...@googlegroups.com
Instead, as a rough workaround we tried bypassing the connection pooling, by creating a new _{{HttpAsyncClientBuilder}}_ against a new _{{PoolingNHttpClientConnectionManager}}_ for each request. This The change is will be leaky - there is nothing to close the new connection pools. We don't know how soon to expect a problem though fairly basic , Jenkins but has stayed enabled Jenkins to stay up so far.

The change is on [ here this fork |https://github.com/xft-devs/jira-plugin /commit/ab0613b85e721364b7bd7d9fded68fae7a2a48e2 ]. (Essentially it extracts a method in the constructor, to create the _{{HttpAsyncClientBuilder}}_, then also calls it from _{{doExecute()}}_.)

As mentioned, this was just a minimal/dirty change to get us up and running for now - it just bypasses connection reuse at the point of the request . For example even an _{{HttpAsyncClient}}_ without connection pooling would be cleaner , and more stable .


For reference this is working against JIRA Cloud.  We're not sure why the connections started failing to close, but this was enough to force our builds through for the time being.

d.france@crossflight.com (JIRA)

unread,
Jan 8, 2020, 7:13:04 AM1/8/20
to jenkinsc...@googlegroups.com
The first hope was that changing the _{{HttpClientOptions.maxConnectionsPerHost}}_ value would stop the connection pool holding the connections, but forcing this (in the code) to either 1 or 2000 had no effect - it was still trying to close the previous (now available) connection .

Instead, as a rough workaround we tried bypassing the connection pooling, by creating a new _{{HttpAsyncClientBuilder}}_ against a new _{{PoolingNHttpClientConnectionManager}}_ for each request. The change is fairly basic, but has enabled Jenkins to stay up so far.

The change is on [this fork|https://github.com/xft-devs/jira-plugin]. (Essentially it extracts a method in the constructor, to create the _{{HttpAsyncClientBuilder}}_, then also calls it from _{{doExecute()}}_.)

As mentioned, this was just a minimal/dirty change to get us up and running for now - it just bypasses connection reuse at the point of the request. For example an _{{HttpAsyncClient}}_ without connection pooling would be cleaner.


For reference this is working against JIRA Cloud.  We're not sure why the connections started failing to close, but this was enough to force our builds through for the time being.

d.france@crossflight.com (JIRA)

unread,
Jan 9, 2020, 3:12:02 AM1/9/20
to jenkinsc...@googlegroups.com
 The problem looks to be a problem with an earlier connection in the connection pool in the _{{HttpAsyncClient}}_. The earlier open connection is failing to be closed before processing the new request ( in see the first thread stack above ).


We did a couple of experiments in the jira-plugin source, and managed to find a way get the connection to complete.

The first hope was that changing the _{{HttpClientOptions.maxConnectionsPerHost}}_ value would stop the connection pool holding the connections, but forcing this (in the code) to either 1 or 2000 had no effect - it was still trying to close the previous (now available) connection.

Instead, as a rough workaround we tried bypassing the connection pooling, by creating a new _{{HttpAsyncClientBuilder}}_ against a new _{{PoolingNHttpClientConnectionManager}}_ for each request. The change is fairly basic, but has enabled Jenkins to stay up so far.

The change is on [this fork|https://github.com/xft-devs/jira-plugin]. (Essentially it extracts a method in the constructor, to create the _{{HttpAsyncClientBuilder}}_, then also calls it from _{{doExecute()}}_.)

As mentioned, this was just a minimal/dirty change to get us up and running for now - it just bypasses connection reuse at the point of the request. For example an _{{HttpAsyncClient}}_ without connection pooling would be cleaner.

For reference this is working against JIRA Cloud.  We're not sure why the connections started failing to close, but this was enough to force our builds through for the time being.

naresh.rayapati@gmail.com (JIRA)

unread,
Jan 28, 2020, 10:10:05 PM1/28/20
to jenkinsc...@googlegroups.com
Naresh Rayapati updated an issue
 
Change By: Naresh Rayapati
Component/s: jira-steps-plugin

rodrigc@FreeBSD.org (JIRA)

unread,
Jan 30, 2020, 9:57:03 AM1/30/20
to jenkinsc...@googlegroups.com
Craig Rodrigues edited a comment on Bug JENKINS-60536
 
Re: Jenkins job hangs in Jira call after git rev-list command
[~dfrance]can you submit a PR of what you've got working to https://github.com/jenkinsci/jira-plugin ?

rodrigc@FreeBSD.org (JIRA)

unread,
Jan 30, 2020, 9:57:04 AM1/30/20
to jenkinsc...@googlegroups.com

David Francecan you submit a PR of what you've got working to https://github.com/jenkinsci/jira-plugin ?

kisipo6270@jetsmails.com (JIRA)

unread,
Feb 24, 2020, 7:08:03 PM2/24/20
to jenkinsc...@googlegroups.com

David France this is affecting our team as well – same stack trace as yours. Your fix did unclog our pipelines temporarily, and then we ran into issues regarding JENKINS-56987 sometime last week which completely broke the plugin again.

The current state of the pull request for 56987 involves deleting all the apache files that you modified in your fork to get the connection to stop freezing. So now we are freezing again. It's been a long few weeks with these plugin problems!

 

rodrigc@FreeBSD.org (JIRA)

unread,
Feb 25, 2020, 6:00:11 AM2/25/20
to jenkinsc...@googlegroups.com

Taylor Nelson David France Take a look at: https://github.com/jenkinsci/jira-plugin/pull/213 and try it out if you want. That is a patch I have which works after the Atlassian GDPR changes in JENKINS-56987 . I'm testing this now against my JIRA Cloud setup.

One thing I did in that patch was I deleted the imported copy of the ApacheAsyncHttpClient code. I'm not sure if that was the right approach, but it allows me to get a setup that worked in my JIRA Cloud setup.

kisipo6270@jetsmails.com (JIRA)

unread,
Feb 25, 2020, 12:00:06 PM2/25/20
to jenkinsc...@googlegroups.com

Craig Rodrigues Yes I definitely saw your pull request. Thanks for all your work on that! I built the plugin with your PR #213 yesterday morning, but this issue here persists even with your changes.

We use the plugin for anything from updating a fixVersion to adding comments on tasks, and for some reason these jobs started occasionally freezing and would require us to restart our Jenkins server to unclog the pipeline. (With the exact stack trace David has above – blocked thread cannot close HTTP connection and another thread waiting forever, etc)

I was going to combine your changes in PR 213 with David's changes on PR 207, but 207 modifies that AsyncHttpClient class that you have deleted in your PR. So now I have to choose to build a plugin that doesn't work, but doesn't freeze OR a plugin that works but freezes and blocks all of our executors – connection freeze fix or GDPR fix, respectively.

I realize I'm not contributing to the fix, I don't have any experience at all building Jenkins plugins. Just hoping my comments will give more context to the problem we are having. I'm sort of surprised this issue isn't affecting more people.

kisipo6270@jetsmails.com (JIRA)

unread,
Feb 25, 2020, 1:23:03 PM2/25/20
to jenkinsc...@googlegroups.com
Taylor Nelson edited a comment on Bug JENKINS-60536
[~rodrigc] Yes I definitely saw your pull request. Thanks for all your work on that! I built the plugin with your PR #213 yesterday morning, but this issue here persists even with your changes.


We use the plugin for anything from updating a fixVersion to adding comments on tasks, and for some reason these jobs started occasionally freezing and would require us to restart our Jenkins server to unclog the pipeline. (With the exact stack trace David has above – blocked thread cannot close HTTP connection and another thread waiting forever, etc)

I was going to combine your changes in PR 213 with David's changes on PR 207, but 207 modifies that AsyncHttpClient class that you have deleted in your PR. So now I have to choose to build a plugin that doesn't work, but doesn't freeze *OR* a plugin that works but freezes and blocks all of our executors – connection freeze fix or GDPR fix, respectively. :)


I realize I'm not contributing to the fix, I don't have any experience at all building Jenkins plugins. Just hoping my comments will give more context to the problem we are having. I'm sort of surprised this issue isn't affecting more people.


 

Edit: I just saw you added some commits related to the apache dependencies being updated. Will try building again

rodrigc@FreeBSD.org (JIRA)

unread,
Feb 25, 2020, 2:42:03 PM2/25/20
to jenkinsc...@googlegroups.com

Cool, let me know how it works for you.

You can also post messages in https://github.com/jenkinsci/jira-plugin

The more people who collaborate on this the better!

rodrigc@FreeBSD.org (JIRA)

unread,
Feb 25, 2020, 2:44:03 PM2/25/20
to jenkinsc...@googlegroups.com
Craig Rodrigues edited a comment on Bug JENKINS-60536
Cool, let me know how it works for you.

You can also post messages in the Gitter chat channel https:// github gitter . com im /jenkinsci/jira-plugin

The more people who collaborate on this the better!

kisipo6270@jetsmails.com (JIRA)

unread,
Feb 26, 2020, 10:29:02 AM2/26/20
to jenkinsc...@googlegroups.com
Taylor Nelson updated an issue
 
Change By: Taylor Nelson
Attachment: image-2020-02-26-08-28-37-010.png

kisipo6270@jetsmails.com (JIRA)

unread,
Feb 26, 2020, 10:30:04 AM2/26/20
to jenkinsc...@googlegroups.com
Taylor Nelson updated an issue
Change By: Taylor Nelson
Attachment: image-2020-02-26-08-29-32-302.png

kisipo6270@jetsmails.com (JIRA)

unread,
Feb 26, 2020, 10:30:04 AM2/26/20
to jenkinsc...@googlegroups.com
Taylor Nelson updated an issue
Change By: Taylor Nelson
Attachment: image-2020-02-26-08-29-08-607.png

kisipo6270@jetsmails.com (JIRA)

unread,
Feb 26, 2020, 10:30:05 AM2/26/20
to jenkinsc...@googlegroups.com
Taylor Nelson updated an issue
Change By: Taylor Nelson
Attachment: image-2020-02-26-08-29-59-154.png

kisipo6270@jetsmails.com (JIRA)

unread,
Feb 26, 2020, 10:38:04 AM2/26/20
to jenkinsc...@googlegroups.com
Taylor Nelson commented on Bug JENKINS-60536
 
Re: Jenkins job hangs in Jira call after git rev-list command

I pulled the latest changes from your pull request 213 and built the plugin. Unfortunately it did not work. It looks like you merged David France's changes into your PR (David's changes solved the freezing problem for me before these GDPR problems) so I'm not sure why it's not working now. Maybe 213 is still using the "default" apache dependencies instead of the custom Jira plugin ones?

Here are the details before I reboot the server again:

 

The BLOCKED thread:

 

The WAITING thread:

d.france@crossflight.com (JIRA)

unread,
Feb 27, 2020, 5:14:04 AM2/27/20
to jenkinsc...@googlegroups.com

Just to report, we've tried merging https://github.com/rodrigc/jira-plugin/tree/jira-rest-client.version_5.2.0 across to https://github.com/xft-devs/jira-plugin

It was able to run a workflow action against Jira Cloud OK again here - the fix for JENKINS-56987 looks good, thanks. Time will tell if it is still stable with the connections...

The comment about deleting ApacheAsyncHttpClient didn't make sense during the merge. The class was still present on the jira-rest-client.version_5.2.0 branch, so the earlier change came across without a problem.

For reference, this is building, with the same test failures before and after:

[ERROR] Failures:
[ERROR] hudson.plugins.jira.JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents(hudson.plugins.jira.JiraCreateIssueNotifierTest)
[ERROR]   Run 1: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR]   Run 2: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR]   Run 3: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR]   Run 4: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR]   Run 5: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[INFO]
[INFO]
[ERROR] Tests run: 199, Failures: 1, Errors: 0, Skipped: 0

I've not investigated these deeper.
This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)
Atlassian logo

d.france@crossflight.com (JIRA)

unread,
Feb 27, 2020, 5:15:03 AM2/27/20
to jenkinsc...@googlegroups.com
David France edited a comment on Bug JENKINS-60536
Just to report, we've tried merging [https://github.com/rodrigc/jira-plugin/tree/jira-rest-client.version_5.2.0] across to [https://github.com/xft-devs/jira-plugin]

It was able to run a workflow action against Jira Cloud OK again here - the fix for JENKINS-56987 looks good, thanks. Time will tell if it is still stable with the connections...

The comment about deleting ApacheAsyncHttpClient didn't make sense during the merge. The class was still present on the jira-rest-client.version_5.2.0 branch, so the earlier change came across without a problem.
----

For reference, this is building, with the same test failures before and after:
{ code:java noformat }

[ERROR] Failures:
[ERROR] hudson.plugins.jira.JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents(hudson.plugins.jira.JiraCreateIssueNotifierTest)
[ERROR]   Run 1: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR]   Run 2: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR]   Run 3: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR]   Run 4: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR]   Run 5: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[INFO]
[INFO]
[ERROR] Tests run: 199, Failures: 1, Errors: 0, Skipped: 0
{ code noformat }

I've not investigated these deeper.

kisipo6270@jetsmails.com (JIRA)

unread,
Mar 4, 2020, 12:39:02 PM3/4/20
to jenkinsc...@googlegroups.com

David France how is the plugin working for you now? Any blocked threads/stuck executors?

 

I merged your changes with Craig's changes and was still getting blocked threads. If it's going smoothly for you, maybe I did the merge incorrectly.

d.france@crossflight.com (JIRA)

unread,
Mar 5, 2020, 3:23:03 AM3/5/20
to jenkinsc...@googlegroups.com

The merged version has been fine for the last week - can't see any stuck threads,  To be fair though, we haven't gone back and checked if the freezing behaviour would come back without this.

 

The merge was pushed up to https://github.com/xft-devs/jira-plugin .  If you haven't already, it may be worth trying pulling or cloning that again.

hugo.sanson@wezen.com (JIRA)

unread,
Apr 15, 2020, 5:01:03 AM4/15/20
to jenkinsc...@googlegroups.com

Bumping up this issue as i believe it is not solved with 3.0.13 nor 3.0.15, 
I remember we had a fix preparing for this issue before the Atlassian GDPR changes rolled out,
and it was working for a week with an open PR on Jira plugin's github, can't find this PR anymore.

 

Our environement, jenkins 2.204.5 with open jdk 11 - Trying to connect to Jira Cloud with API Token and username

Sympthoms are 

First connection after reboot works for validate and jiraSearch

But all subsequent connections threads hang at this point,  with status BLOCKED on 
org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605)

org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#2]
org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605)
org.apache.http.impl.nio.NHttpConnectionBase.close(NHttpConnectionBase.java:511)
org.apache.http.impl.nio.conn.CPoolEntry.closeConnection(CPoolEntry.java:75)
org.apache.http.impl.nio.conn.CPoolEntry.close(CPoolEntry.java:101)
org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:423)
org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseHttpPromiseAsyncClient.java:38)
com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417)
com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364)
com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297)
com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianHttpClientDecorator.java:83)
com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253)
com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.java:69)
com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient.java:109)
com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94)
hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:206)
hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136)
hudson.plugins.jira.pipeline.SearchIssuesStep$SearchStepExecution.run(SearchIssuesStep.java:84)
hudson.plugins.jira.pipeline.SearchIssuesStep$SearchStepExecution.run(SearchIssuesStep.java:61)
org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
hudson.security.ACL.impersonate(ACL.java:290)
org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
java...@11.0.6/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
java...@11.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:264)
java...@11.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java...@11.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java...@11.0.6/java.lang.Thread.run(Thread.java:834)

Issue looks similar to Taylor Nelson's 
Any help would be appreciated, 
I'm willing to fix and build from source given some pointers

hugo.sanson@wezen.com (JIRA)

unread,
Apr 15, 2020, 5:15:07 AM4/15/20
to jenkinsc...@googlegroups.com
Hugo SANSON edited a comment on Bug JENKINS-60536
Bumping up this issue as i believe it is not solved with 3.0.13 nor 3.0.15, 
I remember we had a fix preparing for this issue before the Atlassian GDPR changes rolled out,
and it was working for a week with an open this PR on Jira  [https://github.com/jenkinsci/jira- plugin /pull/207]
but apparently it didn ' s github, can' t find this PR anymore move forward a notch since then .

 

Our environement, jenkins [2.204.5|https://jenkins.io/] with open jdk 11 - Trying to connect to Jira Cloud with API Token and username


Sympthoms are 

First connection after reboot works for validate and jiraSearch

But all subsequent connections threads hang at this point,  with status BLOCKED on 
org.apache.http.nio.reactor.ssl.SSLIOSession.close([SSLIOSession.java:605|http://jenkins.wezen.global/monitoring?part=source&class=org.apache.http.nio.reactor.ssl.SSLIOSession#605])
{code:java}
java...@11.0.6/java.lang.Thread.run(Thread.java:834){code}
Issue looks similar to [~taylornelson]'s 

Any help would be appreciated, 
I'm willing to fix and build from source given some pointers

benoit.guerin@gmail.com (JIRA)

unread,
Apr 15, 2020, 5:56:03 AM4/15/20
to jenkinsc...@googlegroups.com

Agreed, issue is still present

hugo.sanson@wezen.com (JIRA)

unread,
Apr 15, 2020, 1:52:05 PM4/15/20
to jenkinsc...@googlegroups.com

Using this fork you guys made at https://github.com/xft-devs/jira-plugin

The issue looks fixed or at least
Our use cases with Jenkins pipeline JiraSearch and JiraIssueUpdater against Jira Cloud work fine.
No more threads hanging or blocked

 To be able to build this fork, i had to change a few versions in the pom.xml
configuration-as-code.version from 1.8 to 1.36

jackson2-api from 2.8.11.2 to 2.10.2

io.jenkins.configuration-as-code to io.jenkins.configuration-as-code.test-harness
junit from 1.3 to 1.20
org.jenkins-ci.plugins.credentials from 2.1.19 to 2.2.1
These versions changes match the https://github.com/jenkinsci/jira-plugin branch feature/4.0 changes

hugo.sanson@wezen.com (JIRA)

unread,
Apr 15, 2020, 1:53:02 PM4/15/20
to jenkinsc...@googlegroups.com
Hugo SANSON edited a comment on Bug JENKINS-60536
Using this fork you guys made at [https://github.com/xft-devs/jira-plugin]

*The issue looks fixed* or at least

Our use cases with Jenkins pipeline JiraSearch and JiraIssueUpdater against Jira Cloud work fine.
No more threads hanging or blocked

 To be able to build this fork, i had to change a few versions in the pom.xml
_configuration {code:java}
configuration -as-code. version_ version from 1.8 to 1.36
_jackson2 jackson2 - api_ api from 2.8.11.2 to 2.10.2
_io io . jenkins__ jenkins .configuration-as- code_ code to  _io io .jenkins.configuration-as-code.test- harness_ harness
_junit_ junit from 1.3 to 1.20
 _org org .jenkins-ci.plugins. credentials_ credentials from 2.1.19 to 2.2.1
 {code}

These versions changes match the [https://github.com/jenkinsci/jira-plugin] branch feature/4.0 changes

benoit.guerin@gmail.com (JIRA)

unread,
Apr 20, 2020, 6:08:05 AM4/20/20
to jenkinsc...@googlegroups.com

Same here, modifications from xft-devs solve issue. Thanks to them !

I rebase theirs changes on master, here is the PR : https://github.com/jenkinsci/jira-plugin/pull/230
Reply all
Reply to author
Forward
0 new messages