| We've being seeing much the same problem stack that Kieran reported, trying to progress issues by a workflow action in Jira. We first became aware of the problem on 20/12/2019:
"Executor #1 for master : executing deployToUAT #1619" #647 daemon prio=5 os_prio=0 cpu=436.80ms elapsed=236.46s tid=0x000000006
2b92000 nid=0x2590 waiting on condition [0x000000006644d000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java...@11.0.5/Native Method)
- parking to wait for <0x0000000083d46f58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java...@11.0.5/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java...@11.0.5/AbstractQueuedSynchronizer.j
ava:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java...@11.0.5/AbstractQueuedSynchronizer.java:917) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java...@11.0.5/AbstractQueuedSynchronizer.java:1240)
at java.util.concurrent.locks.ReentrantLock.lock(java...@11.0.5/ReentrantLock.java:267)
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.j
ava:295)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseH
ttpPromiseAsyncClient.java:36)
at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417)
at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364)
at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297)
at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianH
ttpClientDecorator.java:83)
at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253)
at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.j
ava:69)
at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient.
java:109)
at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94
)
at hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:196)
at hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136)
at hudson.plugins.jira.JiraSite.progressMatchingIssues(JiraSite.java:1043)
at hudson.plugins.jira.JiraIssueUpdateBuilder.perform(JiraIssueUpdateBuilder.java:105)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1853)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:427)
The problem looks to be a problem with an earlier connection in the connection pool in the HttpAsyncClient. The earlier open connection is failing to be closed before processing the new request. We did a couple of experiments in the jira-plugin source, and managed to find a way get the connection to complete. (This is hardly a submission quality change, but perhaps it offers a clue or two). The first hope was that changing the HttpClientOptions.maxConnectionsPerHost value would stop connection pool holding the connections, but forcing this (in the code) to either 1 or 2000 had no effect. Instead, as a rough workaround we tried bypassing the connection pooling, by creating a new HttpAsyncClientBuilder against a new PoolingNHttpClientConnectionManager for each request. This is will be leaky - there is nothing to close the new connection pools. We don't know how soon to expect a problem though, Jenkins has stayed up so far. The change is here. (Essentially it extracts a method in the constructor, to create the HttpAsyncClientBuilder, then also calls it from doExecute().) For reference this is working against JIRA Cloud. Not sure why the connections started failing to close, but this was enough to force our builds through for the time being. |
