[JIRA] (JENKINS-42136) shared-library abstraction causing RejectedExecutionException when running sh() commands

2 views
Skip to first unread message

jwhitcraft@sugarcrm.com (JIRA)

unread,
Feb 17, 2017, 12:16:02 AM2/17/17
to jenkinsc...@googlegroups.com
Jon Whitcraft created an issue
 
Jenkins / Bug JENKINS-42136
shared-library abstraction causing RejectedExecutionException when running sh() commands
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2017/Feb/17 5:15 AM
Priority: Major Major
Reporter: Jon Whitcraft

Hello,

When using this plug-in plus a rather abstracted shared-pipeline-library and tyring to run a sh() command on the kubernetes node/container, i get the following error

java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@4a046f09 rejected from java.util.concurrent.ThreadPoolExecutor@2314aa63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
	at okhttp3.Dispatcher.enqueue(Dispatcher.java:129)
	at okhttp3.RealCall.enqueue(RealCall.java:78)
	at okhttp3.ws.WebSocketCall.enqueue(WebSocketCall.java:109)
	at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:210)
	at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:54)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:119)
	at hudson.Launcher$ProcStarter.start(Launcher.java:384)
	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:157)
	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:63)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:172)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
	at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1280)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1174)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:18)
	at deployDevBuild.call(/var/jenkins_home/jobs/trynew/builds/52/libs/sugarcrm-pipeline-library/vars/deployDevBuild.groovy:13)
	at helmNode.call(/var/jenkins_home/jobs/trynew/builds/52/libs/sugarcrm-pipeline-library/vars/helmNode.groovy:10)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor806.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:74)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I have my shared library setups and attached here

helmTemplate.groovy
#!/usr/bin/groovy
def call(Map parameters = [:], body) {

    def defaultLabel = "helm.${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_')
    def defaultImage = "my.repo.com/kubernetes/helm-kubectl:latest"
    def label = parameters.get('label', defaultLabel)
    def helmImage = parameters.get('image', defaultImage)
    def inheritFrom = parameters.get('inheritFrom', 'base')

    podTemplate(label: label,  inheritFrom: "${inheritFrom}",
            containers: [
                    [name: 'helm', image: "${helmImage}", command: 'cat', ttyEnabled: true]
            ]) {
        body()
    }
}
helmNode.groovy
#!/usr/bin/groovy
def call(Map parameters = [:], body) {

    def defaultLabel = "helm.${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_')
    def label = parameters.get('label', defaultLabel)

    helmTemplate(parameters) {
        node(label) {
            container('helm') {
                body()
            }
        }
    }
}

and then I call these in my Jenkins file by doing this

helmNode(parameters) {
        sh "helm version"
}

and it just bombs out with that error. It doesn't matter if the jenkins host in my kubernetes cluster or not.

What is weird is that if i don't abstract it out and just put everything in my Jenkinsfile, it works fine, but that isn't an option for us as we want as much reusable code as possible.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Mar 12, 2017, 5:33:01 AM3/12/17
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-42136
 
Re: shared-library abstraction causing RejectedExecutionException when running sh() commands

This has to be a problem in the fabric8 kubernetes client or okhttp, have you tried building the git master version that has an updated k8s client ?

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

jwhitcraft@sugarcrm.com (JIRA)

unread,
Mar 12, 2017, 12:02:02 PM3/12/17
to jenkinsc...@googlegroups.com

jwhitcraft@sugarcrm.com (JIRA)

unread,
Mar 12, 2017, 1:17:01 PM3/12/17
to jenkinsc...@googlegroups.com

Carlos Sanchez that seems to have fixed it, but I didn't do extensive testing yet.  I'll be able to do that tomorrow am.  The only issue i'm seeing now is JENKINS-42048.

Tomorrow I'll do more testing and let you know what I find.  I can work around JENKINS-42048 for now, but this issues seems to be fixed with the latest master.

jwhitcraft@sugarcrm.com (JIRA)

unread,
Mar 22, 2017, 1:57:01 PM3/22/17
to jenkinsc...@googlegroups.com

Carlos Sanchez,

From my extensive testing over the past week, this seems fixed with the latest master from the repo

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 25, 2018, 5:18:01 AM5/25/18
to jenkinsc...@googlegroups.com

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Nov 28, 2018, 5:52:02 AM11/28/18
to jenkinsc...@googlegroups.com

I see RejectedExecutionException when more jobs than "Max connections to Kubernetes API" are running

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

hayden@haydenball.me.uk (JIRA)

unread,
Dec 3, 2018, 10:18:02 AM12/3/18
to jenkinsc...@googlegroups.com

I'm seeing this intermittently when using a shared declarative pipeline (i.e. https://jenkins.io/doc/book/pipeline/shared-libraries/#defining-declarative-pipelines).

Task okhttp3.RealCall$AsyncCall@29e44de7 rejected from java.util.concurrent.ThreadPoolExecutor@19b44afc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 77]

I'm running at well under "Max connections to Kubernetes API" (although I have just bumped that to see if it makes a difference), so I'm not convinced this is the issue.

hayden@haydenball.me.uk (JIRA)

unread,
Dec 3, 2018, 10:25:02 AM12/3/18
to jenkinsc...@googlegroups.com
Hayden Ball edited a comment on Bug JENKINS-42136
I'm seeing this intermittently when using a shared declarative pipeline (i.e. [https://jenkins.io/doc/book/pipeline/shared-libraries/#defining-declarative-pipelines)].
{code:java}
Task okhttp3.RealCall$AsyncCall@29e44de7 rejected from java.util.concurrent.ThreadPoolExecutor@19b44afc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 77]{code}
I
'm should be running at well under "Max connections to Kubernetes API" (although I have just bumped that to see if it makes a difference), so I'm not convinced this is the issue. I'm seeing this with 3 concurrent jobs, with max connections set to 48.

julienduchesne@live.com (JIRA)

unread,
Dec 4, 2018, 6:38:01 AM12/4/18
to jenkinsc...@googlegroups.com

This has started happening systematically recently on our side. I have a job that calls a `sh` every two minutes for ~an hour and it fails after ~30 minutes every time. I have set the Max connections to Kubernetes API to 200 and I am running a single job so it shouldn't be the issue. I am running the latest plugin version (1.13.7)

julienduchesne@live.com (JIRA)

unread,
Dec 4, 2018, 6:41:01 AM12/4/18
to jenkinsc...@googlegroups.com
Julien Duchesne edited a comment on Bug JENKINS-42136
This has started happening systematically recently on our side. I have a job that calls a `sh` every two minutes for ~an hour and it fails after ~30 minutes every time. I have set the Max connections to Kubernetes API to 200 and I am running a single job so it shouldn't be the issue. I am running the latest plugin version (1.13.7) . I am reverting to the version I was before (1.12.1), I'll let you know if that fixes my issue.

julienduchesne@live.com (JIRA)

unread,
Dec 4, 2018, 7:29:02 AM12/4/18
to jenkinsc...@googlegroups.com

Update: I've pinpointed the bug in version 1.13.6.

1.13.6: Systematic issues

1.13.5: Was working fine for weeks

julienduchesne@live.com (JIRA)

unread,
Dec 4, 2018, 7:32:02 AM12/4/18
to jenkinsc...@googlegroups.com
Julien Duchesne edited a comment on Bug JENKINS-42136
Update: I've pinpointed the bug in version 1.13.6.

1.13.6: Systematic issues

1.13.5: Was working fine for weeks (another jenkins instance than the one on 1.12.1 :))

kenjins_bits@timothy.fromnz.net (JIRA)

unread,
Dec 6, 2018, 7:17:06 AM12/6/18
to jenkinsc...@googlegroups.com

kenjins_bits@timothy.fromnz.net (JIRA)

unread,
Dec 6, 2018, 9:06:05 AM12/6/18
to jenkinsc...@googlegroups.com

seckler@in.tum.de (JIRA)

unread,
Dec 7, 2018, 3:19:02 AM12/7/18
to jenkinsc...@googlegroups.com
Steffen Seckler commented on Bug JENKINS-42136
 
Re: shared-library abstraction causing RejectedExecutionException when running sh() commands

can confirm this issue, seeing this with kubernetes plugin 1.13.7

seckler@in.tum.de (JIRA)

unread,
Dec 7, 2018, 3:20:03 AM12/7/18
to jenkinsc...@googlegroups.com
Steffen Seckler edited a comment on Bug JENKINS-42136
can confirm this issue, seeing this with kubernetes plugin 1.13.7 .

Increased the max number to kubernetes api, will check whether this helps.

seckler@in.tum.de (JIRA)

unread,
Dec 7, 2018, 3:22:02 AM12/7/18
to jenkinsc...@googlegroups.com
Steffen Seckler edited a comment on Bug JENKINS-42136
can confirm this issue, seeing this with kubernetes plugin 1.13.7.

Increased the max number to kubernetes api  (according to https://github.com/jenkinsci/kubernetes-plugin/blob/master/CHANGELOG.md#1136) , will check whether this helps.

yoanlin93@gmail.com (JIRA)

unread,
Dec 12, 2018, 4:19:02 PM12/12/18
to jenkinsc...@googlegroups.com

can confirm this issue. this happens even if you updated the kubernetes plugin to 1.13.7

Jenkins 2.150.1

seckler@in.tum.de (JIRA)

unread,
Dec 13, 2018, 4:53:02 AM12/13/18
to jenkinsc...@googlegroups.com
Steffen Seckler edited a comment on Bug JENKINS-42136
can confirm this issue, seeing this with kubernetes plugin 1.13.7.

Increased the max number of connections to the  kubernetes api (according to [ https://github.com/jenkinsci/kubernetes-plugin/blob/master/CHANGELOG.md#1136 ] ), will check whether this helps.

 

*Edit:*

*increasing the max number of connections to the* kubernetes ** api *actually helped*

seckler@in.tum.de (JIRA)

unread,
Dec 13, 2018, 4:53:03 AM12/13/18
to jenkinsc...@googlegroups.com

seckler@in.tum.de (JIRA)

unread,
Dec 13, 2018, 4:53:04 AM12/13/18
to jenkinsc...@googlegroups.com
Steffen Seckler edited a comment on Bug JENKINS-42136
can confirm this issue, seeing this with kubernetes plugin 1.13.7.

Increased the max number of connections to the kubernetes api (according to [https://github.com/jenkinsci/kubernetes-plugin/blob/master/CHANGELOG.md#1136]), will check whether this helps.

 

*Edit:*

*increasing the max number of connections to the* kubernetes * api  * actually helped*

seckler@in.tum.de (JIRA)

unread,
Dec 13, 2018, 4:53:05 AM12/13/18
to jenkinsc...@googlegroups.com
Steffen Seckler edited a comment on Bug JENKINS-42136
can confirm this issue, seeing this with kubernetes plugin 1.13.7.

Increased the max number of connections to the kubernetes api (according to [https://github.com/jenkinsci/kubernetes-plugin/blob/master/CHANGELOG.md#1136]), will check whether this helps.

 

*Edit:*

*increasing the max number of connections to the * kubernetes  * api actually helped*

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 13, 2018, 1:53:04 PM12/13/18
to jenkinsc...@googlegroups.com

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 13, 2018, 1:53:04 PM12/13/18
to jenkinsc...@googlegroups.com

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 13, 2018, 4:58:02 PM12/13/18
to jenkinsc...@googlegroups.com
Carlos Sanchez resolved as Fixed
 

This should have been fixed in 1.13.8

Change By: Carlos Sanchez
Status: Open Resolved
Resolution: Fixed

krzysztof.mulica@etrade.com (JIRA)

unread,
Jan 17, 2019, 10:30:03 AM1/17/19
to jenkinsc...@googlegroups.com
Krzysztof Mulica commented on Bug JENKINS-42136
 
Re: shared-library abstraction causing RejectedExecutionException when running sh() commands

Carlos Sanchez On Kubernetes plugin 1.13.8 and still getting this

Error occurred while checking out Git: Task okhttp3.RealCall$AsyncCall@789b60f9 rejected from java.util.concurrent.ThreadPoolExecutor@67ba2f6e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 14]

 

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Jan 17, 2019, 10:32:04 AM1/17/19
to jenkinsc...@googlegroups.com

krzysztof.mulica@etrade.com (JIRA)

unread,
Jan 17, 2019, 10:37:02 AM1/17/19
to jenkinsc...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages