[JIRA] (JENKINS-59259) sshAgent step does not survive agent disconnect/reconnect

258 views
Skip to first unread message

jon.sten@gmail.com (JIRA)

unread,
Sep 6, 2019, 3:25:02 AM9/6/19
to jenkinsc...@googlegroups.com
Jon Sten created an issue
 
Jenkins / Bug JENKINS-59259
sshAgent step does not survive agent disconnect/reconnect
Issue Type: Bug Bug
Assignee: Unassigned
Components: ssh-agent-plugin
Created: 2019-09-06 07:24
Environment: Jenkins 2.176.2
ssh-agent:1.17
durable-task: 1.30
workflow-durable-task-step:2:32
Priority: Minor Minor
Reporter: Jon Sten

The following pipeline code fails if the connection to the agent is lost and reconnected during execution of the shell script:

node('build-srv28') {
    sshagent(['db1284e3-6df6-4c4f-91dc-d2648957cd3c']) {
        sh '''
for i in `seq 1 120`;
do
    sleep 1
    echo $i
done
'''
        echo "Done"
    }
    echo "Done"
}

The build log is as follows:

Started by user Jon
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on build-srv123 in /some/path/workspace/job/path
[Pipeline] {
[Pipeline] sshagent
[ssh-agent] Using credentials svcjenkins
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-gDb94Onc5Loa/agent.30438
SSH_AGENT_PID=30440
Running ssh-add (command line suppressed)
Identity added: /some/path/workspace/job/path-tmp/private_key_970164985924018321.key (/some/path/workspace/job/path-tmp/private_key_970164985924018321.key)
[ssh-agent] Started.
[Pipeline] {
[Pipeline] sh
+ seq 1 120
+ sleep 1
+ echo 1
1
+ sleep 1
+ echo 2
2
+ sleep 1
+ echo 3
3
+ sleep 1
+ echo 4
4
+ sleep 1
+ echo 5
5
+ sleep 1
Cannot contact build-srv123: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on build-srv123 failed. The channel is closing down or has closed down
+ echo 6
6
+ sleep 1
+ echo 7
7
+ sleep 1
+ echo 8
8
... truncated for your reading sanity...
+ sleep 1
+ echo 119
119
+ sleep 1
+ echo 120
120
[Pipeline] echo
Done
[Pipeline] }
[Pipeline] // sshagent
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 0s
java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
	at hudson.remoting.Command.readFrom(Command.java:140)
	at hudson.remoting.Command.readFrom(Command.java:126)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on build-srv28 failed. The channel is closing down or has closed down
	at hudson.remoting.Channel.call(Channel.java:950)
	at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1060)
	at hudson.Launcher$ProcStarter.start(Launcher.java:455)
	at com.cloudbees.jenkins.plugins.sshagent.exec.ExecRemoteAgent.stop(ExecRemoteAgent.java:130)
	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentStepExecution.cleanUp(SSHAgentStepExecution.java:198)
	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentStepExecution.access$000(SSHAgentStepExecution.java:24)
	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentStepExecution$Callback.finished(SSHAgentStepExecution.java:108)
	at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:118)
	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:377)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:87)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
	at sun.reflect.GeneratedMethodAccessor1513.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:186)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:370)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:93)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:282)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:270)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:66)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

As can be seen in the log, the connection fails, but once the agent comes back online the build log is fetched and continuously pulled until the shell script finishes on the agent. Once the shell script is done SSHAgent throws exception in its teardown phase.

This behavior is different from when the master restarts during ongoing build, in that situation SSHAgent step works as expected. Once the agent comes back online ssh-agent plugin runs necessary commands to restores operation:

...
+ sleep 1
+ echo 17
17
+ sleep 1
Resuming build at Fri Sep 06 08:27:37 CEST 2019 after Jenkins restart
Waiting to resume part of some » job #15: Finished waiting
Ready to run at Fri Sep 06 08:27:40 CEST 2019
[ssh-agent] Using credentials svcjenkins
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-wNzzYEFltayP/agent.10155
SSH_AGENT_PID=10157
Running ssh-add (command line suppressed)
Identity added: /some/path/workspace/job/path-tmp/private_key_155210530577449396.key (/some/path/workspace/job/path-tmp/private_key_155210530577449396.key)
[ssh-agent] Started.
+ echo 18
18
+ sleep 1
+ echo 19
19
+ sleep 1
+ echo 20
....
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

jon.sten@gmail.com (JIRA)

unread,
Sep 6, 2019, 3:26:07 AM9/6/19
to jenkinsc...@googlegroups.com
Jon Sten updated an issue
Change By: Jon Sten
The following pipeline code fails if the connection to the agent is lost and reconnected during execution of the shell script:
{code:groovy}
node ('build-srv28') {

    sshagent(['db1284e3-6df6-4c4f-91dc-d2648957cd3c']) {
        sh '''
for i in `seq 1 120`;
do
    sleep 1
    echo $i
done
'''
        echo "Done"
    }
    echo "Done"
}
{code}

The build log is as follows:

{noformat}
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on build-srv123 failed. The channel is closing down or has closed down
{noformat}

As can be seen in the log, the connection fails, but once the agent comes back online the build log is fetched and continuously pulled until the shell script finishes on the agent. Once the shell script is done SSHAgent throws exception in its teardown phase.

This behavior is different from when the master restarts during ongoing build, in that situation SSHAgent step works as expected. Once the agent comes back online ssh-agent plugin runs necessary commands to restores operation:

{noformat}
{noformat}

jon.sten@gmail.com (JIRA)

unread,
Sep 6, 2019, 3:26:07 AM9/6/19
to jenkinsc...@googlegroups.com
Jon Sten updated an issue
The following pipeline code fails if the connection to the agent is lost and reconnected during execution of the shell script:
{code:groovy}
node('build-srv28') {
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on build- srv28 srv123 failed. The channel is closing down or has closed down

jon.sten@gmail.com (JIRA)

unread,
Sep 12, 2019, 2:31:06 AM9/12/19
to jenkinsc...@googlegroups.com
Jon Sten commented on Bug JENKINS-59259
 
Re: sshAgent step does not survive agent disconnect/reconnect

I did some digging in the code, and to me it looks like the problem is that ExecRemoteAgent and SSHAgentStepExecution caches the launcher object. The code seems to be rather old and would probably become less buggy if StepContext was used instead, cause as far as I know that object is supposed to handle loss of underlying connections/streams. I'll see if I can gather the nervs to fix the code.
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

jon.sten@gmail.com (JIRA)

unread,
Sep 12, 2019, 2:33:03 AM9/12/19
to jenkinsc...@googlegroups.com
Jon Sten edited a comment on Bug JENKINS-59259
I did some digging in the code, and to me it looks like the problem is that ExecRemoteAgent and SSHAgentStepExecution caches the launcher object. The code seems to be rather old and would probably become less buggy if StepContext was used instead, cause as far as I know that object is supposed to handle loss of underlying connections/streams. I'll see if I can gather the nervs to fix the code.

[~stephenc] [~stephenconnolly] am I on the right track?

jon.sten@gmail.com (JIRA)

unread,
Nov 8, 2019, 6:01:02 AM11/8/19
to jenkinsc...@googlegroups.com
Jon Sten updated an issue
Change By: Jon Sten
The following pipeline code fails if the connection to the agent is lost and reconnected during execution of the shell script:
{code :groovy
}
node {
    sshagent([' db1284e3-6df6-4c4f-91dc-d2648957cd3c someCredential ']) {

        sh '''
for i in `seq 1 120`;
do
    sleep 1
    echo $i
done
'''
        echo "Done"
    }
    echo "Done"
}
{code}
The build log is as follows:
{noformat}
Started by user Jon
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on build-srv123 in /some/path/workspace/job/path
[Pipeline] {
[Pipeline] sshagent
[ssh-agent] Using credentials svcjenkins someCredential
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on build-srv123 failed. The channel is closing down or has closed down
[ssh-agent] Using credentials svcjenkins someCredential
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-wNzzYEFltayP/agent.10155
SSH_AGENT_PID=10157
Running ssh-add (command line suppressed)
Identity added: /some/path/workspace/job/path-tmp/private_key_155210530577449396.key (/some/path/workspace/job/path-tmp/private_key_155210530577449396.key)
[ssh-agent] Started.
+ echo 18
18
+ sleep 1
+ echo 19
19
+ sleep 1
+ echo 20
....
{noformat}

jon.sten@gmail.com (JIRA)

unread,
Nov 15, 2019, 5:12:02 AM11/15/19
to jenkinsc...@googlegroups.com
Jon Sten assigned an issue to Jon Sten
Change By: Jon Sten
Assignee: Jon Sten

jon.sten@gmail.com (JIRA)

unread,
Nov 15, 2019, 5:13:02 AM11/15/19
to jenkinsc...@googlegroups.com
Jon Sten commented on Bug JENKINS-59259
 
Re: sshAgent step does not survive agent disconnect/reconnect

Testcase which reproduces the problem:

    @Test
    public void agentConnectionDropTest() throws Exception {

        story.addStep(new Statement() {
            @Override
            public void evaluate() throws Throwable {
                List<String> credentialIds = new ArrayList<String>();
                credentialIds.add(CREDENTIAL_ID);
                SSHUserPrivateKey key = new BasicSSHUserPrivateKey(CredentialsScope.GLOBAL, credentialIds.get(0), "cloudbees",
                        new BasicSSHUserPrivateKey.DirectEntryPrivateKeySource(getPrivateKey()), "cloudbees", "test");
                SystemCredentialsProvider.getInstance().getCredentials().add(key);
                SystemCredentialsProvider.getInstance().save();

                DumbSlave agent = story.j.createSlave(true);
                WorkflowJob job = story.j.jenkins.createProject(WorkflowJob.class, "sshAgentAvailable");
                job.setDefinition(new CpsFlowDefinition(""
                        + "node('" + agent.getNodeName() + "') {\n"
                        + "  sshagent (credentials: ['" + CREDENTIAL_ID + "']) {\n"
                        + "    semaphore 'upAndRunning'\n"
                        + "  }\n"
                        + "}\n", true)
                );

                WorkflowRun run = job.scheduleBuild2(0).getStartCondition().get();

                SemaphoreStep.waitForStart("upAndRunning/1", run);

                story.j.disconnectSlave(agent);
                story.j.waitOnline(agent);

                SemaphoreStep.success("upAndRunning/1", null);

                story.j.waitForCompletion(run);
                story.j.assertBuildStatusSuccess(run);
            }
        });
    }

jon.sten@gmail.com (JIRA)

unread,
Nov 15, 2019, 5:13:03 AM11/15/19
to jenkinsc...@googlegroups.com
Jon Sten started work on Bug JENKINS-59259
 
Change By: Jon Sten
Status: Open In Progress

jon.sten@gmail.com (JIRA)

unread,
Nov 15, 2019, 7:17:02 AM11/15/19
to jenkinsc...@googlegroups.com
Change By: Jon Sten
Status: In Progress Review

jon.sten@gmail.com (JIRA)

unread,
Nov 15, 2019, 7:22:02 AM11/15/19
to jenkinsc...@googlegroups.com
Jon Sten commented on Bug JENKINS-59259

jglick@cloudbees.com (JIRA)

unread,
Jan 10, 2020, 11:47:02 AM1/10/20
to jenkinsc...@googlegroups.com
Change By: Jesse Glick
Status: In Review Fixed but Unreleased
Resolution: Fixed
Released As: 1.18

jglick@cloudbees.com (JIRA)

unread,
Jan 10, 2020, 11:53:01 AM1/10/20
to jenkinsc...@googlegroups.com
Change By: Jesse Glick
Status: Fixed but Unreleased Resolved

snago86@gmail.com (JIRA)

unread,
Jan 14, 2020, 9:12:07 AM1/14/20
to jenkinsc...@googlegroups.com
Robin Karlsson commented on Bug JENKINS-59259
 
Re: sshAgent step does not survive agent disconnect/reconnect

After upgrading to 1.18 we get an error that looks related to this fix:

java.io.NotSerializableException: org.jenkinsci.plugins.docker.workflow.WithContainerStep$Decorator$1
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
	at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at hudson.remoting.UserRequest._serialize(UserRequest.java:263)
	at hudson.remoting.UserRequest.serialize(UserRequest.java:272)
	at hudson.remoting.UserRequest.<init>(UserRequest.java:101)
	at hudson.remoting.Channel.call(Channel.java:949)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
	at com.sun.proxy.$Proxy112.addIdentity(Unknown Source)
	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentStepExecution.initRemoteAgent(SSHAgentStepExecution.java:181)
	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentStepExecution.start(SSHAgentStepExecution.java:63)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:286)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:179)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
	at sun.reflect.GeneratedMethodAccessor570.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:160)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:158)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:162)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:132)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:132)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
Caused: java.io.IOException: Unable to serialize UserRPCRequest:com.cloudbees.jenkins.plugins.sshagent.RemoteAgent.addIdentity[java.lang.String,java.lang.String,java.lang.String,hudson.Launcher,hudson.model.TaskListener](2393)
	at hudson.remoting.UserRequest.serialize(UserRequest.java:274)
	at hudson.remoting.UserRequest.<init>(UserRequest.java:101)
	at hudson.remoting.Channel.call(Channel.java:949)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
	at com.sun.proxy.$Proxy112.addIdentity(Unknown Source)
	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentStepExecution.initRemoteAgent(SSHAgentStepExecution.java:181)
	at com.cloudbees.jenkins.plugins.sshagent.SSHAgentStepExecution.start(SSHAgentStepExecution.java:63)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:286)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:179)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
	at sun.reflect.GeneratedMethodAccessor570.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:160)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:158)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:162)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:132)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:132)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
	at WorkflowScript.run(WorkflowScript:68)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
	at sun.reflect.GeneratedMethodAccessor530.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:405)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:317)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:281)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

After downgrading to 1.17 it works again.

 

jon.sten@gmail.com (JIRA)

unread,
Jan 14, 2020, 9:18:07 AM1/14/20
to jenkinsc...@googlegroups.com
Jon Sten commented on Bug JENKINS-59259

Robin Karlsson ouch, that doesn't look good, what launcher do you use to launch the node where you got this exception. I.e. what I'm looking for is which implementation of RemoteAgent that is causing this.

snago86@gmail.com (JIRA)

unread,
Jan 14, 2020, 9:27:05 AM1/14/20
to jenkinsc...@googlegroups.com

Jon Sten Not 100% sure that this is what you're asking for but we use docker on a linux agent:

agent {
    docker {
        label 'linux'
        image 'python:3.5'
        args '-u root:root'
        label 'linux && baremetal'
    }
}

jon.sten@gmail.com (JIRA)

unread,
Jan 14, 2020, 9:38:06 AM1/14/20
to jenkinsc...@googlegroups.com
Jon Sten updated an issue
Change By: Jon Sten
Attachment: image-2020-01-14-15-37-17-350.png

jon.sten@gmail.com (JIRA)

unread,
Jan 14, 2020, 9:43:07 AM1/14/20
to jenkinsc...@googlegroups.com
Jon Sten commented on Bug JENKINS-59259
 
Re: sshAgent step does not survive agent disconnect/reconnect

I was referring to how the agent where job is launched, i.e.:

 

With that said, think I've found the problem. If JNRRemoteAgent or MinaRemoteAgent is used, then things will break since the agent is serialized and sent to the agent while there only exists a proxy object on the master. This means that all calls needs to be serialized and sent over to the agent side. Which goes bad now that I've changed so that Launcher is included.

Jesse Glick I've just taken a quick look, but a possible plan to fix this is by introduce a "provider" interface which SSHAgentStepExecution and SSHAgentBuildWrapper implements (or sub-class) which is the provided to the constructor of the RemoteAgent implementations. Most implementations, except ExecRemoteAgent can ignore the argument and remain serializable, while ExecRemoteAgent can keep it and use it to recover an up to date version of launcher and listener.

jon.sten@gmail.com (JIRA)

unread,
Jan 14, 2020, 9:45:07 AM1/14/20
to jenkinsc...@googlegroups.com
Jon Sten reopened an issue
 
Change By: Jon Sten
Resolution: Fixed
Status: Resolved Reopened

snago86@gmail.com (JIRA)

unread,
Jan 14, 2020, 9:56:10 AM1/14/20
to jenkinsc...@googlegroups.com
Robin Karlsson commented on Bug JENKINS-59259
 
Re: sshAgent step does not survive agent disconnect/reconnect

Jon Sten Same as in your picture, "Launch agent agents via SSH".

jon.sten@gmail.com (JIRA)

unread,
Jan 14, 2020, 10:00:09 AM1/14/20
to jenkinsc...@googlegroups.com
Jon Sten commented on Bug JENKINS-59259

Okay, strange, I'm guessing that there are other heuristics in play here. In either case, in your situations another RemoteAgent is used to handle the communication. I'm working on a fix.

jglick@cloudbees.com (JIRA)

unread,
Jan 14, 2020, 10:06:07 AM1/14/20
to jenkinsc...@googlegroups.com

Jon Sten your proposal sounds reasonable.

Really I would prefer to delete JNRRemoteAgent and MinaRemoteAgent and just document that users are expected to have an ssh-agent executable in their path.

jon.sten@gmail.com (JIRA)

unread,
Jan 14, 2020, 10:41:10 AM1/14/20
to jenkinsc...@googlegroups.com
Jon Sten updated an issue
 
Change By: Jon Sten
Attachment: ssh-agent.hpi

jon.sten@gmail.com (JIRA)

unread,
Jan 14, 2020, 10:44:06 AM1/14/20
to jenkinsc...@googlegroups.com
Jon Sten commented on Bug JENKINS-59259
 
Re: sshAgent step does not survive agent disconnect/reconnect

Sorry about this mess, the proxy objects weren't on my radar when I implemented the fix.

Anyways, I've made a PR to fix the problem, it gets quite messy, so if any of you se an improvement, please let me know!

Robin Karlsson, if you have the time to test, I've attached a build of the new version. I would be happy if you could test it

Really I would prefer to delete JNRRemoteAgent and MinaRemoteAgent and just document that users are expected to have an ssh-agent executable in their path.

That was my colleges first comment; "why the bleep is it so hard to add ssh credentials to a session". Those classes definitely makes things more complicated...

jon.sten@gmail.com (JIRA)

unread,
Jan 14, 2020, 10:45:08 AM1/14/20
to jenkinsc...@googlegroups.com
Jon Sten started work on Bug JENKINS-59259
 
Change By: Jon Sten
Status: Reopened In Progress

jglick@cloudbees.com (JIRA)

unread,
Jan 14, 2020, 1:34:08 PM1/14/20
to jenkinsc...@googlegroups.com
Jesse Glick resolved as Fixed
 

Thanks for quick turnaround!
Change By: Jesse Glick
Status: In Progress Resolved
Resolution: Fixed
Released As: 1. 18 19

snago86@gmail.com (JIRA)

unread,
Jan 15, 2020, 3:34:07 AM1/15/20
to jenkinsc...@googlegroups.com
Robin Karlsson commented on Bug JENKINS-59259
 
Re: sshAgent step does not survive agent disconnect/reconnect

Jon Sten I've installed the attached build and confirmed that it fixes the problem. Thanks!

I also found the following in the build console log, so your suspicion about JNR being used was correct:

[Pipeline] sshagent
09:26:28  [ssh-agent] Using credentials jenkins (jenkins ssh key)
09:26:28  [ssh-agent] Looking for ssh-agent implementation...
09:26:28  [ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
09:26:28  $ docker exec fd174bb2336438837e17a078f3ed4072e094f1333a515a28e0ddb6f28f5c6907 ssh-agent
09:26:28  [ssh-agent]   Java/JNR ssh-agent
09:26:29  [ssh-agent] Registered BouncyCastle on the remote agent
09:26:30  [ssh-agent] Started.

jon.sten@gmail.com (JIRA)

unread,
Jan 15, 2020, 3:59:18 AM1/15/20
to jenkinsc...@googlegroups.com
Jon Sten commented on Bug JENKINS-59259

Great, good to hear! Jesse made an official release of the fix yesterday, so you could use that one now, version 1.19.
Reply all
Reply to author
Forward
0 new messages