[JIRA] (JENKINS-41854) Contextualize a fresh FilePath after an agent reconnection

8 views
Skip to first unread message

jglick@cloudbees.com (JIRA)

unread,
Feb 8, 2017, 1:10:01 PM2/8/17
to jenkinsc...@googlegroups.com
Jesse Glick created an issue
 
Jenkins / Bug JENKINS-41854
Contextualize a fresh FilePath after an agent reconnection
Issue Type: Bug Bug
Assignee: Unassigned
Components: workflow-durable-task-step-plugin
Created: 2017/Feb/08 6:09 PM
Labels: robustness
Priority: Major Major
Reporter: Jesse Glick

PlaceholderExecutable does not bother listening for a closed connection as such, it just lets any nested step actually using the agent fail if the connection dies in the middle. Usually this is fine, but there is a corner case where it is probably wrong (unconfirmed): if an agent is disconnected and reconnected during the first sh in

node {
  sh 'sleep 999'
  sh 'sleep 999'
}

then the second sh could fail since it would be using the old Channel, even when the first sh succeeds because the DurableTaskStep.Execution recomputes the FilePath after the ChannelClosedException. (But if Jenkins restarted during the first sh after the reconnection then it should work, since the FilePath would be reconstructed from a FilePathPickle.)

The fix may be tricky since FilePath.channel is effectively final, so currently whatever workspace is passed from PlaceholderExecutable will be used for the duration of the block. Perhaps BodyInvoker.withContexts should support offering a Provider of contextual objects—in this case something that caches a FilePath so long as it is valid (!Channel.outClosed?), and otherwise falls back to FilePathUtils.find like FilePathPickle.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

jglick@cloudbees.com (JIRA)

unread,
Apr 4, 2018, 5:56:02 PM4/4/18
to jenkinsc...@googlegroups.com
Jesse Glick commented on Bug JENKINS-41854
 
Re: Contextualize a fresh FilePath after an agent reconnection

Oleg Nenashev reminds me that fixing this would probably involve also acquiring a fresh WorkspaceList lock, after a new Computer becomes available and hence a new WorkspaceList.

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

me@basilcrow.com (JIRA)

unread,
Apr 10, 2018, 9:45:02 PM4/10/18
to jenkinsc...@googlegroups.com

Is there any workaround for this issue? Is there any way for me to run my scripted pipeline on a node without using the problematic PlaceholderExecutable?

jglick@cloudbees.com (JIRA)

unread,
Apr 11, 2018, 7:20:02 AM4/11/18
to jenkinsc...@googlegroups.com

If the issue indeed exists in the described form, I would not expect there to be any general workaround. The build would just fail. Possibly you could retry the second sh. You could consolidate the scripts, if there were no crucial intervening steps of other kinds. You could run each sh in its own node, using stash and unstash as needed.

me@basilcrow.com (JIRA)

unread,
May 21, 2018, 9:18:02 PM5/21/18
to jenkinsc...@googlegroups.com

> If the issue indeed exists in the described form

The issue does indeed exist in the described form, as my production experience shows (described in JENKINS-50504, which I have now marked as a duplicate of this bug). I can also confirm that restarting Jenkins fixes the problem as described above by reconstructing the FilePath from a FilePathPickle. However, between the time that I hit this bug and the time I restart Jenkins, in-use workspaces are handed out to new runs, causing both runs to fail. This is a huge inconvenience for my users.

I have written a reproducible test case.

vivek.pandey@gmail.com (JIRA)

unread,
Nov 16, 2018, 11:09:02 AM11/16/18
to jenkinsc...@googlegroups.com
Vivek Pandey updated an issue
 
Change By: Vivek Pandey
Labels: robustness triaged-2018-11
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

jglick@cloudbees.com (JIRA)

unread,
Mar 28, 2019, 3:23:04 PM3/28/19
to jenkinsc...@googlegroups.com
Jesse Glick started work on Bug JENKINS-41854
 
Change By: Jesse Glick
Status: Open In Progress

jglick@cloudbees.com (JIRA)

unread,
Mar 28, 2019, 3:23:04 PM3/28/19
to jenkinsc...@googlegroups.com

jglick@cloudbees.com (JIRA)

unread,
Apr 1, 2019, 3:56:04 PM4/1/19
to jenkinsc...@googlegroups.com

dnusbaum@cloudbees.com (JIRA)

unread,
Jun 3, 2019, 5:04:03 PM6/3/19
to jenkinsc...@googlegroups.com
Devin Nusbaum updated Bug JENKINS-41854
 

Fixes for this issue have just been released in Pipeline Nodes and Processes Plugin version 2.31 and Pipeline Basic Steps Plugin version 2.17. You must update Pipeline Groovy Plugin to version 2.70 at the same time you update the other plugins.

Change By: Devin Nusbaum
Status: In Review Resolved
Resolution: Fixed
Released As: workflow-durable-task-step 2.31, workflow-basic-steps 2.17

jglick@cloudbees.com (JIRA)

unread,
Jun 3, 2019, 7:45:02 PM6/3/19
to jenkinsc...@googlegroups.com
Jesse Glick commented on Bug JENKINS-41854
 
Re: Contextualize a fresh FilePath after an agent reconnection

(Or you may update Pipeline: Groovy first, and then the other plugins later.)

me@basilcrow.com (JIRA)

unread,
Jun 4, 2019, 12:28:01 AM6/4/19
to jenkinsc...@googlegroups.com

Well done! Thank you for fixing this long-standing robustness issue. This is much appreciated!

dnusbaum@cloudbees.com (JIRA)

unread,
Jun 4, 2019, 5:04:06 PM6/4/19
to jenkinsc...@googlegroups.com
Devin Nusbaum updated an issue
 
Change By: Devin Nusbaum
Released As: workflow-durable-task-step 2.31, workflow-basic-steps 2. 17 18

dnusbaum@cloudbees.com (JIRA)

unread,
Jun 4, 2019, 5:04:06 PM6/4/19
to jenkinsc...@googlegroups.com
Devin Nusbaum edited a comment on Bug JENKINS-41854
 
Re: Contextualize a fresh FilePath after an agent reconnection
Fixes for this issue have just been released in Pipeline Nodes and Processes Plugin version 2.31 and Pipeline Basic Steps Plugin version 2. 17 18 . You *must* update Pipeline Groovy Plugin to version 2.70 at the same time you update the other plugins.
Reply all
Reply to author
Forward
0 new messages