What are the rules for Jenkins Pipeline to allocate workspaces, are ephemeral workspaces possible?

[James Read]

Hey folks,

I'd like to understand how Jenkins Pipeline allocates workspaces - I'm alarmed that subsequent node {} elements in a Jenkinsfile appears to allocate the same workspace. It's causing problems with conflicting or missing files. My Jenkinsfile looks like this; 

    stage "Testing"

    node {

            // This will allocate a "default" workspace.

            writeFile file: "workspace_1st_node_block.txt", text: "${pwd()}"

    }

    node {

            // I think this *should* create another workspace, but it doesn't, it reuses the 1st workspace.

            writeFile file: "workspace_2nd_node_block.txt", text: "${pwd()}"

    }

    node { ws { 

            writeFile file: "workspace_subworkspace.txt", text: "${pwd()}"

    }}

When the build of this Jenksinfile completes, if I do a `tail` on the contents of these workspaces from the Jenkins master, it looks like this; 

    ==> workspace@2/workspace_subworkspace.txt <== 

    /var/lib/jenkins/jobs/jamesread/jobs/testcase-jenkins-pipeline-workspaces-reuse/branches/master/workspace@2 

    ==> workspace/workspace_1st_node_block.txt <== 

    /var/lib/jenkins/jobs/jamesread/jobs/testcase-jenkins-pipeline-workspaces-reuse/branches/master/workspace 

    ==> workspace/workspace_2nd_node_block.txt <== 

    /var/lib/jenkins/jobs/jamesread/jobs/testcase-jenkins-pipeline-workspaces-reuse/branches/master/workspace

I would have expected 3 workspaces to be created, but instead just 2 were created (one by the 1st node{} declaration, one by the ws {} declaration). I've created this as a testcase in the following GitHub repo; https://github.com/jamesread/testcase-jenkins-pipeline-workspaces-reuse/ 

In my real project, I've resorted to wrapping each step in a "node { ws {" to force new workspaces to be created, but now it seems that even these workspaces are being reused! Just to restate, my questions are; 

1) Is workspace re-use expected, and I just have the wrong expectation? 

2) An ideal situation would be that the worksapces are ephemeral, and are deleted when the job completes. But I could not find a reliable way to do this yet in a Jenkinsfile. Is this possible?

Thanks. 

[James Read]

Hey folks,

I went poking around in the Jenkins source, and it seems that WorkspaceList.allocate() allows for workspace re-use if executor threads share the same context. Given that I've got a single master with some executors (small environment) and a single pipeline, I assume that my executor context never changes (or changes rarely). I didn't find out if a single executor is allocated to a pipeline, or node{} calls create new/reuse executors.

I see that in start() WorkspaceStepExecution.java, the call to getWorkspaceList().allocate(baseWorkspace) seems like it would always allow this reuse. Gonna go hunting around on the bug tracker to see if this is just me or I'm doing something horribly wrong though :P 

- James 

[Antonio Muñiz]

Right, the workspace will be reused for the same job in the while running in the same node.

You can use `deleteDir()` step as the very first call on every node block. It will clean the workspace.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/fb31cd9e-ec39-42c8-ab10-ebe94b820916%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Antonio Muñiz

Software Engineer
CloudBees, Inc.

[James Read]

Hi Antonio,

Thanks for confirming what I thought. 

Of course I did think about deleteDir() inside a node{} call, but this is undesirable because we know that workspaces will be shared between any node{} elements, so I would be deleting the contents of a previous node{} block, and not necessarily the workspace contents of the current node{} block only. 

I've made a logical association that a node{} block is like a classical Jenkins Job (with it's allocated workspace). This now seems wrong. It seems that actually the Pipeline makes the whole Jenkinsfile 1x logical job, and the node{} blocks just allow different node selections during that job.

My problem here is that my Jenkinsfile defines 3x very different logical jobs 1) Building code, 2) Packaging (rpm, deb, zip, etc), 3) Publishing the packages to a HTTP API. It seems to me that at the moment with pipeline 1x Git Repository = 1x Jenkinsfile = 1x Logical job (and shared workspaces between node{} elements). Each of these 3x different jobs need clean workspaces that are not shared (because the Packaging node{} block might get a Publishing workspace. 

- James 

[Antonio Muñiz]

On Tue, Jun 21, 2016 at 3:20 PM, James Read <xconsp...@gmail.com> wrote:

It seems that actually the Pipeline makes the whole Jenkinsfile 1x logical job, and the node{} blocks just allow different node selections during that job.

Not. Every `node` call has its workspace (and it can be erased at any time by Jenkins core, so you never know if the workspace contains what you think). Which means that the Jenkinsfile has to be idempotent and don't rely on workspace contents from previous builds. Actually every node block should remove/put whatever it needs (you can move things from one node to another by using `stash`).

If you request for two nodes in the same Jenkinsfile using different labels, you would be using two nodes with two separated workspaces (even in different physical machines!).

[James Read]

Hi Antonio, 

So what I think you're saying, is that every workspace assigned is allowed to contain anything - from other jobs, and always deleteDir() should be used if a clean environment is needed?

Is there a way to request a empty workspace, that is also deleted after the node{} block completes when using the Pipeline plugin? 

- James 

[Antonio Muñiz]

> every workspace assigned is allowed to contain anything - from other jobs

No, only for the same job.

> Is there a way to request a empty workspace

No, you have to use deleteDir() to be sure.

> that is also deleted after the node{} block completes

No, again, use deleteDir() as the last step in `node` or `ws` (however, having deleteDir() as first step you don't need this one).

If you don't want to delete the entire workspace, you can use `dir` step to create a directory inside the workspace (and clean it up by `dir('dir_name') { deleteDir() }`

BTW, all this information is in the snippet generator help tips (navigate to http://[JENKINS]/job/[job_name]/pipeline-syntax/).

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/77432682-e150-469f-bef9-fc28297ddbb1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Björn Pedersen]

Hi,

the pipeline tutorial explains workspace quite well: https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md#allocating-workspaces
The default workspace created by 'node' is 'locked'  to a single run of the appropriate node part. It is not cleared by default.
If you need more than one  workspace on a node 'ws' and/or 'dir' may be handy.

Björn

[Baptiste Mathus]

Also, it would be very easy to define a function like cleanNode that would be a node calling by default deleteDir before running the passed closure.

Possibly this could even be done as a PR on the node step itself, adding a boolean param to ask for that (?).

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/54f41f3e-98e7-418b-af94-44f1ef29b6ca%40googlegroups.com.

[kishor shelke]

For me Jenkins is not working as expected. Even after using ws(SLAVE_ROOT/workspace/JOB_NAME@2') command it is throwing error:

Exception executing "curl -o get-pip.py https://bootstrap.pypa.io/get-pip.py"java.io.IOException: Process working directory '/jenkins/SLAVE_ROOT/workspace/JOB_NAME@2' doesn't exist!

And for some random runs it is working.