[JIRA] (JENKINS-41118) Allow custom workspaces in declarative pipeline

Issue Type:	Improvement
Assignee:	Andrew Bayer
Components:	pipeline-model-definition-plugin
Created:	2017/Jan/17 1:21 AM
Priority:	Major
Reporter:	Sean Sutherland

The declarative pipeline syntax does not appear to allow for custom workspace roots.

Can this configuration be added to either the agent or options blocks?

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

andrew.bayer@gmail.com (JIRA)

unread,

Jan 17, 2017, 3:03:01 PM1/17/17

to jenkinsc...@googlegroups.com

Andrew Bayer commented on

Re: Allow custom workspaces in declarative pipeline

Can you give an example of how you'd want to use this? Thanks!

ssutherland@ea.com (JIRA)

unread,

Jan 17, 2017, 3:59:01 PM1/17/17

to jenkinsc...@googlegroups.com

Sean Sutherland commented on

Re: Allow custom workspaces in declarative pipeline

Since we have fairly large perforce syncs (upwards of 400GB), we design our system to ensure that different jobs running on the same node will use the same workspace.

As a concrete example, I would have 2 jobs: CodeBuild.PlatformA and CodeBuild.PlatformB, building on a pool of N machines with some label. We set the custom workspace of both jobs to the same path (and their p4 client names to the same as well) so that we only have to incrementally sync source between runs of PlatformA and PlatformB.

Without custom workspaces, we either have to allocate syncs for each job (in reality, this gets to upwards of 80 jobs per branch) or sync clean every time.

As a secondary point, with longer job names we have also run into path length issues when using the default workspace path. However, the above use case is definitely the primary motivation.

andrew.bayer@gmail.com (JIRA)

unread,

Jan 17, 2017, 6:59:04 PM1/17/17

to jenkinsc...@googlegroups.com

Andrew Bayer commented on

Re: Allow custom workspaces in declarative pipeline

Ah, ok, so in Scripted Pipeline, you'd be using the ws("some-path") step, I take it?

ssutherland@ea.com (JIRA)

unread,

Jan 18, 2017, 12:47:01 PM1/18/17

to jenkinsc...@googlegroups.com

Sean Sutherland commented on

Re: Allow custom workspaces in declarative pipeline

Exactly. As it turns out, that feature was request by us too

jliao@riotgames.com (JIRA)

unread,

Feb 1, 2017, 7:06:01 PM2/1/17

to jenkinsc...@googlegroups.com

Jianfei Liao commented on

Re: Allow custom workspaces in declarative pipeline

Another use case we run into is that we have a bunch of Go projects that we rely on a specific layout (per the Golang development recommendation: https://golang.org/doc/code.html#Workspaces).

We used to be able to control this with Scripted Pipeline with a top level `ws("some-path")` step. With the Declarative Pipeline, we had to basically wrap a `script {}` block on each step to achieve the same result.

andrew.bayer@gmail.com (JIRA)

unread,

Feb 22, 2017, 6:38:01 PM2/22/17

to jenkinsc...@googlegroups.com

Andrew Bayer commented on

Re: Allow custom workspaces in declarative pipeline

Ok, thinking about this again. For the moment, I'm leaning towards doing this as a top-level option and therefore only working on a top-level agent, but if there's sufficient interest, we could make it a per-stage option too. I thought about doing it as an option on agent but I'm not in love with the idea and can't quite figure out how I'd do it for all possible agent extensions without some frustration going forward.

andrew.bayer@gmail.com (JIRA)

unread,

Feb 22, 2017, 6:40:02 PM2/22/17

to jenkinsc...@googlegroups.com

Andrew Bayer started work on

Change By:	Andrew Bayer
Status:	Open In Progress

nevin2792@gmail.com (JIRA)

unread,

Feb 22, 2017, 6:48:02 PM2/22/17

to jenkinsc...@googlegroups.com

Conor Nevin commented on

Re: Allow custom workspaces in declarative pipeline

I'd definitely vote for having it as a per stage option. From a Golang perspective we need to checkout into a subdirectory and then execute all further commands from that directory. Having this option available only on the top level means that wouldn't be possible

andrew.bayer@gmail.com (JIRA)

unread,

Feb 22, 2017, 7:01:03 PM2/22/17

to jenkinsc...@googlegroups.com

Andrew Bayer commented on

Re: Allow custom workspaces in declarative pipeline

Conor Nevin Do you mean a new agent for each of those stages? Can you give me a sense of what your current setup would look like?

andrew.bayer@gmail.com (JIRA)

unread,

Feb 22, 2017, 7:05:03 PM2/22/17

to jenkinsc...@googlegroups.com

Andrew Bayer commented on

Re: Allow custom workspaces in declarative pipeline

A bit of clarification - if all your stages are running in the same agent, then the top-level option is sufficient - because all the stages would be running side the ws(...) block. A per-stage option would only be needed if either you're running stages on different agents and need to be able to do this on each one, or if you only want some of the stages to be running in the custom workspace, not all of them.

nevin2792@gmail.com (JIRA)

unread,

Feb 22, 2017, 7:06:01 PM2/22/17

to jenkinsc...@googlegroups.com

Conor Nevin commented on

Re: Allow custom workspaces in declarative pipeline

Not as an agent, but as an option on each stage. Currently our Golang scripted pipelines checkout into a subfolder, and the subsequent build steps are executed inside a ws() wrapper pointing towards that subfolder

Ideally, I'd like to be able to

andrew.bayer@gmail.com (JIRA)

unread,

Feb 22, 2017, 7:09:10 PM2/22/17

to jenkinsc...@googlegroups.com

Andrew Bayer commented on

Re: Allow custom workspaces in declarative pipeline

Initial PR up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/123 - this just provides the option at the top-level, but since that's where the checkout happens in most cases, it seemed reasonable enough, and it gives us a place to start further discussion. =)

andrew.bayer@gmail.com (JIRA)

unread,

Feb 22, 2017, 7:09:11 PM2/22/17

to jenkinsc...@googlegroups.com

Andrew Bayer updated

Jenkins /

Change By:	Andrew Bayer
Status:	In Progress Review

andrew.bayer@gmail.com (JIRA)

unread,

Feb 22, 2017, 7:22:04 PM2/22/17

to jenkinsc...@googlegroups.com

Andrew Bayer commented on

Re: Allow custom workspaces in declarative pipeline

And it turns out I need to have it as an agent option anyway in order to have it kick in after we enter the agent's node block but before we do the SCM checkout, so hey, that means it automatically works per-stage too.

andrew.bayer@gmail.com (JIRA)

unread,

Feb 23, 2017, 7:34:01 PM2/23/17

to jenkinsc...@googlegroups.com

Andrew Bayer commented on

Re: Allow custom workspaces in declarative pipeline

FYI, I've just sent the following to the Jenkins dev list:

We're trying out a design review/RFC process for Declarative, with an RFC doc added to a PR for discussion. The RFCs will live in the rfcs subdirectory of the pipeline-model-definition-plugin.git repository, and in this case, is included in the already-in-progress PR (https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/123) at https://github.com/abayer/pipeline-model-definition-plugin/blob/jenkins-41118/rfc/JENKINS-41118-custom-workspaces.md. In the future, RFCs may continue to be included with initial/prototype/proof-of-concept code in a PR, or may be in their own distinct PRs before code comes.

This particular RFC is for https://issues.jenkins-ci.org/browse/JENKINS-41118, adding custom workspace support to Declarative. Comments and thoughts are welcome, either on the RFC doc in the PR or in this mail thread. Thanks!

andrew.bayer@gmail.com (JIRA)

unread,

Feb 23, 2017, 7:38:03 PM2/23/17

to jenkinsc...@googlegroups.com

Andrew Bayer commented on

Re: Allow custom workspaces in declarative pipeline

Forgot to note, you can comment on the RFC doc in the PR itself, at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/123/files#diff-97b9ea96a3eef68edab62f56858e313d

jglick@cloudbees.com (JIRA)

unread,

Feb 24, 2017, 10:00:02 AM2/24/17

to jenkinsc...@googlegroups.com

Jesse Glick commented on

Re: Allow custom workspaces in declarative pipeline

How common is this requirement really? Seems like people who really need this level of control could just stick with scripted Pipelines. You are setting yourself up for forever adding tuning knobs to Declarative that will eventually become unmaintainable.

jdavis@ipswitch.com (JIRA)

unread,

Feb 24, 2017, 10:58:04 AM2/24/17

to jenkinsc...@googlegroups.com

Jason Davis commented on

Re: Allow custom workspaces in declarative pipeline

Without surveying, who knows, but for me it hits about 5-10% of builds. Another mentioned the same above - so I would think that it must be a reasonable requirement. My experience, wherever I work, it seems there's some legacy Windows build that is not going to get re-architected and has a really long path where only a custom workspace will help get the path short enough to compile. Without some way to control the workspace, there's no way to pipeline for those jobs and so I'm stuck with Freestyle, which poses other "how-to" problems like MultiSCM plugin bugs not being fixed due to pipelines. On a side note, reconciling both pipelining methods and providing a standard Jenkins "way" to do this would be even more useful moving forward, but that's clearly a different story...

andrew.bayer@gmail.com (JIRA)

unread,

Feb 24, 2017, 11:24:03 AM2/24/17

to jenkinsc...@googlegroups.com

Andrew Bayer commented on

Re: Allow custom workspaces in declarative pipeline

It is true that an alternative approach could be something like this:

 
                                                                pipeline {
  agent {
    label "something"
  }
  options {
    skipDefaultCheckout()
  }
  stages {
    stage("foo") {
      steps {
        ws("some-path") {
          checkout scm
          // rest of stage
        }
      }
    }
    stage("next") {
      steps {
        ws("some-path") {
          // next stage
        }
      }
    }
  }
}
 
                                                            

jglick@cloudbees.com (JIRA)

unread,

Feb 24, 2017, 1:56:03 PM2/24/17

to jenkinsc...@googlegroups.com

Jesse Glick commented on

Re: Allow custom workspaces in declarative pipeline

has a really long path where only a custom workspace will help get the path short enough to compile

This is a general problem of workspace management in Jenkins, which is already filed. Specifying a custom workspace in a Jenkinsfile is not a great solution; it adds an implicit dependency on the file layout of the node being used for the build (making it harder to scale up with many agents). There should not be collisions between unrelated jobs, though (see JENKINS-36358 for the opposite request).

scm_issue_link@java.net (JIRA)

unread,

Mar 3, 2017, 8:18:03 PM3/3/17

to jenkinsc...@googlegroups.com

SCM/JIRA link daemon commented on

Re: Allow custom workspaces in declarative pipeline

Code changed in jenkins
User: Andrew Bayer
Path:
pipeline-model-definition/pom.xml
pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/options/impl/WorkspaceDir.java
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AbstractModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java
pipeline-model-definition/src/test/resources/inWorkspace.groovy
pom.xml
http://jenkins-ci.org/commit/pipeline-model-definition-plugin/be0509caf79519d6c6c7706982a352d6718a6919
Log:
JENKINS-41118 Allow specifying top level ws(...) block

scm_issue_link@java.net (JIRA)

unread,

Mar 3, 2017, 8:18:04 PM3/3/17

to jenkinsc...@googlegroups.com

SCM/JIRA link daemon commented on

Re: Allow custom workspaces in declarative pipeline

Code changed in jenkins
User: Andrew Bayer
Path:

pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/AbstractDockerAgent.java
pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/Label.java
pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/options/impl/WorkspaceDir.java
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/LabelScript.groovy
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java
pipeline-model-definition/src/test/resources/inCustomWorkspace.groovy
pipeline-model-definition/src/test/resources/inCustomWorkspaceInStage.groovy
pipeline-model-definition/src/test/resources/inWorkspace.groovy
http://jenkins-ci.org/commit/pipeline-model-definition-plugin/bd2b9a673f0f40708d714adf46a21e4f032ad790
Log:
[FIXED JENKINS-41118] Rewrote to make it an agent option

Needed because we need SCM checkout to happen after the ws() call,
while with the initial implementation, SCM checkout happened before
the ws() call. As a bonus, we now can do this for any agent building
off of Label, so that's nice!

andrew.bayer@gmail.com (JIRA)

unread,

Mar 3, 2017, 8:19:02 PM3/3/17

to jenkinsc...@googlegroups.com

Andrew Bayer updated

Jenkins /

Change By:	Andrew Bayer
Status:	In Review Resolved
Resolution:	Fixed

dantran@gmail.com (JIRA)

unread,

Mar 14, 2017, 1:05:02 AM3/14/17

to jenkinsc...@googlegroups.com

dan tran commented on

Re: Allow custom workspaces in declarative pipeline

is there a official doc/howto/sample for this feature?

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

andrew.bayer@gmail.com (JIRA)

unread,

Mar 14, 2017, 1:23:01 AM3/14/17

to jenkinsc...@googlegroups.com

Andrew Bayer commented on

Re: Allow custom workspaces in declarative pipeline

There will be once https://github.com/jenkins-infra/jenkins.io/pull/715 gets merged. =)

scm_issue_link@java.net (JIRA)

unread,

Mar 15, 2017, 8:19:02 PM3/15/17

to jenkinsc...@googlegroups.com

SCM/JIRA link daemon commented on

Re: Allow custom workspaces in declarative pipeline

Code changed in jenkins
User: Andrew Bayer
Path:
content/doc/book/pipeline/syntax.adoc
http://jenkins-ci.org/commit/jenkins.io/57344406ba57e861ae0edc0f6deecb977c7dd06c
Log:
~~JENKINS-41118~~ - add customWorkspace docs

Also adding `node` `agent` type (which is just an alias for `label`,
but hey) and the `reuseNode` option for `docker` and `dockerfile`,
which we should have added earlier but forgot. Whoops.

robert.jurisch-eckardt@dpma.de (JIRA)

unread,

Mar 22, 2017, 5:58:02 AM3/22/17

to jenkinsc...@googlegroups.com

Robert RJE commented on

Re: Allow custom workspaces in declarative pipeline

This worked for me since version 1.1:

 
                                                                pipeline {
    agent {
        label {
            label ""
            customWorkspace "C:/work/${BRANCH_NAME}"
        }
    }
    stages {
        stage("foo") {
            steps {
                echo "Workspace dir is ${pwd()}"
            }
        }
    }
}
 
                                                            

Thanks to Andrew Bayer

mdgunn@gmail.com (JIRA)

unread,

Mar 28, 2017, 9:02:02 AM3/28/17

to jenkinsc...@googlegroups.com

Michael Gunn commented on

Re: Allow custom workspaces in declarative pipeline

Doesn't this solution mean you can't have top level agent as none which seems to be the 'workaround' to avoid inputs blocking?

https://issues.jenkins-ci.org/browse/JENKINS-37515

Seems that either there needs to be a fix so inputs don't block (without the workaround) or this or another fix allows you to set the workspace even if the top level agent is NONE.

If this can already be done (agent of none + custom workspace set) can anyone supply that syntax?

mdgunn@gmail.com (JIRA)

unread,

Mar 28, 2017, 9:42:02 AM3/28/17

to jenkinsc...@googlegroups.com

Michael Gunn updated an issue

Jenkins /