Approaches for sharing workspace in a pipeline

[John D. Ament]

Hi,

I was wondering if anyone had any best practices or tips to share on have a common workspace for a pipeline job.

Basically, I have a series of pipeline jobs and I want them to have a single workspace for the duration of the job chain.  I compile the artifacts once, running unit tests, followed by a suite of integration and BDD tests.  It's a fairly complicated build, including generating an app server and minifying a lot of javascript for our UI.  Some of these steps are pretty long, and in total we have 4 pipeline steps.  I figure by doing this once, I would cut out about 40 minutes of rebuild time in my pipeline.

One idea I had was to use the clone workspace plugin to copy them, https://wiki.jenkins-ci.org/display/JENKINS/Clone+Workspace+SCM+Plugin, but it seems like this isn't pipeline sensitive since each step in the pipeline should be building the same commit.  I also thought about copying artifacts, but it seems like its a huge number of artifacts.  Could I build a zip with the contents?

Any thoughts?

Thanks!

John

[Baptiste Mathus]

Hi John,

Not sure what you call a pipeline job, do you mean 'workflow job'? or do you use the term in a generic way and actually have many (freestyle) jobs you're coordinating?

If the latter, then it really seems like a use case for a workflow job (using the workflow plugin). Using/archiving etc. artifacts and being able to share the ws during the build is gonna be both more natural and more maintainable (and more robust because of the durability).

My 2 cents

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3c25d084-56c8-4242-a6e0-a0b347ea0af8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John D. Ament]

Hi,

I definitely thought about workflow.  It looks promising.

One question though - I can't seem to archive/unarchive everything.  It looks like to use unarchiver you need to know the paths that will be exposed, and instead I'd like to just get everything, including class files.  Is that possible?

John

[Brian Ray]

Isn't the "get everything" Ant regex something like ** or **/*?

If you continue looking at workflow, also check out the stash and unstash steps. Similar purpose and syntax but more applicable to intermediate stages where you don't need to retain the artifacts. You can also refer to the batch of stashed artifacts by an arbitrary logical name for downstream unstashing.

[John D. Ament]

Creating the archive doesn't seem to be an issue.

Its the unarchive step where things don't quite work for me.  If I read this info, it implies that the mapping step is not required.  https://github.com/jenkinsci/workflow-plugin/blob/master/basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/ArtifactUnarchiverStep/config.jelly#L31

However, if mapping isn't set, an exception gets thrown: https://github.com/jenkinsci/workflow-plugin/blob/master/basic-steps/src/main/java/org/jenkinsci/plugins/workflow/steps/ArtifactUnarchiverStepExecution.java#L39

And in fact, the expanding of the archive is keyed off of this mapping config.  https://github.com/jenkinsci/workflow-plugin/blob/master/basic-steps/src/main/java/org/jenkinsci/plugins/workflow/steps/ArtifactUnarchiverStepExecution.java#L42

It looks like it may be straight forward enough to add an unarchive all option, and if that works I may try that out instead.

John

[John D. Ament]

Ok, so I think I got something.  If I use "**/*" on the unarchive step, I get everything.  Not quite what I expected, but it works.

[Brian Ray]

Ah yes, that help seems a little misleading with the wording "May take a mapping parameter ..."

[Martin d'Anjou]

On Wednesday, December 23, 2015 at 9:45:57 PM UTC-5, John D. Ament wrote:

Hi,

I was wondering if anyone had any best practices or tips to share on have a common workspace for a pipeline job.

I use Workflow job definitions, and I compute the location of the workspace in each job of the build chain before the actual build actions take place, using the ws() built step. Subsequent jobs in the build chain use the same workspace (the location can be computed). Subsequent jobs need not to be in the same workflow definition, nor in a formal pipeline (I do not use triggers nor pipeline plugins). I have enough input parameters for the subsequent jobs in the chain to locate the workspace of the previous job. I do not move nor archive data (I have thousands of files and gigabytes of data over NFS so I don't want to spend time moving it or archiving/extracting it).

[Thomas Goeppel]

Hi John,

your approach sounds interesting. Could you please share some Workflow code that shows how you compute the Workspace location, use ws(), and pass workspace locations to subsequent jobs?

When accessing workspaces in Workflow you're also facing the problem of concurrent runs. How do you deal with that?

Thanks in advance!

Greetings,
Thomas

[Thomas Goeppel]

Oh sorry, wrong addressee 

%s/John/Martin/ :-)


On Wednesday, December 30, 2015 at 11:24:26 AM UTC+1, Thomas Goeppel wrote:

Hi John,

[Baptiste Mathus]

Would be great if y'all around could share the cool things you've done with workflow in that examples/tutorial repo: 

https://github.com/jenkinsci/workflow-examples

Thanks!

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/53be2f41-9a45-4af5-a952-c49f53ef94b1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Baptiste <Batmat> MATHUS - http://batmat.net
Sauvez un arbre,
Mangez un castor !

[Martin d'Anjou]

Producer code:
String root = "/opt/martin"
String workspace = "${root}/${env.JOB_NAME}/${env.BUILD_NUMBER}"
node() {
    ws(workspace) {
        writeFile file: 'file.txt', text: "1234"
    }
}

Consumer code:
String root = "/opt/martin"
String folder = env.JOB_NAME.split('/')[0..-2].join('/')
String workspace = "${root}/${folder}/producer/${PRODUCER_BUILD_NUMBER}"
node() {
    ws(workspace) {
        String text = readFile 'file.txt'
        echo text
    }
}

The consumer has an input parameter called PRODUCER_BUILD_NUMBER, and that is of course annoying for users to type it in. What I want for this parameter is to give users the option to choose a producer build number, or if they leave it empty, the most recent successful producer build number is used. Finding this most recent producer build number programmatically is a bit of a challenge currently but in the past with freestyle jobs I have done it with curl. With the Workflow I would prefer to use a plugin based on a groovy RESTClient or the apache fluent api. If I understood how to return an object to the workflow (as opposed to perform a build action) perhaps I would be able to write a decent REST API plugin but I don't write plugins for a living so it would take me a lot of time to learn to do this. I am also aware of Jenkins Workflow - Creating a Class to Wrap Access to a Secured HTTP Endpoint, but it uses the Global Library which is not a good long term solution yet as explained in JENKINS-31155.

Concurrent consumer builds: the workspace has to be copied before the build jumps into action, unless you are certain that no files will collide on write. For me copying the workspace is still cheaper than re-creating it from scratch, so I will go for a full copy. Code is TBD but the easiest is probably a simple unix copy inside an "sh" build step.

Martin