[workflow-plugin] is it possible to use the stage step to serialize execution of builds?

[Alexander Bertram]

Hi there,

I'm using workflow plugin to automate a workflow where new features are submitted as pull requests to the master branch of a git repository. Once the feature has been tested independently we schedule a 'release' workflow job that is meant to:

1. Merges the pull-request locally

2. Builds and verifies the resulting snapshot

3. Deploys the snapshot to a staging server and runs integration tests on the staging server

4. If all is well, push the resulting merge to the remote repo and deploy the build to production.

This is works really well until multiple branches are queued for release at the same time. I've tried to use the stage step to serialize execution of these release builds so that if there are two or three branches to release at the same time, they are processed in sequence. 

I've tried using the 'stage' step, but this will actually abort waiting builds if newer builds enter the queue. That's to avoid successive builds from the same branch, but here I want all builds to run, just in sequence. Workflow jobs don't appear to have the option to limit concurrency through the UI as far as I can tell. 

I'm not sure I fully understand how the concurrency parameter works, though: the TUTORIAL.md doc says that:

A concurrency of one is useful to let you lock a singleton resource, such as deployment to a single target server. Only one build will deploy at a given time: the newest which passed all previous stages.

But it seems from the StepExecution.java that a build queued to enter the stage are still aborted if a newer builds enters the stage. Could we add a parameter to the step to determine whether new builds entering the stage preëmpt older builds?

Thanks,

Alex

[Jesse Glick]

> I've tried using the 'stage' step, but this will actually abort waiting builds if newer builds enter the queue. That's to avoid successive builds from the same branch, but here I want all builds to run, just in sequence.

Right, ‘stage’ is intended for the use case where a newer build would supersede the earlier.

> Workflow jobs don't appear to have the option to limit concurrency through the UI as far as I can tell.

No, not currently. One could be added, I think fairly easily. I initially omitted such an option intentionally, so that ‘stage’ would be used for this. There are probably some legitimate use cases for it though.

An alternative would be a different block step that controls concurrency without aborting. This would be more flexible than a job option, because you can limit the throttling to just the necessary part of the build: in your case, the local testing could presumably be run in parallel.

Or, ‘stage’ could be given an option to throttle concurrency in the way you require: a simple “fair” latch, where the first build enters immediately, and subsequent builds just wait until it is done, entering in the order they started waiting.

You could probably even use ‘stage’ as is: use a _computed_ stage name, based on the branch name or PR number being tested. This might really be what you want. Consider someone who files a PR with one commit. The build starts up and gets into the integration tests on the staging server. Then they push another commit to the PR, triggering a new build; and then in short succession a third. Then the integration tests from the first commit fail (because there was really a mistake that the tests caught, corrected later). The ideal behavior (perhaps) is that the second build is canceled, and the third runs instead. That is what

    stage "ITs for PR #${number}"

should do, without blocking or canceling builds of unrelated PRs. (If this indeed works for you, it would be great to have a pull request strengthening StageStepTest to prove it.)