Workflow question vs. pipelines

[Nigel Magnay]

I'm migrating a lot of fairly hairy infrastructure around to use jerkins-workflow.

At the moment, we have a number of projects that use triggering: -

proj-build

proj-ITU

proj-robot-tests

What I want to migrate is - proj-build builds every commit. proj-ITU is very expensive, so it gets triggered on successful proj-build builds; but - it only builds against master, so multiple triggerings will just queue it to build once.

I'm not sure the best way to model that in workflow (or even if that's the best option).

Something like a stage that says

1) "if there is an 'ITU' stage currently building, wait until it finishes.

2) If there is another build, where the build number is later than us, and that build started the ITU stage (or is, like us, waiting to enter that stage) then exit. (I.E: no point in running integration tests against a version that's now stale).

Is that something do-able in workflow?

[Vincent Latombe]

Hi Nigel,

as far as I understand your statement, I think the multibranch-workflow plugin (currently in beta) could help.

I'd get the current branch and only run the ITU stage only if it's currently working on master.

Vincent

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPYP83RtRLXwCSV8ZqbpoA_cBg4OSzV-G30s9RhMgYqaoP_Y5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Nigel Magnay]

We use the multibranch-workflow. It's ace! :-)

Not sure how it helps though. Our ITU suite is in the same repository on the same branch, it's currently split into a separate job in order to run at a lower cadence than the main build.

e.g:

Build #1: start 09:00. Enters ITU 09:10. Exits ITU 10:00

Build #2 : start 09:10.  09:20 Waits @ ConsiderITU until 10:00

Build #3 : start 09:30.  09:40 Waits @ ConsiderITU until 10:00

At 10:00, Build #2 does not execute ITU step, but build #3 does.

I could probably add a function, something like (pseudocode). Feels like a kind of global threadlock between projects which makes me wonder if there is already a higher-level construct already in existence.

node('build') {

   build();

   if( shouldRunItu() )

      runITU();

   ..

}

runITU() {

  stage "ITU";

}

bool shouldRunItu() {

    stage "ConsiderITU";

    while(true) {

      def builds = get_list_of_builds_for_this_project_and_branch();

      if( builds contains job in stage ITU )

           sleep for a bit;

      else {

         def later_builds = filter_builds_that_are_bigger_buildnumber_than_us(builds);

         def builds_itu = filter_only_items_passed_stage_ConsiderITU(later_builds);

         return ( builds_itu == 0 );

      }

    }

}

 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAH-zGCgaCCSoKzSHFS2i0uiiaBwA%2BQ98Raq%3D-KAmRpTE-n9HWw%40mail.gmail.com.

[Vincent Latombe]

In that case, you could try a stage with concurrency: 1

https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md#creating-stages

Vincent

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPYP83SXD_4C9GhPKRymh7OM%3DwZFdhROcgYEsOtnU7kWSNp%3Dcw%40mail.gmail.com.

[Stephen Connolly]

That sounds like you need the "James Nord operator"

Workflow can do that, because the specific use case was requested by the person it was named after

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPYP83RtRLXwCSV8ZqbpoA_cBg4OSzV-G30s9RhMgYqaoP_Y5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
Sent from my phone

[Nigel Magnay]

Ok. That looks exactly what I want, looking at [1]

So if I've correlated correctly, a

stage 'blah', concurrency: 1

Should give me what I want, from the documentation:

A concurrency of one is useful to let you lock a singleton resource, such as deployment to a single target server. Only one build will deploy at a given time: the newest which passed all previous stages.

Which is great, if a little unexpected. What if I wanted every build to go through a stage, just one at a time?

I.E: The described concurrency:1 behaviour is great - in effect don't waste time deploying already-obsolete versions into production. What about concurrency: 1 where it's just because (say) a testing stage needs access to a resource (server|license|etc) and it's a 'one at a time' - but we want *every* build to run through that stage?

A finite concurrency ≥1 can also be used to prevent slow build stages such as integration tests from overloading the system. Every SCM push can still trigger a separate build of a quicker earlier stage as compilation and unit tests.

This is confusing - it feels like 'concurrency' mixes up two concepts - how many times a stage can run in parallel and termination of jobs at a stage barrier based on a decision (is there a newer build waiting at this stage) ??

[1] https://www.cloudbees.com/sites/default/files/juc_sfo_oct_14-_workflow.pdf

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CA%2BnPnMz7vmh0D8Mbdz3Cb22zrLeRUx3vBSDLkPBjr8Nrt%2BQxOw%40mail.gmail.com.

[Stephen Connolly]

On 13 October 2015 at 08:40, Nigel Magnay <nigel....@gmail.com> wrote:
> Ok. That looks exactly what I want, looking at [1]
>
> So if I've correlated correctly, a
>
> stage 'blah', concurrency: 1
>
> Should give me what I want, from the documentation:
>
>> A concurrency of one is useful to let you lock a singleton resource, such
>> as deployment to a single target server. Only one build will deploy at a
>> given time: the newest which passed all previous stages.
>
>
> Which is great, if a little unexpected. What if I wanted every build to go
> through a stage, just one at a time?
>
> I.E: The described concurrency:1 behaviour is great - in effect don't waste
> time deploying already-obsolete versions into production. What about
> concurrency: 1 where it's just because (say) a testing stage needs access to
> a resource (server|license|etc) and it's a 'one at a time' - but we want
> *every* build to run through that stage?

Welcome to the fun that is the "James Nord operator"

I suggest you have a chat with James, it was his use case after all ;-)

> https://groups.google.com/d/msgid/jenkinsci-users/CAPYP83RA2ZT6m5GJPRRqfiNAjOOuBQGPOj2J8SCjX-uHiKKN_Q%40mail.gmail.com.