Running clean up tasks after failure

[robin.s...@clearpoint.co.nz]

Hi,

I've currently got a three staged pipeline where in stage 1 I tear up a test environment. In stage 2 I run e2e tests against that environment and in stage 3 I tear the environment down. If the tests fail in stage 2 I can't see any way of saying still run stage 3 to tear the environment down. Is that possible?

[Aravind SV]

Stages can't be forced to run on failure, but tasks can be.

This page about concepts in Go might be useful too.

On Thu, Oct 8, 2015 at 2:33 PM, <robin.s...@clearpoint.co.nz> wrote:

Hi,

I've currently got a three staged pipeline where in stage 1 I tear up a test environment. In stage 2 I run e2e tests against that environment and in stage 3 I tear the environment down. If the tests fail in stage 2 I can't see any way of saying still run stage 3 to tear the environment down. Is that possible?

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[robin.s...@clearpoint.co.nz]

Hi,

I've read the concepts and had a look through the build.go.cd pipelines as an example of when to use stages v tasks.

I've got the scenario where I tear up an environment to test against, run the tests and then tear the environment down. The tear up takes a while to do in series - but it could be sped up if things were done in parallel. However if tests fail, or the tear up fails I still want the tear down to run. I've got that going with a single job pipeline that has three tasks, but the only way I can parallelize the tear up is if the different parts are all jobs, but then I'm back to not being able to tear-down.

Does this sound like a valid use case for a feature request to allow subsequent stages to run on failures just like tasks?

[Aravind SV]

On Fri, Oct 23, 2015 at 2:34 PM, <robin.s...@clearpoint.co.nz> wrote:

Does this sound like a valid use case for a feature request to allow subsequent stages to run on failures just like tasks?

I think it might makes sense. Though, I think it might look weird for a different stage to run after failure. Would a post-stage-failure set of tasks (or a job), defined at the stage level work, in this case? I'm thinking setup and teardown tasks (or job) at the stage and pipeline levels.

If an on-failure stage is allowed, the pipeline might look really weird, if stages 1 through 4 are normal stages and stage 5 is an "on-failure" stage, then if stage 2 fails, it'll look like:

Green - Red - Grey - Grey - Green

I think there is a modeling element missing here, but I'm unsure that on-failure stage is the right one. That's why I was thinking of an on-failure job, guaranteed to run, only when one of the other jobs fails and after all other jobs are finished.

[robin.s...@clearpoint.co.nz]

If I'm understanding correctly:

I'd have a stage for setup with a bunch of jobs that can run in parallel and get my environment up quickly. I'd have a test stage running the tests. Then, with your scenario both stages would need to define a post-stage failure job to run (which would be the same job but just to handle possible failure of tear-up AND real test failures). That would work for me, but ideally, to get as fast feedback as possible, the tear-down (post-stage failure job) would also be able to do things in parallel - so in essence would be a Stage.

I wonder if you could define a stage as an "always run" stage and then you wouldn't necessarily have a Green - Red - Grey - Grey - Green, but perhaps a Green - Red - Grey - Grey - Blue or a Green - Red - Grey - Grey - Red if even the "always run" Stage fails? 

[Mark Gibson]

Just to add another voice here ...

I originally had a multi-job deployment stage.  Having multiple jobs gave two things ... 1. Quicker visibility in to where failures occurred.  2. parallelisation of deployment steps where beneficial.

When it came time to plan rollback handling on a failed deployment, I wasn't willing to maintain multiple rollback tasks configured on multiple jobs.  Conceptually, we needed one rollback sequence to be triggered at any point the deployment failed.

What we ended up with is a more concerted investment in Ansible, and a reduction in the use of the multi tiered Pipeline definition.  Our typical deployment pipeline now has a single stage with a single job with the following tasks

1. Fetch Artifact
2. Command: ansible-playbook deploy.yml
3. Command (RunIf Failed): ansible-playbook rollback.yml

And note I'm doing deployment at a pipeline level.  Whilst that "may" be the right thing to do anyway, we actually had no choice as each deployment along the road to production has exactly the same set of instruction (with a different environment).  Because templates can only be applied to Pipelines, we couldn't model a deployment template and apply it to a Stage.

Just in case I'm misinterpreted as complaining - I'm not.  Go CD is a pretty good tool, and I'm still very happy with our decision to use it.  However, we just can't use some of the more complex aspects of Pipeline/Stage configuration due to the above mentioned shortcomings.

Mark

[Fredrik Wendt]

Thanks for sharing!

/ Fredrik

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

+46 702 778511