Pipeline templates and automatic pipeline scheduling

[Manheim]

I currently have 2 pipeline templates for my application: one that builds the binary artifacts, and another downstream pipeline deploys the artifacts into an environment. I'd like to have the downstream deployment pipeline triggered automatically from the first pipeline into an acceptance test environment, where automated tests are run. I'd also like to have a manually triggered instance of the downstream deployment pipeline for deployments into production. Unfortunately, it looks like you can't control automatic pipeline scheduling (or the Stage Type of the first stage) at the pipeline level; it is controlled completely by the template. This means I can't use the same template (which contains 4 fairly complex stages) for both automated and push button deployments. Has someone found a workaround for this limitation? 

[Jason D]

Just want to echo that we, too, would like to see the auto/manual trigger "templatizable" in some fashion, or pulled out of the template all together.

 

I do know Thoughtworks has played with this in the past.  I believe it was even in one of the releases before being pulled out again for some reason.  Maybe someone on that team can add some commentary as to why this might be difficult to implement.

 

As a workaround, you could set all your pipelines to auto trigger and just add a dummy, manual pipeline before the ones you don't want to auto trigger.  This has the potential to explode the number of pipleines you may need which increases the maintenance and related.  It also could alter any stats you may be tring to extract from GO (e.g. pipeline count, success rate of pipelines in prod, etc...).  We started down this route and quickly abandoned it for those reason.  We just manually trigger all our deployment pipelines to every environment which isn't a big deal. 

 

This is how the workaround would look, as an example, for clarity:

BuildPipeline(auto)<-DevPipeline(auto)<-TestPipeline(auto)<-DummyPipeline(manual)<-ProdPipeline(auto)

 

Dev, Test & Prod could all use the same template.  The "DummyPipeline" would need to be manually triggered in order for the ProdPipeline to trigger.

 

The ability to templatize more than just a pipeline (e.g. stage, job, task...) ultimately would solve this as well and add more ability to reduce config size, standardize functionality, among other benefits.

[Sachin Sudheendra]

<history>

As Jason mentioned, Go did for a single release support this use-case by adding the ability to parameterize the type attribute for <approval />. While this did give the flexibility for templates to parameterize the approval type, and the pipelines which consume the template to define the value for the parameter, we could not come up with an intuitive UI for this configuration. We did several iterations on the prototype, but just could not come up with a UI which would convey the intent. The complexity was around the fact that more than one go-concept was tying into this. A novice user would have to understand Go-Parameters, Approval Types & Template-Pipeline relations in order to use the feature as intended. After a while, we took a step back and wanted to verify if parameterization is indeed the only way to achieve this (or) would a better/cleaner implementation could be done for this. Hence, we reverted this in the subsequent release, and unfortunately we did not get back to it post that.

</history>

What do you guys think about the implementation? Do you think we can achieve this in any other way? Also, as Jason suggested, templating more than just a pipeline would, to some extent, add value around this.

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sachin Sudheendra

[Marius Ciotlos]

Hi all, 

From the way I see it, the best way would be to have:

- Default behaviour what we currently have: automatic pipeline scheduling inherited from template

- To override this behaviour to be overwritten with a specially named Pipeline Parameter (something similar to GO_ environment variables). Most users won't need this, and those that do, will read the documentation to find out how that parameter is named, etc. 

From what I've seen current pipeline scheduling is based on the first task in the pipeline (Manual vs Automatic). I'm curious how complex such a change would be. 

Cheers,

Marius

[Damian Stanger]

Is this still how the system works?

reason i ask is that i have many deployment templates (currently 17 and growing every week) deploying to 6 environments. As the other comments in this thread we want to auto deploy to the test environment so we can perform acceptance tests then have push button to the other 5.

but as has been quoted a template can only have either success or manual trigger (it is not based on the pipeline)

i was thinking of a work around that utilised the go api to trigger the test pipeline when all the unit tests are successful

is this a good idea or can you see any downsides to this approach

the value stream map will still be valid and we will still have manual deployments to the more critical environments

thanks

Damian

[Damian Stanger]

Ive created a stage after the test steps with a task in which runs the following powershell command

Invoke-WebRequest -Method post -Uri http://goserver:8153/go/api/pipelines/#{SCHEDULE_PIPELINE}/schedule -Headers @{'Authorization' = '%AUTOMATION_AUTH%'}

where SCHEDULE_PIPELINE is the name of the pipeline to kick off next

and AUTOMATION_AUTH is 'Basic bmljZTp0cnkhIQ==' which is the user name and password base 64 encoded

This gets round the problem of manually triggered templates but one pipleline (deploy to UAT) being triggered automatically after successful build and test 

[Mark Gibson]

Is there any official news on this topic?  I had been considering looking in to fixing this myself, but I see it has a lot of history and complexity which is probably not ideal for a noob GoCD dev to get involved in.

[Damian Stanger]

I've not heard otherwise, we are still doing the hack that I proposed last year.

if there is a better way then id love to hear it but as far as I know the template type still governs the triggering behaviour

On Friday, 26 June 2015 12:38:29 UTC+1, Mark Gibson wrote:

[Feld0]

I found a way to achieve what this thread is about: setting a timer trigger to run the pipeline every second, with the "run only on new material" option enabled (see screenshot below). This works the way I need it to and is a relatively "clean" solution.

The timer string, for easy copy-pasting: * * * * * ? *

[Ketan Padegaonkar]

On Fri, Jan 1, 2016, 9:00 AM Feld0 <pe...@deltchev.com> wrote:

The timer string, for easy copy-pasting: * * * * * ? *

I would probably dial that down to run every minute instead of running every second.