Question: controlling versions of Materials throughout a value stream

[Jason Whittington]

Hi all,

I am having a conceptual block reasoning about how GoCD works with repositories. Please forgive what's probably a noob question.

Suppose I have an svn repo "scripts", and a Nexus one "binaries". I have two pipelines DeployTest and DeployProduction.  

DeployTest has the scripts and binaries repos as Pipeline Materials

DeployProduction has the scripts and binaries repos as materials and also a dependency on some stage.of DeployTest

The behavior then is:

   1) Developer checks scripts or binaries into the repos

   2) DeployTest gets the latest version of the scripts and binaries and executes the scripts in the Test environment.

   3) Tests are executed and on success are cleared for production

   4) DeployProduction is triggered. It gets the latest version of the scripts and binaries and executes the scripts in the Production Environment

Cool!  But then I have a more complicated workflow

   1) Developer checks scripts or binaries into the repos

   2) DeployTest gets the latest version of the scripts and binaries and executes the scripts in the Test environment.

   3) Tests are executed and on success are cleared for production

   4) Developer checks a new version of the scripts or binaries into the repos

   5) DeployProduction  is triggered.  It gets the latest version of the scripts and binaries and (since new changes were checked in in step 4)  executes the new version of the scripts in the Production Environment

This latter workflow is problematic. I can wind up deploying bits to production without having them pass testing!  When DeployProduction is triggered I want to to deploy the bits that passed testing in step 3, not the version that was checked in in step 4.

What is the best way to accomplish that?  I thought maybe I should pass the scripts and binaries from DeployTest to DeployProduction as Artifacts but that doesn't seem to be the right way either.  

Is there a reasonable way to pass version information along the pipeline to control which versions are fetched and used by downstream pipelines?

 

Thanks!

Jason

[Aravind SV]

Hi Jason,

From what I understand, DeployProduction is a manual pipeline and so you trigger it manually. Is that right? Triggering manually will always force the pipeline to pick the latest (successful) revision of all its dependencies and that's why you're seeing what you're seeing.

One approach could be to use "Trigger With Options" to pick the correct revision of DeployTest and the repo material. However, that's tedious and needs you to know the right revisions. Another (better) approach is detailed below. It's from an article, coming up soon, which talks about patterns of modeling pipelines. Here's an excerpt from it, which is related to your question:

----------------

Instead of making the first stage of a pipeline manual, it’s often more convenient to have a dummy first stage and a gate (manual stage) after it. This prevents a user from having to trigger with options and make a mistake in choosing the available upstream dependencies. More importantly, the pipeline history page in GoCD can then be used to see all available builds and choose one to take further. In the configuration, it can look like this:

In the pipeline history page in GoCD, it will look like this:

In a fan-in scenario, this approach becomes even more important. Without this (having the first stage be manual), it will be possible for a user to pick upstream dependencies which are not compatible - meaning, using a run of “Upstream_1” pipeline which ran with SVN revision 1 combined with a run of “Upstream_2” pipeline which ran with SVN revision 2. By using the dummy stage approach, fan-in will apply to the pipeline first and an incompatible combination of the two upstream pipelines will not lead to the “Dummy stage” being triggered at all. So, overall it is safer to use this approach.
----------------

Once you make the first stage of production automatic (and dummy) and the second stage manual, you don't need to trigger the production pipeline, but can instead use the manual gate to allow the pipeline to progress beyond the first stage. Fan-in will take care of your concern regarding version mismatch between the repositories and the upstream (DeployTest) pipeline.

Let me know if that doesn't make sense.

Cheers,

Aravind

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jason Whittington]

 

Thanks for the speedy reply.  Yes, I do have some setups where pipeline B is manually-invoked pipeline and I have wondered if I need to add the "dummy-first-stage-with-manual-gate" as you describe it.  I had one pipeline that was displaying this behavior that I thought was automatically triggered but I just went to look at it and Automatic Pipeline Scheduling is turned off.

The only thing I don't like about this is the fact that UI is not intuitive for manual stages. For example, suppose I have a support person Susan who is responsible for deploying code to production at a certain time.  It's much easier for me to show her "go to this page and click this 'play' button" than it is "go find the pipeline, click it, then find the manual stage and click it to run".  I guess I can build a UI that uses the REST API and hides that from here but that might be something that could be addressed in a future GoCD build.

Thanks again for the help!

Jason

[Aravind SV]

Oops, forgot to reply.

On Wed, Oct 19, 2016 at 2:40 PM, Jason Whittington <jason.wh...@gmail.com> wrote:

It's much easier for me to show her "go to this page and click this 'play' button" than it is "go find the pipeline, click it, then find the manual stage and click it to run".  I guess I can build a UI that uses the REST API and hides that from here but that might be something that could be addressed in a future GoCD build.

Yes, it can definitely be a change that can be in the app (no concrete plans at this time). Some related discussions are in #429 and #1924.

[Andrew van Rooyen]

On Wednesday, October 19, 2016 at 11:40:50 PM UTC+2, Jason Whittington wrote:

It's much easier for me to show her "go to this page and click this 'play' button" than it is "go find the pipeline, click it, then find the manual stage and click it to run".  I guess I can build a UI that uses the REST API and hides that from here but that might be something that could be addressed in a future GoCD build.

Not that this helps you, but I am using an in-house UI similar to what you described. Its job is to track versions in various projects through states like "alpha", "beta", "stable" etc. I have a pipeline in GoCD which talks to this app via a REST api and triggers whichever deployment pipelines it needs to.

For example, with the production environment it would ask the versioning app what the current "stable" is, then compare that to what's actually deployed. If the versions are different, it triggers the deployment pipeline (via the gocd api) with the correct materials. It knows what the pipeline materials are, because I have a script that interrogates the upstream testing pipeline to find the Pipeline_run/Stage_run that corresponds to that particular version.

This might sound quite convoluted, but it works quite well, and non-technical people have a nice dashboard which lets them deploy with the click of a button.

[Aravind SV]

That sounds nice. Would be nice to know more about it.

If you don't mind talking about it, I'd love to talk to you about it sometime as well. Maybe some of those ideas can be rolled into GoCD itself, and help everyone. If you'd like to do that, please send me a message off this list.

Thanks,

Aravind

--

[Jason Whittington]

I agree.  Sounds like exactly what I would want to build. I look forward to cloning the repo and trying it myself :)

To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/1H5My9VP-Ng/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.