Multiple pipelines to trigger another pipeline

[darrin hodges]

Hi,

currently we have  number of pipeline groups in GOCD containing multiple pipelines, so:

Pipelinegroup1
a1 -> a2 -> a3 -> a4
b1 -> b2 -> b3 -> b4
c1 -> c2 -> c3 -> c4

What I'd like to be able to do have the individual pipelines (above) trigger another pipeline that is common to all. This common pipeline
is to deploy code, so when the above pipelines get to their respective stage 4, they can trigger the common pipeline to do the actually deploy as opposed to having a deploy stage for each pipeline, is this possible?

thanks
Darrin

[Marius Ciotlos]

Yes, just set as materials for the deployment pipeline, all the pipelines that you need to trigger it. If i understood correctly, a4, b4, c4 should be set as pipeline material for your deployment pipeline.

Cheers,

Marius

[darrin hodges]

Thanks Marius, I did try that but it creates errors if one (or more) of the upstream pipelines haven't gotten to their repective last stage:

Failed resolution of pipeline Fubar : Cause : Could not find compatible revision for material: DependencyMaterialConfig{pipelineName='mypipeline', stageName='Production_repo_update'}

This seems to stop any other pipeline from triggering the downstream one. Is there a way of handling those errors to it doesn't stop others from triggering the pipeline?

thanks
Darrin

[darrin hodges]

Seems to work now but how does the downstream pipeline know which upstream pipeline triggered it? I have a 'printenv' command printing to the conole but I can't see how it knows which upstrean pipeline was the trigger.

thanks
Darrin

On Thursday, October 23, 2014 1:54:52 AM UTC+11, Marius Ciotlos wrote:

[Lee Benhart]

Marius, How do you prevent the pipeline from triggering until ALL the pipeline materials have reached the correct stage.  In this scenario, if it is desired to promote after a, b, and c have reached the end stage, it is possible the deploy could happen three times, rather than 1.

Lee

[Md. Ali Ejaz]

Go picks up the correct version of upstream dependencies if the pipelines are configured properly. The fan-in feature of Go takes care of this.

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

- Ali

@mdaliejaz

[Marius Ciotlos]

You also need pipeline locking to avoid concurrent triggers

Marius

You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/8CQxCITx0Ac/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.

[Jason D]

Will "fan-in" really handle this situation?  In the scenario presented above there is no common upstream dependency of a1, b1 & c1 as is depicted in the fan-in doc.  It seems that in this case you could certainly end up with the last, common downstream pipeline (the 1 that depends on a4, b4 & c4) triggered up to 3 times as there would be no indication that it should wait for a change to occur in the other 2 pipelines. 

 

Someone please correct me if I'm wrong but that is my understanding.

 

Maybe introducing a common upstream dependency to make it a true "diamond" dependency would throw it into a category of what fan-in would handle?  Not sure if that is possible in this scenario.

[Md. Ali Ejaz]

@Jason You are correct. Fan-in wouldn't solve the problem with current setup and that's why I said 'if the pipelines are configured properly'.

And by pipelines being configured properly I meant exactly what you said 'a common upstream dependency to make it a true "diamond" dependency'. 

I thought it would be easier to explain the configuration, if the OP understands the fan-in concept first. Also, the OP didn't provide any detail on materials. But now I think, I should have gone ahead with the explanation :)

If the pipelines triggers out of the same material, the configuration is quite straight forward.

          -> a1 -> a2 -> a3 -> a4 --

         |                          |

material---> b1 -> b2 -> b3 -> b4 --+-> (the downstream pipeline)

         |                          |

          -> c1 -> c2 -> c3 -> c4 ---

However, if the pipelines trigger out of different materials, the pipelines which don't depend on a material, say M1, could still have M1 as a blacklisted material. This would still acknowledge fan-in and the required configuration could be achieved. The setup should blacklist everything in that material, may be with something like **/*.*

Each material would go as blacklisted material to the other two pipelines which don't depend on the said material. I will try to show it graphically but i'm bad at art!

material1 -> a1 -> a2 -> a3 -> a4 -------------------

  |           ^                                     |

  |           |                                     |

(bcklstd)     |                                     |

  |   ========+========                             |

  |  ||       |       |                             |

  |  ||    (bcklstd)  |                             |

  |  ||    material2 -+ -> b1 -> b2 -> b3 -> b4 ----+----> (the downstream pipeline)

  |  ||    (bcklstd)  |                             |

  |  ||     |         |                             |

  |---------+----------                             |

     ||     |                                       |

(bcklstd)   |                                       |

     ||     |                                       |

material3 --+-> c1 -> c2 -> c3 -> c4 ----------------

There is a caveat though. Pipelines wouldn't reflect changes for the blacklisted material unless the material (the not blacklisted one) triggers the pipeline. 

Example scenario: Let's say pipelines 'a' and 'b' have triggered with version 'x' of material3, and pipeline 'c' has triggered with version 'x+1' of material3, the changes wouldn't come down for version 'x+1' of material3. The blacklisting wouldn't trigger the pipelines 'a' and 'b' with 'x+1' version of material3 unless material1 and material2 trigger pipelines 'a' and 'b' respectively.

Hope this helps. Please feel free to get back if the explanation above is not clear.

Ali

@mdaliejaz