VSM (Dependencies) triggering sequence issues

[Sifu Tian]

Hi Folks,

We have a strange scenario when trying to leverage auto trigger (Trigger on completion of previous stage)

Scenario 1:  We have Production deploy pipeline with 2 upstread dependences that need to be satisfied to trigger the pipeline (UAT Pipeline Success and Common Pipeline Success)

When UAT is triggered by changes and is successful, the pipeline is triggered even though Common has not been triggered in production since a week ago.  When we look at the VSM, it shows the Common pipeline as being successful (green) but that success was from the previous weel.  GoCD, or at least how we have it set up, doesnt incorporate the latest date or commit.  Please see screenshot.  This inadvertently triggered a production deploy which we did not want to happen.

Scenario 2:  To avoid Scenario 1 from happening again, we removed the UAT pipeline as an upstread material requirement which, we thought, seemed to trigger the production pipeline.  Now the only upsteam requirements is the Common pipeline being required successful.  Now the prod pipeline was triggered because changes were commited to our main branch, even though Common prod pipeline has not been triggered for over a week, although, in the VSM, GoCD still shows common as successful (green).

NOTE that each pipeline has its own pluggable SCM with its own specific path criteria.

I am looking for a way to have GoCD, recognize that although an upstream pipeline was successful in the past (> 1 week) it should not trigger a downstream pipeline if the production pipelines have not been triggered or successful.

I hope that makes sense.  Perhaps we have configured it incorrectly however weve had production deploys happen when we didnt want them to.

The first screenshot shows that we manually triggered the common pipeline last week.

The second screenshot shows that the Audit-prod pipeline was triggered by changes the following week, however the upstream pipeline (Common-prod) was not triggered by those changes.

Also imagine that UAT was also an upstream pipeline dependency (Marked in both screenshots which we manually triggered but still triggered prod automatically because it viewed the Common pipeline as successful, even though it was from last weeks build.

Im thinking the issue is due to each pipeline having its own unique pluggable scm with their own trigger path but yet are still all are pointing to our main branch.  

Any help determining the root cause and how to properly set this up so its triggered correctly.

[Ashwanth Kumar]

I'll probably share my side of the story from a similar experience, it was so long ago so I don't remember the exact details but we were unable to get pipelines with multiple upstream dependencies work as expected when we wanted logical AND type conditions across them. The way we solved them was to move to manual triggers and use the API to trigger the production pipelines. For example, after UAT and Common, we would have a Pre-Prod which would depend on both UAT and Common and use artifacts (version of upstream labels) to check if they've changed since the last time they ran. The idea is, both the versions should change simultaneously for the API trigger to happen. We do this via a simple bash script that exposes a new artifact only if both the labels have changed, else it just returns the same existing artifact as the output which will be consumed in the next run of the pipeline. If both the versions change at the same time, then we would use the /schedule API of the Prod pipeline to trigger it.

I'm not even sure if this is the right way, but this approach made reasoning very easy for us and a better audit on who and how our prod pipeline was triggered.

Thanks,

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/d3083bda-9a0f-40a4-b997-b4265d71feabn%40googlegroups.com.

--

Ashwanth Kumar / ashwanthkumar.in

[Jason Smyth]

Hi there,

My understanding, which Ashwanth's description seems to support, is that GoCD's design is such that a Pipeline should be triggered if any of its materials have changed, i.e., regardless of the number of dependencies a Pipeline has the answer to "has there been a change" is "no" only if none of the materials have changed.

The GoCD documentation discusses the concepts of fan-out and fan-in. Based on what I read there, I believe that fan-in only creates a logical AND condition if there is an upstream fan-out. This means that, if you want Pipeline C to depend on Pipelines A and B and only be triggered if both upstream Pipelines succeed, then Pipelines A and B must have a shared upstream dependency (e.g., Pipeline Z). Without the upstream fan-out, GoCD views Pipelines A and B as 2 distinct sources and will trigger an instance of Pipeline C when either of its sources changes.

I believe GoCD's design is correct in this regard. The principles of Continuous Integration, Deployment and Delivery state that we should test every change, no matter how small. These principles are founded on the idea of obtaining fast feedback so that we can fix mistakes as quickly as possible. With these design principles in mind, I think triggering a Pipeline if any of its materials change is the correct behaviour.

>  I am looking for a way to have GoCD, recognize that although an upstream pipeline was successful in the past (> 1 week) it should not trigger a downstream pipeline if the production pipelines have not been triggered or successful.

I reviewed the screenshots you provided and I am going to assume that there are 8 Pipelines that are relevant to this conversation: 4 each (Build, Deploy, UAT, Prod) for 2 applications (Common, Audit). I have never tried to implement a fan-in scenario, but I think the following might work for you:

Configure GoCD such that each Audit Pipeline depends on the corresponding Common Pipeline. I.E., the Audit-Build Pipeline should depend on the Audit source code and the Common-Build Pipeline, the Audit-Deploy Pipeline should depend on the Audit-Build Pipeline and the Common-Deploy Pipeline, and so on. The theory is that this design creates the necessary upstream fan-outs required for GoCD to enforce the downstream logical AND on fan-in.

One caveat here is that a successful code change in Common will cause a deployment in Audit, regardless of whether or not Audit has been changed. This is because each Audit deployment can be thought of as a combination of (Audit Code + Common Code) and the CI/CD principles described above say that we should deploy (and test) a change to either component inside our combined bundle. If this caveat is not acceptable for your use case(s), then this design probably won't work for you but may give you a decent jumping off point.

Hope this helps,

Jason Smyth

[Sifu Tian]

HI Jason,

Thanks for your feedback.

The scenerio as you described is exactly how we have it set up and the problem is indeed the successful code change will trigger the audit pipeline or any pipeline we have set up in this manner.  Since we are using the Kubernetes plugin, we havent found a way to leverage the existing clone repository at build time in the subsequent pipelines instead of listening to changes in master.

Thanks Ashwanth for your input as well.

I will try to test the manual trigger with labels to see if we can get some better results.