Pipeline not waiting for previous test pipeline to complete

[Joel Bodenmann]

Hello folks,

Consider the following scenario:

• Template T_A: Builds some binaries and stores them as an artifact.
• Template T_B: Fetches binaries from the upstream pipeline T_A and runs tests. To achieve this,  T_B has a Fetch Artifact task with the upstream pipeline as a parameter #{UPSTREAM_BUILD_PIPELINE}.
• Template T_C: Fetches binaries from upstream pipeline T_A and creates packages. To achieve this,  T_C has a Fetch Artifact task with the upstream pipeline as a parameter #{UPSTREAM_BUILD_PIPELINE}.
• Pipeline A: Uses template T_A.
• Pipeline B: Uses template T_B. Sets UPSTREAM_BUILD_PIPELINE  to pipeline A.
• Pipeline C: Uses template T_C. Sets UPSTREAM_BUILD_PIPELINE  to pipeline A.

This results in the following VSM:

The problem I have is that pipeline C will be triggered twice: Once after A finishes and once after B finishes. The expected behavior (or the behavior I wish to achieve) is that C will only run after B finishes successfully.

How can I make sure that C only runs after successful completion of B (and therefore implicitly A)?

Attached is the corresponding XML fragment of my templates & pipelines where:

• T_A -> Build-CMake-OS-Compiler
• T_B -> Test-OS
• T_C -> Package-OS
• A -> GPDS-Build-FreeBSD-GCC
• B -> GPDS-Test-FreeBSD-GCC
• C -> GPDS-Package-FreeBSD

[Joel Bodenmann]

On Friday, November 27, 2020 at 3:17:36 PM UTC+1 Joel Bodenmann wrote:

Adding the VSM image didn't seem to work, here we go again:

[Ketan Padegaonkar]

If pipeline "C" requires a binary from "A", there's no need to add an explicit dependency. You can specify an upstream pipeline as part of your fetch artifact task.

In either case, C should not trigger twice. Pipeline C should wait for B to be complete before kicking off.

Also — what you're looking at is called "fan-in" in gocd terminology. This document might help.

- Ketan

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/0a8a6dc4-0b49-4036-a573-44d0feca32b9n%40googlegroups.com.

[Joel Bodenmann]

Thank you for your reply, Ketan.

I've come across the fan-in & fan-out capabilities - which is one of the main reasons I decided to give GoCD a try and hopefully switch to it eventually.

I've read the documentation you've linked prior to creating this conversation. However, I'm unclear as to how to modify my pipeline C (or rather template T_C) so there's no more explicit dependency on the upstream pipeline. In my opinion I did this by having the Fetch Artifact task in T_C which allows C which uses T_C to specify the upstream pipeline with #{UPSTREAM_BUILD_PIPELINE}.
What am I missing / doing wrong?

In my pipeline C -> Materials I am unable to remove the dependency to the upstream build pipeline A. As far as I can tell this has been added automatically when creating pipeline A from template T_A as T_A requires the #{UPSTREAM_BUILD_PIPELINE} parameter for the Fetch Artifact task.

~ Joel

[Jason Smyth]

Hi Joel,

I think the missing piece may be that #{UPSTREAM_BUILD_PIPELINE} does not need to be a direct parent of the Pipeline where the Fetch Artifact Task resides.

I recommend the following changes to Pipeline C:

1. Change the value of #{UPSTREAM_BUILD_PIPELINE} from Pipeline A to Pipeline A/Pipeline B.
2. Remove the Pipeline A Material.

I modified your original XML document to show what I mean.

Hope this helps,

Jason

[Joel Bodenmann]

Jason,

Thank you very much for clarifying.
I understand what caused the issue and I managed to modify my pipelines according to your guidance and it appears to be working as per my liking now.

Thanks a lot guys!

Best regards,

~ Joel