[ashok reddy C k]

Hi Team,

I am reaching out to bring attention to a critical issue we are experiencing with our VSM runs. We have observed that the VSM is frequently stalling at random points across multiple pipelines, occurring approximately 90% of the time.

While the VSM setup appears to be correctly configured—confirmed by the proper VSM linkage display in the UI—the process is not progressing as expected and has not triggered the pipelines.


We would appreciate your support in diagnosing and resolving this issue as soon as possible.

Please let us know if you require any additional information or logs for further investigation.

Please find the below screenshots:

[Chad Wilson]

There's lots of reasons in your pipeline design that this might happen so would be guessing without being able to see the pipeline config or having a description of the relevant dependencies and materials and what you are intending to do.

The VSM view (when focused on an upstream node) won't always show all the inputs/leaves to a non-triggered downstream node so it's a bit hard to tell.

Perhaps you can include a screenshot from the perspective of the pipeline that gets 'stuck' for one of the runs where it *does* trigger so it's easier to see all the upstream dependencies.

Generally the server logs will log something if a pipeline is being scheduled but hasn't met the requirements, so could check what it says there too.

Perhaps you can include 

Does the pipeline that is *not* triggering

- have manual approval set on its stage?

- have other source code materials or pipeline dependencies as input triggers which have not completed and are not shown in your screenshots?

- share the same repo/material as input with an earlier pipeline in the VSM (so you are trying to do a fan-in?)

The way you describe the problem, it sounds like a pipeline fan-in design problem but I can't see that from the diagram. This can commonly happen if the same repository material is being used as a trigger for multiple upstream pipelines, but you have ignore/exclude/blacklist conditions configured.

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAJCwbnX4XRpRD6xiqsce2veEsZ53nrzF60sH5qLH3voscA6XSg%40mail.gmail.com.

[ashok reddy C k]

Hi Chad,

i will give more info about pipeline VSM here on  below 

Materials is configured as pipeline to trigger next pipeline

see the below screenshot

1) Pipeline A and passed (Pipeline name :  Prospect-AWS-CreateAndDrop-Databricks3-jobs-VSM)


2) B pipeline (pipeline name :  Prospect-AWS-Cache-Registry2-VSM ) , connection is established but not run the pipeline, her material as previous pipeline name .




this is fan-in pipeline VSM

If the pipeline names are changed, the pipeline triggers successfully; however, in subsequent runs, it gets VSM stuck and does not trigger again.

1. We are not manually triggering all pipelines—only the first pipeline has a manual trigger button.

2. I have shared screenshots where the pipeline names correspond to material names.

3. I have reviewed the settings, and there are no options to blacklist or exclude users

all these pipelines are created by yaml files.

Previously, it was working fine, but recently, it has been getting stuck in VSM

Please give a few suggestions, what actions will be taken after reviewing the above screenshots.  

To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAA1RwH_ofHrOsJL-J98h__RjjJ1qNep0-tNmFygY56b5fb6vkw%40mail.gmail.com.

[Chad Wilson]

Unfortunately it's not possible to conclude anything from this. I also requested specific information/screenshots which you haven't supplied, please re-check.

- Did you check the server logs as I suggested?

- Need to see the upstream pipelines and materials (to the left) - not the downstream pipelines (to the right). Fan-in works on the basis of all the inputs back through the pipeline chain, not just the immediate dependency.

- Need to see the full VSM view (all the way left, or a redacted text representation of the dependencies and materials) for an instance run where the "sometimes stuck" pipeline does trigger correctly

- Need to know the model for ancestor materials, to determine whether you are trying to do a fan-in that has got stuck.

-Chad

To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAJCwbnVRqVv2t4Ti%2BM0i0f6TY2SOx6fcUdQbf1Cok7d4hq9uJg%40mail.gmail.com.

[ashok reddy C k]

  Hi Chad,

Please find the required info on below,

- Need to see the upstream pipelines and materials (to the left) - not the downstream pipelines (to the right). Fan-in works on the basis of all the inputs back through the pipeline chain, not just the immediate dependency.

<Vittal> : We are customizing our pipelines using materials, where each pipeline has a material designated as its immediate parent. I am unsure if there have been any recent changes in GoCD, as it was functioning properly until January 2025.

- Need to see the full VSM view (all the way left, or a redacted text representation of the dependencies and materials) for an instance run where the "sometimes stuck" pipeline does trigger correctly

<Vittal> : You can compile and share the materials associated with each pipeline. However, it is crucial to note that the problematic pipeline should ideally trigger immediately after the successful completion of its parent pipeline. Despite each pipeline being configured with the appropriate settings to ensure sequential triggering upon the completion of its predecessor, this is not occurring as expected. This anomaly persists even though the dependencies are specified solely through materials.

- Need to know the model for ancestor materials, to determine whether you are trying to do a fan-in that has got stuck.

<vittal> Our VSM is constructed by linking materials to their preceding pipelines. It appears that the previous pipeline is failing to communicate its success to the subsequent, non-triggered pipeline.



The pipeline "Prospect-AWS-Cache-Registry2-VSM" is configured with the material "Prospect-AWS-CreateAndDrop-Databricks3-jobs-VSM." The expectation is that the successful execution of this preceding pipeline should trigger the subsequent one. However, this is not occurring as anticipated.

 

It is intriguing to observe that other upstream pipelines are influencing the execution of "Prospect-AWS-Cache-Registry2-VSM." Despite this influence, all upstream pipelines have successfully completed before reaching the non-triggering pipeline.

To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAA1RwH_s618_WA752KRhi9OLibyxACKH5wqfjubPW%2B3UNAuanQ%40mail.gmail.com.

[Chad Wilson]

It looks like your GoCD version is 23.4.0 from 2023 which means by definition your change in behaviour cannot be due to a change in GoCD?

Perhaps the structure of your VSM changed, the materials on the upstream pipelines changed, whitelists/allowlists on source code materials changed etc. If you have a precise date that the behaviour changed and you confirm there was no GoCD upgrade, you could ask an admin to look through the GoCD XML configuration history to see if anything changed in the configuration of the relevant pipelines or the upstream ancestors.

In any case, I don't have sufficient additional information from the below to give any more insight.

To understand why I'd need a full ancestor pipeline material and pipeline view (from the perspective of Prospect-AWS-Cache-Registry2-VSM in a case when it DOES trigger) to rule out certain configuration problems, please take some time to read https://docs.gocd.org/current/advanced_usage/fan_in.html or https://preview.gocd.org/getting-started/part-3/#fan_out_fan_in to understand how a "fan in" might prevent triggering of a downstream pipeline, and whether or not you are relying on this mechanism.

If you manually trigger Prospect-AWS-Cache-Registry2-VSM in a case where it did not automatically trigger and you then see warnings such as "built from incompatible revisions" when you view the VSM or stage, then that is another way to confirm you have a "fan-in problem".

By "upstream pipelines and materials" I mean a view going back to the origin materials (roots of the graph/VSM), e.g from the below focused on the "installers" pipeline, I can see that the git repository at top left triggers all of build-linux, plugins and installers pipelines. However due to the dependencies, this implies "fan-in" so installers will only trigger when build-linux and plugins pipelines have successfully run against the same revision of the upstream git material.

[ashok reddy C k]

Hi Chad,

we have identified issue on the revisions side (for not triggered child pipelines)

Note: all pipelines are created by yaml file and here to trigger configured Pipeline name as material name for previous pipeline)

config repo is sync up with new commit id
 

• The pipeline flow is:
  Git → Pipeline A (passed) → B (passed) → C (passed) → D (not triggered)

• Pipelines A, B, and C are all successfully picking up the latest revisions (April or today commits), and are passing as expected.

• However, pipeline D (or similar downstream pipelines) are still referencing older revisions (e.g., March 22nd commits), and as a result, are not being triggered correctly.

Since pipeline B uses pipeline A as a dependency material (pipeline name), and similar setups are used throughout, we believe this might be causing a mismatch in revision propagation.

Question:

Is there a way to force update or refresh the revisions in downstream pipelines (e.g., pipeline D), either via:

• Updating configuration repositories /revisions?

• Resetting/refreshing dependency materials?

below screenshot for not triggered child pipelines due to still they looking into old revions.

[Sriram Narayanan]

A few questions:

1. Could you select "Prospect-AWS-Cache-Registry2-VSM" and show a VSM from its perspective? 

You will need to click that pipeline, select the latest green version, and then click the VSM button next to it.

2. In the screenshot that you have shared, the third stage is already red and the stage after it has not run.

What are the materials of the pipeline "Prospect-AWS-Cache-Registry2-VSM" ?

3. Could you try to trigger the pipeline "Prospect-AWS-Cache-Registry2-VSM" manually? What does it say?

Chad had stated "If you manually trigger Prospect-AWS-Cache-Registry2-VSM in a case where it did not automatically trigger and you then see warnings such as "built from incompatible revisions" when you view the VSM or stage, then that is another way to confirm you have a "fan-in problem"."

-- Sriram

To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAJCwbnWQcaj-Rc0Z4D%2BzaAF89L3AWrhes8CfPqq8DC1SYcjAfw%40mail.gmail.com.