Material branch check exclusion?

[Chris Gillatt]

Hi all

Is there a way to exclude a pipeline or pipeline group from the material checks?

We have a use-case in my business where a release pipeline's material is pointed at a specific, temporary branch in GitHub.  The branch is pushed to, and the pipeline triggers via a push from a GitHub webhook (opposed to a pull from the GoCD poller). The webhook in GitHub is configured to only notify on push changes.  Once the branch triggers the pipeline and it succeeds, the branch is deleted in GitHub. Until a new temporary branch is pushed to the repository, the existing one set in GoCD becomes invalid.  GoCD somehow notices this, and reports this material problem in the warning/error button found on the top right of the GUI.  For us, the problem is that warnings/errors we care about are sometimes lost in this list of ones we don't.  It'd be nice if we could exclude some pipelines from this check.  There doesn't appear to be an obvious way to do this, and I suspect that it's just not a supported feature.

Cheers

Chris

[Chad Wilson]

In theory if you have disabled polling on all usages of the material (via its URL/branch combination), and there are no triggers from webhooks, APIs or elsewhere then it shouldn't be trying to do modification checks that generate such an error.

Having said that, I wouldn't entirely be surprised if there is some other kind of trigger for an update/modification check, e.g as a side effect of an upstream pipeline completing (if there is one) or a server restart, even if that is not intended.

If you have disabled that polling everywhere and are still seeing the error appear, it'd be interesting to know if you can figure out when it is triggered.

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/b0a6f0aa-44c6-4e76-b65b-715b7c45ad55n%40googlegroups.com.

[Chris Gillatt]

Thanks Chad

I've done some more investigation and you're right - there's some upstream pipelines set to "Regularly fetch updates to this repository" rather than our company strategy of "Fetch updates only on webhook or manual trigger".  I wasn't aware that upstream material configurations could affect downstream ones.  We rely on templates for the majority of our pipeline creations, and use an in-house tool (relying on GoCD APIs) that explicitly specify 'autoUpdate="false"'.  It's been a while since I've created a pipeline from the GoCD wizard or from scratch, so not sure if this has always been the case, but through the GoCD GUI Create Pipeline workflow, the material configuration is selected before a template is chosen, meaning templates cannot control advanced material configuration, such as autoUpdate="false".  Unless the user selects this from the material Advanced menu at the time, the default is essentially 'autoUpdate="true"'.  I think that a bunch of these has crept in since I last cleared them up.

For posterity for anyone else reading this, some contextual info is useful to know:

1. When "Regularly fetch updates to this repository" is selected, it is not explicitly expressed in the XML.  Its absence on the line beginning <git url=" under the material configuration defines it, somewhat unintuitively. For instance, if you set  autoUpdate="true", GoCD will simply delete the key value pair, and "Regularly fetch updates to this repository" is selected.

2. When "Fetch updates to this repository only on webhook or manual trigger" is selected, on the line beginning <git url=" the property of  autoUpdate="false" appears to show this in the XML.

3. This property must be the same over all pipelines which use the same material.  You cannot have one pipeline which pulls and one which receives a push for the same material.

[Chad Wilson]

I think it wouldn't be so much that the material polling on an upstream would affect a downstream pipeline, but more that if a build runs upstream (regardless of trigger type), when it triggers the downstream pipeline it possibly triggers a material check on the other materials that pipeline has as inputs, to determine which revision the other materials should use. Kind of like when you manually trigger a pipeline (with the "play" button) GoCD will trigger with the latest revisions of the materials on that pipeline, regardless of whether you have polling enabled or disabled - meaning it triggers a modification check.

However if the same logical Git material (generally determined by identical URL and branch) is used in different pipelines whether connected to each other or not, then the polling setting of one can definitely affect others, I believe. Generally speaking GoCD complains at you when there are conflicting values for the same logical material and asks you to make them all the same, but I have seen cases where it gets confused or ends up allowing inconsistent values here, and behaviour becomes indeterminate. This is one of the testing challenges with fixing/allowing the change in https://github.com/gocd/gocd/pull/11636 which would make it easier to support such autoUpdate=false repos from pipelines-as-code. If the concern is the triggering (rather than spurious errors from deleting branches as in your case) the workaround some folks use for this is to put **/* material deny lists on pipelines, so even if the material is polled and updated it shouldn't cause a trigger since the denylists/allowlists are processed in the context of an individual pipeline, rather than the (de-duplicated) material itself.

But yes, the "default" is autoUpdate=true/polling.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/19d266e2-05b3-4c1c-a624-920da5eb6026n%40googlegroups.com.

[Alexey Savchkov]

Chris, did you try the Git branch plugin (https://github.com/ashwanthkumar/gocd-build-github-pull-requests)? GoCD is built around Trunk Based Development so branches are not an inherent part of its design. This plugin however polls branches of a repository according to plugin filters based on branch names. It makes no assumption about the prior existence of a particular branch.

суббота, 18 ноября 2023 г. в 06:33:40 UTC+7, Chad Wilson:

[Chris Gillatt]

Thanks Alexey - I haven't yet considered that plugin, but I'll give it a go.