How to struct GO-CD in multi materials condition

[Xavier Quick]

I have more then one products, and created build  pipeline for each product(material). After build ok, will upload artifact. And I just have a package pipeline for all products, it will be triggered whill any of previous build pipelines successed. But I don't know how the package pipeline fetch the artifact! For package pipeline, it regardless of which upsteam trigger it, just fetch artifact, pack them and upload installer. Are there any recommendations?

[Xavier Quick]

like below picture, BUILD1, BUILD4 and Package, The Package pipeline don't know which upstream pipeline triggered it, and how to get correct artifact。

[Chad Wilson]

I think this is possibly a misunderstanding of the purpose of fan-in and pipeline/Value Stream design.

In the diagram you shared, the intent is to take the sum/aggregation of the outputs from the 3 source code repos, and 4 builds and produce a single package, while ensuring that the "package" pipeline is triggered by pipeline runs of build 2/build 3/acc 2/acc 3 based on the same revision of source code from the middle source code repository.

If you want a single "package" pipeline, from a modelling perspective it would be expected every run to package all of the upstream "things" that are triggering it each time it runs, not just the trigger source, and thus to fetch the artifact independently for all upstream pipelines it is dependent on.

If you have independent pipelines producing their own artifacts/products, you generally are not trying to do "fan-in" even if you want them to be deployed the same way. Generally you would want independent `deploy` pipelines, or stages that are part of other independent pipelines and to use approaches such as pipeline templates or pipelines-as-code to reduce duplication. That way every time such a  "package" pipeline or stage runs it is clear in the visualisation exactly what is being packaged, rather than the action/result varying based on the upstream trigger source. This might be conceptually different to what you might be used to in some other build/deploy automation.

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/b4824314-7576-4e4a-a292-6c96af16ecc3n%40googlegroups.com.

[Xavier Quick]

thanks, in past and present, I did just like you say. It run for 1 year well, but as long as the products increased, the pipeline's count become more and more. The dash board seemed comfixed now. Ok, since the go-cd is designed in this way, I no longer tangled.
thanks for your answer again

[Chad Wilson]

If your pipelines are just straight through linear triggers like A -> B -> C you might want to consider whether A, B, C need to be different pipelines, or could just be stages within a single pipeline.

Otherwise the number of pipelines shouldn't be a concern unless you have thousands. I'm not sure what "comfixed" means, but if it's just the dashboard view that is confusing, consider grouping pipelines and/or applying permissions to restrict how many pipelines people see. Or have users set up dashboard filters to see only the ones that are relevant to them.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/283ae3a0-c83e-41c6-b2ff-a0730488feeen%40googlegroups.com.

[Xavier Quick]

sorry, "comfixed" is not right word, should be Complex. In my company, pipelines are not always like A-->B-->C.  During the code building, may insert result of other pipeline.
Another question, can fetch artifact command be set from pipeline template? How to set pipeline name, stage name and job name in the template if can?

[Chad Wilson]

Yeah, if using pipeline templates, you'd generally set the pipeline name in the fetch artifact task to a pipeline parameter such as #{upstream-pipeline-to-fetch-from} - or include a parameter for the part that varies, if you use some kind of convention.

When you configure a new instance of that pipeline from the template you will be asked to set the fetch pipeline path using the "upstream-pipeline-to-fetch-from" variable you defined earlier.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/b3761b6a-cb6e-4b06-af0d-04cd76f14794n%40googlegroups.com.

[Xavier Quick]

thank you,  I tried it just now. it's worked well for me.