best practices for maintaining go pipelines

[pan...@mammoth.io]

Hi community,

Every developer in our organization gets a pipeline of his own to do the full build and test process. Very frequently we are making changes that have to be replicated across all pipelines, for example adding a new material, adding an additional step, minor changes in the command syntax of one of the steps in pipeline. Since each developer is supposed to maintain their own pipeline, they realize it only when the pipeline breaks because of this change and we are in a continuous maintenance loop.

Our need is to easily replicate these changes across all pipelines. Most configuration is same except the name of the pipeline is different. Whats the community recommendation for this. How else to manage this situation?

Warm regards,
Pankaj

[Chad Wilson]

Hi Pankaj

How do you maintain the pipeline config now? Via the UI? Via API? Via one of the pipelines-as-code/confg repo plugins?

In any case, I would say it depends a bit on

• what the content of your pipelines are
• how much variation there is between these pipelines
• when you say developers "maintain" their pipelines, does that include changing its config and definition? Or just making sure it's green?
  
• when folks need to change the pipelines, will it be acceptable (or desirable) for them to be forced to do it via source control?
  

...but in terms of general approaches to reduce duplication and factor out commonality in pipelines, there are a few approaches, which have upsides and downsides. Perhaps you can take a look at them and ask some more specific questions based on your needs?

• Pipeline templates https://docs.gocd.org/current/configuration/pipeline_templates.html - maintained via UI|API IIRC, but not really customisable by those pipelines that use the template, except by parameters/env/materials
• One of the Pipelines as Code plugins (aka Config Repository plugins) https://docs.gocd.org/current/advanced_usage/pipelines_as_code.html
• At least with Pipelines-as-Code you can find-and-replace en-masse, but...
  
• With the YAML plugin you can use YAML anchors to factor out commonality in a limited way.
• With either the YAML or JSON plugins you could conceptually template the YAML/JSON and generate it using a tool as well, although I have never done this and have no opinion on whether it's a good idea
  
• With the Groovy plugin you can write Groovy script to express your pipelines using a DSL, which could potentially allow you to model the differences between them, dependencies etc.
  
• This may or may not work for depending on how you plan to supporting "self-service" and developers making changes to their own pipelines, unless you work with some kind of PR approval model into a Git repository you centrally maintain
• Write some tooling against the GoCD API https://api.gocd.org/current/#pipeline-config
• There were some client libraries to support this such as gomatic, but I haven't personally used those, and not sure what state they are in.
  

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/f66ff769-da55-4383-9d55-7d5c201153fdn%40googlegroups.com.

[Michael Day]

I've used the JSON plugin and a jsonnet (https://jsonnet.org/) library to achieve this before.

It worked pretty well, but did require some discipline on behalf of the programmers, i.e. to provide make targets to build, push to the container repo and deploy to the test cluster, which had uniform names etc.

M

[pan...@mammoth.io]

Thanks Chad and M,

I would look into some of the options you have provided. Good to know that I am not missing something out of the box and this needs some creative thinking.

All pipelines under consideration differ just in the name and the material branch names. Everything else is exactly the same. We use this pretty much only through the UI, not using API or pipeline as code as yet. One option we are also considering is to templatise the cruise-config.xml and generate some relevant sections and re-upload when such need arises but I think the options suggested by you are better things to consider.

I would keep the group posted as we solve this.

Warm regards,
Pankaj

[Ketan Padegaonkar]

The groovy DSL plugin lets you do most of what jsonnet does without the additional compile step. Some examples here.

- Ketan

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/74675668-1f37-4466-9e59-a4efa67023ben%40googlegroups.com.

[Jason Smyth]

Hi Pankaj,

If you are sure that the only difference between the Pipelines is the name of the Pipeline and the name of the Git branch, you may want to look at Templates.

If each developer has their own Pipeline that is based on the same Template then if and when the process (steps taken during execution) need to change then you only have to update the Template and all associated Pipelines will be updated automatically.

This won't solve Material-related issues but, depending on how often the Materials change, it could be a good first step while you look into other options.

Hope this helps,

Jason