Staging von Pipelines as Code

[Helge Walter]

Hello,

we use GoCD as a shared service for our companies internal application provider and we develop scripting, pipelines and plugins for common usage. So we have a GoCD test environment and a GoCD production environment.

What we want to achieve is that we develop scripting (in a development branch) and pipelines as code (also in the development branch) and for production we want to have the same setup but using the master branch. The scripting repository is referenced in the pipelines as code as materials.

Here we can setup the config repository in the different GoCD environments with different branches. But the pipelines as code directly contain the branch to be used for the materials. How can I achive this?:

• Different material branches across different GoCD instances
• Different material user/PWs across different GoCD instances
• Encrypted values across different GoCD instances (encryption is specific per GoCD instance)

Is there some facility to specify GoCD instance specific properties which can be used in pipelines as code?

Is there another solution how to make it possible to test pipelines as code  w/o affecting production use?

Thanks in advance!

Helge

[Aravind SV]

Hello Helge,

If the scripting code and the pipelines-as-code code is in the same branch of the same repository, then the config-repo-material type might be useful. That will allow you to reference and use the material and branch of the config repository inside the YAML or JSON.

If not, and if you are comfortable using Groovy, maybe the groovy config-repo plugin will give you more flexibility.

The plugin endpoint (plugin infrastructure) from the GoCD server side does allow plugins to accept arbitrary configuration ("instance specific properties" as you said), but the plugin will then need to accept it and make sure it is available in the context to be used in the YAML or JSON, for instance. It'll be a conversation you need to have over in the issue section of the plugins, to see if there is appetite to do that. Maybe a PR will be accepted. Templating, which is related to what you're asking about has been talked about before, for the YAML plugin. Maybe that conversation is relevant to you.

Finally, about encrypted values across different instances, you should probably look at secrets management.

Cheers,

Aravind

[Helge Walter]

Hello Aravind

Thank you for your advice.

I think the groovy config-repo plugin is worth a look. There it would be possible to read such parameters programmatically, e.g. from a file.

Another idea is some kind of instance specific parameters as used in pipelines but defined on GoCD instance level -- maybe provided by some kind of parameter plugin like secret plugins.

I reviewed the secrets management page and found it very useful. So would it be possible to abuse the secrets plugin to provide the branch for the materials and configuration repositories? Can all configuration values use the secrets from secret manager plugins?

Regards,

Helge

[Ketan Padegaonkar]

On Thu, Jun 11, 2020 at 3:30 PM Helge Walter <bewal...@gmail.com> wrote:

Hello Aravind

Thank you for your advice.

I think the groovy config-repo plugin is worth a look. There it would be possible to read such parameters programmatically, e.g. from a file.

 

This is explicitly prohibited for reasons of security (imagine someone sending a PR that allows reading all you secrets from gocd config). But can be enabled with some code changes on the plugin.

[Aravind SV]

Hello Helge,

On Thu, Jun 11, 2020 at 03:00:28 -0700, Helge Walter wrote:
> Thank you for your advice.
> I think the groovy config-repo plugin is worth a look. There it would be
> possible to read such parameters programmatically, e.g. from a file.

As Ketan mentioned, please think about the security implications of it. If you're using it publicly, you need to be careful since it can be powerful and (so) dangerous.

> Another idea is some kind of instance specific parameters as used in pipelines but defined on GoCD instance level -- maybe provided by some kind of parameter plugin like secret plugins.

> I reviewed the secrets management page and found it very useful. So would it be possible to abuse the secrets plugin to provide the branch for the materials and configuration repositories? Can all configuration values use the secrets from secret manager plugins?

I thought about that. However, based on the documentation, the only places secrets can be used are the password field of a material and environment variables (and so, within a task in a job as well). So, it doesn't seem like you can use it arbitrarily, which should mean that you can't use it to provide the branch, unfortunately.

Another piece of work that is going on now which might be relevant to you: https://github.com/gocd/gocd/issues/6123

It implements some form of templating which might be useful to you. It's a work in progress though and you'll need to wait.

Cheers,
Aravind