Configuring pipeline configuration repo

67 views
Skip to first unread message

HUSSEIN KADIRI

unread,
May 7, 2021, 8:14:19 PM5/7/21
to go-cd
https://docs.gocd.org/current/advanced_usage/pipelines_as_code.html talks about configuring pipeline configuration repo via the UI. 

Can the configuration be done programatically?

Maybe by updating the helm chart or some API?

Marques Lee

unread,
May 7, 2021, 8:20:34 PM5/7/21
to go...@googlegroups.com
Yes! We have several plugins to do pipelines as code.

Officially, we support Groovy DSL, YAML, and JSON.

The JSON and YAML plugins are built in, but groovy needs to be downloaded and copied into the plugins dir. the Groovy plugin offers the most flexibility since you have a general purpose programming language, but YAML and JSON are still quite capable for most.

Docs/Source code:

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/4039487e-ccef-46b1-af80-cb8466b50600n%40googlegroups.com.

HUSSEIN KADIRI

unread,
May 7, 2021, 8:27:40 PM5/7/21
to go...@googlegroups.com
not referring to configuring pipelines.

All the examples in the yaml plugin point to configure pipelines. 

I want to configure the GoCD server itself in preferably yaml.format. Like configuring the elastic agent profiles. Etc. Are there examples of those?
.

You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/kvs4Gd0BCKg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAPKX9jY1YmFsBMOHRK3nLx3Oj6VvfTw7R_eROPSQHYqtEgNBWA%40mail.gmail.com.

HUSSEIN KADIRI

unread,
May 7, 2021, 8:29:34 PM5/7/21
to go...@googlegroups.com
Looking at the helm chart, I don't see configuration of cluster profiles via yaml or json. I see an API call to GoCD server passing in yaml.

Is that how it's typically done? 

Marques Lee

unread,
May 7, 2021, 8:31:24 PM5/7/21
to go...@googlegroups.com
Oh I see, so server config.

You can just create the cruise-config.xml file and provision that in the server config dir.

Yes, I know it’s XML (yuck) and not something nicer, but it works. That’s where GoCD keeps its canonical configuration (the UI changes result in direct changes to this xml file)

-Marques

Marques Lee

unread,
May 7, 2021, 8:33:10 PM5/7/21
to go...@googlegroups.com
What you can do is configure things in the UI and then copy the resulting XML file.

We also document all of this configuration here: 

HUSSEIN KADIRI

unread,
May 7, 2021, 8:36:51 PM5/7/21
to go...@googlegroups.com
Yeah XML is a bummer :( Any plans of replacing it with yaml :)?

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAPKX9jYw20_mrN5Q6%2BogxpLcfqrKUACzYvZ79i2p02WkniheLA%40mail.gmail.com.

Marques Lee

unread,
May 7, 2021, 8:40:40 PM5/7/21
to go...@googlegroups.com
I think that has been talked about for a long time (actually, in the context of doing doing full server config as code) but hasn’t materialized unfortunately :(. Probably won’t happen for a long time until someone gets enough time on their hands (and maybe an external committer at that — that’s actually how we initially gained support for pipelines as config).

So while I’d love to tell you “yes we’re working on it” the reality is nobody’s taken the time to truly explore it since XML does work and is doing “good enough.” 🤷🏻‍♂️

-Marques

HUSSEIN KADIRI

unread,
May 9, 2021, 1:07:12 AM5/9/21
to go...@googlegroups.com
Thanks Marques.
Curious what's the best/common/recommended practice on  configs around configuring from scratch and making incremental changes?

 

Marques Lee

unread,
May 9, 2021, 1:25:16 AM5/9/21
to go...@googlegroups.com
Hmm, well you could always keep your cruise config in source control of course, just like you might do with any other service config.

However, what we do internally for build.gocd.org is to utilize the backup feature of GoCD — it’s not just the server config that you’ll want to track, but also your data in the database that keeps track of pipeline history and such. The backup feature will save all of that for you to an archive. We persist them nightly to S3 IIRC. Our config folder and flyweight folders, etc are all bind-mounted into the container at provisioning.

Is that the kind of info you're looking for?

-Marques

HUSSEIN KADIRI

unread,
May 9, 2021, 1:36:44 AM5/9/21
to go...@googlegroups.com
So starting from a bare Installation, there's probably nothing in the DB. How do you first populate the DB with config? Do you do the use config.xml or API calls or UI?

If the config is stored in version control how does GoCD pick up config changes from version control? do you rely on a config management system like puppet or helm to get the config from version control to the server? 

For subsequent incremental changes, what do you do? Update config file and redeploy service? UI change? some other system.



To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAPKX9jZAbCBV4W1Qjp_r4dOdM5c_rC0G75B8u9gHBppdXxzNFQ%40mail.gmail.com.

Marques Lee

unread,
May 9, 2021, 1:37:12 AM5/9/21
to go...@googlegroups.com
Certainly possible that I misinterpreted your question. If it was more about “how do I get started to configure GoCD and learn to how make changes thereafter” I’d just start with the vanilla settings that the UI creates. There’s an option in the admin menu to view the current effective config xml and even edit it right there (validating any changes before saving). Making bulk changes in the UI is inefficient so sometimes we configure something once, open the config xml view/edit page, copy the xml our favorite code editor (eg, sublime, vscode) and use the editor’s advance text manipulation facilities to replicate config blocks to however many we need and then paste it back into config xml GoCD page to validate/save.

HUSSEIN KADIRI

unread,
May 9, 2021, 1:45:09 AM5/9/21
to go...@googlegroups.com
ok got it. This approach works if you're managing a few goCD servers but it  doesn't really scale if you have to manage a lot of GoCD servers. We have a setup where each team has their own CI system. It would be quite painful managing each system this way. Also we aim to have one touch push button deployments. Having to interact with the UI to set things up won't quite work there. 

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAPKX9jbiE_A-behXj4uQkPknmOoY%2Bu02iDnnEZfJrb4DGBeeSQ%40mail.gmail.com.

HUSSEIN KADIRI

unread,
May 9, 2021, 1:49:11 AM5/9/21
to go...@googlegroups.com
Or am I just using GoCD wrong? I know there's the concept of environments. Maybe we should have one GoCD server with an environment for each team instead of a  GoCD servers for a team.

Ashwanth Kumar

unread,
May 9, 2021, 1:56:48 AM5/9/21
to go...@googlegroups.com
I've personally used one GoCD instance per team and eventually combined them into one a few years later. There are few pros / cons with both, but may be that's for another day. 

For getting started the very first time, just setup things via UI. From there on use the backup at periodic intervals (we use nightly) and use that to restore and do one click deployment at a later point in time. Think of each backup as a Snapshot of the system. You can do a fresh install of GoCD and then resotre the latest backup and you're good to go. 

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAFD%2B7D%3Db-4OWDBe7s0Kufoet1mkvzry9_SkF_CyW9xPGk%3DAfrQ%40mail.gmail.com.

Marques Lee

unread,
May 9, 2021, 2:03:33 AM5/9/21
to go...@googlegroups.com
Hard to say without knowing more.

Environments do allow you to keep pipelines, agents, and such all in a single grouping/isolation. If you’re planning to have thousands of pipelines, you’ll have to probably monitor and tweak parameters to keep adequate performance if there are a lot if concurrent builds happening at any given time. Things like how many threads will poll scm repos, and the like.

Marques Lee

unread,
May 9, 2021, 2:03:33 AM5/9/21
to go...@googlegroups.com
Usually I just let the UI do the initial setup and I tinker with it until I have something basic working. It’s just convenient that way. One could of course use APIs or fiddle directly with config.xml, but for all the boiler plate stuff, I personally find it more efficient to just let GOCD do its thing through the UI.

If it’s in version control, it’s still up to you to place the file in the config dir before starting the server. We aren’t doing that for our use case. While we keep backups on S3 in case we need to restore an old snapshot, for normal day to day, we just keep our configs on a persistent disk and that gets bindmounted to the config directories when the docker containers start/restart. So the config files and DB are persisted between deploys. We’re not using puppet or anything like that. Just docker and some simple bash scripts to control start/stop, and fetching of new app/plugin versions. Configuration happens by way of the bindmounts ensuring that the configs exist in the expected locations.

Most of the incremental changes are done through the UI (whether that’s using the “actual” UI, or the copy-edit-paste method I described earlier. It’s not “sexy” like using the API or some other config tool, but it feels natural and works for us. Most of the time we’re just adding/updating a new elastic profile or a config repository (for pipelines as code), and it’s pretty straightforward to do in the UI. For pipeline configs, we usually use the pipelines as code feature and config that in source control. We mostly use the YAML and Groovy syntaxes for that. 

Marques Lee

unread,
May 9, 2021, 2:03:33 AM5/9/21
to go...@googlegroups.com
In that case, you might want to use version control and use whatever tool you are comfortable with to apply those on before startup.

Perhaps you could extract a basic config xml, templatize it, and render the template as appropriate for all the nodes. You could then deploy this config before startup using whichever tool suits it best — puppet, and I Le, terraform (yea, not really a config tool but it could work) or even a simple rsync/ssh workflow could do?

You’d still want to keep around your database and artifact directories, etc though, so somewhere you’ll need to persist disk or at least restore from a tar all or something, I would imagine. Otherwise no pipeline history or build artifacts.

Reply all
Reply to author
Forward
0 new messages