Exploring GoCD for bitbucket pipeline replacement

[Vignesh Mohanasundaram]

Hi. I’m not sure if this is the right forum for it, if not please direct me to right forum.

We are exploring GoCD for our CI/CD pipeline. Currently we are using Bitbucket Pipelines.

I tried locally and have few questions regarding how-to,

1. Can we configure a pipeline without git dependency / material? We have a pipeline which runs on a schedule and the pipeline is just pure shell script (and it does not depend on any code from any repo).

2. How to configure multi-branch pipeline? For example, we have an frontend repo with different branch for different environment, like dev, prod, … so that when triggering the pipeline manually, we can select which branch to trigger from.

3. Here is a complex scenarios which we trying to resolve. We have 2 repo, 1 for frontend and 2 for backend. The backend pipeline will be dependent on frontend pipeline. Whenever the frontend pipeline build is successful, it should trigger backend pipeline. Also if there is any change in backend repo, it should trigger backend pipeline. I’m able to configure this by adding multiple dependencies for backend pipeline. The complexity here is, we generated random key when building frontend (For eg.: KEY=hash-BUILD-1), and the same needs to be passed to backend pipeline. 

a. How to trigger backend pipeline with custom build parameter (KEY)? From pipeline 1 it should pass parameter and value for pipeline 2 build. 

b. How to persist param value (KEY) in backend pipeline, so that subsequent build from backend repo (commits, poll scm) trigger will use the param value (KEY) which is already set. Param value changes only when frontend pipeline creates a new build.

Thanks, V

[Jason Smyth]

Hi V,

Here are some thoughts on the questions you raised:

1. Configure Pipeline without Git Material.

I believe GoCD requires that Pipelines have at least one Material. For the Pipeline model you describe, why not put the "pure shell script", as you describe it, into source control and then key the Pipeline off of that? Alternately, you could put the shell script into source control and use a GoCD Pipeline to test and then deploy the scripts to somewhere else to have them run outside GoCD. Regardless of the path chosen, I believe the scripts you alluded to should be under source control somewhere and, assuming they are, I don't see a need to create a Pipeline in GoCD without a Material.

2. Specify Git branch at Pipeline runtime.

The closest thing I can think of to this model would be to use Pipeline Parameters in GoCD. You can reference a Pipeline's Parameters in a Material definition so it should be possible to use a Parameter to specify the desired branch of a Git repository.

With that said, if I understand the situation you describe correctly, I do not recommend this approach. I think you are describing a pattern where you have multiple long-lived branches in Git and each one maps to a corresponding long-lived environment. So there is a "prod" branch that gets deployed to the "prod" environment; the "uat" branch gets deployed to the "uat" environment, and so on.

If this is accurate then using Parameters should be safe in that using a single Parameter to control both source control branch and destination environment would make it impossible to accidentally deploy the wrong code to the wrong environment. However, using a single Pipeline to manage all of your different environments would likely make it difficult to understand exactly what has been deployed where at any given point in time.

Instead, I recommend a templating pattern where you have a parameterized process defined somewhere (preferably in source control) and then reference that process in multiple Pipelines, with each Pipeline linking a particular source branch to its respective environment.

3a. Pass data from Pipeline 1 to Pipeline 2.

GoCD's Artifacts system works well for this. Configure Pipeline 1 so that it writes the key to a file that gets uploaded to GoCD Artifact repository. Create Pipeline 2 with Pipeline 1 as (one of) its Material(s). Configure Pipeline 2 to Fetch the saved Artifact from Pipeline 1, read the key from the file and does whatever work is required from there.

3b. Persist Pipeline data.

This happens automatically with GoCD's Artifacts system. Every time Pipeline 2 is run, it will run with a specific version of Pipeline 1 (and therefore a specific copy of the key) as its source. By default, this should be the most recent instance of Pipeline 1, but there is always the option of triggering with an older instance if that becomes necessary.

Hope you find some of this useful.

Regards,

Jason