GoCD Artifactory best pratices

[Anh Le]

Hello everyone,

I'm learning GoCD, I only know how to use the Files and Directories out of the box as Artifactory.

I have few question about artifacts management best practices or your preferred approach

Assumption:

 - Already has a build pipeline (to generate artifact)

 - Deploy to QA is manual deploy and depend on build pipeline success

 - Deploy to Prod is manual deploy and depend on Deploy to QA pipeline success

I'm thinking about things below but something doesn't feel right:

 - In the build pipeline, after run build tasks, I push the artifact to an Artifactory

    + How to fetch artifact for "Deploy to QA" and "Deploy to Production"? if I add build as material (so I can get the artifact) somehow I feel it is weird because Production can access to something not already in QA. If I publish artifact again in QA to break the dependency of Prod to build then I feel that the artifact is duplicated

What are your suggestions?

[Marques Lee]

It’s been a few years since I really looked at GoCD, so my memory is hazy on this.

You should be able to store artifacts on the GoCD server itself by configuring a build artifact path. Then downstream stages or pipelines can grab them from gocd server directly. There’s an S3 plugin as well if you wanted to persist them in AWS. Then you can upload to artifactory when you’ve had an artifact pass certain stages for confidence.

You should be able to pass artifacts through stages so that a downstream stage can fetch an artifact that passed the previous stage. If a stage depends on the previous stage passing, it cannot run until those conditions are met, so they won’t run with artifacts that didn’t succeed in upstream stages.

Once it’s passed production, you can then publish it to artifactory such that artifactory only has production quality packages.

Does that help?

-Marques

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/d7155810-ffd1-4ecd-822a-0bddfb7ae419n%40googlegroups.com.

[Anh Le]

1. So you are suggesting there is no way to avoid temporarily duplicating the artifact between pipelines following my use case right? I'm still new to GoCD and haven't play much with plugins

Not sure if I get your idea correctly, does that mean if I model my pipelines like this:

build pipeline - deploy to QA pipeline - mark passed QA Gate pipeline - deploy to Production pipeline

If I have a perfect artifact, I will need to Publishing artifacts and Fetch Artifacts for each pipeline

Since storage is cheap, I'm perfectly fine with this approach. But having to upload the same file every time could sound not optimal

2. Is it possible to avoid that duplication?  (but it could be over-engineer, I'm not sure, haven't spent much time doing this after reading the Continuous Delivery book)

I'm thinking is there any vendor/tool/plugins that does artifact management strategy and could be integrated well with GoCD OOTB: 

- An artifact (it is a file generated from the build step anyway) should have some states:

        + Deployed to QA

        + Passed QA Gate

Mark passed QA Gate pipeline and Deploy to Production pipeline should fetch artifacts based on that state from a central Artifactory

[Marques Lee]

You could upload it from every stage that passes, but I don't see a benefit to doing that -- and as you said it's duplicative. What I'm suggesting is that every downstream stage/pipeline can use the build stage as a material so you can fetch artifacts from it.

Something like:

build-and-test (generates artifact) --> deploy QA --> validate QA --> promote-to-prod-manual-gate --> deploy-prod

Every stage that needs to use the built artifact can include build-and-test as a material so that you can do a Fetch Artifact task. I believe you had a concern/question that it meant stages downstream would have access to an artifact that possibly didn't pass validations? I don't think that's an issue because you would have more than 1 material. there's an implicit dependency on the prior stage succeeding by default. For instance, "deploy-prod" would only run if all the prior stages succeeded, I believe, so it wouldn't actually deploy without going through all of your validation stages (deploy-QA, validate-QA, promote-to-prod-manual-gate).

So in other words, deploy-prod would need to depend on both promote-to-prod-manual-gate and build-and-test.

If you can't get it working, let me dust off my GoCD hat and look at it tonight after working hours.

Best,

Marques

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/042f88ea-2e9e-4433-a2a8-af77cbbb294cn%40googlegroups.com.

[Jason Smyth]

Hi there,

I think the best way to do this in GoCD is using build artifacts, pipeline materials and the Fetch Artifact task.

The build pipeline should publish the artifact to Artifactory and generate a GoCD build artifact that references the Artifactory location. The QA pipeline should have the build pipeline as a material and should fetch the GoCD artifact using the Fetch Artifact task with a Pipeline value of "build". The Prod pipeline should have the QA pipeline as a material and should fetch the GoCD artifact using the Fetch Artifact task with a Pipeline value of "build/QA".

This ensures that it is impossible to deploy an artifact to production without first successfully deploying it to QA.

Hope this helps,

Jason