Use Container Registry as Material?

[Jason Smyth]

Hi everyone,

Is it possible to configure GoCD to use a container registry (we are using ECR on AWS in particular) as a Material?

The thought process here is that I would like to possibly trigger a Pipeline whenever a new version of a particular image is published.

Thank you in advance for the feedback.

Regards,

Jason Smyth

[Chad Wilson]

Hi Jason

Personally, I'm not aware of there being a way to do this within the material model of GoCD.

From a design perspective I think there would be some challenges to a general purpose approach here, since to my knowledge container registries don't typically have any way to link together multiple versions of a container to one another in a clear way compared to a VCS.

• A tag is just a tag; the same registry can have multiple different variants of a container, but whose lineages are separate (e.g alpine and debian variants of the same image).
• Tags while useful and human-readable don't really imply ordering of versions except by, say semver convention - and may or may not be immutable
  
• Monitoring a mutable tag for changes in the underlying image hash wouldn't necessarily allow the reproducibility that the GoCD material subsystem aims for except by capturing and referring to the digest of the image which might not be very friendly for usage in an environment since I believe that you can't get the tag metadata along with that.

Perhaps one would want something like OpenShift's image streams concept, which IIRC is rather opinionated?

The way we approached this problem (on ECR/AWS) was to have the pipelines that produce or push docker images outside GoCD to also produce a small manifest with the registry name and target tag etc, similar to https://github.com/gocd/docker-registry-artifact-plugin (the plugin which you could use if you are producing images inside GoCD and thus can use pipeline dependency materials). The manifest was committed and pushed to a Git Repo which could be monitored by a regular Git SCM material or a Git Path Material (to avoid many-to-many repo explosion); with some simple scripting to parse the manifest and decide what to do with that new image (e.g override an image.tag in a Helm chart's values). By using a Git repo & SCM material, we could pass along context from the originating system about the changes included between one Docker image build and the next, and provide more context for investigating issues and comparing VSMs, but this need was really due to doing some custom Docker image builds outside of GoCD rather than within.

Another option, which I never evaluated but contemplated, would possibly be to subscribe to the ECR repository's CloudWatch 'push' events and run a lamba which invokes the GoCD API to schedule the relevant pipeline; passing the relevant image tag etc as an environment variable. I felt this would be less manageable, externalized even more "glue" outside GoCD, thus more opaque and difficult to decide whether a trigger was necessary/desirable inside such lambdas, but was an option all the same.

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/08105e21-3ffa-4167-acb2-2d39c3530961n%40googlegroups.com.

[Jason Smyth]

Hi Chad,

Thank you for the feedback.

You raise a lot of good points.

I will need to think about this some more and come up with a pattern that fits our specific use cases.

Regards,

Jason