Go CD with Amazon ECS

[ranjeet gupta]

I m a bit lost trying figuring out how to create a build pipeline for deploying docker images to Amazons container services. Basically I have a bunch of micro-services, which ideally will have there own build-pipeline and release cycle.    However, before pushing a release to production, I would like to do an integration test in an environment where all the micro-services will be deployed. I plan to have the "integration" job (which triggers the integration test suit) every couple of hours.

My requirements are

1)Integration job should identify which all micro-services have a new version available.

2)Upon, successful completion of integration tests, fetch versions of docker images deployed in Amazon ECS and compare (calculate delta) with the new versions available. Mark all the delta set of services to be a release candidate.

3) Deploy (Manual trigger)- This one is already figured out.

I am not sure if GO has something which can help me here. I have a jenkins setup. As a last resort, I plan to calculate the delta step using a custom script(Python ?).Could someone please help and suggest how to model this requirement.

Thanks

Ranjeet K Gupta

[Derek Kastner]

Hi Ranjeet,

We have exactly this setup at Faraday, the company I work with. We're currently working on a way to calculate the delta and only update services that need it, but for now each deploy does a full refresh of all ECS services.

Essentially we have an intermediary step that aggregates all of our service builds and generates a `gocd.yml` that is a key/value mapping of service names to version numbers. Each version number corresponds to a private tagged docker image, eg `mycompany/service_a:1234`. This gocd.yml is used to generate docker-compose.yml files for acceptance testing and production. The acceptance test runs a full test cluster in memory and we run cucumber/browser tests against it. The production docker-compose.yml is then translated into ECS task definitions using our publicly available ecs_compose gem https://github.com/faradayio/ecs_compose. once we have the new task definitions we just use the aws-cli tool to deploy the new tasks.

I have some more info on the gocd.yml here: https://dkastner.github.io/gocd-docker-pres/index.html#1

I also blogged about our overall journey to GoCD/docker here: https://medium.com/@dkastner/accidental-continuous-delivery-and-docker-aecec7deb8f1#.gkj4pg2ym

[ranjeet gupta]

HI Derek, 

Thanks a lot for sharing your experience. I was hoping that GOCD can help me calculating the delta, however will go with custom Python scripts for that. Would be great if you share your way of calculating the delta.

Thanks

Ranjeet K Gupta

[Fredrik Wendt]

If you use a SCM as material, go will give you the FROM commit/changeset and TO commit/changeset. Perhaps you can calculate your deltas from that? Perhaps you can produce the delta early upstream in the pipe, and simply use that (as an artifact) when you get to deployment. 

/ Fredrik

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

+46 702 778511

[David Rice]

Hi Ranjeet -

I'm not completely sure what you mean by "calculating the delta," but I think you mean this:

- you're building a bunch of microservices

- you have some integration tests

- each time the integration tests pass, you want to know which microservices have new versions since the last push to production

If that's the case, Go CD should be able to give you what you want with it's compare builds and fan-in features. 

If you model things this way:

- each services has its own pipeline

- the service pipelines all fan-in to an integration pipeline

You could then compare any two runs of the integration pipeline.  You'd see upstream pipeline labels/versions as well as have the ability to drill into SCM detail of the upstream changes.

But all that said... if you are only calculating the delta to figure out which services to deploy, Go's fan-in and artifact propagation would likely allow you to skip that bit altogether.  Assuming you fetch all the deployable service artifacts for the integration pipeline, you could deploy downstream from the integration testing without worrying about which individual service needs to be deployed.

Obviously, I don't know much of the details of what you are looking to do, so some of that might be  off.  But perhaps it is helpful.

--

[ranjeet gupta]

Thanks Fredrik and David ! I will ponder on these two approaches. Btw, by delta what I mean is , check if in the docker repository,  there is any new version available (for a bunch of images). Compare those tags/versions with what is deployed on ECS and push the updated ones.

Thanks

Ranjeet K Gupta