Continuous deployment strategies with nomad

[Brian Lalor]

I'd like to hear how folks are doing continuous deployment with nomad. I'm struggling to reconcile a couple of concepts in my mind. The primary method for deploying a jobspec is to write the file and submit it. But it seems that when deploying a new version of a container image, I just want to modify the image reference and resubmit it. But that makes the jobspec ephemeral and the jobspec file and config in the server diverge quickly. I could version the spec files in git and have each deploy consist of a commit, but that seems a bit heavy.

Any best practices or resources I can learn from?

Thanks,
Brian

--
Brian Lalor
bla...@bravo5.org

[Alex Dadgar]

Hey Brian,

You could put a placeholder on the Docker version and then use a tool like envsubst to dynamically update the job file and then submit it.

Thanks,
Alex Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/70A37356-53C6-4306-9A75-A4F3FEF1798D%40bravo5.org.
For more options, visit https://groups.google.com/d/optout.

[james....@made.com]

We are using Jenkins & Ansible to deploy our jobs. The job spec is an Ansible template.

Jenkins is triggered and builds the container and pushes to our repo tagging with the build #

Jenkins then calls a provisioning playbook in the project which builds the spec with the build # of the job

Ansible then plans and runs the job on the first node in the group.

It's still early days and haven't thought properly about blue / green other than every change to the job spec is a change in git, which is no bad thing.

May move to using Jenkins pipelines to have differents stages for blue / green depending on acceptance tests.

We are only using non-critical non-customer facing apps at the moment so many of the requirements for releasing haven't been fully worked out yet.

James

[Craig Sawyer]

We have Jenkins that runs CI/CD, the repository it builds from has a .nomad job file with this as the image:
image = "registry.service.consul:5000/IMAGENAME:@@BUILD_NUMBER@@"

I use sed to replace @@BUILD_NUMBER@@, and then calls nomad run.

I also have a python script that runs after, that checks nomad status for a running status for the allocation(s) on that job. If it doesn't find one, it subtracts one from the BUILD_NUMBER, and resubmits the job to nomad, it will do this up to 5 times before giving up and failing the jenkins job.

The script can be seen here: https://gist.github.com/csawyerYumaed/076bbc6a0d5056d20ad45095b434b02d

I'm open to better solutions, but it's been working well for us so far.

-Craig

[Yuvaraj Loganathan]

We make all the nomad job files as consul template. Lot of parameters of the nomad job file for ex : count , memory , cpu , network and docker image version are values fetched from consul and  nomad job files are updated whenever we change the values 
by the consul-template and nomad file are submitted to nomad by consul-template cmmand option. 

[Adam Vollrath]

I'd like to resurrect this for 2019. How are people continuously delivering and deploying things to Nomad now?

Jenkins is fine, but can't really be configured as code.

[Rod]

We aren’t doing it quite yet, but we’re getting close to trying to use Gitlab CI to manage our submit-to-deploy strategy.

Given the ability of gitlab-runner to run arbitrary scripts and commands, we can mount the nomad client and cents into a container and execute commands as needed.  We keep a gitlab-runner container running as a service in the cluster, it auto-registers with Gitlab on startup, and waits for jobs.  Gitlab calls the CI component and divvies jobs out to runners based on settings at the group/project (repo) level.

We already use CI to build docker images and push them to the docker repo, but after that it’s a manual process.  That’s the secret sauce we’re looking to integrate.

Rod

Sent from my iPad

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/8aece658-6969-42ec-b22a-5025c24c12b0%40googlegroups.com.

[Shantanu Gadgil]

There are a couple of strategies which come to mind 2019:

1. The usual (most flexible)
Template (source code) ->Job -> Jenkins/GoCD -> Run

2. Jobfile (source) -> Git+Atlantis -> Apply
(Stick with TF 0.11.14 for now)

HTH,
Shantanu Gadgil

[David Muñoz]

I've been playing for a few months with CD and Nomad, what I am doing is that the jobspec is:

- The jobspec is a template stored in the git project itself (I'm using ERB but any templating system should work)

- Jenkins uses the commit ID as $VERSION

- Jenkins generates the artifact 'whatever-$VERSION and stores it in somewhere (S3 in my case)

- Jenkins renders the template, which references to 'whatever-$VERSION', and submits it to Nomad.

There are probably much better options but in my case -which is not terribly complex- it works well.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/dde2d73e-4e04-4576-bb42-ac2016ac5ed0%40googlegroups.com.

[Chris Raborg]

Docker images are built using Drone (but you can use any tool). Builds are kicked off on deploys to master. Every image is tagged with it's commit sha. 

To deploy we use Terraform. You can inject the desired git sha into the jobspec and then apply the changes all within terraform apply. If we want controlled rollouts we'll use the built-in update stanza. This is the only manual process right now, but we want to start using an orchestrator tool to chain together deploys for all of our environments (e.g., deploy QA, check if healthy, deploy stage, check if healthy, deploy prod, ...).