Redeploying docker images with the same (latest) tag

[Simon Sparks]

Hi,

I am working a platform which uses Nomad to provision Docker-based services across a cluster of AWS nodes.

We have been really impressed with how easy it is to set up such a capable microservice architecture using Nomad + Consul (+ Packer + Terraform); great work guys!

An instance of our platform is available for testing services in a continuous integration environment.

In this scenario, when a new Docker image is successfully built from the "develop" branch, we want to deploy it with a Nomad job that references the "latest" tag in the job's Docker driver configuration.

The problem we are finding is that, if a previous "latest" version of the service is currently running on the platform, re-running the Nomad job doesn't cause the most recent version to be retrieved from the Docker repository for redeployment.

Because the Nomad job spec appears to be unchanged, we need to explicitly stop the previous allocations (with docker.cleanup.container and docker.cleanup.image set on the Nomad client) before redeploying the new instance.

Is there a way we can configure Nomad's Docker driver to check hash codes from the Docker repository so that new versions of the "latest" image would cause a redeployment of services?

If not, would this behaviour make a valid feature request?

[Alex Dadgar]

Hey Simon,

Glad you have been enjoying our tools! 

We generally recommend not using the latest tag for this reason as well as the fact that you don't know what version of the container is in production.

A work around to this problem is that you can add an env into the task that is unique per run (timestamp or something along those lines). That way when the job is run again, there is a diff and Nomad will know to stop the old and start the new.

Thanks,

Alex

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/455a7223-ec7d-469a-9bce-43963398940e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Simon Sparks]

Hi Alex,

Thanks for your reply.

I agree that, in our production environment, we would want to fix on specific stable versions of our deployable services.

In the testing/staging environment though we want to continuously build, deploy and test off "HEAD" before we decide to tag and release.

Thanks for your suggestion of using an arbitrary env var to trigger the change; it sounds like a good idea.

I think I'll see if I can make something work from our CI setup using git commit ids as the differentiator.

Thanks again,

Simon

[Cameron Davison]

@Alex is there a way to do "env that is unique per run" inside of nomad? I have checked my nomad configuration files into git, and could put some kind of token string into the file that sed could replace, but was kind of hoping not to have to wrap the nomad files in a bash script in order to give them to nomad.

[Diptanu Choudhury]

Hi Cameron,

If you are asking if Nomad could put a unique environment variable in your job definition, then the answer is no. But we will support string interpolation of dynamic variables in job files in the future, so this should be possible when we get there. For the time being, you will have to do the replacement and put the unique values yourself.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/3fbe9404-e922-4e40-a010-6d36d057d6ec%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Diptanu Choudhury

Web - www.linkedin.com/in/diptanu

Twitter - @diptanu

[DJ Enriquez]

Hi all,

Just looking for clarification on this subject. We are using Nomad v0.4.1 and it appears Nomad has a feature where it WILL check for diffs in latest tag when updating the job using the same job spec according to this: https://github.com/hashicorp/nomad/issues/639. However, that feature doesn't look to be working for us, even when the tag latest is explicitly set.

When I run nomad status <job>, the evaluation changes but the created at stays the same. Then checking on nomad alloc-status <ALLOC>, nothing new is happening. So its not pulling the diff from the latest. Should Nomad be checking for diffs and pulling the latest image as of v0.4.1?

Thanks,

DJ

[Alex Dadgar]

Hey DJ,

Nomad clients will pull the latest image but what is happening is that nothing in your job is changing such that the servers think there is nothing to do. What you have to do is changed something about the job spec such that the servers add/update the allocs. To do that most people are setting and changing a dummy environment variable.

Thanks,

Alex

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/db583277-5ae1-4b46-816b-e70764f57849%40googlegroups.com.

[Ilin T]

On Tuesday, 1 November 2016 23:07:07 UTC+1, Alex Dadgar wrote:

Hey DJ,

Nomad clients will pull the latest image but what is happening is that nothing in your job is changing such that the servers think there is nothing to do. What you have to do is changed something about the job spec such that the servers add/update the allocs. To do that most people are setting and changing a dummy environment variable.

I am having problem pulling the same tag even after adding a dummy variable if there are 2+ allocations (count = 2+) running on a same node. This is likely because the image is not remove since there is still an allocation that is using it, so the new one starts with the same image.

[Alex Dadgar]

Hey Ilin,

Could you file an issue for that and also paste the logs of the client when that error occurs and maybe some reproduction steps!

Thanks,
Alex

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/f178581e-1b57-49d3-be01-21ae78f9bb68%40googlegroups.com.

[Nick Turi]

I am new to Docker and using Bamboo for my build/deploy pipeline. I am trying to figure out a strategy to label my Docker images. For example if I deploy image 123 to Production, is there a way to retroactively go back in my dev and test environments and tag image 123 with the "latest" tag? I want "latest" tag to represent my latest known stable image. Thanks!

[Chuck Danger]

Is there a way to set a dummy variable such that I don't have to physically edit the job spec for each deployment?  In other words, can the job spec be made to interpret a env variable that is set externally?  

To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.

[Alex Dadgar]

Hey Chuck,

At the moment, there is not. You will have to do as you mentioned and modify a dummy variable.

Thanks,
Alex Dadgar

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/0fcc7798-c106-45fa-a24a-ac0769bea5aa%40googlegroups.com.