ECR Login With Credential Store

[Joshua Banton]

I'm trying to pull an image from Amazon ECR. I have setup docker-credential-ecr-login along with an IAM EC2 Role to give permission to access ECR. I can "docker pull 278325265250.dkr.ecr.us-east-1.amazonaws.com/nginx:latest" with no problem.

My nomad client config is:

# Set log level
log_level = "DEBUG"


# Setup data dir
data_dir = "/var/nomad"


#Set datacenter
datacenter = "us-east-1"


# Enable the server
client {
    enabled = true
options  {
    "docker.auth.config" = "/etc/nomad.d/docker_auth.json"
  }
}

My job config is:

job "nginx" {
  datacenters = ["us-east-1"]
  type = "service"


  update {
    stagger = "10s"
    max_parallel = 1
  }
 
  group "nginx" {
    count = 1


    restart {
      attempts = 10
      interval = "5m"
      delay = "25s"
      mode = "delay"
    }


    task "nginx" {
      driver = "docker"
      config {
        image = "278325265250.dkr.ecr.us-east-1.amazonaws.com/nginx"
        port_map {
          http = 80
        }
    }
     
    resources {


    network {
      mbits = 10
      port "http" {}
    }
    }


    service {
    name = "nginx"
    tags = ["global", "cache"]
    port = "http"
    check {
      name     = "alive"
      type     = "tcp"
      interval = "10s"
      timeout  = "2s"
}}}}}

The DEBUG output says:

  2017/02/16 20:09:21.399976 [DEBUG] driver.docker: did not find docker auth for repo "278325265250.dkr.ecr.us-east-1.amazonaws.com/nginx"

    2017/02/16 20:09:21.399997 [DEBUG] client: driver event for alloc "23f86334-5d4a-9d76-8f98-7a78d0688ec8": Downloading image 278325265250.dkr.ecr.us-east-1.amazonaws.com/nginx:latest

    2017/02/16 20:09:21.435711 [ERR] driver.docker: failed pulling container 278325265250.dkr.ecr.us-east-1.amazonaws.com/nginx:latest: unauthorized: authentication required

    2017/02/16 20:09:21.435741 [WARN] client: error from prestart: failed to initialize task "nginx" for alloc "23f86334-5d4a-9d76-8f98-7a78d0688ec8": Failed to pull `278325265250.dkr.ecr.us-east-1.amazonaws.com/nginx`: unauthorized: authentication required

I have the dockercfg setup in /etc/nomad.d/docker_auth.json as:

{

    "credsStore": "ecr-login"

}

[Joshua Banton]

I took a closer look at this, and if I'm reading it correctly Credential Store/Helper support needs to be built into the client, so either Nomad or the docker client they use go-dockerclient, and so far the support is in neither of those places. Until support is added I'm going to go with a hacky solution of just logging into docker with the aws cli tool every 8 hours, and pointing Nomad to the config file that logging in creates.