Not able to mount docker volume

[Manish Kumar]

I am trying to create a docker volume and getting following error. Everything works if I don't create mount.

2019-08-07T13:48:27.564-0400 [ERROR] client.driver_mgr.docker: failed to create container: driver=docker error="no such image"
2019-08-07T13:48:27.565-0400 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=3623c0cf-64f4-bf12-3108-88e07617028a task=kafka error="failed to create container: no such image"

Please make a note that Nomad complaints about missing docker image that I am trying to deploy but I can assure you that it exists. As soon as I stop mounting volume, everything start working fine.  I am referring to documentation here - https://www.nomadproject.io/docs/drivers/docker.html#mounts

Reproduction steps

Job file (if appropriate)

job "kafka" {
region = "global"
datacenters = ["dc1"]
type = "service"
constraint {
attribute = "${attr.kernel.name}"
value = "darwin"
}
update {
stagger = "10s"
max_parallel = 1
}
group "kafka" {
count = 1
restart {
interval = "5m"
attempts = 10
delay = "25s"
mode = "delay"
}
task "kafka" {
driver = "docker"
logs {
max_files = 10
max_file_size = 10
}
config {
image = "itspanwar/kafka:2.2.1"
network_mode = "host"
mounts = [
{
type = "volume"
target = "/etc/telegraf/telegraf.d"
source = "kafka-telegraf-config"
readonly = false
volume_options {
no_copy = false
labels {
foo = "bar"
}
driver_config {
name = "pxd"
options = {
foo = "bar"
}
}
}
}
]
}
env {
ZOOKEEPER_PORT = "2181"
CONSUL_DNS_SERVICE_PORT = "8600"
KAFKA_ZOOKEEPER_CONNECTION_CONSUL_SERVICE = "zk.service.consul"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP = "INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT"
KAFKA_ADVERTISED_LISTENERS = "INSIDE://{HOSTNAME_COMMAND}:9094,OUTSIDE://{HOSTNAME_COMMAND}:9092"
KAFKA_LISTENERS = "INSIDE://:9094,OUTSIDE://:9092"
KAFKA_INTER_BROKER_LISTENER_NAME = "INSIDE"
BROKER_ID_COMMAND = "echo '${NOMAD_IP_kafka_port}' | cut -d . -f 4"
HOSTNAME_COMMAND = "echo '${NOMAD_IP_kafka_port}'"
KAFKA_HEAP_OPTS = "-Xmx8G -Xms6G"
JMX_PORT = "${NOMAD_PORT_jolokia_jmx_port}"
KAFKA_JMX_OPTS = "-javaagent:/opt/kafka/libs/jolokia-jvm-agent.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=${NOMAD_IP_kafka_port} -Dcom.sun.management.jmxremote.rmi.port=${JMX_PORT}"
}
service {
name = "kafka"
tags = [ "kafka"]
port = "kafka_port"
address_mode = "host"
check {
type = "script"
interval = "15s"
timeout = "5s"
command = "/usr/bin/kafka-healthcheck.sh"
}
}
resources {
cpu = 1024 # Mhz
memory = 1024 # MB
network {
port "kafka_port" {
static = 9092
}
port "jolokia_jmx_port" {
static = 9999
}
port "jolokia_http_port" {
static = 8778
}
}
}
}
}
}

Nomad Server logs (if appropriate)

2019-08-07T13:52:02.230-0400 [DEBUG] http: request complete: method=PUT path=/v1/jobs?region=global duration=2.710175ms
2019-08-07T13:52:02.230-0400 [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=e96e699d-7bb3-db3f-8479-7a1fc5777989 job_id=kafka namespace=default results="Total changes: (place 1) (destructive 0) (inplace 0) (stop 0)
Created Deployment: "e655a109-cb5b-a0ff-8c08-3fa972a6d972"
Desired Changes for "kafka": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)"
2019-08-07T13:52:02.232-0400 [DEBUG] http: request complete: method=GET path=/v1/evaluation/e96e699d-7bb3-db3f-8479-7a1fc5777989?region=global duration=664.244µs
2019-08-07T13:52:02.232-0400 [DEBUG] worker: submitted plan for evaluation: eval_id=e96e699d-7bb3-db3f-8479-7a1fc5777989
2019-08-07T13:52:02.233-0400 [DEBUG] worker.service_sched: setting eval status: eval_id=e96e699d-7bb3-db3f-8479-7a1fc5777989 job_id=kafka namespace=default status=complete
2019-08-07T13:52:02.233-0400 [DEBUG] worker: updated evaluation: eval="<Eval "e96e699d-7bb3-db3f-8479-7a1fc5777989" JobID: "kafka" Namespace: "default">"
2019-08-07T13:52:02.233-0400 [DEBUG] worker: ack evaluation: eval_id=e96e699d-7bb3-db3f-8479-7a1fc5777989
2019-08-07T13:52:02.235-0400 [DEBUG] client: updated allocations: index=12 total=1 pulled=1 filtered=0
2019-08-07T13:52:02.235-0400 [DEBUG] client: allocation updates: added=1 removed=0 updated=0 ignored=0
2019-08-07T13:52:02.235-0400 [DEBUG] http: request complete: method=GET path=/v1/evaluation/e96e699d-7bb3-db3f-8479-7a1fc5777989/allocations?region=global duration=334.321µs
2019-08-07T13:52:02.236-0400 [DEBUG] client: allocation updates applied: added=1 removed=0 updated=0 ignored=0 errors=0
2019-08-07T13:52:02.239-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=320da715-de45-7da0-12cb-2b052fea02db task=kafka path=/Users/manishk/development/nomad args="[/Users/manishk/development/nomad logmon]"
2019-08-07T13:52:02.241-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=320da715-de45-7da0-12cb-2b052fea02db task=kafka path=/Users/manishk/development/nomad pid=48604
2019-08-07T13:52:02.241-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=320da715-de45-7da0-12cb-2b052fea02db task=kafka path=/Users/manishk/development/nomad
2019-08-07T13:52:02.283-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=320da715-de45-7da0-12cb-2b052fea02db task=kafka @module=logmon address=/var/folders/38/n72qm7yj0c75hzs_ybwzy2q80000gr/T/plugin689846539 network=unix timestamp=2019-08-07T13:52:02.283-0400
2019-08-07T13:52:02.283-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=320da715-de45-7da0-12cb-2b052fea02db task=kafka version=2
2019-08-07T13:52:02.285-0400 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=320da715-de45-7da0-12cb-2b052fea02db task=kafka @module=logmon path=/private/var/folders/38/n72qm7yj0c75hzs_ybwzy2q80000gr/T/NomadClient443266025/320da715-de45-7da0-12cb-2b052fea02db/alloc/logs/.kafka.stdout.fifo timestamp=2019-08-07T13:52:02.285-0400
2019-08-07T13:52:02.285-0400 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=320da715-de45-7da0-12cb-2b052fea02db task=kafka @module=logmon path=/private/var/folders/38/n72qm7yj0c75hzs_ybwzy2q80000gr/T/NomadClient443266025/320da715-de45-7da0-12cb-2b052fea02db/alloc/logs/.kafka.stderr.fifo timestamp=2019-08-07T13:52:02.285-0400
2019-08-07T13:52:02.303-0400 [DEBUG] client.driver_mgr.docker: image reference count incremented: driver=docker image_name=itspanwar/kafka:2.2.1 image_id=sha256:96100d3672f4251bf1ccc87c47c836e13df8208fedfec2abb1658105e60e283e references=1
2019-08-07T13:52:02.303-0400 [DEBUG] client.driver_mgr.docker: configured resources: driver=docker task_name=kafka memory=1073741824 cpu_shares=1024 cpu_quota=0 cpu_period=0
2019-08-07T13:52:02.303-0400 [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=kafka binds="[]string{"/private/var/folders/38/n72qm7yj0c75hzs_ybwzy2q80000gr/T/NomadClient443266025/320da715-de45-7da0-12cb-2b052fea02db/alloc:/alloc", "/private/var/folders/38/n72qm7yj0c75hzs_ybwzy2q80000gr/T/NomadClient443266025/320da715-de45-7da0-12cb-2b052fea02db/kafka/local:/local", "/private/var/folders/38/n72qm7yj0c75hzs_ybwzy2q80000gr/T/NomadClient443266025/320da715-de45-7da0-12cb-2b052fea02db/kafka/secrets:/secrets"}"
2019-08-07T13:52:02.303-0400 [DEBUG] client.driver_mgr.docker: allocated static port: driver=docker task_name=kafka ip=127.0.0.1 port=9092
2019-08-07T13:52:02.303-0400 [DEBUG] client.driver_mgr.docker: exposed port: driver=docker task_name=kafka port=9092
2019-08-07T13:52:02.303-0400 [DEBUG] client.driver_mgr.docker: allocated static port: driver=docker task_name=kafka ip=127.0.0.1 port=9999
2019-08-07T13:52:02.303-0400 [DEBUG] client.driver_mgr.docker: exposed port: driver=docker task_name=kafka port=9999
2019-08-07T13:52:02.303-0400 [DEBUG] client.driver_mgr.docker: allocated static port: driver=docker task_name=kafka ip=127.0.0.1 port=8778
2019-08-07T13:52:02.303-0400 [DEBUG] client.driver_mgr.docker: exposed port: driver=docker task_name=kafka port=8778
2019-08-07T13:52:02.303-0400 [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=kafka container_name=kafka-320da715-de45-7da0-12cb-2b052fea02db
2019-08-07T13:52:02.306-0400 [DEBUG] client: updated allocations: index=14 total=1 pulled=0 filtered=1
2019-08-07T13:52:02.307-0400 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1
2019-08-07T13:52:02.307-0400 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0
2019-08-07T13:52:03.238-0400 [DEBUG] http: request complete: method=GET path=/v1/evaluation/e96e699d-7bb3-db3f-8479-7a1fc5777989?region=global duration=826.08µs
2019-08-07T13:52:03.240-0400 [DEBUG] http: request complete: method=GET path=/v1/evaluation/e96e699d-7bb3-db3f-8479-7a1fc5777989/allocations?region=global duration=177.417µs
2019-08-07T13:52:17.383-0400 [DEBUG] client.driver_mgr.docker: failed to create container: driver=docker container_name=kafka-320da715-de45-7da0-12cb-2b052fea02db image_name=itspanwar/kafka:2.2.1 image_id=itspanwar/kafka:2.2.1 attempt=1 error="no such image"
2019-08-07T13:52:17.383-0400 [ERROR] client.driver_mgr.docker: failed to create container: driver=docker error="no such image"
2019-08-07T13:52:17.383-0400 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=320da715-de45-7da0-12cb-2b052fea02db task=kafka error="failed to create container: no such image"
2019-08-07T13:52:17.383-0400 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=320da715-de45-7da0-12cb-2b052fea02db task=kafka reason="Restart within policy" delay=27.251689022s
2019-08-07T13:52:17.511-0400 [DEBUG] client: updated allocations: index=15 total=1 pulled=0 filtered=1
2019-08-07T13:52:17.511-0400 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1

2019-08-07T13:52:17.511-0400 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0 

[Lang Martin]

I think the next step is to make sure that `kafka-telegraf-config` is an existing docker volume. The `no such image` error may be returned from the attempt to mount the volume, but reported with the name of the container image. In particular, the portworx driver requires that the volume exists before it can be mounted.

Lang

[Manish Kumar]

I have tried creating the volume manually and still the same problem. Anyways I don't think one should need to create the volume beforehand as Docker don't force you to do that eaither - you specify the volume and Docker create it automatically.

[Lang Martin]

My understanding is that portworx, set by `driver_config = pxd` imposes the requirement that the volume has to exist ahead of time. Have you tried mounting the image with docker directly with the portworx image mounted?

[Manish Kumar]

I didn't realized that portworx has to be explicitly mounted. Can't Nomad create mounted volume the way you can do in Docker? e.g.  `docker run --rm -v telegraf-config:/etc/telegraf/telegraf.d/kafka itspanwar/telegraf-wavefront:1.11.3`

[Manish Kumar]

I got it working by removing the pxd driver.