Why does docker.sock inspect of Docker image in a mesos-slave (0.28.1-2.20) fail?

[David Vogel]

When I try to run this Marathon application on a Mesos cluster in AWS GovCloud:

{
  "id": "gogs",
  "cpus": 0.5,
  "mem": 64.0,
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "docker": {
      "network": "HOST",
      "image": "gogs/gogs",
      "portMappings": [ "10022:22", "10030:3000" ]
    },
    "volumes": [
      {
        "containerPath": "/var/run/docker.sock",
        "hostPath": "/var/run/docker.sock",
        "mode": "RW"
      },
      {
        "containerPath": "/usr/local/bin/docker",
        "hostPath": "/usr/bin/docker",
        "mode": "RW"
      },
      {
        "containerPath": "/data",
        "hostPath": "/tmp",
        "mode": "RW"
      }
    ]
  }
}

I get the following error on the mesos-slave executor:

WARNING: Your kernel does not support swap limit capabilities, memory limited without swap.
I0524 18:34:39.936663     6 logging.cpp:193] Logging to STDERR
I0524 18:34:39.938588     6 process.cpp:986] libprocess is initialized on 10.0.0.115:45874 for 8 cpus
I0524 18:34:39.939527     6 exec.cpp:143] Version: 0.28.1
I0524 18:34:39.941130    13 exec.cpp:193] Executor started at: executor(1)@10.0.0.115:45874 with pid 6
I0524 18:34:39.942661    11 exec.cpp:217] Executor registered on slave 3f1ecc5c-124d-4433-b3ea-16f4e1df2400-S1
I0524 18:34:39.943501    11 exec.cpp:229] Executor::registered took 103369ns
I0524 18:34:39.943758    11 exec.cpp:304] Executor asked to run task 'gogs.25fa84b7-21de-11e6-a598-0242a81cec92'
I0524 18:34:39.943835    11 exec.cpp:313] Executor::launchTask took 55162ns
I0524 18:34:39.944028    14 docker.cpp:799] Running docker -H unix:///var/run/docker.sock inspect mesos-3f1ecc5c-124d-4433-b3ea-16f4e1df2400-S1.4bb9b9fe-cc81\
-4bd6-b795-248563596cab
I0524 18:34:40.044986    14 docker.cpp:846] Retrying inspect with non-zero status code. cmd: 'docker -H unix:///var/run/docker.sock inspect mesos-3f1ecc5c-12\
4d-4433-b3ea-16f4e1df2400-S1.4bb9b9fe-cc81-4bd6-b795-248563596cab', interval: 500ms
I0524 18:34:40.545833    15 docker.cpp:799] Running docker -H unix:///var/run/docker.sock inspect mesos-3f1ecc5c-124d-4433-b3ea-16f4e1df2400-S1.4bb9b9fe-cc81\
-
4bd6-b795-248563596cab
I0524 18:34:40.648707     7 docker.cpp:846] Retrying inspect with non-zero status code. cmd: 'docker -H unix:///var/run/docker.sock inspect mesos-3f1ecc5c-12\
4d-4433-b3ea-16f4e1df2400-S1.4bb9b9fe-cc81-4bd6-b795-248563596cab', interval: 500ms
I0524 18:34:41.149451    15 docker.cpp:799] Running docker -H unix:///var/run/docker.sock inspect mesos-3f1ecc5c-124d-4433-b3ea-16f4e1df2400-S1.4bb9b9fe-cc81\
-4bd6-b795-248563596cab

      and Mesos keeps retrying to run docker.sock inspect on the image.

I can run a different, simpler, Marathon application without getting an error:

{
  "id": "dvogel-0524-test2-4",
  "cpus": 1,
  "mem": 32.0,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "tutum/hello-world"
    }
  }
}

        Here is how I start the mesos-slave, inside a Mesosphere Docker image, on an AWS Ubuntu 14.04 instance:

docker run --name mesos-slave -d --net=host --privileged --pid=host \
  -e GLOG_v=1 \
  -e USER=root \
  -e MESOS_PORT=5051 \
  -e MESOS_MASTER=zk://ip-10-0-0-117.us-gov-west-1.compute.internal:2181/mesos \
  -e MESOS_SWITCH_USER=0 \
  -e MESOS_CONTAINERIZERS=docker \
  -e MESOS_LOG_DIR=/var/log/mesos \
  -e MESOS_WORK_DIR=/var/tmp/mesos \
  -e "MESOS_RESOURCES=ports:[1-65000]" \
  -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=300000 \
  -v "/var/log/mesos:/var/log/mesos" \
  -v "/var/tmp/mesos:/var/tmp/mesos" \
  -v /var/run/docker.sock:/run/docker.sock \
  -v /sys/fs/cgroup:/sys/fs/cgroup \
  -v /sys:/sys \
mesosphere/mesos-slave:0.28.1-2.0.20.ubuntu1404