container not found by a unit service and unable to find image from systemd

[Jordi Cabré]

I'm trying to set a QA environtment up running my Docker images and setting up a single CoreOS VM with Vagrant with VirtualBox.

In order to do so I've solved several issues:

1. Create my owned Docker images (my web apis) and push them on my private registry.
   
2. These Docker images are using some backend technologies I set up using their official Docker images. Between them mongo.
3. Once I'm able to kick off these images, I need to feed these backend containers with some data. In order to do that, I've created an docker image, where I initialize some data on my mongo container.

My problem is related to how to configure my CoreOS instance in order to start my containers tidily and taking care of dependencies.
I'm using 4 backend containers:

• 2 memcached containers (official image)
  
• 1 elasticsearch container (official image)
• 1 mongo container (official image)
• 1 mongo-seed container in order to intialize data (private image)
• 1 webapi container (private image)

Briefly:

• Start backends engines.
  
• Feed mongo engine.
• Start webapi container.
  

Moreover, I've my webapi container that depends of the last ones. So, I need to kick off the other ones after starting it.

Until now, I've been able to weave this cloud-config:

#cloud-config

---
coreos:
  etcd2:
    discovery: https://discovery.etcd.io/2a869d708001e4b595ac273ff69d9de8
    advertise-client-urls: http://$public_ipv4:2379
    initial-advertise-peer-urls: http://$private_ipv4:2380
    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    listen-peer-urls: http://$private_ipv4:2380,http://$private_ipv4:7001
  fleet:
    public-ip: "$public_ipv4"
  flannel:
    interface: "$public_ipv4"
  units:
  - name: docker.service
    drop-ins:
    - name: 50-insecure-registry.conf
      content: |
        [Service]
        Environment=DOCKER_OPTS='--insecure-registry="private-registry.example.com"'
  - name: etcd2.service
    command: start
  - name: fleet.service
    command: start
  - name: flanneld.service
    drop-ins:
    - name: 50-network-config.conf
      content: |
        [Service]
        ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.1.0.0/16" }'
    command: start
  - name: docker-tcp.socket
    command: start
    enable: true
    content: |
      [Unit]
      Description=Docker Socket for the API

      [Socket]
      ListenStream=2375
      Service=docker.service
      BindIPv6Only=both

      [Install]
      WantedBy=sockets.target
  - name: mc1.service
    command: start
    enable: true
    content: |-
      [Unit]
      Description=MemCache-1
      Author=Living
      After=docker.service

      [Service]
      Restart=always
      ExecStartPre=-/usr/bin/docker stop mc1
      ExecStartPre=-/usr/bin/docker rm mc1
      ExecStart=/usr/bin/docker run --name mc1 -p 11211:11211 --hostname mc1 memcached -m 128 -p 11211
      ExecStop=/usr/bin/docker stop mc1
  - name: mc2.service
    command: start
    enable: true
    content: |-
      [Unit]
      Description=MemCache-2
      Author=Living
      After=docker.service

      [Service]
      Restart=always
      ExecStartPre=-/usr/bin/docker stop mc2
      ExecStartPre=-/usr/bin/docker rm mc2
      ExecStart=/usr/bin/docker run --name mc2 -p 11212:11212 --hostname mc2 memcached -m 128 -p 11212
      ExecStop=/usr/bin/docker stop mc2
  - name: mongo.service
    command: start
    enable: true
    content: |-
      [Unit]
      Description=Mongo
      Author=Living
      After=docker.service

      [Service]
      Restart=always
      RestartSec=10s
      ExecStartPre=-/usr/bin/docker stop mongo
      ExecStartPre=-/usr/bin/docker rm mongo
      ExecStart=/usr/bin/docker run --name mongo -p 27017:27017 --hostname mongo mongo:2.6
      ExecStop=/usr/bin/docker stop mongo
      ExecStopPost=-/usr/bin/docker rm mongo
  - name: mongo-seed.service
    command: start
    enable: true
    content: |-
      [Unit]
      Description=Mongo Seed
      Author=Living
      After=mongo.service

      [Service]
      Restart=always
      ExecStartPre=-/usr/bin/docker stop mongo-seed
      ExecStartPre=-/usr/bin/docker rm mongo-seed
      ExecStart=/usr/bin/docker run --name mongo-seed --link mongo:mongo private-registry.example.com/mongo-seed:v1
      ExecStop=/usr/bin/docker stop mongo-seed
  - name: es.service
    command: start
    enable: true
    content: |-
      [Unit]
      Description=ElasticSearch
      Author=Living
      After=docker.service

      [Service]
      Restart=always
      ExecStartPre=-/usr/bin/docker stop es
      ExecStartPre=-/usr/bin/docker rm es
      ExecStart=/usr/bin/docker run --name es -p 9200:9200 -p 9300:9300 --hostname es elasticsearch:2.4 -Des.node.name="esn1" -Des.cluster.name="cluster"
      ExecStop=/usr/bin/docker stop es
  - name: webapi.service
    command: start
    enable: true
    content: |-
      [Unit]
      Description=Living WebApi Platform
      Author=Living
      Wants=mongo.service es.service
      Requires=mongo.service es.service
      After=mongo.service es.service

      [Service]
      User=core
      Restart=always
      ExecStartPre=-/usr/bin/docker stop webapi
      ExecStartPre=-/usr/bin/docker rm webapi
      ExecStart=/usr/bin/docker run --name webapi -p 8080:8080 -p 9990:9990 --link mongo:mongo --link es:es --link mc1:mc1 --link mc2:mc2 --hostname webapi private-registry.example.com/webapi:v1
      ExecStop=/usr/bin/docker stop webapi

After having performed a vagrant up, only 4 container are running:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES
51e6002d16b6        mongo:2.6           "/entrypoint.sh mongo"   22 minutes ago      Up 22 minutes       0.0.0.0:27017->27017/tcp                         mongo
38475c8102e8        elasticsearch:2.4   "/docker-entrypoint.s"   22 minutes ago      Up 22 minutes       0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   es
f8ac880567b2        memcached           "docker-entrypoint.sh"   24 minutes ago      Up 24 minutes       11211/tcp, 0.0.0.0:11212->11212/tcp              mc2
a019d28630c6        memcached           "docker-entrypoint.sh"   24 minutes ago      Up 24 minutes       0.0.0.0:11211->11211/tcp                         mc1

I've watched out the logs of the systemd mongo-seed unit:

Unable to find image 'private-registry.example.com/mongo-seed:v1' locally
Sep 12 08:00:23 core-01 docker[2412]: docker: Error response from daemon: Error parsing HTTP response: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolo
Sep 12 08:00:23 core-01 docker[2412]: See '/usr/bin/docker run --help'.
Sep 12 08:00:23 core-01 systemd[1]: mongo-seed.service: Main process exited, code=exited, status=125/n/a
Sep 12 08:00:23 core-01 docker[2419]: Failed to stop container (mongo-seed): Error response from daemon: No such container: mongo-seed

However, If I perform this docker command on shell it works fine:

docker run --name mongo-seed --link mongo:mongo registry.living-digital-way.com/mongo_demo:v1
Unable to find image 'registry.living-digital-way.com/mongo_demo:v1' locally
v1: Pulling from mongo_demo
046d0f015c61: Already exists
ba95eb02831f: Already exists
53dc8636c4de: Already exists
a1ba40c46d70: Already exists
58b7d37cc7a7: Already exists
6fc4041cef29: Already exists
4cb494f83a39: Already exists
29839a673e80: Pull complete
cc731752cc1a: Pull complete
4241c596e82f: Pull complete
Digest: sha256:55688cc6bfff59f816bb26030d1098afbf56dfe30b2b0e6a9cedcd8742d8e9a5
Status: Downloaded newer image for private-registry.example.com/mongo_demo:v1
/entrypoint.sh: "Well imported"

What am I doing wrong?

Moreover, when I perform 'journalctl -u webapi.service' is telling me that docker is unable to find mongo container, even thought I set it started after

docker: Error response from daemon: Could not get container for mongo.

[Brandon Philips]

Hey Jordi-

I don't really see the issue, sorry.

Once you get to this level of complexity you should consider using Kubernetes: https://coreos.com/kubernetes/docs/latest/

Cheers,

Brandon

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.