Docker fails with dependency (iptables?)

[Pan Cake]

Hi,

I'm on CoreOS 1235.4.0 (stable), on a DigitalOcean droplet.

I've been using the same cloud-config file for months, but now I can't get the docker service to run automatically at startup.

systemctl --failed
UNIT                           LOAD   ACTIVE SUB           DESCRIPTION
● docker.service               loaded failed failed        Docker Application Container Engine
● iptables-restore.service     loaded failed failed        iptables rules
● docker.socket                loaded failed failed        Docker Socket for the API

As you can see, 3 units failed to start.

journalctl -u docker
systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE

systemd[1]: docker.service: Unit entered failed state.

systemd[1]: docker.service: Failed with result 'exit-code'.

systemd[1]: Dependency failed for Docker Application Container Engine.

systemd[1]: docker.service: Job docker.service/start failed with result 'dependency'.
...

Here, I want to know why Docker is failing.  Apparently because of a dependency.

What is this dependency?  I suspect a problem with iptables, since it's also failing.

journalctl -u iptables-restore
systemd[1]: Starting iptables rules...
systemd[1]: Started iptables rules.

systemd[1]: iptables-restore.service: Start request repeated too quickly.

systemd[1]: Failed to start iptables rules.

systemd[1]: iptables-restore.service: Unit entered failed state.

systemd[1]: iptables-restore.service: Failed with result 'start-limit-hit'.

systemd[1]: iptables-restore.service: Start request repeated too quickly.

systemd[1]: Failed to start iptables rules.

...

It's really weird, because:

sudo iptables -S
-P INPUT DROP
-P FORWARD DROP
-P OUTPUT ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -d 127.0.0.0/24 -p tcp -m tcp --dport 2375 -j ACCEPT
-A INPUT -d 172.17.0.0/24 -p tcp -m tcp --dport 2375 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT

My rules are loaded!  So what has been wrong in the startup process?

systemctl cat docker.service
# /etc/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=docker.socket early-docker.target network.target iptables-restore.service
Requires=docker.socket early-docker.target iptables-restore.service


[Service]
Environment="DOCKER_CGROUPS=--exec-opt native.cgroupdriver=systemd"
EnvironmentFile=-/run/flannel_docker_opts.env
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
ExecStart=/usr/lib/coreos/dockerd daemon --ip=0.0.0.0 --host=fd:// --insecure-registry 0.0.0.0:5000 $DOCKER_OPTS $DOCKER_CGROUPS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ


[Install]
WantedBy=multi-user.target


# /etc/systemd/system/docker.service.d/10-debug.conf
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
# /etc/systemd/system/docker.service.d/30-increase-ulimit.conf
[Service]
LimitMEMLOCK=infinity

Since iptables-restore.service is REQUIRED for Docker to start, I guess this is why Docker isn't starting...

At this point, can anyone tell me how I can investigate any further?

My cloud-config file is working very well with older versions of CoreOS.  For example, I have CoreOS 1185.5.0 working perfectly fine, and has been built with this cloud-config file.

Oh, and by the way, there's the cloud-config in question:

#cloud-config


coreos:
  update:
    reboot-strategy: "off"
    group: "stable"

  units:
    - name: update-engine.service
      command: stop

    - name: locksmithd.service
      command: stop

    - name: iptables-restore.service
      command: start
      enable: true
      content: |
        [Unit]
        Description=iptables rules
        After=network-online.target
        Requires=network-online.target
        Before=docker.service
       
        [Service]
        Type=oneshot
        ExecStart=/sbin/iptables-restore /var/lib/iptables/rules-save

    - name: docker.service
      command: restart
      enable: true
      content: |
        [Unit]
        Description=Docker Application Container Engine
        Documentation=http://docs.docker.com
        After=docker.socket early-docker.target network.target iptables-restore.service
        Requires=docker.socket early-docker.target iptables-restore.service

        [Service]
        Environment="DOCKER_CGROUPS=--exec-opt native.cgroupdriver=systemd"
        EnvironmentFile=-/run/flannel_docker_opts.env
        MountFlags=slave
        LimitNOFILE=1048576
        LimitNPROC=1048576
        ExecStart=/usr/lib/coreos/dockerd daemon --ip=0.0.0.0 --host=fd:// --insecure-registry 0.0.0.0:5000 $DOCKER_OPTS $DOCKER_CGROUPS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ

        [Install]
        WantedBy=multi-user.target

    - name: docker.service
      drop-ins:
        - name: 10-debug.conf
          content: |
            [Service]
            Environment=SYSTEMD_LOG_LEVEL=debug
        - name: 30-increase-ulimit.conf
          content: |
            [Service]
            LimitMEMLOCK=infinity
      command: restart

    - name: docker.socket
      command: start
      drop-ins:
        - name: 30-ListenStream.conf
          content: |
            [Socket]
            ListenStream=2375

    - name: env.service
      command: start
      enable: false
      content: |
        [Unit]
        Description=Some ENV variables to set at startup


        [Service]
        Type=oneshot
        ExecStart=/bin/sh -c "export HOSTNAME=$(curl -s http://169.254.169.254/metadata/v1/hostname); export PUBLIC_IPV4=$(curl -s http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address);"

write_files:
    - path: /var/lib/iptables/rules-save
      permissions: 0644
      owner: root:root
      content: |
        *filter
        :INPUT DROP [0:0]
        :FORWARD DROP [0:0]
        :OUTPUT ACCEPT [0:0]

        # Accept all loopback traffic:
        -A INPUT -i lo -j ACCEPT

        # Accept all TCP/IP traffic to these ports:
        -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

        # Keep existing connections alive:
        -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

        # Accept all local traffic to the Docker Remote API
        -A INPUT -p tcp -m tcp --dst 127.0.0.1/24 --dport 2375 -j ACCEPT
        -A INPUT -p tcp -m tcp --dst 172.17.0.1/24 --dport 2375 -j ACCEPT

        # Accept pings:
        -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
        -A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT
        -A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT
        COMMIT

It's been a while since I updated my cloud-config file.  I guess recent changes in CoreOS (or docker itself) might prevent my droplet for starting required services automatically.

Thanks for your help!

[Alex Crawford]

On 01/08, Pan Cake wrote:
> Here, I want to know why Docker is failing. Apparently because of a
> dependency.
> What is this dependency? I suspect a problem with iptables, since it's
> also failing.

You can use `systemctl list-dependencies docker` to list the
dependencies and view their statuses. Once you've done that, you can
check the journal for the failed services and determine why they failed.

> My rules are loaded! So what has been wrong in the startup process?

You can add the `--verbose` flag to the command (maybe we should do that
by default) to get a bit more information.


> *systemctl cat docker.service*
> # /etc/systemd/system/docker.service

You generally don't want to supercede the docker.service we ship in the
OS. All of the customizations you'll need to make can be done via
drop-ins. As we change the OS and docker, this might break your service
(e.g. we are removing early-docker.target and have split out containerd
into a seperate service).

> # /etc/systemd/system/docker.service.d/10-debug.conf
> [Service]
> Environment=SYSTEMD_LOG_LEVEL=debug

I don't think this will have any effect on docker. Most (if not all) of
the systemd utilities respect this variable, but docker doesn't know or
care about it.

> coreos:
> update:
> reboot-strategy: "off"

This makes me sad. Can I ask why you are disabling automatic updates?

> - name: iptables-restore.service
> command: start
> enable: true
> content: |
> [Unit]
> Description=iptables rules
> After=network-online.target
> Requires=network-online.target
> Before=docker.service
>
> [Service]
> Type=oneshot
> ExecStart=/sbin/iptables-restore /var/lib/iptables/rules-save

This seems suspect. The default service is run before network.target,
but in your case, it is run after.

> - name: env.service
> command: start
> enable: false
> content: |
> [Unit]
> Description=Some ENV variables to set at startup
>
>
> [Service]
> Type=oneshot
> ExecStart=/bin/sh -c "export HOSTNAME=$(curl -s
> http://169.254.169.254/metadata/v1/hostname); export PUBLIC_IPV4=$(curl -s
> http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address);"

This functionality is already provided by coreos-cloudinit (written to
/etc/environment) and coreos-metadata (written to /run/metadata/coreos).

-Alex

[Pan Cake]

Alex,

I finally found the problem!

It's because of the ExecStart line for the main docker.service file:

ExecStart=/usr/lib/coreos/dockerd daemon --ip=0.0.0.0 --host=fd:// --insecure-registry 0.0.0.0:5000 $DOCKER_OPTS $DOCKER_CGROUPS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ

This is wrong.  Since newer versions of Docker, the "daemon" parameter is deprecated.  We should not use it anymore.

I did not know this.  So I thought that my iptables-restore.service was keeping docker from starting, but I was wrong!

Removing "daemon" parameter from this command line, and everything is ok now!

On Wednesday, January 18, 2017 at 11:33:02 AM UTC-5, Alex Crawford wrote:

On 01/08, Pan Cake wrote:
> Here, I want to know why Docker is failing.  Apparently because of a
> dependency.
> What is this dependency?  I suspect a problem with iptables, since it's
> also failing.

You can use `systemctl list-dependencies docker` to list the
dependencies and view their statuses. Once you've done that, you can
check the journal for the failed services and determine why they failed.

systemctl list-dependencies docker: Thanks, I did not know about this one!

Very useful...

I also found this one, which helps me get more details about what's failing:

journalctl _PID=1234

where 1234 is the port number of the process to investigate, usually found with:

systemctl status [servicename]

 

> My rules are loaded!  So what has been wrong in the startup process?

You can add the `--verbose` flag to the command (maybe we should do that
by default) to get a bit more information.

Good idea!  iptables will seldom be a source of problems, so in this case adding the --verbose flag will help diagnose these problems.

 

> *systemctl cat docker.service*
> # /etc/systemd/system/docker.service

You generally don't want to supercede the docker.service we ship in the
OS. All of the customizations you'll need to make can be done via
drop-ins. As we change the OS and docker, this might break your service
(e.g. we are removing early-docker.target and have split out containerd
into a seperate service).

Well, believe it or not, I was adding some drop-ins I found in tutorials here and there on the net, thinking it would be useful for me.

But the thing is, I never knew the real reason of using "drop-ins".

Now with your explanation, I really understand why people should use them!

So, any customization to Docker should be done via drop-ins, to avoid service breakage after an update.  Logic!

 

> # /etc/systemd/system/docker.service.d/10-debug.conf
> [Service]
> Environment=SYSTEMD_LOG_LEVEL=debug

I don't think this will have any effect on docker. Most (if not all) of
the systemd utilities respect this variable, but docker doesn't know or
care about it.

I'll keep this one for other services then, and forget about debug for Docker.

This will help keep my cloud-config smaller  ;)

 

> coreos:
>   update:
>     reboot-strategy: "off"

This makes me sad. Can I ask why you are disabling automatic updates?

Well, 3 reasons:

1) I'm still a newbie with CoreOS

2) This server was only a test-server (not one I use in production)

3) I'm more of a conservative type of person, I prefer to keep things as-is, and usually I will update things only if NEEDED or only if a major security update is impacting me.

Note: After reading your reply, now I can "trust" automatic updates, because I will use drop-ins for any customization to the Docker service (or any other systemd utilities).

I won't fear service breakage anymore!

 

>     - name: iptables-restore.service
>       command: start
>       enable: true
>       content: |
>         [Unit]
>         Description=iptables rules
>         After=network-online.target
>         Requires=network-online.target
>         Before=docker.service
>        
>         [Service]
>         Type=oneshot
>         ExecStart=/sbin/iptables-restore /var/lib/iptables/rules-save

This seems suspect. The default service is run before network.target,
but in your case, it is run after.

 

Should I run iptables-restore.service BEFORE network-online.target?

>     - name: env.service
>       command: start
>       enable: false
>       content: |
>         [Unit]
>         Description=Some ENV variables to set at startup
>
>
>         [Service]
>         Type=oneshot
>         ExecStart=/bin/sh -c "export HOSTNAME=$(curl -s
> http://169.254.169.254/metadata/v1/hostname); export PUBLIC_IPV4=$(curl -s
> http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address);"

This functionality is already provided by coreos-cloudinit (written to
/etc/environment) and coreos-metadata (written to /run/metadata/coreos).

-Alex

Again, some stuff to remove, to lighten my cloud-config file  :D 

Alex, your answer has been very valuable to me.  I've learned many many things, just from what you said.

Thanks a lot for your help, this is really appreciated!