Docker NAT interaction with iptables-persistent package

[Brian Candler]

This is on an Ubuntu 14.04 host.

There appears to be a bad interaction between Docker's NAT setup and the iptables-persistent package. On a system reboot, docker rules are lost. I believe that docker is setting up NAT *before* iptables-persistent reloads its ruleset, and therefore the Docker rules are lost.

To demonstrate:

1. set up some rules for the iptables-persistent package (they are stored in /etc/iptables/rules.v4). I have the following:

root@kit1:~# cat /etc/iptables/rules.v4

# Generated by iptables-save v1.4.12 on Wed Oct 23 14:45:33 2013

*nat

:PREROUTING ACCEPT [0:0]

:INPUT ACCEPT [0:0]

:OUTPUT ACCEPT [0:0]

:POSTROUTING ACCEPT [0:0]

-A POSTROUTING -s 10.0.0.0/8 -d 10.0.0.0/8 -j ACCEPT

-A POSTROUTING -s 10.0.0.0/8 -d 172.16.0.0/12 -j ACCEPT

-A POSTROUTING -s 10.0.0.0/8 -d 192.168.0.0/16 -j ACCEPT

-A POSTROUTING -s 10.0.0.0/8 -o br-wan -j MASQUERADE

-A POSTROUTING -s 10.0.0.0/8 -o wlan0 -j MASQUERADE

COMMIT

# Completed on Wed Oct 23 14:45:33 2013

# Generated by iptables-save v1.4.12 on Wed Oct 23 14:45:33 2013

*filter

:INPUT ACCEPT [899:65540]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [571:78761]

COMMIT

# Completed on Wed Oct 23 14:45:33 2013

2. install docker. Everything is fine, and docker adds its NAT rules

3. reboot. The docker NAT rules are gone :-(

4. "service docker restart" and the docker NAT rules are back again.

(See below for output from steps 3 and 4)

I suspect the solution is to fiddle with boot order dependencies so that docker starts *after* iptables-persistent, but I don't really know how to do that.

Regards,

Brian.

root@kit1:~# iptables -L -n -v
Chain INPUT (policy ACCEPT 4411 packets, 334K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 4363 packets, 328K bytes)
 pkts bytes target     prot opt in     out     source               destination
root@kit1:~# iptables -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 14 packets, 1548 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 8 packets, 1194 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 833 packets, 52128 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 669 packets, 42028 bytes)
 pkts bytes target     prot opt in     out     source               destination
  161  9684 ACCEPT     all  --  *      *       10.0.0.0/8           10.0.0.0/8
    0     0 ACCEPT     all  --  *      *       10.0.0.0/8           172.16.0.0/12
    0     0 ACCEPT     all  --  *      *       10.0.0.0/8           192.168.0.0/16
    3   416 MASQUERADE  all  --  *      br-wan  10.0.0.0/8           0.0.0.0/0
    0     0 MASQUERADE  all  --  *      wlan0   10.0.0.0/8           0.0.0.0/0
root@kit1:~# service docker restart
docker stop/waiting
docker start/running, process 21015
root@kit1:~# iptables -L -n -v
Chain INPUT (policy ACCEPT 49 packets, 3525 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 28 packets, 2769 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination
root@kit1:~# iptables -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
  161  9684 ACCEPT     all  --  *      *       10.0.0.0/8           10.0.0.0/8
    0     0 ACCEPT     all  --  *      *       10.0.0.0/8           172.16.0.0/12
    0     0 ACCEPT     all  --  *      *       10.0.0.0/8           192.168.0.0/16
   67  5387 MASQUERADE  all  --  *      br-wan  10.0.0.0/8           0.0.0.0/0
    0     0 MASQUERADE  all  --  *      wlan0   10.0.0.0/8           0.0.0.0/0

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination

[Dave Tucker]

If you only want to use iptables-persistent for the rules you have saved
already, it should just be a case of changing Docker's upstart script to
include "start on started iptables-persistent" (assuming you are using
upstart).

If you want to save the DOCKER chain too, you should 'iptables-save >
/etc/iptables/rules.v4' after Docker is installed and running. I'd only
recommend doing this is you know *exactly* what you are doing though as
I can think of cases where it may have some unexpected results.

-- Dave

> --
> You received this message because you are subscribed to the Google
> Groups "docker-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to docker-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Brian Candler]

Thank you for your suggestions.

/etc/init/docker.conf had:

    start on (local-filesystems and net-device-up IFACE!=lo)

I tried adding

    start on iptables-persistent

as an additional line, and I also tried changing it to

    start on (local-filesystems and net-device-up IFACE!=lo and started iptables-persistent)

but in both cases the docker daemon didn't start on reboot.

Looking at the iptables-persistent package, it looks like it hasn't been converted to upstart:

$ dpkg-query -L iptables-persistent | grep etc

/etc

/etc/iptables

/etc/init.d

/etc/init.d/iptables-persistent

Regards,

Brian.

[Olivier Voortman]

Hello,

I had the same issue.
Docker daemon should start after iptables-persistent.

To do this I changed /etc/init/docker.conf with this (as suggested) :

start on (local-filesystems and net-device-up IFACE!=lo and started iptables-persistent)
 

And added some emits in /etc/init.d/iptables-persistent to signal upstart it has started :

At the end of start section :
initctl emit -n started JOB=iptables-persistent

At the beginning of stop section :
initctl emit stopping JOB=iptables-persistent

It works properly now.

Regards,

Olivier