Inefficient docker0 mtu on GKE: sets it as 1500 when eth0 is 1460, TLS performance impact

[Don Bowman]

Google Cloud has an MTU of 1460. And this is propagated via DHCP so that when you create an Instance, it has that properly on eth0.

But, when setting up GKE, it sets up a MTU of 1500 on the docker0 bridge.

This means that containers are doing path-mtu-discovery (PMTU) as they route back, and, getting ICMP would frag from their own docker0 address.

In practise this leads to poor performance on e.g. TLS. We can see this in the image below.

Non-image:

 ClientIP -> ClusterIP SYN

 ClusterIP -> ClientIP SYN/ACK

 ClientIP -> ClusterIP ACK

 ClientIP -> ClusterIP Client Hello

 ClusterIP -> ClientIP ACK

 ClusterIP -> ClientIP Server Hello [>1460 payload]

 docker0 -> ClusterIP ICMP Would Fragment

 ClusterIP -> CLientIP Server Hello Done [this is an error, it still has the hello in flight]

 ClusterIP -> ClientIP : out of order, now re-segment and resent server hello

Now, If i login to each of the nodes that were created by the GKE template, I can set the MTU manually on docker0.

What is the right way to do this? It has a fairly big affect on connection setup performance.

Is this only an issue on GKE? or more general?

this is with calico if it matters (not sure it does, this is internal to the node).

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP group default qlen 1000

    link/ether 42:01:0a:a2:00:03 brd ff:ff:ff:ff:ff:ff

    inet 10.162.0.3/32 brd 10.162.0.3 scope global dynamic eth0

       valid_lft 58015sec preferred_lft 58015sec

    inet6 fe80::4001:aff:fea2:3/64 scope link 

       valid_lft forever preferred_lft forever

3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 

    link/ether 02:42:93:c1:14:a1 brd ff:ff:ff:ff:ff:ff

    inet 169.254.123.1/24 scope global docker0

       valid_lft forever preferred_lft forever

[Tim Hockin]

We don't use docker0.  It's an artifact of docker starting up and creating it.  Kubernetes on GKE uses either cbr0 (Calico NetworkPolicy disabled) or no bridge at all (Calico enabled).

If you `ifconfig docker0` you should see something like this:

```

$ ifconfig docker0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500

        inet 169.254.123.1  netmask 255.255.255.0  broadcast 0.0.0.0

        ether 02:42:76:62:c8:98  txqueuelen 0  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 0  bytes 0 (0.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

```

Note "0 bytes" all over.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Don Bowman]

On Monday, June 11, 2018 at 11:48:25 AM UTC-4, Tim Hockin wrote:

We don't use docker0.  It's an artifact of docker starting up and creating it.  Kubernetes on GKE uses either cbr0 (Calico NetworkPolicy disabled) or no bridge at all (Calico enabled).

If you `ifconfig docker0` you should see something like this:

OK thank you, you are correct.

But, the container ifconfig is '1500' and the host is 1460, so this problem exists. So I'm not sure where the container gets the 1500 from.

I guess an iptables rule on the nodes doing MSS clamping would be another option. Currently it advertises MSS of 1460, it should be 1420 for GKE.

In a container:

# ip a

3: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 

    link/ether 06:85:37:8c:e7:be brd ff:ff:ff:ff:ff:ff link-netnsid 0

    inet 10.16.1.10/32 scope global eth0

       valid_lft forever preferred_lft forever

    inet6 fe80::485:37ff:fe8c:e7be/64 scope link 

       valid_lft forever preferred_lft forever

 

On the node:

# ip a

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP group default qlen 1000

    link/ether 42:01:0a:a2:00:03 brd ff:ff:ff:ff:ff:ff

    inet 10.162.0.3/32 brd 10.162.0.3 scope global dynamic eth0

       valid_lft 49712sec preferred_lft 49712sec

[Tim Hockin]

Are you running with calico on or off?

[Don Bowman]

This kubernetes cluster is using calico. it is not set to enforce tho.

[Tim Hockin]

Thanks, That aligns with what I am seeing. We'll investigate.