arp incomplete

[Chad Cropper]

Has anyone dealt with network issues where the assigned MetalLB IPs are left in arp incomplete state? Setup is below:

3x Physical Servers ArchLinux (bridge with bonding 2x10G LACP)

LXD 4.2 Cluster

Containers: kmaster1 + kworkers[1-3] (Centos7) with single eth0 each

Docker 19.03

kubeadm installation + flannel

Configured Dashboard

Installed MetalLB via yaml from GIT repo tag v0.9.3

Config Map

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 191.168.203.191-192.168.203.200

Added nginx deployment

# kubectl get all --all-namespaces
NAMESPACE              NAME                                            READY   STATUS    RESTARTS   AGE
default                pod/nginx-6db489d4b7-4zmrw                      1/1     Running   1          43m
kube-system            pod/coredns-6955765f44-h94kh                    1/1     Running   1          19h
kube-system            pod/coredns-6955765f44-nj9j8                    1/1     Running   1          19h
kube-system            pod/etcd-kmaster1                               1/1     Running   1          19h
kube-system            pod/kube-apiserver-kmaster1                     1/1     Running   1          19h
kube-system            pod/kube-controller-manager-kmaster1            1/1     Running   1          19h
kube-system            pod/kube-flannel-ds-amd64-j62ws                 1/1     Running   1          19h
kube-system            pod/kube-flannel-ds-amd64-mtk98                 1/1     Running   3          19h
kube-system            pod/kube-flannel-ds-amd64-n5nv4                 1/1     Running   5          19h
kube-system            pod/kube-flannel-ds-amd64-sw4mc                 1/1     Running   1          19h
kube-system            pod/kube-proxy-5mvf8                            1/1     Running   2          19h
kube-system            pod/kube-proxy-g4zvh                            1/1     Running   1          19h
kube-system            pod/kube-proxy-hr6wz                            1/1     Running   1          19h
kube-system            pod/kube-proxy-mjhfb                            1/1     Running   2          19h
kube-system            pod/kube-scheduler-kmaster1                     1/1     Running   1          19h
kubernetes-dashboard   pod/dashboard-metrics-scraper-c79c65bb7-k5jkb   1/1     Running   1          19h
kubernetes-dashboard   pod/kubernetes-dashboard-55fd8c78bd-fgrjt       1/1     Running   1          19h
metallb-system         pod/controller-5c9894b5cd-2tsmv                 1/1     Running   1          19h
metallb-system         pod/speaker-6tkxm                               1/1     Running   1          19h
metallb-system         pod/speaker-bt7zk                               1/1     Running   2          19h
metallb-system         pod/speaker-lpcfv                               1/1     Running   1          19h
metallb-system         pod/speaker-t92hr                               1/1     Running   2          19h

NAMESPACE              NAME                                TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)                  AGE
default                service/kubernetes                  ClusterIP      10.96.0.1        <none>            443/TCP                  19h
default                service/nginx                       LoadBalancer   10.104.106.69    191.168.203.191   80:32305/TCP             16h
kube-system            service/kube-dns                    ClusterIP      10.96.0.10       <none>            53/UDP,53/TCP,9153/TCP   19h
kubernetes-dashboard   service/dashboard-metrics-scraper   ClusterIP      10.105.229.140   <none>            8000/TCP                 19h
kubernetes-dashboard   service/kubernetes-dashboard        ClusterIP      10.101.188.210   <none>            443/TCP                  19h

NAMESPACE        NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-system      daemonset.apps/kube-flannel-ds-amd64     4         4         4       4            4           <none>                        19h
kube-system      daemonset.apps/kube-flannel-ds-arm       0         0         0       0            0           <none>                        19h
kube-system      daemonset.apps/kube-flannel-ds-arm64     0         0         0       0            0           <none>                        19h
kube-system      daemonset.apps/kube-flannel-ds-ppc64le   0         0         0       0            0           <none>                        19h
kube-system      daemonset.apps/kube-flannel-ds-s390x     0         0         0       0            0           <none>                        19h
kube-system      daemonset.apps/kube-proxy                4         4         4       4            4           beta.kubernetes.io/os=linux   19h
metallb-system   daemonset.apps/speaker                   4         4         4       4            4           beta.kubernetes.io/os=linux   19h

NAMESPACE              NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
default                deployment.apps/nginx                       1/1     1            1           16h
kube-system            deployment.apps/coredns                     2/2     2            2           19h
kubernetes-dashboard   deployment.apps/dashboard-metrics-scraper   1/1     1            1           19h
kubernetes-dashboard   deployment.apps/kubernetes-dashboard        1/1     1            1           19h
metallb-system         deployment.apps/controller                  1/1     1            1           19h

NAMESPACE              NAME                                                  DESIRED   CURRENT   READY   AGE
default                replicaset.apps/nginx-6db489d4b7                      1         1         1       16h
kube-system            replicaset.apps/coredns-6955765f44                    2         2         2       19h
kubernetes-dashboard   replicaset.apps/dashboard-metrics-scraper-c79c65bb7   1         1         1       19h
kubernetes-dashboard   replicaset.apps/kubernetes-dashboard-55fd8c78bd       1         1         1       19h

metallb-system         replicaset.apps/controller-5c9894b5cd                 1         1         1       19h

kmaster1

arp -a

? (10.244.0.5) at 7e:6c:02:b8:fb:1a [ether] on cni0
? (10.244.2.0) at c6:ff:eb:d7:91:32 [ether] PERM on flannel.1
gateway (192.168.203.1) at 1c:df:0f:d8:40:43 [ether] on eth0
kworker3 (192.168.203.80) at 00:16:3e:4b:00:d4 [ether] on eth0
? (10.244.0.4) at da:e6:5f:8f:79:9a [ether] on cni0
kworker1 (192.168.203.78) at 00:16:3e:43:a2:c6 [ether] on eth0
kworker2 (192.168.203.79) at 00:16:3e:7e:91:4c [ether] on eth0
? (10.244.1.0) at da:47:94:02:86:3b [ether] PERM on flannel.1
? (192.168.203.191) at <incomplete> on eth0
? (10.244.3.0) at 4e:54:3f:bb:d0:15 [ether] PERM on flannel.1

[Makrand]

Hi Chad,

Is incomplete causing you any issues accessing the nginx web server on external IP (from the browser)?

I did some testing in my lab (1 master, 2 workers. cluster running on LXD containers). Apparently, when you ping the external IP from master, the MAC address of the worker node hosting the pod should come up in arp. 

makrand@mint-gl63:~$ lxc exec kmaster -- ping 10.70.241.51 -c 4
PING 10.70.241.51 (10.70.241.51) 56(84) bytes of data.
From 10.70.241.207: icmp_seq=2 Redirect Host(New nexthop: 10.70.241.51)
From 10.70.241.207: icmp_seq=3 Redirect Host(New nexthop: 10.70.241.51)
From 10.70.241.207 icmp_seq=1 Destination Host Unreachable
From 10.70.241.207 icmp_seq=4 Destination Host Unreachable

makrand@mint-gl63:~$ lxc exec kmaster -- arp -a
? (169.254.169.254) at <incomplete> on eth0
? (10.244.0.39) at d2:fc:44:26:c8:94 [ether] on cni0
kworker2.lxd (10.70.241.207) at 00:16:3e:50:b2:7f [ether] on eth0
? (10.70.241.51) at 00:16:3e:50:b2:7f [ether] on eth0
kworker1.lxd (10.70.241.148) at 00:16:3e:6a:72:10 [ether] on eth0
mint-gl63 (10.70.241.1) at fe:13:f8:58:75:33 [ether] on eth0
? (10.244.0.38) at 16:b6:da:50:47:25 [ether] on cni0
? (10.244.2.0) at 4e:92:06:40:cc:02 [ether] PERM on flannel.1
? (10.244.1.0) at fe:93:e9:8a:86:8b [ether] PERM on flannel.1

In a pure network scene...incomplete means the destination host is not responding on network/LAN. 

Have you tried moving the POD to another worker node?

--

Makrand

--
You received this message because you are subscribed to the Google Groups "metallb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metallb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/metallb-users/1d70c76c-e66a-488a-851a-8a154082efa2o%40googlegroups.com.

[Chad Cropper]

Issue was resolved by setting the kube-proxy to ipvs and setting strictarp=true

Thanks.