External IP in pending

[Darin Tracy]

I'm using kind and using the steps listed here:  https://kind.sigs.k8s.io/docs/user/loadbalancer/

Very simple steps.

I was running just fine for months.  I'm not sure what has happened .  But now when I setup metallb and create a service as loadbalancer i just get pending for external IP.

Things I've looked at:

docker network inspect -f '{{.IPAM.Config}}' kind
[{172.18.0.0/16  172.18.0.1 map[]} {fc00:f853:ccd:e793::/64  fc00:f853:ccd:e793::1 map[]}]

Configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 172.18.255.200-172.18.255.250

And kubectl logs from the controler pod. (TWO things MISSING).  What is the problem?

kubectl logs -n metallb-system controller-57b74d8656-gtjxl
{"branch":"main","caller":"level.go:63","commit":"1b0a48fb","goversion":"gc / go1.16.15 / amd64","level":"info","msg":"MetalLB controller starting (commit 1b0a48fb, branch main)","ts":"2022-03-11T05:05:46.44958418Z","version":""}
{"Starting Reloader":"(MISSING)","caller":"level.go:63","level":"info","ts":"2022-03-11T05:05:47.106480567Z"}
{"Starting Manager":"(MISSING)","caller":"level.go:63","level":"info","ts":"2022-03-11T05:05:47.106518116Z"}
{"caller":"level.go:63","controller":"ServiceReconciler","level":"info","start reconcile":"default/kubernetes","ts":"2022-03-11T05:05:47.307092157Z"}
{"caller":"level.go:63","controller":"ServiceReconciler","end reconcile":"default/kubernetes","level":"info","ts":"2022-03-11T05:05:47.307349982Z"}
{"caller":"level.go:63","controller":"ServiceReconciler","level":"info","start reconcile":"kube-system/kube-dns","ts":"2022-03-11T05:05:47.307389874Z"}
{"caller":"level.go:63","controller":"ServiceReconciler","end reconcile":"kube-system/kube-dns","level":"info","ts":"2022-03-11T05:05:47.307461411Z"}
{"caller":"level.go:63","controller":"ServiceReconciler","level":"info","start reconcile":"helm-testing/foo-service","ts":"2022-03-11T05:06:19.496010896Z"}
{"caller":"level.go:63","controller":"ServiceReconciler","end reconcile":"helm-testing/foo-service","level":"info","ts":"2022-03-11T05:06:19.496472356Z"}

Any ideas/help is greatly appreciated.

Darin

[Darin Tracy]

Update:

kubectl logs from a speaker:  show this error:

"caller":"level.go:63","event":"nodeLabelsChanged","level":"info","msg":"Node labels changed, resyncing BGP peers","ts":"2022-03-11T05:49:45.546565208Z"}
{"caller":"level.go:63","controller":"NodeReconciler","end reconcile":"/kind-worker","level":"info","ts":"2022-03-11T05:49:45.546616266Z"}
{"caller":"level.go:63","error":"1 error occurred:\n\t* Failed to join 172.18.0.4: dial tcp 172.18.0.4:7946: connect: connection refused\n\n","expected":1,"joined":0,"level":"error","msg":"partial join","op":"memberDiscovery","ts":"2022-03-11T05:49:45.546652685Z"}
{"caller":"level.go:63","error":"1 error occurred:\n\t* Failed to join 172.18.0.4: dial tcp 172.18.0.4:7946: connect: connection refused\n\n","expected":1,"joined":0,"level":"error","msg":"partial join","op":"memberDiscovery","ts":"2022-03-11T05:49:45.547146957Z"}

Has something happend to my system, that I'm not aware of that has caused this not to work anymore?  connection refused.  is not good.

[Rodrigo Campos]

Yeah, the connection refused seems like the speaker pod on that node
(the one on the IP that you can't connect to) is not running or not
receiving connections.

Is that the case? The speaker pod should be running on all nodes :)

> --
> You received this message because you are subscribed to the Google Groups "metallb-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to metallb-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/metallb-users/23f42e6d-616b-4770-ae18-d1681a5e0fc1n%40googlegroups.com.



--
Rodrigo Campos
---
Kinvolk GmbH | Adalbertstr.6a, 10999 Berlin | tel: +491755589364
Geschäftsführer/Directors: Benjamin Owen Orndorff
Registergericht/Court of registration: Amtsgericht Charlottenburg
Registernummer/Registration number: HRB 171414 B
Ust-ID-Nummer/VAT ID number: DE302207000

[Darin Tracy]

It seems the speaker pods are running on all nodes.  I have 4 worker nodes and 1 control plane.

k get pods -n metallb-system -o wide
NAME                          READY   STATUS    RESTARTS   AGE     IP           NODE                 NOMINATED NODE   READINESS GATES
controller-57b74d8656-j6nhj   1/1     Running   0          6m31s   10.244.3.2   kind-worker4         <none>           <none>
speaker-92sxp                           1/1     Running   0          6m31s   172.18.0.5   kind-worker4         <none>           <none>
speaker-gl2ll                              1/1     Running   0          6m31s   172.18.0.4   kind-worker3         <none>           <none>
speaker-m59n6                         1/1     Running   0          6m31s   172.18.0.6   kind-control-plane   <none>           <none>
speaker-mg6q2                         1/1     Running   0          6m31s   172.18.0.3   kind-worker          <none>           <none>
speaker-r865c                            1/1     Running   0          6m31s   172.18.0.2   kind-worker2         <none>           <none>

This was working, for months, but something seems to have "broken" it in my environment.

[Darin Tracy]

I also noticed this warning in the speaker pods.  Wondering if this might be related to the not being able to connect.  And if so, what I need to fix?

W0314 14:50:05.145345       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice

I'm using kind + metallb with instructions listed here.

https://kind.sigs.k8s.io/docs/user/loadbalancer/

[Darin Tracy]

And More Info.

I tried to patch the service and set the external IP address.  I did this and I tried a curl on the ip/port.  Which should work based on the sample service/pod I setup.  And I get

Failed to connect to:  172.18.255.204 port 5678: No route to host

To view this discussion on the web visit https://groups.google.com/d/msgid/metallb-users/ffb19430-84c7-4fdc-b516-f926a45a1bafn%40googlegroups.com.

[Darin Tracy]

IMPORTANT:

I found that if I use v0.11.0 manifests to install metallb it still works.  So it seems like the "recent" change has broken metallb+kind

Darin