kube proxy ... over internal service endpoints....?

[jay vyas]

Hi folks: This might be a naive question .... as im not privy to all the history around how kube-proxy deals with hostNetwork services and headless services and so on but...

I guess my high level question is ... how do people normally start kube-proxy with an in-cluster configuration ?  isnt there a fundamental chicken or egg problem ? 

Details

in general, kube-proxy i guess sets up routing rules for the internal kubernetes service (i.e. usually 10.0.96.1)... now *i think* the way this works is that the apiserver writes is own  endpoints rules i.e.  it sais "my ip address is 

subsets:

- addresses:

  - ip: 172.18.0.5

  ports:

  - name: https

    port: 6443

    protocol: TCP

So then, when KP comes online, it sees this endpoint, and writes routing rules for 10.96.0.1 -> 172.18.0.5

 

Theres an internal service for k8s
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y

Theres a service endpoint for that dang internal service
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-OJCTP5LCEHQJ3D72

The service endpoint points to a host IP address
-A KUBE-SEP-OJCTP5LCEHQJ3D72 -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 172.18.0.5:6443

OK Great, now pods can access the apiserver through 10.0.96.1.... however .....  should kube proxy generally  be smart  enough to write this rule before it actually connects to the apiserver?  

IF WE DID allowed kube-proxy to do this,  then i think...

- in the context of a next gen kube proxy you'd get to bootstrap the kpng apiserver reflector without it needing to read the default service endpoints object

- in the context of load balancing, you could have kube proxy itself access the apiserver through a loadbalanced endpoint, i.e. --api-servers=1.2.3.4, 2.3.4.5, ... 

 

[Antonio Ojea]

kubeadm deploys kube-proxy as a daemonset, it passes a kubeconfig with the "real" IP address of the apiserver.

It may be a load balancer IP in front of the apiservers or an apiserver IP if you only have one node, but it has to be reachable to avoid the chicken and egg problem you mention.

This is the main problem behind https://github.com/kubernetes/kubernetes/issues/18174

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/2a23bf24-dd76-4a05-9c02-99afd8aa39a5n%40googlegroups.com.

[Dan Winship]

On 4/19/21 9:27 AM, jay vyas wrote:
> Hi folks: This might be a naive question .... as im not privy to all the
> history around how kube-proxy deals with hostNetwork services and
> headless services and so on but...
>

> *I guess my high level question is ... how do people normally start

> kube-proxy with an in-cluster configuration ?  isnt there a fundamental

> chicken or egg problem ? *

Yes. You can't run kube-proxy with the (default) in-cluster
configuration. You have to either pass --server or --kubeconfig to give
it an actual apiserver name/IP to use, or else override the value of
KUBERNETES_SERVER_HOST set by kubelet in its environment.

> OK Great, now pods can access the apiserver through 10.0.96.1....

> however .....  should kube proxy *generally * be smart  enough to write
> this rule *before* it actually connects to the apiserver?  

But before it connects to the apiserver, it doesn't know what the
endpoints are...

More generally, if something goes wrong (eg, iptables-restore failure),
it seems like it's better to have kube-proxy talking to the apiserver
directly rather than depending on its own output.

> - in the context of load balancing, you could have kube proxy itself
> access the apiserver through a loadbalanced endpoint, i.e.
> --api-servers=1.2.3.4, 2.3.4.5, ... 

Someone brought up this idea recently... in the SIG meeting or in a KEP?

In theory, kube-proxy could be extended to allow you to pass multiple
apiserver IPs to it so it could try to loadbalance on its own.

However, kube-proxy isn't the only component that needs to do this. Eg,
if you are running kube-proxy in a pod, then kubelet will also have
needed to know the non-service-network IP(s) of the apiservers. So it is
better to have an external load balancer, or round-robin DNS name.

However, doing that entirely within Kubernetes is tricky, because of the
chicken-and-egg issues... This is more of an install/deployment tool
(kubeadm/openshift-install/etc) sort of thing than a
kubelet/kube-proxy/kube-apiserver sort of thing.

-- Dan