Re: [k8s-sig-net] Kubelet Node Addresses and External Cloud Providers [Revisited][Important]

[Antonio Ojea]

Hi,

Following up on this topic as the existing behavior has broken some deployments https://github.com/kubernetes/kubernetes/issues/125348 (I apologize for this :( )

It seems there is consensus on using https://github.com/kubernetes/kubernetes/pull/125337 to fix it.

In addition, and due to how brittle is this area of the codebase, I've created the following document to try to consolidate and document the existing problems https://docs.google.com/document/d/1mqdVLQHIYjrzjy8Hq-FMysthHRL7ht0guk2uWI0FxKM/edit .

I encourage all the SIG leads and people working on downstream Kubernetes deployments that depend on external cloud providers to chime in.

Regards,

Antonio Ojea

On Mon, 9 Oct 2023 at 11:49, Antonio Ojea <antonio.o...@gmail.com> wrote:

Good question Joel, thanks for bringing this up, it is no breaking change, the previous behavior was clearly a bug because assumed that external cloud-provider and kubelet autodiscovery will be in sync, and we heavily discussed the problems of mutating PodIPs on hostNetwork pods.

You don't have to specify node-ip flags, pods will be deployed as always and will run normally, just the hostNetwork pods , PodIPs will be updated later ...  if your pod startup logic depends on the PodIPs somehow, as using the downward API to get the value, then of course you'll have to wait, but it will be wrong for the CCM that is deployed in a Pod to depend on the PodIPs since it is the one that has to provide them.

Below example shows how a daemonset with hostNetwork: true, kindnet-nzgbk, runs in a node "external-worker" with the kubelet using external cloud-provider and no node-ip flags added

```

$ kubectl get pods -A -o wide
NAMESPACE            NAME                                             READY   STATUS    RESTARTS   AGE     IP            NODE                     NOMINATED NODE   READINESS GATES
kube-system          kindnet-4m67t                                    1/1     Running   0          4m1s    192.168.8.7   external-control-plane   <none>           <none>
kube-system          kindnet-5bn5q                                    1/1     Running   0          3m57s   192.168.8.8   external-worker2         <none>           <none>
kube-system          kindnet-nzgbk                                    1/1     Running   0          3m57s   <none>        external-worker          <none>           <none>

```

On Mon, 9 Oct 2023 at 10:14, Joel Speed <jsp...@redhat.com> wrote:

Hey Antonio,

With your solution, IIUC I think, if a topology relies on the CCM being a host network pod on the control plane nodes, this means that control plane Nodes will need to specify `—node-ip` always, is that correct?

And is that not considered to be a breaking change?

I expect at the moment there are a number of deployments of the CCM where there is an expectation that the CCM can run on your first, uninitialized control plane node. If these deployments haven’t set the `—node-ip` flag up to this point, when they are upgraded to 1.29 they will be required to change the cluster installation process to set the `—node-ip` flag going forward.

Thanks,

Joel

Joel Speed

He/Him/His

Principal Software Engineer, OpenShift

Red Hat

On 6 Oct 2023, at 15:13, Antonio Ojea <antonio.o...@gmail.com> wrote:

I think that I've found a solution that solves the problem without breaking the compatibility https://github.com/kubernetes/kubernetes/pull/121028

Please review if you are impacted by this problem, this release we are moving to beta the KEP-2395 Removing In-Tree Cloud Providers and it is critical that we fix this behavior ASAP.

Thanks 

On Sat, 30 Sept 2023 at 22:07, Antonio Ojea <antonio.o...@gmail.com> wrote:

Yes, that's correct, thanks for catching that 

On Sat, Sep 30, 2023, 4:47 PM Dan Winship <danwi...@redhat.com> wrote:

On 9/28/23 15:00, Antonio Ojea wrote:
> It is important to mention that this is especially problematic because
> the hostNetwork Pods uses the node Addresses to populate the HostIPs and
> pod.status.PodIPs with all the implications that it has: Endpoints,
> Downward API, ... Another important detail is  that we decided that the
> kubelet will not mutate the HostIPs, it has to be
> restarted https://github.com/kubernetes/enhancements/issues/2999
> <https://github.com/kubernetes/enhancements/issues/2999>, and this will
> make the race to happen again.

Mmm, no, the race only happens the *first* time kubelet starts, when
there is no existing Node object. If you restart kubelet after the Node
exists and the cloud provider has filled in .Status.Addresses, then when
kubelet starts up again it will see that there are already addresses
there and so it won't run its own address-detecting code again.

-- Dan

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cloud-provider" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cloud...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cloud-provider/CABhP%3Dtae8utFJxJH5zvRQB2Eo5b78bntQxotUZXOW5NMHduP3w%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cloud-provider" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cloud...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cloud-provider/4747C861-E8D3-4642-918F-50B27F73E50F%40redhat.com.

[Michael McCune]

On Sun, Jun 16, 2024 at 6:04 PM Antonio Ojea <antonio.o...@gmail.com> wrote:

Hi,

Following up on this topic as the existing behavior has broken some deployments https://github.com/kubernetes/kubernetes/issues/125348 (I apologize for this :( )

It seems there is consensus on using https://github.com/kubernetes/kubernetes/pull/125337 to fix it.

In addition, and due to how brittle is this area of the codebase, I've created the following document to try to consolidate and document the existing problems https://docs.google.com/document/d/1mqdVLQHIYjrzjy8Hq-FMysthHRL7ht0guk2uWI0FxKM/edit .

I encourage all the SIG leads and people working on downstream Kubernetes deployments that depend on external cloud providers to chime in.

excellent summary in the doc Antonio, thank you for putting this together.

i like the notion of extending the "0.0.0.0" or "::" discovery behavior to work in a similar manner with external cloud providers. although, i guess this would also put pressure on the CCM to choose the proper interface through configuration (which i would like) or some other mechanism. but, given the discussion on https://github.com/kubernetes/kubernetes/pull/125300 , it makes me think there will need to be some changes in the cloud-provider framework, and potentially the CCMs, to allow the discovery behavior. is that accurate? have i missed some details?

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cloud-provider/CABhP%3DtbnMeLFkcNbH94rOsEsmNTqZ15kqXfwEMmBCWSft6L8rQ%40mail.gmail.com.

[Antonio Ojea]

On Tue, 18 Jun 2024 at 10:52, Michael McCune <elm...@redhat.com> wrote:

On Sun, Jun 16, 2024 at 6:04 PM Antonio Ojea <antonio.o...@gmail.com> wrote:

Hi,

Following up on this topic as the existing behavior has broken some deployments https://github.com/kubernetes/kubernetes/issues/125348 (I apologize for this :( )

It seems there is consensus on using https://github.com/kubernetes/kubernetes/pull/125337 to fix it.

In addition, and due to how brittle is this area of the codebase, I've created the following document to try to consolidate and document the existing problems https://docs.google.com/document/d/1mqdVLQHIYjrzjy8Hq-FMysthHRL7ht0guk2uWI0FxKM/edit .

I encourage all the SIG leads and people working on downstream Kubernetes deployments that depend on external cloud providers to chime in.

excellent summary in the doc Antonio, thank you for putting this together.

i like the notion of extending the "0.0.0.0" or "::" discovery behavior to work in a similar manner with external cloud providers. although, i guess this would also put pressure on the CCM to choose the proper interface through configuration (which i would like) or some other mechanism. but, given the discussion on https://github.com/kubernetes/kubernetes/pull/125300 , it makes me think there will need to be some changes in the cloud-provider framework, and potentially the CCMs, to allow the discovery behavior. is that accurate? have i missed some details?

I don't fully understand this question, the unspecified address indicates to the kubelet "use the IP address associated with the interface used by the default gateway", and it adds it to the node.status.addresses when it creates the node object. Once the node object is created, it is processed by the external cloud provider and is up to the cloud provider implementation to decide what node addresses it reports in status.

[Michael McCune]

On Sun, Jun 30, 2024 at 11:41 PM Antonio Ojea <antonio.o...@gmail.com> wrote:

On Tue, 18 Jun 2024 at 10:52, Michael McCune <elm...@redhat.com> wrote:

On Sun, Jun 16, 2024 at 6:04 PM Antonio Ojea <antonio.o...@gmail.com> wrote:

Hi,

Following up on this topic as the existing behavior has broken some deployments https://github.com/kubernetes/kubernetes/issues/125348 (I apologize for this :( )

It seems there is consensus on using https://github.com/kubernetes/kubernetes/pull/125337 to fix it.

In addition, and due to how brittle is this area of the codebase, I've created the following document to try to consolidate and document the existing problems https://docs.google.com/document/d/1mqdVLQHIYjrzjy8Hq-FMysthHRL7ht0guk2uWI0FxKM/edit .

I encourage all the SIG leads and people working on downstream Kubernetes deployments that depend on external cloud providers to chime in.

excellent summary in the doc Antonio, thank you for putting this together.

i like the notion of extending the "0.0.0.0" or "::" discovery behavior to work in a similar manner with external cloud providers. although, i guess this would also put pressure on the CCM to choose the proper interface through configuration (which i would like) or some other mechanism. but, given the discussion on https://github.com/kubernetes/kubernetes/pull/125300 , it makes me think there will need to be some changes in the cloud-provider framework, and potentially the CCMs, to allow the discovery behavior. is that accurate? have i missed some details?

I don't fully understand this question, the unspecified address indicates to the kubelet "use the IP address associated with the interface used by the default gateway", and it adds it to the node.status.addresses when it creates the node object. Once the node object is created, it is processed by the external cloud provider and is up to the cloud provider implementation to decide what node addresses it reports in status.

 

i was wondering if we needed to make any changes in the cloud-provider framework for the "0.0.0.0" and "::" addresses, but it sounds like (if i understand correctly) the addresses (0.0.0.0 and/or ::) would be populated in the node.status.addresses, and as such the framework should already have the proper logic.

do i have that correct?

peace o/

[Antonio Ojea]

On Mon, 8 Jul 2024 at 11:41, Michael McCune <elm...@redhat.com> wrote:

On Sun, Jun 30, 2024 at 11:41 PM Antonio Ojea <antonio.o...@gmail.com> wrote:

On Tue, 18 Jun 2024 at 10:52, Michael McCune <elm...@redhat.com> wrote:

On Sun, Jun 16, 2024 at 6:04 PM Antonio Ojea <antonio.o...@gmail.com> wrote:

Hi,

Following up on this topic as the existing behavior has broken some deployments https://github.com/kubernetes/kubernetes/issues/125348 (I apologize for this :( )

It seems there is consensus on using https://github.com/kubernetes/kubernetes/pull/125337 to fix it.

In addition, and due to how brittle is this area of the codebase, I've created the following document to try to consolidate and document the existing problems https://docs.google.com/document/d/1mqdVLQHIYjrzjy8Hq-FMysthHRL7ht0guk2uWI0FxKM/edit .

I encourage all the SIG leads and people working on downstream Kubernetes deployments that depend on external cloud providers to chime in.

excellent summary in the doc Antonio, thank you for putting this together.

i like the notion of extending the "0.0.0.0" or "::" discovery behavior to work in a similar manner with external cloud providers. although, i guess this would also put pressure on the CCM to choose the proper interface through configuration (which i would like) or some other mechanism. but, given the discussion on https://github.com/kubernetes/kubernetes/pull/125300 , it makes me think there will need to be some changes in the cloud-provider framework, and potentially the CCMs, to allow the discovery behavior. is that accurate? have i missed some details?

I don't fully understand this question, the unspecified address indicates to the kubelet "use the IP address associated with the interface used by the default gateway", and it adds it to the node.status.addresses when it creates the node object. Once the node object is created, it is processed by the external cloud provider and is up to the cloud provider implementation to decide what node addresses it reports in status.

 

i was wondering if we needed to make any changes in the cloud-provider framework for the "0.0.0.0" and "::" addresses, but it sounds like (if i understand correctly) the addresses (0.0.0.0 and/or ::) would be populated in the node.status.addresses, and as such the framework should already have the proper logic.

ah, no, 0.0.0.0 and :: means "discovery, use the IP that exists in the interface with the default gateway", they are not populated and represented in the API

[Michael McCune]

On Mon, Jul 8, 2024 at 1:23 PM Antonio Ojea <antonio.o...@gmail.com> wrote:

On Mon, 8 Jul 2024 at 11:41, Michael McCune <elm...@redhat.com> wrote:

On Sun, Jun 30, 2024 at 11:41 PM Antonio Ojea <antonio.o...@gmail.com> wrote:

On Tue, 18 Jun 2024 at 10:52, Michael McCune <elm...@redhat.com> wrote:

On Sun, Jun 16, 2024 at 6:04 PM Antonio Ojea <antonio.o...@gmail.com> wrote:

Hi,

Following up on this topic as the existing behavior has broken some deployments https://github.com/kubernetes/kubernetes/issues/125348 (I apologize for this :( )

It seems there is consensus on using https://github.com/kubernetes/kubernetes/pull/125337 to fix it.

In addition, and due to how brittle is this area of the codebase, I've created the following document to try to consolidate and document the existing problems https://docs.google.com/document/d/1mqdVLQHIYjrzjy8Hq-FMysthHRL7ht0guk2uWI0FxKM/edit .

I encourage all the SIG leads and people working on downstream Kubernetes deployments that depend on external cloud providers to chime in.

excellent summary in the doc Antonio, thank you for putting this together.

i like the notion of extending the "0.0.0.0" or "::" discovery behavior to work in a similar manner with external cloud providers. although, i guess this would also put pressure on the CCM to choose the proper interface through configuration (which i would like) or some other mechanism. but, given the discussion on https://github.com/kubernetes/kubernetes/pull/125300 , it makes me think there will need to be some changes in the cloud-provider framework, and potentially the CCMs, to allow the discovery behavior. is that accurate? have i missed some details?

I don't fully understand this question, the unspecified address indicates to the kubelet "use the IP address associated with the interface used by the default gateway", and it adds it to the node.status.addresses when it creates the node object. Once the node object is created, it is processed by the external cloud provider and is up to the cloud provider implementation to decide what node addresses it reports in status.

 

i was wondering if we needed to make any changes in the cloud-provider framework for the "0.0.0.0" and "::" addresses, but it sounds like (if i understand correctly) the addresses (0.0.0.0 and/or ::) would be populated in the node.status.addresses, and as such the framework should already have the proper logic.

ah, no, 0.0.0.0 and :: means "discovery, use the IP that exists in the interface with the default gateway", they are not populated and represented in the API

ok, i think this is helping me come closer to relieving my confusion. let me create an example:

1. user starts kubelet with `--cloud-provider=external` and `--node-ip=0.0.0.0`

2. kubelet knows there will be an external provider and that discovery should be used for the ip address

3. does kubelet use the `alpha.kubernetes.io/provided-node-ip` annotation to pass "0.0.0.0" to the CCM?

is accurate?

do we need to add anything to help this?

[Dan Winship]

On 7/8/24 05:41, Michael McCune wrote:
> i was wondering if we needed to make any changes in the cloud-provider
> framework for the "0.0.0.0" and "::" addresses

no; when you pass "--node-ip 0.0.0.0" or "--node-ip ::" to kubelet, you
are not asking the cloud provider to do anything different than it would
have done. You are just asking kubelet to do something different
_before_ the cloud-provider does what it was going to do.

-- Dan

[Michael McCune]

ack, thanks Dan!

 

-- Dan