Determining what IP families the infrastructure needs to support for a given cluster
84 views
Skip to first unread message
David McClure
May 12, 2021, 9:41:12 AM5/12/21
to kubernetes-...@googlegroups.com
Hey folks,
While working on
adding IPv6 support to the docker cluster-api provider, we
discussed the tradeoffs of auto-detecting the IP family to configure at the infrastructure layer (in this case, the docker machines), vs. requiring additional configuration.
In Cluster API, the Cluster object has
two optional fields for Pod CIDRs and Service CIDRs that allow us to do this in most cases, but there are
a few ambiguous cases (namely when the Pod CIDRs field is not set). We've decided this is a "good enough" approach for now, but we may revisit it in the future.
Putting Cluster API aside and just looking at Kubernetes itself, I wondered: is it possible to determine what IP families the infrastructure needs to support for a given cluster?
I started looking into this and discussed it with Antonio briefly
on Slack.
If I understand correctly, since pod CIDRs are optional and the CNI really determines the requirements for the infrastructure, it is not possible to determine the infrastructure requirements at the time the cluster is being created.
Is that a fair statement?
Do others see value in being able to determine what is required by the infrastructure earlier in the lifecycle of a cluster and/or in a specified way?
Cheers,
Dave
Dave
Tim Hockin
May 12, 2021, 10:25:08 AM5/12/21
to David McClure, kubernetes-sig-network
I don't think you can know automatically. Lots of machines have ipv6 enabled but they may not want it for their clusters.
Also, we should talk about the work being done for multi&l-CIDR, to make sure CAPI ready :)
--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/BL0PR05MB54607311F5D6D5131FBF08E6A8529%40BL0PR05MB5460.namprd05.prod.outlook.com.
David McClure
May 12, 2021, 8:39:39 PM5/12/21
to Tim Hockin, kubernetes-sig-network
Lots of machines have ipv6 enabled but they may not want it for their clusters.
True.
I think I'm looking at this the other way around though: "oh, you want a cluster with [spec], let me get some machines for you that meet your requirements."
Also, we should talk about the work being done for multi&l-CIDR, to make sure CAPI ready :)
Where can I go
to educate myself about this work?
From: 'Tim Hockin' via kubernetes-sig-network <kubernetes-...@googlegroups.com>
Sent: Wednesday, May 12, 2021 10:24 AM
To: David McClure <dmcc...@vmware.com>
Cc: kubernetes-sig-network <kubernetes-...@googlegroups.com>
Subject: Re: [k8s-sig-net] Determining what IP families the infrastructure needs to support for a given cluster
Sent: Wednesday, May 12, 2021 10:24 AM
To: David McClure <dmcc...@vmware.com>
Cc: kubernetes-sig-network <kubernetes-...@googlegroups.com>
Subject: Re: [k8s-sig-net] Determining what IP families the infrastructure needs to support for a given cluster
To view this discussion on the web visit
https://groups.google.com/d/msgid/kubernetes-sig-network/CAO_RewYMbj9-uHsk4%3DD%2BrKbqNSG%2BXe%2BP3NBvW2jU%2BTjOEOdcvQ%40mail.gmail.com.
Tim Hockin
May 13, 2021, 12:58:32 AM5/13/21
to David McClure, kubernetes-sig-network
Antonio Ojea
May 13, 2021, 5:05:34 AM5/13/21
to David McClure, Tim Hockin, kubernetes-sig-network
On Thu, 13 May 2021 at 02:39, David McClure <dmcc...@vmware.com> wrote:
Lots of machines have ipv6 enabled but they may not want it for their clusters.
True. I think I'm looking at this the other way around though: "oh, you want a cluster with [spec], let me get some machines for you that meet your requirements."
I think that this point is important, "technically" dual-stack is not a thing, dual-stack means you have 2 IP families, but in Kubernetes you can have it at pod level and/or at service level (and also at the machine level as Tim said). There is no validation, ie. having 2 dual-stack service cidrs and 1 pod cidr is valid, or dual stack pods with single-stack services, ...
However, we went through this in KIND, and we ended giving "dual-stack" a meaning for "installation/deployment", so when an user deploys a cluster and selects "dual-stack", it means that he will install a cluster with "dual-stack" Service - Cidrs and Pod-Cidrs, aka a "full-dual-stack" cluster XD
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/BL0PR05MB5460FA2EACC5E2A65687D1EEA8519%40BL0PR05MB5460.namprd05.prod.outlook.com.
David McClure
May 13, 2021, 9:30:57 AM5/13/21
to Antonio Ojea, Tim Hockin, kubernetes-sig-network
Thanks! Will keep an eye on these and make sure others working on Cluster API are aware as well.
in Kubernetes you can have it at pod level and/or at service level (and also at the machine level as Tim said). There is no validation, ie. having 2 dual-stack service cidrs and 1 pod cidr is valid, or dual stack pods with single-stack services, ...
I'd like to understand this better. What is the expected
behavior of a cluster in scenarios where (as I think of it, anyway), a layer "above" has an IP family that does not exist in the layer "below"
- My cluster has dual-stack services, but pods only have IPv4 addresses
- My cluster/CNI is configured to create pods with IPv4 and IPv6 addresses, but worker nodes only have IPv4 addresses
From: Antonio Ojea <antonio.o...@gmail.com>
Sent: Thursday, May 13, 2021 5:04 AM
To: David McClure <dmcc...@vmware.com>
Cc: Tim Hockin <tho...@google.com>; kubernetes-sig-network <kubernetes-...@googlegroups.com>
Sent: Thursday, May 13, 2021 5:04 AM
To: David McClure <dmcc...@vmware.com>
Cc: Tim Hockin <tho...@google.com>; kubernetes-sig-network <kubernetes-...@googlegroups.com>
Dan Winship
May 13, 2021, 9:43:27 AM5/13/21
to David McClure, Antonio Ojea, Tim Hockin, kubernetes-sig-network
On 5/13/21 9:30 AM, David McClure wrote:
> I'd like to understand this better. What is the expected behavior of a
> cluster in scenarios where (as I think of it, anyway), a layer "above"
> has an IP family that does not exist in the layer "below"
>
> 1. My cluster has dual-stack services, but pods only have IPv4 addresses
> I'd like to understand this better. What is the expected behavior of a
> cluster in scenarios where (as I think of it, anyway), a layer "above"
> has an IP family that does not exist in the layer "below"
>
So, this configuration would be dumb, so the expectation is that nobody
would actually configure their cluster this way. But it would mean that
you could create dual-stack Services, but their IPv6 ClusterIPs wouldn't
accept connections because they had no IPv6 endpoints. (I guess you
could create dual-stack Services pointing to IPv6 addresses outside the
cluster... except that your pods wouldn't be able to access those IPv6
addresses since the pods themselves had no IPv6 addresses.)
So... it works in the obvious way, which is to say that it's totally
useless.
> 2. My cluster/CNI is configured to create pods with IPv4 and IPv6
> addresses, but worker nodes only have IPv4 addresses
This depends on the details of the network plugin, but you'd most likely
have dual-stack traffic between pods on the same node, and probably also
have dual-stack traffic between pods on different nodes, but probably
only IPv4 cluster ingress and egress traffic.
-- Dan
> ------------------------------------------------------------------------
> *From:* Antonio Ojea <antonio.o...@gmail.com>
> *Sent:* Thursday, May 13, 2021 5:04 AM
> *To:* David McClure <dmcc...@vmware.com>
> *Cc:* Tim Hockin <tho...@google.com>; kubernetes-sig-network
> <kubernetes-...@googlegroups.com>
> *Subject:* Re: [k8s-sig-net] Determining what IP families the
> infrastructure needs to support for a given cluster
>
>
>
> On Thu, 13 May 2021 at 02:39, David McClure <dmcc...@vmware.com
> <mailto:dmcc...@vmware.com>> wrote:
>
> Lots of machines have ipv6 enabled but they may not want it for
> their clusters.
>
> True. I think I'm looking at this the other way around though: "oh,
> you want a cluster with [spec], let me get some machines for you
> that meet your requirements."
>
>
>
> I think that this point is important, "technically" dual-stack is not a
> thing, dual-stack means you have 2 IP families, but in
> Kubernetes you can have it at pod level and/or at service level (and
> also at the machine level as Tim said). There is no validation, ie.
> having 2 dual-stack service cidrs and 1 pod cidr is valid, or dual stack
> pods with single-stack services, ...
>
> However, we went through this in KIND, and we ended giving "dual-stack"
> a meaning for "installation/deployment", so when an user deploys a
> cluster and selects "dual-stack", it means that he will install a
> cluster with "dual-stack" Service - Cidrs and Pod-Cidrs, aka a
> "full-dual-stack" cluster XD
>
>
> Also, we should talk about the work being done for multi&l-CIDR,
> to make sure CAPI ready :)
>
> Where can I go to educate myself about this work?
> ------------------------------------------------------------------------
>
>
>
> On Thu, 13 May 2021 at 02:39, David McClure <dmcc...@vmware.com
> <mailto:dmcc...@vmware.com>> wrote:
>
> Lots of machines have ipv6 enabled but they may not want it for
> their clusters.
>
> True. I think I'm looking at this the other way around though: "oh,
> you want a cluster with [spec], let me get some machines for you
> that meet your requirements."
>
>
>
> I think that this point is important, "technically" dual-stack is not a
> thing, dual-stack means you have 2 IP families, but in
> Kubernetes you can have it at pod level and/or at service level (and
> also at the machine level as Tim said). There is no validation, ie.
> having 2 dual-stack service cidrs and 1 pod cidr is valid, or dual stack
> pods with single-stack services, ...
>
> However, we went through this in KIND, and we ended giving "dual-stack"
> a meaning for "installation/deployment", so when an user deploys a
> cluster and selects "dual-stack", it means that he will install a
> cluster with "dual-stack" Service - Cidrs and Pod-Cidrs, aka a
> "full-dual-stack" cluster XD
>
>
> Also, we should talk about the work being done for multi&l-CIDR,
> to make sure CAPI ready :)
>
> Where can I go to educate myself about this work?
> *From:* 'Tim Hockin' via kubernetes-sig-network
> <kubernetes-...@googlegroups.com
> <mailto:kubernetes-...@googlegroups.com>>
> *Sent:* Wednesday, May 12, 2021 10:24 AM
> *To:* David McClure <dmcc...@vmware.com <mailto:dmcc...@vmware.com>>
> *Cc:* kubernetes-sig-network
> <kubernetes-...@googlegroups.com
> <mailto:kubernetes-...@googlegroups.com>>
> *Subject:* Re: [k8s-sig-net] Determining what IP families the
> infrastructure needs to support for a given cluster
>
> I don't think you can know automatically. Lots of machines have
> ipv6 enabled but they may not want it for their clusters.
>
> Also, we should talk about the work being done for multi&l-CIDR, to
> make sure CAPI ready :)
>
> On Wed, May 12, 2021, 6:41 AM David McClure <dmcc...@vmware.com
> <mailto:dmcc...@vmware.com>> wrote:
>
> Hey folks,
>
> While working on adding IPv6 support to the docker cluster-api
> provider
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes-sigs%2Fcluster-api%2Fpull%2F4558&data=04%7C01%7Cdmcclure%40vmware.com%7C9971f9e62d384ff0be8208d915ee4388%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637564935364559820%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ussZ8RQQcMCoKQfFU2xaGBZk8TEBiRB6IMadSUdmX6E%3D&reserved=0>,
>
> I don't think you can know automatically. Lots of machines have
> ipv6 enabled but they may not want it for their clusters.
>
> Also, we should talk about the work being done for multi&l-CIDR, to
> make sure CAPI ready :)
>
> On Wed, May 12, 2021, 6:41 AM David McClure <dmcc...@vmware.com
> <mailto:dmcc...@vmware.com>> wrote:
>
> Hey folks,
>
> While working on adding IPv6 support to the docker cluster-api
> provider
> we discussed the tradeoffs
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes-sigs%2Fcluster-api%2Fpull%2F4558%2Ffiles%23r625226177&data=04%7C01%7Cdmcclure%40vmware.com%7C9971f9e62d384ff0be8208d915ee4388%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637564935364569815%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Fc5HnkCG7FdC5HYWtbZPRAC%2FTxtX0rjhncdPjefJlJk%3D&reserved=0>
> of auto-detecting the IP family to configure at the
> infrastructure layer (in this case, the docker machines), vs.
> requiring additional configuration.
>
> In Cluster API, the Cluster object has two optional fields
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes-sigs%2Fcluster-api%2Fblob%2F321ec3976b18200742b4ddb5b706e28b5d5657a7%2Fapi%2Fv1alpha4%2Fcluster_types.go%23L74-L80&data=04%7C01%7Cdmcclure%40vmware.com%7C9971f9e62d384ff0be8208d915ee4388%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637564935364579811%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=q9VF50%2BRFIBsKIMztMkYWFRErqhpYDzQ%2FwCQvW4U9nA%3D&reserved=0>
> infrastructure layer (in this case, the docker machines), vs.
> requiring additional configuration.
>
> In Cluster API, the Cluster object has two optional fields
> for Pod CIDRs and Service CIDRs that allow us to do this in most
> cases, but there are a few ambiguous cases
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes-sigs%2Fcluster-api%2Fpull%2F4558%2Ffiles%23diff-688ca7c8bf1a51eb8dfe432ca4f13a395c9cc9ad26bbe80369361824fc7d80d6R102-R125&data=04%7C01%7Cdmcclure%40vmware.com%7C9971f9e62d384ff0be8208d915ee4388%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637564935364579811%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=b6wgICAOFN0mRYkWTSuiWxh9ytRElypiVfOS3w6p4Yk%3D&reserved=0>
> cases, but there are a few ambiguous cases
> (namely when the Pod CIDRs field is not set). We've decided this
> is a "good enough" approach for now, but we may revisit it in
> the future.
>
> Putting Cluster API aside and just looking at Kubernetes itself,
> I wondered: is it possible to determine what IP families the
> infrastructure needs to support for a given cluster?
>
> I started looking into this and discussed it with Antonio
> briefly on Slack
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkubernetes.slack.com%2Farchives%2FC09QYUH5W%2Fp1620240705491200%3Fthread_ts%3D1620240684.490600%26cid%3DC09QYUH5W&data=04%7C01%7Cdmcclure%40vmware.com%7C9971f9e62d384ff0be8208d915ee4388%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637564935364589802%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=j13PdCOKe3%2FNOOZxRv8A8VqSOmpZfg6JMK1aRDtDj1c%3D&reserved=0>.
> is a "good enough" approach for now, but we may revisit it in
> the future.
>
> Putting Cluster API aside and just looking at Kubernetes itself,
> I wondered: is it possible to determine what IP families the
> infrastructure needs to support for a given cluster?
>
> I started looking into this and discussed it with Antonio
> briefly on Slack
>
> If I understand correctly, since pod CIDRs are optional and the
> CNI really determines the requirements for the infrastructure,
> it is not possible to determine the infrastructure requirements
> at the time the cluster is being created.
>
> Is that a fair statement?
>
> Do others see value in being able to determine what is required
> by the infrastructure earlier in the lifecycle of a cluster
> and/or in a specified way?
>
> Cheers,
> Dave
>
>
>
>
>
> --
> You received this message because you are subscribed to the
> Google Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to
> kubernetes-sig-ne...@googlegroups.com
> <mailto:kubernetes-sig-ne...@googlegroups.com>.
> If I understand correctly, since pod CIDRs are optional and the
> CNI really determines the requirements for the infrastructure,
> it is not possible to determine the infrastructure requirements
> at the time the cluster is being created.
>
> Is that a fair statement?
>
> Do others see value in being able to determine what is required
> by the infrastructure earlier in the lifecycle of a cluster
> and/or in a specified way?
>
> Cheers,
> Dave
>
>
>
>
>
> --
> You received this message because you are subscribed to the
> Google Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to
> kubernetes-sig-ne...@googlegroups.com
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubernetes-sig-network/BL0PR05MB54607311F5D6D5131FBF08E6A8529%40BL0PR05MB5460.namprd05.prod.outlook.com
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FBL0PR05MB54607311F5D6D5131FBF08E6A8529%2540BL0PR05MB5460.namprd05.prod.outlook.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cdmcclure%40vmware.com%7C9971f9e62d384ff0be8208d915ee4388%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637564935364589802%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=K3mcB5n5ZSI5CUWFylAMiHJXvQB%2B7E9DV42Teo3CBfw%3D&reserved=0>.
> https://groups.google.com/d/msgid/kubernetes-sig-network/BL0PR05MB54607311F5D6D5131FBF08E6A8529%40BL0PR05MB5460.namprd05.prod.outlook.com
>
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kubernetes-sig-ne...@googlegroups.com
> <mailto:kubernetes-sig-ne...@googlegroups.com>.
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kubernetes-sig-ne...@googlegroups.com
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubernetes-sig-network/CAO_RewYMbj9-uHsk4%3DD%2BrKbqNSG%2BXe%2BP3NBvW2jU%2BTjOEOdcvQ%40mail.gmail.com
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FCAO_RewYMbj9-uHsk4%253DD%252BrKbqNSG%252BXe%252BP3NBvW2jU%252BTjOEOdcvQ%2540mail.gmail.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cdmcclure%40vmware.com%7C9971f9e62d384ff0be8208d915ee4388%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637564935364599800%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EsU9oBmFZaxtKNjisjyqUrqsJITpk%2BaOYgcRS0jiIaE%3D&reserved=0>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kubernetes-sig-ne...@googlegroups.com
> <mailto:kubernetes-sig-ne...@googlegroups.com>.
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kubernetes-sig-ne...@googlegroups.com
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubernetes-sig-network/BL0PR05MB5460FA2EACC5E2A65687D1EEA8519%40BL0PR05MB5460.namprd05.prod.outlook.com
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FBL0PR05MB5460FA2EACC5E2A65687D1EEA8519%2540BL0PR05MB5460.namprd05.prod.outlook.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cdmcclure%40vmware.com%7C9971f9e62d384ff0be8208d915ee4388%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637564935364609791%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=vyq95fu0aDscElzLb3bCJN%2FcqZsjIyeUJD6O5n%2FvPbY%3D&reserved=0>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kubernetes-sig-ne...@googlegroups.com
> <mailto:kubernetes-sig-ne...@googlegroups.com>.
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kubernetes-sig-ne...@googlegroups.com
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubernetes-sig-network/BL0PR05MB5460FF9591F4E1DA5705AA2BA8519%40BL0PR05MB5460.namprd05.prod.outlook.com
> <https://groups.google.com/d/msgid/kubernetes-sig-network/BL0PR05MB5460FF9591F4E1DA5705AA2BA8519%40BL0PR05MB5460.namprd05.prod.outlook.com?utm_medium=email&utm_source=footer>.
David McClure
May 13, 2021, 9:51:46 AM5/13/21
to Dan Winship, Antonio Ojea, Tim Hockin, kubernetes-sig-network
> 2. My cluster/CNI is configured to create pods with IPv4 and IPv6
> addresses, but worker nodes only have IPv4 addresses
This depends on the details of the network plugin, but you'd most likely
have dual-stack traffic between pods on the same node, and probably also
have dual-stack traffic between pods on different nodes, but probably
only IPv4 cluster ingress and egress traffic.
Perhaps this is "less broken" than the first example, but I'm also struggling to imagine why anyone would want to do this.
From: Dan Winship <dwin...@redhat.com>
Sent: Thursday, May 13, 2021 9:43 AM
To: David McClure <dmcc...@vmware.com>; Antonio Ojea <antonio.o...@gmail.com>
Cc: Tim Hockin <tho...@google.com>; kubernetes-sig-network <kubernetes-...@googlegroups.com>
Subject: Re: [k8s-sig-net] Determining what IP families the infrastructure needs to support for a given cluster
Sent: Thursday, May 13, 2021 9:43 AM
To: David McClure <dmcc...@vmware.com>; Antonio Ojea <antonio.o...@gmail.com>
Cc: Tim Hockin <tho...@google.com>; kubernetes-sig-network <kubernetes-...@googlegroups.com>
Subject: Re: [k8s-sig-net] Determining what IP families the infrastructure needs to support for a given cluster
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes-sigs%2Fcluster-api%2Fpull%2F4558&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069716586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3i0Omsr%2BGzfMsxSIrUEW0oL2xyZCzy7eCY45Xp%2BlEds%3D&reserved=0>,
> we discussed the tradeoffs
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes-sigs%2Fcluster-api%2Fpull%2F4558%2Ffiles%23r625226177&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069716586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=kqVcmUeXYg%2B4sSFcqjRaONA9Hbrm2PcxBt0KFBj4xfA%3D&reserved=0>
> we discussed the tradeoffs
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes-sigs%2Fcluster-api%2Fpull%2F4558%2Ffiles%23r625226177&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069716586%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=kqVcmUeXYg%2B4sSFcqjRaONA9Hbrm2PcxBt0KFBj4xfA%3D&reserved=0>
> of auto-detecting the IP family to configure at the
> infrastructure layer (in this case, the docker machines), vs.
> requiring additional configuration.
>
> In Cluster API, the Cluster object has two optional fields
> for Pod CIDRs and Service CIDRs that allow us to do this in most
> cases, but there are a few ambiguous cases
> (namely when the Pod CIDRs field is not set). We've decided this
> is a "good enough" approach for now, but we may revisit it in
> the future.
>
> Putting Cluster API aside and just looking at Kubernetes itself,
> I wondered: is it possible to determine what IP families the
> infrastructure needs to support for a given cluster?
>
> I started looking into this and discussed it with Antonio
> briefly on Slack
>
> If I understand correctly, since pod CIDRs are optional and the
> CNI really determines the requirements for the infrastructure,
> it is not possible to determine the infrastructure requirements
> at the time the cluster is being created.
>
> Is that a fair statement?
>
> Do others see value in being able to determine what is required
> by the infrastructure earlier in the lifecycle of a cluster
> and/or in a specified way?
>
> Cheers,
> Dave
>
>
>
>
>
> --
> You received this message because you are subscribed to the
> Google Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to
> kubernetes-sig-ne...@googlegroups.com
> <mailto:kubernetes-sig-ne...@googlegroups.com>.
> To view this discussion on the web visit
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FBL0PR05MB54607311F5D6D5131FBF08E6A8529%2540BL0PR05MB5460.namprd05.prod.outlook.com&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069726583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=IhWBu7a4FC7b7zZ%2B2A12w8KhHJWWEfKJDf889WprCaE%3D&reserved=0
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FBL0PR05MB54607311F5D6D5131FBF08E6A8529%2540BL0PR05MB5460.namprd05.prod.outlook.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069726583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3emTchqDLMGCfElImtdVKMUe5RShtZAbj55H94%2Fpzh8%3D&reserved=0>.
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FBL0PR05MB54607311F5D6D5131FBF08E6A8529%2540BL0PR05MB5460.namprd05.prod.outlook.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069726583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3emTchqDLMGCfElImtdVKMUe5RShtZAbj55H94%2Fpzh8%3D&reserved=0>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kubernetes-sig-ne...@googlegroups.com
> <mailto:kubernetes-sig-ne...@googlegroups.com>.
> To view this discussion on the web visit
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FCAO_RewYMbj9-uHsk4%253DD%252BrKbqNSG%252BXe%252BP3NBvW2jU%252BTjOEOdcvQ%2540mail.gmail.com&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069726583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SH416AhcxXMfWKdxV7kA52x2G0KGR5Sqd6hfeZQoO2A%3D&reserved=0
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FCAO_RewYMbj9-uHsk4%253DD%252BrKbqNSG%252BXe%252BP3NBvW2jU%252BTjOEOdcvQ%2540mail.gmail.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069726583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=REP%2F6E5Vk%2BY%2Fk10O69Drq3tgTKVoSRmj5mg4b6fctKw%3D&reserved=0>.
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FCAO_RewYMbj9-uHsk4%253DD%252BrKbqNSG%252BXe%252BP3NBvW2jU%252BTjOEOdcvQ%2540mail.gmail.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069726583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=REP%2F6E5Vk%2BY%2Fk10O69Drq3tgTKVoSRmj5mg4b6fctKw%3D&reserved=0>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kubernetes-sig-ne...@googlegroups.com
> <mailto:kubernetes-sig-ne...@googlegroups.com>.
> To view this discussion on the web visit
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FBL0PR05MB5460FA2EACC5E2A65687D1EEA8519%2540BL0PR05MB5460.namprd05.prod.outlook.com&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069726583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=OUCPxU2t4%2Bcei3sunnagxn%2BfWQNZ%2BcTS458NWhnM20w%3D&reserved=0
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FBL0PR05MB5460FA2EACC5E2A65687D1EEA8519%2540BL0PR05MB5460.namprd05.prod.outlook.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069726583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rT3Bqr%2FDSYgatHGCwblLZL7m9%2FaN9oV60rXt2QeZRRE%3D&reserved=0>.
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FBL0PR05MB5460FA2EACC5E2A65687D1EEA8519%2540BL0PR05MB5460.namprd05.prod.outlook.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069726583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rT3Bqr%2FDSYgatHGCwblLZL7m9%2FaN9oV60rXt2QeZRRE%3D&reserved=0>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kubernetes-sig-ne...@googlegroups.com
> <mailto:kubernetes-sig-ne...@googlegroups.com>.
> To view this discussion on the web visit
>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FBL0PR05MB5460FF9591F4E1DA5705AA2BA8519%2540BL0PR05MB5460.namprd05.prod.outlook.com&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069726583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=zwm%2F2mKxBfIO2TAfJGkxYB1Jgaoz5R4wg0eahCc0BeI%3D&reserved=0
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FBL0PR05MB5460FF9591F4E1DA5705AA2BA8519%2540BL0PR05MB5460.namprd05.prod.outlook.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069726583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=czMjiJvV%2FOTlTIy6xKgIUk1tgSYHCqDCkqDq9cyzAI0%3D&reserved=0>.
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-network%2FBL0PR05MB5460FF9591F4E1DA5705AA2BA8519%2540BL0PR05MB5460.namprd05.prod.outlook.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cdmcclure%40vmware.com%7C58676f1986734acbd62108d91615142d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637565102069726583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=czMjiJvV%2FOTlTIy6xKgIUk1tgSYHCqDCkqDq9cyzAI0%3D&reserved=0>.
Dan Williams
May 13, 2021, 1:00:32 PM5/13/21
to David McClure, Dan Winship, Antonio Ojea, Tim Hockin, kubernetes-sig-network
On Thu, 2021-05-13 at 13:51 +0000, David McClure wrote:
> > > 2. My cluster/CNI is configured to create pods with IPv4 and
> > > IPv6
> > > addresses, but worker nodes only have IPv4 addresses
> >
> > This depends on the details of the network plugin, but you'd most
> > likely
> > have dual-stack traffic between pods on the same node, and probably
> > also
> > have dual-stack traffic between pods on different nodes, but
> > probably
> > only IPv4 cluster ingress and egress traffic.
> Perhaps this is "less broken" than the first example, but I'm also
> struggling to imagine why anyone would want to do this.
The other alternative is that the network plugin "handles" this kind of
> > > 2. My cluster/CNI is configured to create pods with IPv4 and
> > > IPv6
> > > addresses, but worker nodes only have IPv4 addresses
> >
> > This depends on the details of the network plugin, but you'd most
> > likely
> > have dual-stack traffic between pods on the same node, and probably
> > also
> > have dual-stack traffic between pods on different nodes, but
> > probably
> > only IPv4 cluster ingress and egress traffic.
> Perhaps this is "less broken" than the first example, but I'm also
> struggling to imagine why anyone would want to do this.
thing for you by using IP tunnels or IP translation so that if
something does try to talk to an IPv6 ClusterIP, the network plugin
somehow converts that to IPv4 and picks a pod from the IPv4 endpoints
and sends it along, then converts back to v6 on egress.
But that seems like a ton of work and complexity for something that
(like Dan said) is just silly to configure. Better to keep your cluster
v4-only and have your ingress LB do the 6to4 translation at the edge,
or something like that.
Dan
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubernetes-sig-network/BL0PR05MB5460AE7A7D7FFA5855E0AED8A8519%40BL0PR05MB5460.namprd05.prod.outlook.com
> .
Reply all
Reply to author
Forward
0 new messages