requirement of multiple nodes in conformance

[Brian Grant]

Issue escalated due to a DaemonSet test (https://github.com/kubernetes/kubernetes/issues/69601), but I believe it's of general relevance.

I don't know how many certified providers would be affected (though it's not obvious that any would be) or how many tests currently have the requirement, implicitly or explicitly, but I think users would benefit most from providers that support multiple nodes with the expected scheduling and networking behaviors.

So I think it makes sense for 2+ (untainted) nodes to be a requirement. (Yes, this would exclude minikube, which isn't certified today.)

Any other thoughts/opinions?

[Brendan Burns]

I feel like Joe mentioned last week this is a good opportunity to start to exercise the profiles work-flow.

What if we marked every test that can pass on one-node clusters as "base" and marked that daemonset test as the "multi-node" profile.

I agree that maybe there are so few providers that are single-node that this may be unnecessary busy-work, but at the same time it's clear to me that we're going to need to rip the band-aid on profiles sooner rather than later...

--brendan

---

From: 'Brian Grant' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Sent: Tuesday, October 30, 2018 1:22 PM
To: kubernetes-si...@googlegroups.com
Subject: requirement of multiple nodes in conformance

 

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAKCBhs49BpbCYXxD1i9Xug0Gn5A%2BpK6V6rpszJDK82qH2OJmmQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Clayton Coleman]

I would be ok with either suggestion.  2+ nodes is not unreasonable.  Maybe we should consider the current profile as the normal one and have a single node profile?  

Many of the scheduling tests don’t fit well with a tenanted user executing conformance anyway

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CY4PR21MB05041BE3C418C17FEF3BDC70DBCC0%40CY4PR21MB0504.namprd21.prod.outlook.com.

[Brian Grant]

I am in favor of categorizing tests, as was discussed in the conformance criteria PR (https://github.com/kubernetes/community/pull/2459):

https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md#kinds-of-tests

Once we identify important attributes, we'd have a much better sense of the potential impact of proposed profiles.

We just need someone to do that work.

[Brian Grant]

On Tue, Oct 30, 2018 at 4:09 PM Clayton Coleman <ccol...@redhat.com> wrote:

I would be ok with either suggestion.  2+ nodes is not unreasonable.  Maybe we should consider the current profile as the normal one and have a single node profile?  

Looking at the current certified providers, I don't even know that there is demand for a single-node profile.

Also, at a technical level, the most recent proposal was to make profiles purely additive, rather than subtractive, since the latter is more problematic when combining profiles.

Many of the scheduling tests don’t fit well with a tenanted user executing conformance anyway

Yes, we'll need to tackle that at some point.

[Daniel Smith]

I think requiring 2+ nodes shouldn't be problematic no matter how nodes are implemented by the provider.

Even if the provider is doing something like "virtual kubelet" I think it's still correct to make more than one node to represent separate failure domains so the rest of the logic can properly anticipate the expected fail over behavior.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAKCBhs4iUCdiqDyJ%2BkHMBOQA05g10_wO7-pfXQDbmSp%2BiWoiwQ%40mail.gmail.com.

[Davanum Srinivas]

+1 (reluctantly as i love testing with local-up-cluster.sh).

Is it 2 worker nodes (and 1 with the control plane processes)? (total of 3)

-- Dims

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3bZPDzVrRNQFqVDNbyKsW1b%3Dy_7R1Vst7uBWMQu-DFg1Fg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Davanum Srinivas :: https://twitter.com/dims

[Tim St. Clair]

I've been thinking a lot recently about the plethora of conformance
issues and questions. We've spent a lot of time trying to retro-fit
ideas into a legacy system that was built in an ad-hoc fashion.

I'd really prefer for us to start to define a set of requirements and
try to divide the work into meaningful chunks as a sub-project of
sig-arch. This was my intent with testing-commons, but there seems to
be more vested interest in the conformance area.

Instead of bikeshedding every sig-arch call, does it make more sense
to break this into a sub-project where we comprehensively try to
address these issues and report back?

Cheers,
Tim

> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CANw6fcE7Kz4LmnYKTWZkpWgODh4trN-uDs9wE2CHfT5-Jv3%3DRA%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.



--

Cheers,
Timothy St. Clair

“Do all the good you can. By all the means you can. In all the ways
you can. In all the places you can. At all the times you can. To all
the people you can. As long as ever you can.”

[Brendan Burns]

Yes I think that's a great idea. I'm happy to help lead the sub project...

--brendan

---

From: Tim St. Clair <timo...@gmail.com>
Sent: Friday, November 2, 2018 6:49:57 AM
To: Davanum Srinivas
Cc: Daniel Smith; Brian Grant; Clayton Coleman; Brendan Burns; kubernetes-si...@googlegroups.com
Subject: Re: requirement of multiple nodes in conformance

 

>>>> Issue escalated due to a DaemonSet test (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fkubernetes%2Fissues%2F69601&amp;data=02%7C01%7Cbburns%40microsoft.com%7C1ba0439be8e543b76a5808d640ca1b6e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636767634125066090&amp;sdata=Aah053KTTX65aAlvKWczTRQqkmsx%2FYt9Z0nAdpsPX5o%3D&amp;reserved=0), but I believe it's of general relevance.

>>>>
>>>> I don't know how many certified providers would be affected (though it's not obvious that any would be) or how many tests currently have the requirement, implicitly or explicitly, but I think users would benefit most from providers that support multiple nodes with the expected scheduling and networking behaviors.
>>>>
>>>> So I think it makes sense for 2+ (untainted) nodes to be a requirement. (Yes, this would exclude minikube, which isn't certified today.)
>>>>
>>>> Any other thoughts/opinions?
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
>>>> To post to this group, send email to kubernetes-si...@googlegroups.com.

>>>> To view this discussion on the web visit https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-architecture%2FCAKCBhs49BpbCYXxD1i9Xug0Gn5A%252BpK6V6rpszJDK82qH2OJmmQ%2540mail.gmail.com&amp;data=02%7C01%7Cbburns%40microsoft.com%7C1ba0439be8e543b76a5808d640ca1b6e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636767634125066090&amp;sdata=HHJMH0evI7Id3pKV8GXoPpRXA%2FbuZsNyc%2BW29020BSw%3D&amp;reserved=0.
>>>> For more options, visit https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Foptout&amp;data=02%7C01%7Cbburns%40microsoft.com%7C1ba0439be8e543b76a5808d640ca1b6e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636767634125066090&amp;sdata=SjC2RXykjGQShcESgSiuNPVA6T64ACuZOQFjyiWoAnU%3D&amp;reserved=0.

>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
>>>> To post to this group, send email to kubernetes-si...@googlegroups.com.

>>>> To view this discussion on the web visit https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-architecture%2FCY4PR21MB05041BE3C418C17FEF3BDC70DBCC0%2540CY4PR21MB0504.namprd21.prod.outlook.com&amp;data=02%7C01%7Cbburns%40microsoft.com%7C1ba0439be8e543b76a5808d640ca1b6e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636767634125066090&amp;sdata=2T%2B7w6zAElKkoCAbAyZXTttGl1dwPRiRIVvYLNDhepc%3D&amp;reserved=0.
>>>> For more options, visit https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Foptout&amp;data=02%7C01%7Cbburns%40microsoft.com%7C1ba0439be8e543b76a5808d640ca1b6e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636767634125066090&amp;sdata=SjC2RXykjGQShcESgSiuNPVA6T64ACuZOQFjyiWoAnU%3D&amp;reserved=0.

>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
>>> To post to this group, send email to kubernetes-si...@googlegroups.com.

>>> To view this discussion on the web visit https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-architecture%2FCAKCBhs4iUCdiqDyJ%252BkHMBOQA05g10_wO7-pfXQDbmSp%252BiWoiwQ%2540mail.gmail.com&amp;data=02%7C01%7Cbburns%40microsoft.com%7C1ba0439be8e543b76a5808d640ca1b6e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636767634125066090&amp;sdata=SK39qZC1eFsDvAYOXKtnSJpzIEgXgqKQfRYemgPfdcU%3D&amp;reserved=0.
>>> For more options, visit https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Foptout&amp;data=02%7C01%7Cbburns%40microsoft.com%7C1ba0439be8e543b76a5808d640ca1b6e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636767634125076099&amp;sdata=uh%2FzBGj73%2BQRPXTe4JNqEqLU3pabyFpgVZPOUBR1vqg%3D&amp;reserved=0.

>>
>> --
>> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
>> To post to this group, send email to kubernetes-si...@googlegroups.com.

>> To view this discussion on the web visit https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-architecture%2FCAB_J3bZPDzVrRNQFqVDNbyKsW1b%253Dy_7R1Vst7uBWMQu-DFg1Fg%2540mail.gmail.com&amp;data=02%7C01%7Cbburns%40microsoft.com%7C1ba0439be8e543b76a5808d640ca1b6e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636767634125076099&amp;sdata=Ji3I%2FzlNVS3hV9SGYqd7r3rrjILpoEGaUvUChIhAAwQ%3D&amp;reserved=0.
>> For more options, visit https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Foptout&amp;data=02%7C01%7Cbburns%40microsoft.com%7C1ba0439be8e543b76a5808d640ca1b6e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636767634125076099&amp;sdata=uh%2FzBGj73%2BQRPXTe4JNqEqLU3pabyFpgVZPOUBR1vqg%3D&amp;reserved=0.
>
>
>
> --
> Davanum Srinivas :: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fdims&amp;data=02%7C01%7Cbburns%40microsoft.com%7C1ba0439be8e543b76a5808d640ca1b6e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636767634125076099&amp;sdata=eRvH6v%2BiS2EzWsHWOoTkFfPweap%2BIBF%2FJs4hfrmKYmM%3D&amp;reserved=0

>
> --
> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
> To post to this group, send email to kubernetes-si...@googlegroups.com.

> To view this discussion on the web visit https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-architecture%2FCANw6fcE7Kz4LmnYKTWZkpWgODh4trN-uDs9wE2CHfT5-Jv3%253DRA%2540mail.gmail.com&amp;data=02%7C01%7Cbburns%40microsoft.com%7C1ba0439be8e543b76a5808d640ca1b6e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636767634125076099&amp;sdata=LSNEvLlcsMePyO0HbfwLHOX%2Bi2ZmBlxKlq4O1WZfTU0%3D&amp;reserved=0.
> For more options, visit https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Foptout&amp;data=02%7C01%7Cbburns%40microsoft.com%7C1ba0439be8e543b76a5808d640ca1b6e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636767634125076099&amp;sdata=uh%2FzBGj73%2BQRPXTe4JNqEqLU3pabyFpgVZPOUBR1vqg%3D&amp;reserved=0.

[Brian Grant]

The conformance effort is a subproject with people working on it.

If you'd like to help, great! 

It sounds like what is being requested is a discussion forum other than SIG Arch meetings and mailing list, plus github PRs and issues. We could use the conformance WG mailing list, which is very low-traffic.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CY4PR21MB0504016D87220FE8B0AE9C4ADBCF0%40CY4PR21MB0504.namprd21.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

[Davanum Srinivas]

Brian,

We need to close https://github.com/kubernetes/kubernetes/issues/69601 with the consensus here. 

Thanks,

Dims

[Brian Grant]

Dims, why should that issue be closed? The test is currently skipped in single-node configurations. It shouldn't be, at least when running as part of the conformance suite. It should fail instead.

[Davanum Srinivas]

Brian,

sorry used wrong words. i meant "bring closure" by documenting the consensus and providing guidance on how to proceed. 

I'll file a PR (or request Katherine) to not skip the test on single node cluster.

Thanks,

Dims

[Gareth Rushgrove]

I'd just like to note that there are 2 current conformant Kubernetes
distributions which target single nodes, Docker Desktop and MicroK8s.

I obviously work for Docker so I can be regarded as biased here, but
both of those (as well as Minikube which targets a single node but
it's yet conformant I believe?) are useful and widely used in the
Kubernetes community.

From Arun Gupta's recent survey of >1000 folks, Docker Desktop was the
most popular solution, used by ~40% of respondents.
https://twitter.com/arungupta/status/1073421250305892352

Do we have an opinion on this usecase or are we saying explicitly
"single node clusters cannot be conformant"?

Gareth

On Mon, 7 Jan 2019 at 17:24, 'Brian Grant' via

> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAKCBhs4a%2BRJLyADMi_41fqkMTiF1JqYRZgRO%3D3N_eE4bm0RPpg%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.



--

Gareth Rushgrove
@garethr

devopsweekly.com
morethanseven.net
garethrushgrove.com

[Tim Hockin]

I find the idea that a single node can't be conformant distressing,
but more in the words than the intention. If conformance includes
behaviors around multi-node scenarios, we simply can't confirm that a
1-node cluster is conformant, but to call it non-conformant has a
particularly negative connotation and real implications wrt trademark
and naming.

I think we can retain the meaning (can't confirm) without yanking
trademark allowances. After all, a user of a 1-node cluster can't
reasonably expect multi-node behaviors to work, so it does conform to
realistic expectations.

Do we have a list of the things that require multi-node?

> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAFi_6yKhf6MR404gv90-8FLAZGDJMvBCoQNOyTA2UN8SR-A11Q%40mail.gmail.com.

[Brian Grant]

On Tue, Jan 8, 2019 at 5:51 AM Gareth Rushgrove <gar...@morethanseven.net> wrote:

I'd just like to note that there are 2 current conformant Kubernetes
distributions which target single nodes, Docker Desktop and MicroK8s.

I didn't see these in the list when I looked, but it doesn't change my position. 

I obviously work for Docker so I can be regarded as biased here, but
both of those (as well as Minikube which targets a single node but
it's yet conformant I believe?) are useful and widely used in the
Kubernetes community.

From Arun Gupta's recent survey of >1000 folks, Docker Desktop was the
most popular solution, used by ~40% of respondents.
https://twitter.com/arungupta/status/1073421250305892352

Do we have an opinion on this usecase or are we saying explicitly
"single node clusters cannot be conformant"?

The latter. Or rather providers that can't support multiple nodes aren't conformant.

Have you explored supporting multi-node clusters? Single-node clusters are of fairly limited value and, more generally, the more closely that these local environments can emulate production environments, the more value they have to users.

[Brian Grant]

On Tue, Jan 8, 2019 at 9:05 AM Tim Hockin <tho...@google.com> wrote:

I find the idea that a single node can't be conformant distressing,
but more in the words than the intention.  If conformance includes
behaviors around multi-node scenarios, we simply can't confirm that a
1-node cluster is conformant, but to call it non-conformant has a
particularly negative connotation and real implications wrt trademark
and naming.

I'm unconvinced that the burden of supporting multiple nodes is excessive and that the value of single-node clusters is high enough to carve out an exception for this.  

I think we can retain the meaning (can't confirm) without yanking
trademark allowances.  After all, a user of a 1-node cluster can't
reasonably expect multi-node behaviors to work, so it does conform to
realistic expectations. 

Do we have a list of the things that require multi-node?

Pod networking

A number of DaemonSet features and behaviors

A number of scheduling features (e.g., node and pod affinity / anti-affinity)

Some PV behaviors

Probably other things

[Tim Hockin]

On Tue, Jan 8, 2019 at 9:12 AM Brian Grant <brian...@google.com> wrote:
>
> On Tue, Jan 8, 2019 at 9:05 AM Tim Hockin <tho...@google.com> wrote:
>>
>> I find the idea that a single node can't be conformant distressing,
>> but more in the words than the intention. If conformance includes
>> behaviors around multi-node scenarios, we simply can't confirm that a
>> 1-node cluster is conformant, but to call it non-conformant has a
>> particularly negative connotation and real implications wrt trademark
>> and naming.
>
>
> I'm unconvinced that the burden of supporting multiple nodes is excessive and that the value of single-node clusters is high enough to carve out an exception for this.
>>
>>
>> I think we can retain the meaning (can't confirm) without yanking
>> trademark allowances. After all, a user of a 1-node cluster can't
>> reasonably expect multi-node behaviors to work, so it does conform to
>> realistic expectations.
>>
>>
>> Do we have a list of the things that require multi-node?
>
>
> Pod networking

I can verify pod networking on a single node. It doesn't prove that
the same config would work for multi-node, but the requirement is only
that pods can reach pods.

> A number of DaemonSet features and behaviors

I think user expectations of DaemonSet are being met in a single node
(unless I am missing something). We just can't verify multi-node. I
have one node. Did I get a pod on every node?

> A number of scheduling features (e.g., node and pod affinity / anti-affinity)

You can verify that anti-affinity works by ensuring the second pod is
not on the same node as the first (even if that means pending).
Affinity is possible to test false positive, but that is true in any
case, isn't it?

> Some PV behaviors

I am trying to think what, but I expect it is similar to the pending
case above. I can verify that a PV got detached, even if there's
nowhere else to re-attach it.

> Probably other things

I think I am saying that some of these tests are relying on a second
node when they could instead rely on "not the first node" instead.

If we could fix those tests, are there lingering objections based on
principles/philosophy or just based on practicality?

[Gareth Rushgrove]

On Tue, 8 Jan 2019 at 09:08, Brian Grant <brian...@google.com> wrote:
>
> On Tue, Jan 8, 2019 at 5:51 AM Gareth Rushgrove <gar...@morethanseven.net> wrote:
>>
>> I'd just like to note that there are 2 current conformant Kubernetes
>> distributions which target single nodes, Docker Desktop and MicroK8s.
>
>
> I didn't see these in the list when I looked, but it doesn't change my position.
>>
>>
>> I obviously work for Docker so I can be regarded as biased here, but
>> both of those (as well as Minikube which targets a single node but
>> it's yet conformant I believe?) are useful and widely used in the
>> Kubernetes community.
>>
>> From Arun Gupta's recent survey of >1000 folks, Docker Desktop was the
>> most popular solution, used by ~40% of respondents.
>> https://twitter.com/arungupta/status/1073421250305892352
>>
>> Do we have an opinion on this usecase or are we saying explicitly
>> "single node clusters cannot be conformant"?
>
>
> The latter. Or rather providers that can't support multiple nodes aren't conformant.
>
> Have you explored supporting multi-node clusters?

Yes. This is obviously technically possible, but it's also more
expensive in terms of resource usage and more complex to
write/manage/maintain. This does come up as a request for a small
fraction of Docker Desktop users, but it's incredibly rare and most
common amongst folks who rolled there own setup with Vagrant
previously, rather than because of API compatibility.

> Single-node clusters are of fairly limited value and, more generally, the more closely that these local environments can emulate production environments, the more value they have to users.
>

I'd disagree with this statement. There are likely more than ~100k
active users between Kubernetes on Docker Desktop, Minikube and
microk8s. That's a lot of folks finding some value in the single node
cluster. I think Arun's tweet is relevant here too. There are also
lots of users who use Kubernetes and don't want or see value in a
local single node cluster. I think that typically points to different
teams with different sensibilities, rather than one being the ultimate
best option for all.

I think it follows that if single node clusters are of value to users,
the conformance tests have value for single node clusters.

Gareth

[Brian Grant]

On Tue, Jan 8, 2019 at 9:23 AM Tim Hockin <tho...@google.com> wrote:

On Tue, Jan 8, 2019 at 9:12 AM Brian Grant <brian...@google.com> wrote:
>
> On Tue, Jan 8, 2019 at 9:05 AM Tim Hockin <tho...@google.com> wrote:
>>
>> I find the idea that a single node can't be conformant distressing,
>> but more in the words than the intention.  If conformance includes
>> behaviors around multi-node scenarios, we simply can't confirm that a
>> 1-node cluster is conformant, but to call it non-conformant has a
>> particularly negative connotation and real implications wrt trademark
>> and naming.
>
>
> I'm unconvinced that the burden of supporting multiple nodes is excessive and that the value of single-node clusters is high enough to carve out an exception for this.
>>
>>
>> I think we can retain the meaning (can't confirm) without yanking
>> trademark allowances.  After all, a user of a 1-node cluster can't
>> reasonably expect multi-node behaviors to work, so it does conform to
>> realistic expectations.
>>
>>
>> Do we have a list of the things that require multi-node?
>
>
> Pod networking

I can verify pod networking on a single node.  It doesn't prove that
the same config would work for multi-node, but the requirement is only
that pods can reach pods.

> A number of DaemonSet features and behaviors

I think user expectations of DaemonSet are being met in a single node
(unless I am missing something).  We just can't verify multi-node.  I
have one node.  Did I get a pod on every node?

Rolling updates 

> A number of scheduling features (e.g., node and pod affinity / anti-affinity)

You can verify that anti-affinity works by ensuring the second pod is
not on the same node as the first (even if that means pending).
Affinity is possible to test false positive, but that is true in any
case, isn't it?

It would probably mean that new clusters would have to be created for more tests.

> Some PV behaviors

I am trying to think what, but I expect it is similar to the pending
case above.  I can verify that a PV got detached, even if there's
nowhere else to re-attach it.

ReadWriteOnce?

We don't have PV tests in conformance yet, though, so this isn't yet a blocker.

> Probably other things

I think I am saying that some of these tests are relying on a second
node when they could instead rely on "not the first node" instead.

If we could fix those tests, are there lingering objections based on
principles/philosophy or just based on practicality?

Practicality, utility, and simplicity for users.

[Brian Grant]

On Tue, Jan 8, 2019 at 9:35 AM Gareth Rushgrove <gar...@morethanseven.net> wrote:

On Tue, 8 Jan 2019 at 09:08, Brian Grant <brian...@google.com> wrote:
>
> On Tue, Jan 8, 2019 at 5:51 AM Gareth Rushgrove <gar...@morethanseven.net> wrote:
>>
>> I'd just like to note that there are 2 current conformant Kubernetes
>> distributions which target single nodes, Docker Desktop and MicroK8s.
>
>
> I didn't see these in the list when I looked, but it doesn't change my position.

Ah, it's listed in the spreadsheet, but not on the CNCF website, and not prominently mentioned on the Docker Desktop site.

Some value, sure.

Supporting this case in conformance would require some amount of work to:

• Change policies regarding tests of clusters with varying capabilities
• Fix existing tests that accidentally rely on multiple nodes
• Ensure that future tests don't accidentally rely on multiple nodes
• Ensure that providers capable of supporting multiple nodes actually exercise multiple nodes so as to avoid effectively circumventing relevant tests (even accidentally)
• Ensure more generally that variable test behaviors don't become a loophole, even an accidental one
• Figure out how differences in capabilities, such as single vs multi-node clusters, among conformant providers should be conveyed to users, without complicating or diluting the meaning for the 98% of certified providers that support multiple nodes today

So it's worth asking about the value to users of conformant clusters generally that only support one node, the value of local environments being conformant, and the value of single-node-only local development environments vs. having the option to provision multiple nodes.

[Tim Hockin]

On Tue, Jan 8, 2019 at 11:52 AM 'Brian Grant' via

kubernetes-sig-architecture
<kubernetes-si...@googlegroups.com> wrote:

> So it's worth asking about the value to users of conformant clusters generally that only support one node, the value of local environments being conformant, and the value of single-node-only local development environments vs. having the option to provision multiple nodes.

Do 1-node Docker "clusters" represent "local environments" (e.g. dev,
not prod) exclusively? If so, I agree that maybe the value of being
conformant is overblown and we maybe just need better words.

[Brian Grant]

Yes, AFAIK, there are just the 2 mentioned here: Docker Desktop and Microk8s. Minikube isn't certified, but is also local. 

KinD supports multiple nodes, which we need to run our full suite of e2e tests.

[Dan Kohn]

On Tue, Jan 8, 2019 at 2:52 PM 'Brian Grant' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com> wrote:

On Tue, Jan 8, 2019 at 9:35 AM Gareth Rushgrove <gar...@morethanseven.net> wrote:

On Tue, 8 Jan 2019 at 09:08, Brian Grant <brian...@google.com> wrote:

 

>> I'd just like to note that there are 2 current conformant Kubernetes
>> distributions which target single nodes, Docker Desktop and MicroK8s.

 

> I didn't see these in the list when I looked, but it doesn't change my position.

Ah, it's listed in the spreadsheet, but not on the CNCF website, and not prominently mentioned on the Docker Desktop site.

Microk8s was already listed: https://landscape.cncf.io/category=certified-kubernetes-distribution,certified-kubernetes-hosted,certified-kubernetes-installer&grouping=category&selected=canonical-microk8s

But, we had missed Docker Desktop due to an oversight. It's there now and we've confirmed that the website now matches the spreadsheet: https://landscape.cncf.io/category=certified-kubernetes-distribution,certified-kubernetes-hosted,certified-kubernetes-installer&grouping=category&selected=docker-desktop
--

Dan Kohn <d...@linuxfoundation.org>

Executive Director, Cloud Native Computing Foundation https://www.cncf.io

+1-415-233-1000 https://www.dankohn.com

 

[Mehdy Bohlool]

my 2 cents:

If the point of mentioned single-node clusters is dev, then there is value in supporting multiple virtual nodes in dev environment *even if users don't ask for it*. They may simply missing the point of those features.

For example, in case of DaemonSet, having a one node cluster does not help understand the feature, nor test anything with it. You may assume if you have N pods when there are N nodes, there will be assigned 1-1 but there is no way to test it nor to understand how it works with a single node cluster.

The point of dev cluster is to mimic production in a resource-friendly-less-performant way not to drop features to make it more performant.

[Justin Cormack]

The point of dev is not to mimic production: users do not ask us (Docker Desktop) to mimic production, they use it for small scale testing, and for learning Kubernetes. Kubernetes performance is already a problem on typical laptops (memory usage and CPU), and running multiple nodes makes this much worse, many developers have 8 or 16GB laptops and Kubernetes already uses too much resources with a single node. We are disappointed that Docker Desktop cannot be certified as conformant any more, we have been part of the conformance program for many years. We are unlikely to implement multi node support as there is no significant customer ask for it, and the only realistic way would be to move to something like Kind which is much less like a production setup.

Development environments for Kubernetes such as Docker Desktop and Minikube are extremely widely used and are the way that many people learn about Kubernetes. A single node is not "realistic for production" but Kubernetes as an API is the same, and we are seeing all sorts of use cases of Kubernetes on Edge and other places that do not look exactly like the original model either. 

Justin Cormack

[John Belamaric]

Hi Justin,

I understand your disappointment. Ultimately, the conformance program is intended to ensure portability of user workloads, and multiple nodes are an important factor in that goal. However, we have also recognized that there are many, many deployment scenarios for k8s. We have entertained, on multiple occasions, the idea of "conformance profiles" which would capture sets of optional behaviors or different classes of cluster. If we implemented this, we could have a "dev" or "single-node" profile that would allow conformance within those constraints. We have not really been able to agree on how those profiles would be defined, unfortunately, and so it has not moved forward. 

Here are some thoughts I put together the last time we looked at this, it also links out to a few other places:

https://docs.google.com/document/d/1eVEVAH8-kt95hbSTJi8qRosgvSvBAEgKz4Tx1xsjt4I/edit

If it's something you're interested in pursuing, we'd love to hear from you about it. Feel free to come to the next conformance meeting on 2/9 at 1pm Pacific.

John

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/3caf57bc-b19d-4475-8a59-7298b27f61adn%40googlegroups.com.