Inconsistently shipping (CPU) architectures for which we have no testing

[Benjamin Elder]

Hello SIG Architecture (SIG Release & Testing CCed),

Today in the code organization meeting the somewhat off-topic question of why we ship & maintain platforms for which we have no test signal came up.

For context, currently we ship builds of some or all of Kubernetes for the following at least somewhat tested builds:

- linux/amd64
- windows/amd64 *

* "node" binaries and kubectl only

And the following wholly untested builds (to my knowledge)

- linux/386 *
- linux/arm
- linux/arm64
- linux/s390x
- linux/ppc64le
- windows/386 *
- darwin/amd64 *

* kubectl only

Many of these I doubt even have post-release conformance submissions from downstream (and Windows certainly doesn't but that's another topic).

Not too distantly we blocked linux/RISC-V builds on the premise that this would require a KEP and that we did not want to build for platforms that we do not have signal for.

The question now is: Should we continue to ship builds for platforms we have zero testing for?

I can understand why some of these would be sorely missed if removed, but I also think if that's the case then someone with a vested interest in them could ensure that they are tested.
As it stands we're 6 years into the project and these platforms still do not have any signal, yet we ship them and take at least some support load as well as not-super-trivial CI resources & increased release time building these.

Should we allow the RISC-V folks to add their build with the same level of support as e.g. s390x?
Or should we stop shipping ourselves some or all of these platforms for which we have no tests ...

Thank you for your time and consideration,

- Ben

[Calvin Hartwell]

Hi Ben, 

We are currently testing (internally) the release of our k8s distro on all of the above mentioned architectures, outside of 386, Windows, ppc64le and darwin. 

Is there any way we can help? 

Cheers,

    - Calvin / Кэльвин / كالفين / かるびん / קלווין

Calvin Hartwell Director of Field Engineering EMEA | Canonical Ltd. Mobile: +44 (0) 7534801542 |Email:calvin....@canonical.com

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-release" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-re...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-release/CAOZRXm_Kfd0oj%2ByFeqmokcjxKxEqxaTB%2Bs%3DWfGwzdz-QoOJLCA%40mail.gmail.com.

[Derek Carr]

let's queue this up for discussion in next sig-arch meeting.

if volunteers interested in each architecture can speak up prior to that meeting and see if we can coalesce a group to address the gap, it would be great.

You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAF9mF2YL1MGGeXLCBH-v6W0zAxTdW1x6cV0c3JNhOQE_hBBBDg%40mail.gmail.com.

[Cindy Lee]

Hi Ben, When is the next sig-arch meeting? My team has been heavily involved in s390x arch and depends on what it takes to enable test, we can see how we could help.

[Benjamin Elder]

> We are currently testing (internally) the release of our k8s distro on all of the above mentioned architectures, outside of 386, Windows, ppc64le and darwin. 

>

> Is there any way we can help? 

Yes, please come join #sig-testing and let's discuss.
In the meantime you might be interested in a program I setup for contributing third-party results, though we tend to prefer contributing resources and tools / upstream work to get tests running on our community CI where contributors can reconfigure by pull request to some yaml, etc..

----------------

> let's queue this up for discussion in next sig-arch meeting.

+1.

> if volunteers interested in each architecture can speak up prior to that meeting and see if we can coalesce a group to address the gap, it would be great.

While I appreciate efforts to close the gap, this has long been the case with previous efforts producing a lot of noise but nothing we can count on to ship releases based on ... and isn't really what I'm asking in this thread.

What I'm asking for right now is to establish policy for platform support, so as to stop holding platforms we ship upstream to inconsistent standards and give new platforms a consistent set of expectations.

Bringing the platforms we do ship up to speed would be fantastic, but that is a related, yet different conversation.

We should still set less than arbitrary expectations for what platforms we're willing to include in upstream builds even if we manage to bring all current platforms up to the same unwritten standard.

----------------

> Hi Ben, When is the next sig-arch meeting? My team has been heavily involved in s390x arch and depends on what it takes to enable test, we can see how we could help.

Hi Cindy, I appreciate your responses and interest in helping!

You can find all Kubernetes community details in https://github.com/kubernetes/community, SIG Architecture's next meeting in particular is Thursday the 18th.
All members of the (publicly joinable) mailing list should receive invites automatically, and there are also direct links to the zoom and agenda doc in kubernetes/community, as well as the SIG slack channel.

----------------

One correction and some further notes:

We do have some very limited, third-party CI results on testgrid.k8s.io for linux/arm64 and linux/ppc64le that are currently still functional.
I was not aware that any were currently functioning but I did some digging and found ~3 jobs that are still posting limited results.
Given the dashboards these are in, I don't believe these are currently monitored upstream.


A concrete example of why this situation is a problem beyond just being inconsistent for new platform requests is the kube-proxy issue I believe Cindy alluded to.
Our kube-proxy image from 1.17+ development did not function on s390x up through release, with a bug report coming in 36 days after the release ...
It was not fixed until 1.17.6, due to lack of upstream resources.
1.17.0 - 1.17.5 official Kubernetes builds did not work on s390x per the bug.


An additional related but perhaps different topic those reading this thread may or may not yet be aware of is the recent ask to port Kubernetes to Illumos ...

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/1ceb2771-2fcb-4409-b3c5-16994cef5f64o%40googlegroups.com.

[Pradipta Banerjee]

Hi Ben,
Thanks for bringing this up. Few comments specific to Power (ppc64le) CPU arch.

One correction and some further notes:

We do have some very limited, third-party CI results on testgrid.k8s.io for linux/arm64 and linux/ppc64le that are currently still functional.
I was not aware that any were currently functioning but I did some digging and found ~3 jobs that are still posting limited results.
Given the dashboards these are in, I don't believe these are currently monitored upstream.

For ppc64le arch we are following the guidelines for contributing 3rd party results as described in the following link. As you had already pointed out, we have limited CI for ppc64le arch. Specifically we are running conformance tests for daily builds and also running unit tests [1,2]. Additionally we are also working to augment it with support for additional tests. Manju is the key focal for ppc64le arch.  

We will continue to work with the community to address any concerns.

1. https://testgrid.k8s.io/sig-node-ppc64le#Summary

2. https://cs.k8s.io/?q=conformance-ppc64le&i=nope&files=&repos=kubernetes/test-infra

Thanks,

Pradipta (IBM)

 

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsub...@googlegroups.com.

[Cindy Lee]

Thanks Ben, I got your invite, will join the call. I'm trying to dig out some history from my(s390x) side too because we've tried to add e2e testing before(at least a year ago though) but got stuck. Will find out more to see whether the same blocker still applies or not.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsub...@googlegroups.com.

[John Belamaric]

It is on the agenda for tomorrow, Derek had requested that, we just hadn't added it yet. It's there now.

On Tue, Jun 16, 2020 at 5:56 PM Tim Pepper <tpe...@vmware.com> wrote:

In SIG Release we’ve gotten pinged this month about the possibility of IllumOS support in the project, opening another variation of the os/arch discussion in this thread.

 

Has any conversation happened out of band on this?

 

Will this be on Thursday’s agenda?

(not currently listed at https://docs.google.com/document/d/1BlmHq5uPyBUDlppYqAAzslVbAO8hilgjqZUTaNXUhKM/edit#bookmark=kix.mvoqmhsy4ulz )

 

-- 

Tim Pepper

Orchestration Lead

VMware Open Source Technology Center

 

From: <kubernetes-...@googlegroups.com> on behalf of Derek Carr <dec...@redhat.com>
Date: Thursday, June 11, 2020 at 12:43 PM
To: Calvin Hartwell <calvin....@canonical.com>
Cc: Benjamin Elder <benth...@google.com>, kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>, kubernetes-sig-release <kubernetes-...@googlegroups.com>, kubernetes-sig-testing <kubernetes-...@googlegroups.com>
Subject: Re: Inconsistently shipping (CPU) architectures for which we have no testing

 

let's queue this up for discussion in next sig-arch meeting.

if volunteers interested in each architecture can speak up prior to that meeting and see if we can coalesce a group to address the gap, it would be great.

 

On Thu, Jun 11, 2020 at 3:37 PM Calvin Hartwell <calvin....@canonical.com> wrote:

Hi Ben, 

 

We are currently testing (internally) the release of our k8s distro on all of the above mentioned architectures, outside of 386, Windows, ppc64le and darwin. 

 

Is there any way we can help? 

Cheers,

    - Calvin / Кэльвин / كالفين / かるびん / קלווין

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-release/CAHROWxQty4X5CWpFG5%2B1U5mUXgwke3j3mh6mYeB15pq7pXmoEQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-release" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-re...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-release/04732607-B4A5-4032-87F0-B35B78E2F2C2%40vmware.com.