Update regarding proposal to stop building and releasing artifacts for mips64le, ppc64le, and s390x

[Jeremy Rickard]

Hello Kubernetes Community,

Tl;dr:

• We’ll stop producing kube-cross image for ppc64le and s390x architectures effective immediately

• We’ll continue releasing Kubernetes artifacts (binaries, container images, and packages) for ppc64le and s390x architectures for Kubernetes versions up to (and including) v1.32, as long as those Kubernetes releases are supported

• We’ll stop publishing all release artifacts for ppc64le and s390x architectures starting with Kubernetes 1.33

• We’ll not change the libc dependency/requirement at this time, but this might change in future Kubernetes minor releases

We’d like to thank everyone for their feedback. In the SIG Release meeting on Tuesday, October 8, 2024[1], we had a very active discussion about this topic as well. One approach that we discussed was dropping the problematic architectures from our cross image[2], which is used to cross compile the various release artifacts. Thank you to Manjunath Kumatagi for suggesting this. This could potentially have an impact on any downstream consumers that are directly consuming cross image for these architectures, as they will now need to produce them. 

We have confirmed that during the release process, we only make use of the amd64 architecture, so doing this would allow us move forward with the Go version bump. Additionally, we can continue to produce these architectures for the currently supported release branches (1.31, 1.30, 1.29, 1.28) as well as the in progress 1.32 release. 

The suggestion to bump our cross image to bookworm is not something we can do at this time. The impact from this change would impact far more users than dropping the architectures in question. The kubelet binary is not statically linked, and bumping the build to use bookworm would increase the libc dependency to a version which has not been widely adopted. We also do not have bandwidth to begin producing our own base images to mitigate this problem. We will begin looking at this soon in order to identify solutions that allow us to minimize or avoid breaking downstream consumers  but we do not think we should force this change now in order to continue producing the cross image for these affected architectures. 

It is important to mention that these architectures are currently not tested in a manner that aligns with our criteria[3] for release blocking jobs. There are also associated costs, both in terms of compute and the time of release engineering volunteers, for offering less-common architectures. For these reasons, we have decided to drop these architectures from the published release artifacts starting with the 1.33 release. We are happy to work with downstream consumers to help them continue building artifacts for these architectures, and if an effort forms to produce “community” artifacts for these architectures outside of the official Kubernetes release process, we can link to these as has been done in the past[4]. If community managed infrastructure becomes available in the 1.33 or later timeframes, along with appropriate commitments to maintain release informing/release blocking tests, we can revisit this decision. 

Thank you again,

Jeremy Rickard // on behalf of SIG Release

—-

[1] https://bit.ly/k8s-sig-release-meeting

[2] https://github.com/kubernetes/release/blob/451fd1f88417adc41f8836e99cfcee92bbfaa785/images/build/cross/Makefile#L56

[3] https://github.com/kubernetes/sig-release/blob/master/release-blocking-jobs.md#release-blocking-criteria-and-dashboard

[4] https://kubernetes.io/blog/2020/01/15/kubernetes-on-mips/

[Jordan Liggitt]

Thanks for the update, I appreciate the effort to preserve behavior on existing release branches to the greatest extent possible.

[Manjunath Kumatagi]

Thank you for the detailed update and for continuing to produce artifacts for the released branches as well as the upcoming release(1.32). We also appreciate your acceptance of the proposed workaround to unblock the release.

We acknowledge and understand the requirements for release-blocking jobs and affirm our intent to add IBM architectures to these jobs as soon as the criteria are met. We are actively exploring options for contributing IBM infrastructure to CNCF so that we can run the necessary tests on community-managed infrastructure to meet the required conditions for releasing the artifacts. We've already initiated conversations with CNCF on this front and will keep you posted as we make progress.

We believe that with continued community involvement and infrastructure contributions, we can sustain the release of artifacts for these architectures in future Kubernetes versions.

Thank you for your consideration, and we look forward to collaborating on this moving forward.

Thanks,

Manjunath Kumatagi