Switching container images to debian 12 (bookworm)

[Sascha Grunert]

Hey folks,

Debian 12 got released 12 days ago, so I would like to bring up the
point of switching our base images to the new distribution.

The default Docker Hub golang:1.20/19 image tags already refer to the
new version, which may cause implicit issues in CI stability if we
don't pin them to a specific distribution like golang:1.20-bullseye.

I created a tracking issue for the k/release images [0] and started
working on porting debian-base [1].

I'd like to set a lazy consensus until July 1st to open-up room for
input and discussions before we move forward. I appreciate any
thoughts on that topic.

Have a wonderful day and all the best,
Sascha
(SIG Release)

[0]: https://github.com/kubernetes/release/issues/3128
[1]: https://github.com/kubernetes/release/pull/3127

[Verónica López]

+1, we did at work for the same reason you mention concerning CI, which was already having issues. 

[Carlos Tadeu Panato Jr]

+1

--
You received this message because you are subscribed to the Google Groups "release-managers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to release-manage...@kubernetes.io.
To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/release-managers/487da8dd-b40d-40a1-bfc5-fd6e73b006dan%40kubernetes.io.

[Marko Mudrinić]

+1

To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/release-managers/CAOxYG4zzeyY2pYW-kuwAPJTcOi1h9sZbkwsG4ZFbM0xr_CKHHA%40mail.gmail.com.

[Sascha Grunert]

Hey folks,

We finally reached consensus.

registry.k8s.io/build-image/debian-base:bookworm-v1.0.0 is now
available for y'all.

Tracking issue: https://github.com/kubernetes/release/issues/3128

All the best,
Sascha

[Tim Hockin]

I tried applying this git-sync and found the resulting image is about
40 MB larger.

```
gcr.io/k8s-staging-git-sync/git-sync
v4.0.0-rc2-3-g0753bd5__linux_amd64 fb8730cea360 10 seconds
ago 85.5MB
gcr.io/k8s-staging-git-sync/git-sync
v4.0.0-rc2-3-g0753bd5-dirty__linux_amd64 3a4af9e802e8 10 minutes
ago 112MB
```
That's a 50% increase. I don't know what is larger within the image yet.

> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAPre7xBj3OPmhFgQKVzuzJK6WfmyJfk_xUPxnjKjZfxd0yoZjg%40mail.gmail.com.

[Tim Hockin]

I....may have screwed up my one world. Belay this email, please.

[Tim Hockin]

Yeah, it was me. Sorry.

[David Porter]

Sorry for the late note, but one concern to note - debian bookworm is upgraded to glibc 2.34 while bullseye is on glibc 2.31. 

I'm not super familiar with the build process for binaries, but if it's using the kube-cross image in particular, it may result in an issue because the kubelet dynamically links against glibc and is not statically built (xref). As a result, those binaries would only run on systems which have the newer glibc, so this would be a problem because kubelet built on debian bookworm would result in kubelet not being able to run on older distros that have older glibc. 

Thanks,

David

You received this message because you are subscribed to the Google Groups "kubernetes-sig-release" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-re...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-release/CAO_RewZSa9jq-4g_bTB2Xa4f1vgknceSbe03CxbbsDd%2B56rZ-g%40mail.gmail.com.

[Benjamin Elder]

This is a good point, though it's not a new problem and I don't think we've previously blocked on this?

We definitely want to compile against the oldest glibc we support to be certain, glibc is backwards compatible but not necessarily forwards compatible.

https://abi-laboratory.pro/?view=timeline&l=glibc

Debian stable has usually been a reasonable enough target for this.

Even RHEL 9 is on 2.34 for example, but extended support for RHEL 6 still exists and it's only on glibc 2.12 AFAICT ...

Distros and users can compile their own kubelet builds against an older glibc if they need.

To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/release-managers/CAKjGtWsNgG-ye8nALqOEGqCL%2B7qHOBPm7mnveN9mqF2eMrrwPw%40mail.gmail.com.

[Marko Mudrinić]

I agree with Ben in a large part. We need to move forward at some point and should't block on it for a long time.

The only remark I have is that we should make sure that we don't introduce such changes as a part of patch releases.

Users should be able to easily upgrade from a patch release to another especially because of potential security issues and CVEs.

Other than that, I think if we announce this before a new minor release and deliver it as part of that upcoming minor release, it should be fine.

Given that, announcing this on kubernetes-announce might be a good idea so end users are aware of it.

To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/release-managers/CAOZRXm9Z%2B1xGa2NuPvvYyTJDeWBvtP2orJGf%2B9FVMjnHXW1WnQ%40mail.gmail.com.

[Tim Hockin]

I may be taking back my take-back. After fixing my own problem (I think) the result is still significantly bigger than bullseye. I am still debugging in my free time (ha!) but we should pay attention to the size, I think. 

[Benjamin Elder]

We could keep kube-cross on debian 11 for existing minor releases and just put a release notes entry for the rest?

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-release/CAEtDNhoex_9MxYtH%2B9QWS99W6Sr4xD4g7UTvv-iF5AVTE4272g%40mail.gmail.com.

[Marko Mudrinić]

I'm thinking if just kube-cross is enough or should we keep other/all images (e.g. go-runner, setcap, iptables, k8s-cloud-builder, etc.).

[Benjamin Elder]

The other images don't involve interacting with the host glibc at all.
We only care about the build environment for non-static non-containerized binaries.

[Sascha Grunert]

On Thu, Jul 6, 2023 at 9:28 PM Benjamin Elder <benth...@google.com> wrote:
>
> The other images don't involve interacting with the host glibc at all.
> We only care about the build environment for non-static non-containerized binaries.
>
> On Thu, Jul 6, 2023 at 12:27 PM Marko Mudrinić <mudrin...@gmail.com> wrote:
>>
>> I'm thinking if just kube-cross is enough or should we keep other/all images (e.g. go-runner, setcap, iptables, k8s-cloud-builder, etc.).
>>
>> On Thu, Jul 6, 2023 at 9:25 PM Benjamin Elder <benth...@google.com> wrote:
>>>
>>> We could keep kube-cross on debian 11 for existing minor releases and just put a release notes entry for the rest?

Sounds good to me!