Switching container images to debian 12 (bookworm)

[Sascha Grunert]

Hey folks,

Debian 12 got released 12 days ago, so I would like to bring up the
point of switching our base images to the new distribution.

The default Docker Hub golang:1.20/19 image tags already refer to the
new version, which may cause implicit issues in CI stability if we
don't pin them to a specific distribution like golang:1.20-bullseye.

I created a tracking issue for the k/release images [0] and started
working on porting debian-base [1].

I'd like to set a lazy consensus until July 1st to open-up room for
input and discussions before we move forward. I appreciate any
thoughts on that topic.

Have a wonderful day and all the best,
Sascha
(SIG Release)

[0]: https://github.com/kubernetes/release/issues/3128
[1]: https://github.com/kubernetes/release/pull/3127

[Sascha Grunert]

Hey folks,

We finally reached consensus.

registry.k8s.io/build-image/debian-base:bookworm-v1.0.0 is now
available for y'all.

Tracking issue: https://github.com/kubernetes/release/issues/3128

All the best,
Sascha

On Tue, Jul 4, 2023 at 12:33 PM Marko Mudrinić <mudrin...@gmail.com> wrote:
>
> +1
>
> On Tue, Jun 27, 2023 at 9:53 AM Carlos Tadeu Panato Jr <cta...@gmail.com> wrote:
>>
>> +1
>>
>> Em qui., 22 de jun. de 2023 às 12:12, Verónica López <gvero...@gmail.com> escreveu:
>>>
>>> +1, we did at work for the same reason you mention concerning CI, which was already having issues.

>>> --
>>> You received this message because you are subscribed to the Google Groups "release-managers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to release-manage...@kubernetes.io.
>>> To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/release-managers/487da8dd-b40d-40a1-bfc5-fd6e73b006dan%40kubernetes.io.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "release-managers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to release-manage...@kubernetes.io.
>> To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/release-managers/CAOxYG4zzeyY2pYW-kuwAPJTcOi1h9sZbkwsG4ZFbM0xr_CKHHA%40mail.gmail.com.

[Tim Hockin]

I tried applying this git-sync and found the resulting image is about
40 MB larger.

```
gcr.io/k8s-staging-git-sync/git-sync
v4.0.0-rc2-3-g0753bd5__linux_amd64 fb8730cea360 10 seconds
ago 85.5MB
gcr.io/k8s-staging-git-sync/git-sync
v4.0.0-rc2-3-g0753bd5-dirty__linux_amd64 3a4af9e802e8 10 minutes
ago 112MB
```
That's a 50% increase. I don't know what is larger within the image yet.

> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAPre7xBj3OPmhFgQKVzuzJK6WfmyJfk_xUPxnjKjZfxd0yoZjg%40mail.gmail.com.

[Tim Hockin]

I....may have screwed up my one world. Belay this email, please.

[Tim Hockin]

Yeah, it was me. Sorry.

[Benjamin Elder]

This is a good point, though it's not a new problem and I don't think we've previously blocked on this?

We definitely want to compile against the oldest glibc we support to be certain, glibc is backwards compatible but not necessarily forwards compatible.

https://abi-laboratory.pro/?view=timeline&l=glibc

Debian stable has usually been a reasonable enough target for this.

Even RHEL 9 is on 2.34 for example, but extended support for RHEL 6 still exists and it's only on glibc 2.12 AFAICT ...

Distros and users can compile their own kubelet builds against an older glibc if they need.

On Wed, Jul 5, 2023 at 5:44 PM 'David Porter' via release-managers <release-...@kubernetes.io> wrote:

Sorry for the late note, but one concern to note - debian bookworm is upgraded to glibc 2.34 while bullseye is on glibc 2.31. 

I'm not super familiar with the build process for binaries, but if it's using the kube-cross image in particular, it may result in an issue because the kubelet dynamically links against glibc and is not statically built (xref). As a result, those binaries would only run on systems which have the newer glibc, so this would be a problem because kubelet built on debian bookworm would result in kubelet not being able to run on older distros that have older glibc. 

Thanks,

David

You received this message because you are subscribed to the Google Groups "kubernetes-sig-release" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-re...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-release/CAO_RewZSa9jq-4g_bTB2Xa4f1vgknceSbe03CxbbsDd%2B56rZ-g%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "release-managers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to release-manage...@kubernetes.io.

To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/release-managers/CAKjGtWsNgG-ye8nALqOEGqCL%2B7qHOBPm7mnveN9mqF2eMrrwPw%40mail.gmail.com.

[Tim Hockin]

I may be taking back my take-back. After fixing my own problem (I think) the result is still significantly bigger than bullseye. I am still debugging in my free time (ha!) but we should pay attention to the size, I think. 

[Benjamin Elder]

We could keep kube-cross on debian 11 for existing minor releases and just put a release notes entry for the rest?

On Thu, Jul 6, 2023 at 12:04 PM Marko Mudrinić <mudrin...@gmail.com> wrote:

I agree with Ben in a large part. We need to move forward at some point and should't block on it for a long time.

The only remark I have is that we should make sure that we don't introduce such changes as a part of patch releases.

Users should be able to easily upgrade from a patch release to another especially because of potential security issues and CVEs.

Other than that, I think if we announce this before a new minor release and deliver it as part of that upcoming minor release, it should be fine.

Given that, announcing this on kubernetes-announce might be a good idea so end users are aware of it.

To view this discussion on the web visit https://groups.google.com/a/kubernetes.io/d/msgid/release-managers/CAOZRXm9Z%2B1xGa2NuPvvYyTJDeWBvtP2orJGf%2B9FVMjnHXW1WnQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-release" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-re...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-release/CAEtDNhoex_9MxYtH%2B9QWS99W6Sr4xD4g7UTvv-iF5AVTE4272g%40mail.gmail.com.

[Benjamin Elder]

The other images don't involve interacting with the host glibc at all.
We only care about the build environment for non-static non-containerized binaries.

On Thu, Jul 6, 2023 at 12:27 PM Marko Mudrinić <mudrin...@gmail.com> wrote:

I'm thinking if just kube-cross is enough or should we keep other/all images (e.g. go-runner, setcap, iptables, k8s-cloud-builder, etc.).

[Sascha Grunert]

On Thu, Jul 6, 2023 at 9:28 PM Benjamin Elder <benth...@google.com> wrote:
>
> The other images don't involve interacting with the host glibc at all.
> We only care about the build environment for non-static non-containerized binaries.
>
> On Thu, Jul 6, 2023 at 12:27 PM Marko Mudrinić <mudrin...@gmail.com> wrote:
>>
>> I'm thinking if just kube-cross is enough or should we keep other/all images (e.g. go-runner, setcap, iptables, k8s-cloud-builder, etc.).
>>
>> On Thu, Jul 6, 2023 at 9:25 PM Benjamin Elder <benth...@google.com> wrote:
>>>
>>> We could keep kube-cross on debian 11 for existing minor releases and just put a release notes entry for the rest?

Sounds good to me!