[IMPORTANT] Issue with container image signatures for Kubernetes 1.26.2, 1.25.7, 1.24.11, 1.23.17 patch releases

[Marko Mudrinić]

Dear Kubernetes Community,

We’re happy to announce that Kubernetes patch releases for February (1.26.2, 1.25.7, 1.24.11, 1.23.17) are available. However, we would like to point out an issue with container image signatures affecting those releases.

We’re signing all container images using cosign[1]. Users can check and validate integrity of container images (and binary artifacts) using cosign as described in docs[2].

While signing container images for those patch releases, we got affected by rate limits on the Sigstore side. We were NOT able to sign all container images for those patch releases. This means that if you try to verify affected images, cosign will report that those images are not signed.

This only affects users who are automatically or manually validating container images using cosign. The patch releases are fully functional and you can run them as usual if this doesn’t affect you. Additionally, binary artifacts are completely signed.

We plan on retroactively signing those container images in the upcoming days and we’ll keep this thread updated with more details. We’ll also provide more details about the issue.

If you have any questions, please feel free to reach out to Release Managers[3] on #release-management channel on Kubernetes Slack[4].

Thank you for understanding!

Marko Mudrinić,

on behalf of Kubernetes Release Managers

[1]: https://docs.sigstore.dev/cosign/overview/

[2]: https://kubernetes.io/docs/tasks/administer-cluster/verify-signed-artifacts/

[3]: https://kubernetes.io/releases/release-managers/

[4]: https://kubernetes.slack.com/archives/CJH2GBF7Y

[Marko Mudrinić]

Dear Kubernetes Community,

We want to follow up on this and provide you some updates.

We released Kubernetes 1.26.3, 1.25.8, and 1.24.12 today. However, those releases are also affected by this issue, i.e. some container images are missing signatures. This time the root cause is different. We hit rate limits imposed on the Google Artifact Registry (AR) that we’re publishing our images to. Those rate limits failed the image promotion process which is responsible for publishing and signing images. With that said, some container images for those releases are missing signatures.

As mentioned in the previous email, we’re working on tooling to allow us to retroactively sign container images in such cases. We made good progress since the original email -- the tooling needed to fix those images is now in place. The next step is to test the tooling and if tests are successful, we’ll be able to fix images for all affected releases.

At the time of writing this email, affected releases are:

• 1.26.2 - 1.26.3

• 1.25.7 - 1.25.8

• 1.24.11 - 1.24.12

• 1.23.17

This only affects users who are automatically or manually validating container images using cosign. All affected patch releases are fully functional and you can run them as usual if this issue doesn’t affect you. Additionally, binary artifacts are completely signed.

For more information about the progress, please see the following GitHub issue[1]. We’ll also send another update here once the issue is fixed.

If you have any questions, please feel free to reply on the GitHub issue or reach out to Release Managers[2] on #release-management channel on Kubernetes Slack[3].

Thank you for your patience and understanding!

Marko Mudrinić,

on behalf of Kubernetes Release Managers

[1]: https://github.com/kubernetes/release/issues/2962

[2]: https://kubernetes.io/releases/release-managers/

[3]: https://kubernetes.slack.com/archives/CJH2GBF7Y