Sigstore container image signature validation support

236 views
Skip to first unread message

Sascha Grunert

unread,
Apr 27, 2023, 8:39:25 AM4/27/23
to kubernete...@googlegroups.com, kubernetes-sig-architecture
Hey folks,

I'm having an enhancement in mind to be able to validate sigstore
signatures of container images before pulling it within the image
manager of the kubelet [0]. The idea is to add a CRI method for the
validation, which passes the policy down to the container runtime and
can handle errors correctly. Policies define which signatures are
allowed for a subset of images and will also provide the main user
interface.

I'm wondering where the policies could fit into Kubernetes. Using
admission controllers, or just the kubelet configuration?

I'd appreciate any thoughts on the feature.

Thank you and all the best,
Sascha

[0]: https://github.com/kubernetes/kubernetes/blob/041bb9a/pkg/kubelet/images/image_manager.go#L155

Peter Hunt

unread,
Apr 27, 2023, 9:44:14 AM4/27/23
to Sascha Grunert, kubernete...@googlegroups.com, kubernetes-sig-architecture
How would the sig validation fit into admission? I'm not very familiar with admission controllers: is there a precedent for an admission controller to be able to ask kubelet to ask CRI to do something (verify a signature in this case) before the pod is admitted. It seems like it would be simplest to have that validation in the kubelet because of the piece of stack that does that validation.

Though, at that point, the validation would happen at about the same time as it would if it was tied into PullImage anyway, which could mean it could be an internal CRI configuration as signature validation is today. Since CRI knows best the image pulling (for instance, how would this validation tie into CRI image mirroring), it seems to me the best spot is where it currently is. An enhancement that could be useful is having the CRI tell the Kubelet when an image pull error is fatal and not transient, so it doesn't retry forever unnecessarily, and instead reports up through the API the image failed because signature validation failed.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/CAPre7xDda%2BiTDHEbg-SNZ6HeNT9xy8XuBVNeLt-exgstVmBKxw%40mail.gmail.com.

Clayton

unread,
Apr 27, 2023, 11:56:52 AM4/27/23
to Sascha Grunert, kubernete...@googlegroups.com, kubernetes-sig-architecture
Historically the runtimes validated signatures - is there data / policy you want to carry along with the pod that CRI doesn’t get?

OpenShift did validate signatures at admission time at one point and did check deeper pod data.

One feature that came up from time to time was validating the entire pod spec along with the image signature - having the node validate the spec meant that you could have a separate entity deciding which pods could run on a node independent of the control plane, which allowed mitigation of control plane compromises to a degree.

> On Apr 27, 2023, at 8:39 AM, Sascha Grunert <sascha...@gmail.com> wrote:
>
> Hey folks,
> --
> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAPre7xDda%2BiTDHEbg-SNZ6HeNT9xy8XuBVNeLt-exgstVmBKxw%40mail.gmail.com.

Benjamin Elder

unread,
Apr 27, 2023, 12:46:28 PM4/27/23
to Brendan Burns, Clayton, Sascha Grunert, kubernete...@googlegroups.com, kubernetes-sig-architecture

On Thu, Apr 27, 2023 at 9:28 AM 'Brendan Burns' via kubernetes-sig-architecture <kubernetes-si...@googlegroups.com> wrote:
It's worth checking out the Ratify project:


Which can do signature validation as an admission controller.


From: kubernetes-si...@googlegroups.com <kubernetes-si...@googlegroups.com> on behalf of Clayton <smarter...@gmail.com>
Sent: Thursday, April 27, 2023 8:56 AM
To: Sascha Grunert <sascha...@gmail.com>
Cc: kubernete...@googlegroups.com <kubernete...@googlegroups.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Subject: [EXTERNAL] Re: Sigstore container image signature validation support
 
Historically the runtimes validated signatures - is there data / policy you want to carry along with the pod that CRI doesn’t get?

OpenShift did validate signatures at admission time at one point and did check deeper pod data.

One feature that came up from time to time was validating the entire pod spec along with the image signature - having the node validate the spec meant that you could have a separate entity deciding which pods could run on a node independent of the control plane, which allowed mitigation of control plane compromises to a degree.

> On Apr 27, 2023, at 8:39 AM, Sascha Grunert <sascha...@gmail.com> wrote:
>
> Hey folks,
>
> I'm having an enhancement in mind to be able to validate sigstore
> signatures of container images before pulling it within the image
> manager of the kubelet [0]. The idea is to add a CRI method for the
> validation, which passes the policy down to the container runtime and
> can handle errors correctly. Policies define which signatures are
> allowed for a subset of images and will also provide the main user
> interface.
>
> I'm wondering where the policies could fit into Kubernetes. Using
> admission controllers, or just the kubelet configuration?
>
> I'd appreciate any thoughts on the feature.
>
> Thank you and all the best,
> Sascha
>

>
> --
> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

Sascha Grunert

unread,
May 2, 2023, 4:14:12 AM5/2/23
to kubernete...@googlegroups.com, kubernetes-sig-architecture, Peter Hunt
Thank you for all the input!

Yes, my idea was to unify the behavior for sigstore signatures through
the CRI by having a single configuration pattern in Kubernetes and not
having a need for configuring each runtime.

> Historically the runtimes validated signatures - is there data / policy you want to carry along with the pod that CRI doesn’t get?

For example in sigstore, I'd like to configure Kuberentes to allow a
specific certificate identity, OIDC issuer, public key/signature or
the rekor (transparency log) instance for a subset of container
images. I can imagine that we pass that data through the CRI and let
the kubelet image pull behave accordingly as @Peter Hunt mentioned.
Users could define the policies as separate Kubernetes resources or
they could be integrated into an existing API.

How could that fit into Kubernetes?

All the best,
Sascha

Peter Hunt

unread,
May 2, 2023, 9:33:05 AM5/2/23
to Sascha Grunert, kubernete...@googlegroups.com, kubernetes-sig-architecture
A way this could be achieved could mirror how the SPO does this, where a CRD is defined for profiles that are applied directly to the nodes. Technically, Kubelet doesn't become aware of these until the CRI processes them, and it could look the same. Then, this entity that holds the CRD could create the correct configurations based on which CRI implementation is on the node. This would mean minimal changes in the Kubelet and runtimes but still have a kubernetes object that can be referenced.

Sascha Grunert

unread,
May 3, 2023, 11:00:52 AM5/3/23
to kubernete...@googlegroups.com, kubernetes-sig-architecture
Adding a new CRI RPC would also require a new Kubernetes API to be
able to work together out of the box.

We could merge this into an admission plugin like the image policy
webhook [0], but I'm not sure this is the way to go for Kubernetes.

[0]: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#imagepolicywebhook
Reply all
Reply to author
Forward
0 new messages