Sigstore container image signature validation support

484 views
Skip to first unread message

Sascha Grunert

unread,
Apr 27, 2023, 8:39:24 AM4/27/23
to kubernete...@googlegroups.com, kubernetes-sig-architecture
Hey folks,

I'm having an enhancement in mind to be able to validate sigstore
signatures of container images before pulling it within the image
manager of the kubelet [0]. The idea is to add a CRI method for the
validation, which passes the policy down to the container runtime and
can handle errors correctly. Policies define which signatures are
allowed for a subset of images and will also provide the main user
interface.

I'm wondering where the policies could fit into Kubernetes. Using
admission controllers, or just the kubelet configuration?

I'd appreciate any thoughts on the feature.

Thank you and all the best,
Sascha

[0]: https://github.com/kubernetes/kubernetes/blob/041bb9a/pkg/kubelet/images/image_manager.go#L155

Clayton

unread,
Apr 27, 2023, 11:56:52 AM4/27/23
to Sascha Grunert, kubernete...@googlegroups.com, kubernetes-sig-architecture
Historically the runtimes validated signatures - is there data / policy you want to carry along with the pod that CRI doesn’t get?

OpenShift did validate signatures at admission time at one point and did check deeper pod data.

One feature that came up from time to time was validating the entire pod spec along with the image signature - having the node validate the spec meant that you could have a separate entity deciding which pods could run on a node independent of the control plane, which allowed mitigation of control plane compromises to a degree.

> On Apr 27, 2023, at 8:39 AM, Sascha Grunert <sascha...@gmail.com> wrote:
>
> Hey folks,
> --
> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAPre7xDda%2BiTDHEbg-SNZ6HeNT9xy8XuBVNeLt-exgstVmBKxw%40mail.gmail.com.

Brendan Burns

unread,
Apr 27, 2023, 12:28:14 PM4/27/23
to Clayton, Sascha Grunert, kubernete...@googlegroups.com, kubernetes-sig-architecture
It's worth checking out the Ratify project:


Which can do signature validation as an admission controller.


From: kubernetes-si...@googlegroups.com <kubernetes-si...@googlegroups.com> on behalf of Clayton <smarter...@gmail.com>
Sent: Thursday, April 27, 2023 8:56 AM
To: Sascha Grunert <sascha...@gmail.com>
Cc: kubernete...@googlegroups.com <kubernete...@googlegroups.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Subject: [EXTERNAL] Re: Sigstore container image signature validation support
 
Historically the runtimes validated signatures - is there data / policy you want to carry along with the pod that CRI doesn’t get?

OpenShift did validate signatures at admission time at one point and did check deeper pod data.

One feature that came up from time to time was validating the entire pod spec along with the image signature - having the node validate the spec meant that you could have a separate entity deciding which pods could run on a node independent of the control plane, which allowed mitigation of control plane compromises to a degree.

> On Apr 27, 2023, at 8:39 AM, Sascha Grunert <sascha...@gmail.com> wrote:
>
> Hey folks,
>
> I'm having an enhancement in mind to be able to validate sigstore
> signatures of container images before pulling it within the image
> manager of the kubelet [0]. The idea is to add a CRI method for the
> validation, which passes the policy down to the container runtime and
> can handle errors correctly. Policies define which signatures are
> allowed for a subset of images and will also provide the main user
> interface.
>
> I'm wondering where the policies could fit into Kubernetes. Using
> admission controllers, or just the kubelet configuration?
>
> I'd appreciate any thoughts on the feature.
>
> Thank you and all the best,
> Sascha
>

>
> --
> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

Benjamin Elder

unread,
Apr 27, 2023, 12:46:28 PM4/27/23
to Brendan Burns, Clayton, Sascha Grunert, kubernete...@googlegroups.com, kubernetes-sig-architecture

Sascha Grunert

unread,
May 2, 2023, 4:14:11 AM5/2/23
to kubernete...@googlegroups.com, kubernetes-sig-architecture, Peter Hunt
Thank you for all the input!

Yes, my idea was to unify the behavior for sigstore signatures through
the CRI by having a single configuration pattern in Kubernetes and not
having a need for configuring each runtime.

> Historically the runtimes validated signatures - is there data / policy you want to carry along with the pod that CRI doesn’t get?

For example in sigstore, I'd like to configure Kuberentes to allow a
specific certificate identity, OIDC issuer, public key/signature or
the rekor (transparency log) instance for a subset of container
images. I can imagine that we pass that data through the CRI and let
the kubelet image pull behave accordingly as @Peter Hunt mentioned.
Users could define the policies as separate Kubernetes resources or
they could be integrated into an existing API.

How could that fit into Kubernetes?

All the best,
Sascha

Sascha Grunert

unread,
May 3, 2023, 11:00:51 AM5/3/23
to kubernete...@googlegroups.com, kubernetes-sig-architecture
Adding a new CRI RPC would also require a new Kubernetes API to be
able to work together out of the box.

We could merge this into an admission plugin like the image policy
webhook [0], but I'm not sure this is the way to go for Kubernetes.

[0]: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#imagepolicywebhook

On Tue, May 2, 2023 at 3:33 PM Peter Hunt <peh...@redhat.com> wrote:
>
> A way this could be achieved could mirror how the SPO does this, where a CRD is defined for profiles that are applied directly to the nodes. Technically, Kubelet doesn't become aware of these until the CRI processes them, and it could look the same. Then, this entity that holds the CRD could create the correct configurations based on which CRI implementation is on the node. This would mean minimal changes in the Kubelet and runtimes but still have a kubernetes object that can be referenced.

Brendan Burns

unread,
May 3, 2023, 4:32:05 PM5/3/23
to Sascha Grunert, kubernete...@googlegroups.com, kubernetes-sig-architecture
As far as I know, the CRI API already preserves and passes all Pod annotations down to the CRI runtime as part of the existing API.

The Kubernetes pattern for this sort of thing has generally been to define an annotation key (e.g. "sigstore.io/validation") and value that your CRI runtime knows how to interpret.

--brendan


From: kubernetes-si...@googlegroups.com <kubernetes-si...@googlegroups.com> on behalf of Sascha Grunert <sascha...@gmail.com>
Sent: Wednesday, May 3, 2023 8:00 AM
To: kubernete...@googlegroups.com <kubernete...@googlegroups.com>; kubernetes-sig-architecture <kubernetes-si...@googlegroups.com>
Subject: Re: [EXTERNAL] Re: Sigstore container image signature validation support
 
Adding a new CRI RPC would also require a new Kubernetes API to be
able to work together out of the box.

We could merge this into an admission plugin like the image policy
webhook [0], but I'm not sure this is the way to go for Kubernetes.



On Tue, May 2, 2023 at 3:33 PM Peter Hunt <peh...@redhat.com> wrote:
>
> A way this could be achieved could mirror how the SPO does this, where a CRD is defined for profiles that are applied directly to the nodes. Technically, Kubelet doesn't become aware of these until the CRI processes them, and it could look the same. Then, this entity that holds the CRD could create the correct configurations based on which CRI implementation is on the node. This would mean minimal changes in the Kubelet and runtimes but still have a kubernetes object that can be referenced.
>
> On Tue, May 2, 2023 at 4:14 AM Sascha Grunert <sascha...@gmail.com> wrote:
>>
>> Thank you for all the input!
>>
>> Yes, my idea was to unify the behavior for sigstore signatures through
>> the CRI by having a single configuration pattern in Kubernetes and not
>> having a need for configuring each runtime.
>>
>> > Historically the runtimes validated signatures - is there data / policy you want to carry along with the pod that CRI doesn’t get?
>>
>> For example in sigstore, I'd like to configure Kuberentes to allow a
>> specific certificate identity, OIDC issuer, public key/signature or
>> the rekor (transparency log) instance for a subset of container
>> images. I can imagine that we pass that data through the CRI and let
>> the kubelet image pull behave accordingly as @Peter Hunt mentioned.
>> Users could define the policies as separate Kubernetes resources or
>> they could be integrated into an existing API.
>>
>> How could that fit into Kubernetes?
>>
>> All the best,
>> Sascha
>>
>> On Thu, Apr 27, 2023 at 6:46 PM Benjamin Elder <benth...@google.com> wrote:
>> >
>> > See also: More specific pointers re "runtimes do/can do this"
>> >
>> > containerd: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcontainerd%2Fcontainerd%2Fissues%2F6691&data=05%7C01%7Cbburns%40microsoft.com%7Cce5d0aef905c40d8062c08db4be72ff9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638187229827476644%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C7000%7C%7C%7C&sdata=l%2F5e%2BccXQpQ6TD8cXNNTHy17lKWnSCC510sMZMlZ0Vc%3D&reserved=0
>> > cri-o: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F10%2F29%2Fverifying-signatures-of-red-hat-container-images&data=05%7C01%7Cbburns%40microsoft.com%7Cce5d0aef905c40d8062c08db4be72ff9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638187229827476644%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C7000%7C%7C%7C&sdata=EigPP2uiwMlooT%2BrKcJ4vzs2yEM5PgVP%2F1JEtXCJsmg%3D&reserved=0
>> >
>> > And webhooks:
>> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsigstore%2Fpolicy-controller&data=05%7C01%7Cbburns%40microsoft.com%7Cce5d0aef905c40d8062c08db4be72ff9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638187229827476644%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C7000%7C%7C%7C&sdata=UIqo0kkPXo3h%2BEhxtMA5R6gBNAz%2BSIaFp0WSBizeOZE%3D&reserved=0

>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
>> >> > To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.

>>

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages