Adding webhooks to CRD validation...

54 views
Skip to first unread message

Brendan Burns

unread,
Nov 14, 2017, 11:32:43 PM11/14/17
to K8s API Machinery SIG
I'm looking for a new project, and adding webhooks to custom resource definition validation seems to be a decent place to start.

Before I started in on the design, I thought I'd double check and make sure no one else is working on it.

Thanks
--brendan

Clayton Coleman

unread,
Nov 14, 2017, 11:40:09 PM11/14/17
to Brendan Burns, K8s API Machinery SIG
Isn't this just covered by admission control extensions on the api server side?  You'd just register an admission webhook for your CRD type and then reject if it's not valid

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/c1665711-8c5d-4057-b5c4-eade1178175b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Daniel Smith

unread,
Nov 14, 2017, 11:56:33 PM11/14/17
to Clayton Coleman, Brendan Burns, K8s API Machinery SIG
Yeah, agreed, dynamic admission webhooks will work for CRDs and should be beta in 1.9.

On Tue, Nov 14, 2017 at 8:40 PM, Clayton Coleman <ccol...@redhat.com> wrote:
Isn't this just covered by admission control extensions on the api server side?  You'd just register an admission webhook for your CRD type and then reject if it's not valid
On Tue, Nov 14, 2017 at 11:32 PM, Brendan Burns <brendan...@gmail.com> wrote:
I'm looking for a new project, and adding webhooks to custom resource definition validation seems to be a decent place to start.

Before I started in on the design, I thought I'd double check and make sure no one else is working on it.

Thanks
--brendan

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAH16Sh%2BCocdMc9%3D4xZyxL_kVbBxzD0CgfX1FMAT34_o%3D%2BL72tA%40mail.gmail.com.

Brendan Burns

unread,
Nov 15, 2017, 12:25:50 AM11/15/17
to K8s API Machinery SIG
While this works, I don't think that we should force people into this mechanism, it's decoupled from the main CRD flow, has ownership/RBAC problems and it introduces cognitive hurdles for the end-user.

If I'm a CRD user, I want to have all of my validation stuff in the CRD api object, not have to learn about custom admission web-hooks.

Additionally, for most cluster owners, I think they will want different RBAC for admission web-hooks vs CRD validation.

There's nothing (as far as I can tell) in the RBAC you could do on an admission controller that would prevent a CRD from registering a "sniff and report" admission controller for all of my Pods.

Where-as if you restrict validation to the CRD type that is added by the CRD-based plugin, you're much more comfortable that the CRD is only seeing the stuff relevant to the CRD.

As an example use case:

I install the etcd-operator, if I'm forced to use admission control, I have to give the etcd-operator RBAC to admission controllers (not good) or I can not package my operator as a single Deployment (also not good)

Hope that helps clarify why CRD validation web-hooks are needed.

--brendan


On Tuesday, November 14, 2017 at 8:56:33 PM UTC-8, lavalamp wrote:
Yeah, agreed, dynamic admission webhooks will work for CRDs and should be beta in 1.9.
On Tue, Nov 14, 2017 at 8:40 PM, Clayton Coleman <ccol...@redhat.com> wrote:
Isn't this just covered by admission control extensions on the api server side?  You'd just register an admission webhook for your CRD type and then reject if it's not valid
On Tue, Nov 14, 2017 at 11:32 PM, Brendan Burns <brendan...@gmail.com> wrote:
I'm looking for a new project, and adding webhooks to custom resource definition validation seems to be a decent place to start.

Before I started in on the design, I thought I'd double check and make sure no one else is working on it.

Thanks
--brendan

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

Daniel Smith

unread,
Nov 15, 2017, 12:42:48 AM11/15/17
to Brendan Burns, K8s API Machinery SIG
Have you looked at the design? You can easily constrain an admission webhook to run only on particular types. In 1.9 you will be able to modify the object with them if desired. I am really not sure what a CRD validation webhook would do differently.

It is also (at least intended to be) trivial to implement the webhook.

RBAC rules also aren't a thing webhook authors need to worry about generally. As long as you have permission to POST the admission registration object, you're good.

At most, I could see setting up a shim for CRDs to auto-file a "known-safe" admission registration object for people, but before thinking about that I'd want experience with the planned webhooks and evidence that they're too hard or something.


On Tue, Nov 14, 2017 at 9:25 PM, Brendan Burns <brendan...@gmail.com> wrote:
While this works, I don't think that we should force people into this mechanism, it's decoupled from the main CRD flow, has ownership/RBAC problems and it introduces cognitive hurdles for the end-user.

If I'm a CRD user, I want to have all of my validation stuff in the CRD api object, not have to learn about custom admission web-hooks.

Additionally, for most cluster owners, I think they will want different RBAC for admission web-hooks vs CRD validation.

There's nothing (as far as I can tell) in the RBAC you could do on an admission controller that would prevent a CRD from registering a "sniff and report" admission controller for all of my Pods.

Where-as if you restrict validation to the CRD type that is added by the CRD-based plugin, you're much more comfortable that the CRD is only seeing the stuff relevant to the CRD.

As an example use case:

I install the etcd-operator, if I'm forced to use admission control, I have to give the etcd-operator RBAC to admission controllers (not good) or I can not package my operator as a single Deployment (also not good)

Hope that helps clarify why CRD validation web-hooks are needed.

--brendan


On Tuesday, November 14, 2017 at 8:56:33 PM UTC-8, lavalamp wrote:
Yeah, agreed, dynamic admission webhooks will work for CRDs and should be beta in 1.9.
On Tue, Nov 14, 2017 at 8:40 PM, Clayton Coleman <ccol...@redhat.com> wrote:
Isn't this just covered by admission control extensions on the api server side?  You'd just register an admission webhook for your CRD type and then reject if it's not valid
On Tue, Nov 14, 2017 at 11:32 PM, Brendan Burns <brendan...@gmail.com> wrote:
I'm looking for a new project, and adding webhooks to custom resource definition validation seems to be a decent place to start.

Before I started in on the design, I thought I'd double check and make sure no one else is working on it.

Thanks
--brendan

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/3c0a15e4-d52b-42b2-aebc-d45c53baca61%40googlegroups.com.

Daniel Smith

unread,
Nov 15, 2017, 12:44:44 AM11/15/17
to Brendan Burns, K8s API Machinery SIG

To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.

Brian Grant

unread,
Nov 15, 2017, 1:06:13 AM11/15/17
to Daniel Smith, Brendan Burns, K8s API Machinery SIG
We don't have design/review bandwidth to start this right now. 



To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.

To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.

Brendan Burns

unread,
Nov 15, 2017, 12:40:44 PM11/15/17
to K8s API Machinery SIG
Daniel:

To be clear on the RBAC issue:
 
It's not that I can't restrict an admission controller to a particular type, clearly I can.

However, in the operator pattern, the operator is third-party Pod/code running in my cluster.  The operator pod itself is the thing that registers the CRD (and would also be the entity that needs to register the admission controller webhook). That means that the operator Pod needs to have RBAC permission to create admission controllers, and I have no ability to enforce that the admission controllers that the operator Pod only creates admission controller objects that are restricted to that CRD. Thus a malicious operator could register admission controllers for arbitrary types.

Regarding design bandwidth, the design has already been done, the work is in the discussion about Schema validation, and in the design of admission control web hooks. (there's no reason we can't abstract/re-use the hook API object)

it's unclear to me what more needs to be designed.

--brendan

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

Daniel Smith

unread,
Nov 15, 2017, 12:52:47 PM11/15/17
to Brendan Burns, K8s API Machinery SIG
On Wed, Nov 15, 2017 at 9:40 AM, Brendan Burns <brendan...@gmail.com> wrote:
Daniel:

To be clear on the RBAC issue:
 
It's not that I can't restrict an admission controller to a particular type, clearly I can.

However, in the operator pattern, the operator is third-party Pod/code running in my cluster.  The operator pod itself is the thing that registers the CRD (and would also be the entity that needs to register the admission controller webhook). That means that the operator Pod needs to have RBAC permission to create admission controllers, and I have no ability to enforce that the admission controllers that the operator Pod only creates admission controller objects that are restricted to that CRD. Thus a malicious operator could register admission controllers for arbitrary types.

Yes, for that scenario, to quote myself, "I could see setting up a shim for CRDs to auto-file a "known-safe" admission registration object for people". I don't think we want a separate webhook mechanism just for this; adjusting the configuration mechanism is sufficient. Also I want experience with this before we add an additional config mechanism. Let's see how the beta goes and we can revisit in Q1?
 
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.

To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.

Brendan Burns

unread,
Nov 15, 2017, 1:00:32 PM11/15/17
to K8s API Machinery SIG
I'm not sure if I quite follow the "shim" idea, is that a field in the CRD object which would auto-create a new AdmissionController object via a controller? If that's the design that seems fine to me.

I'm happy to update the existing validation doc with that design and have it reviewed whenever api-machinery has bandwidth.

--brendan
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

Daniel Smith

unread,
Nov 15, 2017, 1:01:56 PM11/15/17
to Brendan Burns, K8s API Machinery SIG
On Wed, Nov 15, 2017 at 10:00 AM, Brendan Burns <brendan...@gmail.com> wrote:
I'm not sure if I quite follow the "shim" idea, is that a field in the CRD object which would auto-create a new AdmissionController object via a controller? If that's the design that seems fine to me.

Yeah, something along those lines.
 
I'm happy to update the existing validation doc with that design and have it reviewed whenever api-machinery has bandwidth.

SGTM, thanks.
 

--brendan
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.

To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.

To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.

Brendan Burns

unread,
Dec 13, 2017, 4:02:31 PM12/13/17
to K8s API Machinery SIG

--brendan
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages