Doc on extending kubernetes with TPR and Aggregated API Servers.

193 views
Skip to first unread message

Eric Tune

unread,
Feb 6, 2017, 4:45:45 PM2/6/17
to kubernete...@googlegroups.com, kubernetes-sig...@googlegroups.com
We have two options planned for adding new REST endpoints to the Kubernetes API.
Some people I have talked to have said they are confused about the need for two ways and which to use.

I wrote this doc that tries to answer those questions.

I plan to get it into a kubernetes.io doc once the work on Aggregated API Servers is released.  I thought I would share it now.  

Anyone with this link has comment permissions.
Feel free to comment!  Thanks.


Brandon Philips

unread,
Feb 6, 2017, 4:55:39 PM2/6/17
to Eric Tune, kubernete...@googlegroups.com, kubernetes-sig...@googlegroups.com
Thanks for putting this together Eric. 

Is there a request flow diagram for the kube-aggregator? I am really confused at how this changes a users deployment. https://github.com/kubernetes/community/blob/master/contributors/design-proposals/aggregated-api-servers.md



--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To post to this group, send email to kubernete...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/CAASt_VE8gfqdZmW_61Qw54vNZeUUFafgiPUy7h5331mi6-gPHg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Daniel Smith

unread,
Feb 6, 2017, 5:01:38 PM2/6/17
to Brandon Philips, Eric Tune, kubernete...@googlegroups.com, K8s API Machinery SIG
We are having a debate right now about whether it should be a separate binary or we should just build it into the core apiserver. The latter is operationally much simpler and is probably going to win the debate, if we can convince deads2k that we won't fill up his code with ugly special cases. :)

On Mon, Feb 6, 2017 at 1:55 PM, Brandon Philips <brandon...@coreos.com> wrote:
Thanks for putting this together Eric. 

Is there a request flow diagram for the kube-aggregator? I am really confused at how this changes a users deployment. https://github.com/kubernetes/community/blob/master/contributors/design-proposals/aggregated-api-servers.md



On Mon, Feb 6, 2017 at 1:45 PM 'Eric Tune' via kubernetes-sig-apps <kubernetes-sig-apps@googlegroups.com> wrote:
We have two options planned for adding new REST endpoints to the Kubernetes API.
Some people I have talked to have said they are confused about the need for two ways and which to use.

I wrote this doc that tries to answer those questions.

I plan to get it into a kubernetes.io doc once the work on Aggregated API Servers is released.  I thought I would share it now.  

Anyone with this link has comment permissions.
Feel free to comment!  Thanks.


--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-apps@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/CAASt_VE8gfqdZmW_61Qw54vNZeUUFafgiPUy7h5331mi6-gPHg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAD2oYtNJDRRdsDxHMYa7BX0XHorfHjoStyRBp_21sf%2Bj2s-SWQ%40mail.gmail.com.

Brandon Philips

unread,
Feb 6, 2017, 5:04:38 PM2/6/17
to Daniel Smith, Eric Tune, kubernete...@googlegroups.com, K8s API Machinery SIG
Where is this debate happening? I really think we should stay laser focused on simplifying our deployments as much as possible. We are spawning deployment complexity faster than we are providing solutions to users.

On Mon, Feb 6, 2017 at 2:01 PM Daniel Smith <dbs...@google.com> wrote:
We are having a debate right now about whether it should be a separate binary or we should just build it into the core apiserver. The latter is operationally much simpler and is probably going to win the debate, if we can convince deads2k that we won't fill up his code with ugly special cases. :)

On Mon, Feb 6, 2017 at 1:55 PM, Brandon Philips <brandon...@coreos.com> wrote:
Thanks for putting this together Eric. 

Is there a request flow diagram for the kube-aggregator? I am really confused at how this changes a users deployment. https://github.com/kubernetes/community/blob/master/contributors/design-proposals/aggregated-api-servers.md



On Mon, Feb 6, 2017 at 1:45 PM 'Eric Tune' via kubernetes-sig-apps <kubernete...@googlegroups.com> wrote:
We have two options planned for adding new REST endpoints to the Kubernetes API.
Some people I have talked to have said they are confused about the need for two ways and which to use.

I wrote this doc that tries to answer those questions.

I plan to get it into a kubernetes.io doc once the work on Aggregated API Servers is released.  I thought I would share it now.  

Anyone with this link has comment permissions.
Feel free to comment!  Thanks.


--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To post to this group, send email to kubernete...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/CAASt_VE8gfqdZmW_61Qw54vNZeUUFafgiPUy7h5331mi6-gPHg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAD2oYtNJDRRdsDxHMYa7BX0XHorfHjoStyRBp_21sf%2Bj2s-SWQ%40mail.gmail.com.

xian...@coreos.com

unread,
Feb 6, 2017, 6:52:27 PM2/6/17
to K8s API Machinery SIG, kubernete...@googlegroups.com, et...@google.com
I read the doc, and have a few concerns.

We are developing etcd operator at CoreOS, which heavily relies on TPR. We expose TPR to end users, which means we do not control the clients. 
We were hoping that TRP can support hooks to help things like admission, validation, and defaults. 

There is an issue created for TPR here: https://github.com/kubernetes/features/issues/95. It covers most of the things we care about TPR. And we were hoping them to be included eventually.

From the doc shared in the thread, it states (in the comparison section) that TPR will not support: validation, customized admission, printer, etc. It also suggests that "advanced user" should use UAS throughout the doc. Basically, this would "force" us to drop TPR completely, and switch from TPR to UAS as soon as possible. Without validation, admission, things like etcd operator cannot be stable. Users can break spec, and there is no way to recover.

I am wondering what stop us from executing the original planed features for TPR. I would expect us to make TPR stable and feature complete. Then we can start to work on a more flexible alternative.

I feel we are dropping features of TPR because of UAS. This is a surprise to me at least.

Clayton Coleman

unread,
Feb 6, 2017, 7:05:09 PM2/6/17
to xian...@coreos.com, K8s API Machinery SIG, kubernete...@googlegroups.com, et...@google.com
Hrm, I read that as "doesn't support now".  I do think defaulting and validation would be useful for tpr, although I don't know that anything other than "openapi spec" comes close to being simple or easy.  We can't even represent label or annotation validation without code, so part of that is a reflection that someone has to step up to help push validation through and commit significant resources.  Most of the cleanup and refactoring happening in API machinery is a prereq for that, but there's still lots of work.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

brendan...@gmail.com

unread,
Feb 6, 2017, 7:06:17 PM2/6/17
to K8s API Machinery SIG, kubernete...@googlegroups.com, et...@google.com, xian...@coreos.com
I have some concerns with the direction I see in this document. There are lots of things in there that are listed as "no" for TPR when in fact they are possible, but we haven't been willing invest the engineering to improve them. It seems like it has a forgone conclusion that API extensibility is required for "real" APIs, and it dramatically plays down the increased complexity both for the operator and the Kubernetes code base as a whole.

I think there is clear utility in both third party resources and federated API servers. But we need to make them both first-class options. This notion that the only way to get a "real" API is to have a delegated API server is not true, in my opinion.

And the fact that federated API servers are talking about doubling back to use third party resources as a storage option is simply proving the point that there is real value in third-party resources.

Rather than forcing people into API delegation, and then doubling back into TPR if they don't want to manage storage, I think that we should (and actually are already) putting extensibility in the right places so that we can use webhooks for admission control, validation, etc. These hooks should
be seen as independent of TPR, UAS or even "Core" APIs, they are generic API extensibility, and as such they will light up customization and extensibility regardless of how the API is created or managed.

anyway, that's my $0.02.

--brendan

brendan...@gmail.com

unread,
Feb 6, 2017, 7:55:33 PM2/6/17
to K8s API Machinery SIG, kubernete...@googlegroups.com, et...@google.com, xian...@coreos.com, brendan...@gmail.com
I like Clayton's conclusion from the doc better than mine :)

I hope that is true. I would very much like to see a core commitment from sig-apimachinery (and Kubernetes writ large) that both third-party-resources and user-apiservers are strongly supported for "real" "production" APIs.

I agree that finding the finding people to do the work is always the challenge (I'm as guilty as anyone of slacking here) but I think we can find the resources between myself, CoreOS and the work that RedHat folks have been doing to add the necessary support. 

I think one of the things always holding things back was the sense that such changes weren't valued or taken seriously by other folks in sig-apimachinery. If we are will to say as a SIG that both paths are valid paths forward, and that as a SIG we are committed to TPRs as a peer to user apiservers, I think that would go a long way to helping unlock the effort necessary. People don't want to do work if they're not sure it will be valued or accepted.

--brendan

Brandon Philips

unread,
Feb 6, 2017, 8:58:24 PM2/6/17
to brendan...@gmail.com, K8s API Machinery SIG, kubernete...@googlegroups.com, et...@google.com, xian...@coreos.com
On Mon, Feb 6, 2017 at 4:55 PM <brendan...@gmail.com> wrote:
I think one of the things always holding things back was the sense that such changes weren't valued or taken seriously by other folks in sig-apimachinery. If we are will to say as a SIG that both paths are valid paths forward, and that as a SIG we are committed to TPRs as a peer to user apiservers, I think that would go a long way to helping unlock the effort necessary. People don't want to do work if they're not sure it will be valued or accepted.

+1 I think we should just declare this in the doc.

If Brendan and I drew these conclusions but it wasn't the intention then something is missing. I guess the things to declare are the ultimate architecture we are shooting for, the steps we imagine it will take to get there, and links to the features we want to complete to do it.

Thanks again, this is a critical discussion.

Brandon
 
On Monday, February 6, 2017 at 4:06:17 PM UTC-8, brendan...@gmail.com wrote:
I have some concerns with the direction I see in this document. There are lots of things in there that are listed as "no" for TPR when in fact they are possible, but we haven't been willing invest the engineering to improve them. It seems like it has a forgone conclusion that API extensibility is required for "real" APIs, and it dramatically plays down the increased complexity both for the operator and the Kubernetes code base as a whole.

I think there is clear utility in both third party resources and federated API servers. But we need to make them both first-class options. This notion that the only way to get a "real" API is to have a delegated API server is not true, in my opinion.

And the fact that federated API servers are talking about doubling back to use third party resources as a storage option is simply proving the point that there is real value in third-party resources.

Rather than forcing people into API delegation, and then doubling back into TPR if they don't want to manage storage, I think that we should (and actually are already) putting extensibility in the right places so that we can use webhooks for admission control, validation, etc. These hooks should
be seen as independent of TPR, UAS or even "Core" APIs, they are generic API extensibility, and as such they will light up customization and extensibility regardless of how the API is created or managed.

anyway, that's my $0.02.

--brendan

On Monday, February 6, 2017 at 3:52:27 PM UTC-8, xian...@coreos.com wrote:
I read the doc, and have a few concerns.

We are developing etcd operator at CoreOS, which heavily relies on TPR. We expose TPR to end users, which means we do not control the clients. 
We were hoping that TRP can support hooks to help things like admission, validation, and defaults. 

There is an issue created for TPR here: https://github.com/kubernetes/features/issues/95. It covers most of the things we care about TPR. And we were hoping them to be included eventually.

From the doc shared in the thread, it states (in the comparison section) that TPR will not support: validation, customized admission, printer, etc. It also suggests that "advanced user" should use UAS throughout the doc. Basically, this would "force" us to drop TPR completely, and switch from TPR to UAS as soon as possible. Without validation, admission, things like etcd operator cannot be stable. Users can break spec, and there is no way to recover.

I am wondering what stop us from executing the original planed features for TPR. I would expect us to make TPR stable and feature complete. Then we can start to work on a more flexible alternative.

I feel we are dropping features of TPR because of UAS. This is a surprise to me at least.



On Monday, February 6, 2017 at 1:45:45 PM UTC-8, Eric Tune wrote:
We have two options planned for adding new REST endpoints to the Kubernetes API.
Some people I have talked to have said they are confused about the need for two ways and which to use.

I wrote this doc that tries to answer those questions.

I plan to get it into a kubernetes.io doc once the work on Aggregated API Servers is released.  I thought I would share it now.  

Anyone with this link has comment permissions.
Feel free to comment!  Thanks.


--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To post to this group, send email to kubernete...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/0a1f318d-2180-4d8c-8ff8-5462349919f7%40googlegroups.com.

derekwa...@gmail.com

unread,
Feb 6, 2017, 9:50:54 PM2/6/17
to K8s API Machinery SIG, brendan...@gmail.com, kubernete...@googlegroups.com, et...@google.com, xian...@coreos.com
The initial TPR proposal [1] explicitly deferred many of the items (spec vs status, support for subresources, validation, admission hooks, etc.) that it now feels like the part of the community expects to have included by default.  It basically asserted that TPR are opaque data blobs (much like ConfigMap), but were intended to support custom controllers that handled the deferred items client-side.  I think its a mistake to try and retro-actively fix TPR or grow its scope to align with the community expectation being stated in this thread.  If we want to support more than opaque data blobs, we should design a TPRv2 that accounts for the expanded usage patterns.

- Derek

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

brendan...@gmail.com

unread,
Feb 6, 2017, 10:39:49 PM2/6/17
to K8s API Machinery SIG, brendan...@gmail.com, kubernete...@googlegroups.com, et...@google.com, xian...@coreos.com, derekwa...@gmail.com
Derek,
I don't think that's an accurate characterization of the PR discussion. 

Spec/Status:
I had a proposal for having Kubernetes-native (spec/status) vs. Blob objects:

Schema and validation:
Schema and validation were dropped due to discussions around the PR:

Admission control and defaulting:
Extensible admissions control wasn't a thing, so customizable initialization/defaulting/admission control weren't possible and it didn't seem worth blocking the feature on them.

Subresources:
Subresources are definitely something that require a full user added API server (which I've repeatedly said is a useful and valid alternative) so explicitly out of scope.

So I think that most of these things have always been part of the roadmap and the goal for third-party resources, and one is still not on the roadmap.

I'd reiterate that my goal is that as a SIG, apimachinery says:

   * We are fully committed to supporting both third party and user apiservers as first-class ways to add complete APIs to Kubernetes.
   * We are committed to ensuring that they are peers in terms of functionality, except in the few places (e.g. subresoruces) where we explicitly believe them to be out of scope.

--brendan

brendan...@gmail.com

unread,
Feb 6, 2017, 10:43:21 PM2/6/17
to K8s API Machinery SIG, brendan...@gmail.com, kubernete...@googlegroups.com, et...@google.com, xian...@coreos.com, derekwa...@gmail.com
And I don't think that implementing any of these things requires a re-design, which in my mind will take too long and distract from fixing/improving the thing that is already in widespread use.

Many of the items (extensible admission control/initializers) are already in the works due to other effort, and the remainder are not difficult to add.

--brendan

brendan...@gmail.com

unread,
Feb 6, 2017, 11:28:24 PM2/6/17
to K8s API Machinery SIG, brendan...@gmail.com, kubernete...@googlegroups.com, et...@google.com, xian...@coreos.com, derekwa...@gmail.com
I've been thinking about this a bunch:

I think we're all basically in violent agreement about this.

I think that we will drive to a better form of agreement if we recognize that third-party-resources and user-apiservers are actually instances of the same thing.

We've defined the two extreme ends of a spectrum.  They are far enough apart that we believe that they are different things, 
but in reality they are the same, just at different ends of the continuum.

Basically here's my view of the continuum:

Easy End
Existing third-party resources

Medium End
Existing third-party resources plus web hooks for validation, admission control, conversion, etc.
Webhooks are far easier than user apiservers because on all major clouds you can simply host a snippet of javascript as a webhook (e.g. Lambda)

Upper End
Run your own user-apiserver, but delegate back to third-party-resources for storage
Running your own servers not _that_ bad, but storage is tricky, and its helpful if it's all in one place.

Complete End
Full on user-apiserver, run your own server, run your own storage.

I think that it's helpful to think in this way, that actually they are all the same.  Once you realize this, it becomes clear that there should be one API that can span the complete use case from existing TPR, all the way through complete API federation. The challenge that we face is how do we iterate toward that complete solution, and what exactly does the middle ground look like.

As a stake in the ground, I think we should start with the existing third-party object for API registration. It already exists, it already has a controller, and I don't think that there are any blockers to adapt it to all of these different use cases.

I think if we do that, then we stand a chance of maintaining support for all of the current users and use cases, while iterating towards a complete solution that enables users to define extensions in the manner that is best suited to their use case.

--brendan

de...@redhat.com

unread,
Feb 7, 2017, 7:41:47 AM2/7/17
to K8s API Machinery SIG, brendan...@gmail.com, kubernete...@googlegroups.com, et...@google.com, xian...@coreos.com, derekwa...@gmail.com
I do not think we should start from TPR.  TPR is poorly organized for gathering discrete API servers and unifying them. Even something as simple as naming doesn't work well in the scheme, since its more reasonable to aggregate API servers than individual resources.

The aggregator exists and works using a separate, API (much different from TPR), here: https://github.com/kubernetes/kubernetes/tree/master/cmd/kube-aggregator and the implementation fell out much more cleanly than TPR one.

Brian Grant

unread,
Feb 7, 2017, 9:32:32 AM2/7/17
to Xiang Li, K8s API Machinery SIG, kubernetes-sig-apps, Eric Tune
On Mon, Feb 6, 2017 at 3:52 PM, <xian...@coreos.com> wrote:
I read the doc, and have a few concerns.

We are developing etcd operator at CoreOS, which heavily relies on TPR. We expose TPR to end users, which means we do not control the clients. 
We were hoping that TRP can support hooks to help things like admission, validation, and defaults. 

That issue said that there was no owner.
 

There is an issue created for TPR here: https://github.com/kubernetes/features/issues/95. It covers most of the things we care about TPR. And we were hoping them to be included eventually.

From the doc shared in the thread, it states (in the comparison section) that TPR will not support: validation, customized admission, printer, etc. It also suggests that "advanced user" should use UAS throughout the doc. Basically, this would "force" us to drop TPR completely, and switch from TPR to UAS as soon as possible. Without validation, admission, things like etcd operator cannot be stable. Users can break spec, and there is no way to recover.

I am wondering what stop us from executing the original planed features for TPR. I would expect us to make TPR stable and feature complete. Then we can start to work on a more flexible alternative.

I feel we are dropping features of TPR because of UAS. This is a surprise to me at least.



On Monday, February 6, 2017 at 1:45:45 PM UTC-8, Eric Tune wrote:
We have two options planned for adding new REST endpoints to the Kubernetes API.
Some people I have talked to have said they are confused about the need for two ways and which to use.

I wrote this doc that tries to answer those questions.

I plan to get it into a kubernetes.io doc once the work on Aggregated API Servers is released.  I thought I would share it now.  

Anyone with this link has comment permissions.
Feel free to comment!  Thanks.


--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

Brian Grant

unread,
Feb 7, 2017, 9:54:09 AM2/7/17
to Clayton Coleman, xian...@coreos.com, K8s API Machinery SIG, kubernete...@googlegroups.com, et...@google.com
On Mon, Feb 6, 2017 at 4:05 PM, Clayton Coleman <ccol...@redhat.com> wrote:
Hrm, I read that as "doesn't support now".  I do think defaulting and validation would be useful for tpr, although I don't know that anything other than "openapi spec" comes close to being simple or easy.  We can't even represent label or annotation validation without code, so part of that is a reflection that someone has to step up to help push validation through and commit significant resources.  Most of the cleanup and refactoring happening in API machinery is a prereq for that, but there's still lots of work.

Some bugs:


Features that users think are bugs because they work for other resources:

 

On Feb 6, 2017, at 6:52 PM, "xian...@coreos.com" <xian...@coreos.com> wrote:

I read the doc, and have a few concerns.

We are developing etcd operator at CoreOS, which heavily relies on TPR. We expose TPR to end users, which means we do not control the clients. 
We were hoping that TRP can support hooks to help things like admission, validation, and defaults. 

There is an issue created for TPR here: https://github.com/kubernetes/features/issues/95. It covers most of the things we care about TPR. And we were hoping them to be included eventually.

From the doc shared in the thread, it states (in the comparison section) that TPR will not support: validation, customized admission, printer, etc. It also suggests that "advanced user" should use UAS throughout the doc. Basically, this would "force" us to drop TPR completely, and switch from TPR to UAS as soon as possible. Without validation, admission, things like etcd operator cannot be stable. Users can break spec, and there is no way to recover.

I am wondering what stop us from executing the original planed features for TPR. I would expect us to make TPR stable and feature complete. Then we can start to work on a more flexible alternative.

I feel we are dropping features of TPR because of UAS. This is a surprise to me at least.



On Monday, February 6, 2017 at 1:45:45 PM UTC-8, Eric Tune wrote:
We have two options planned for adding new REST endpoints to the Kubernetes API.
Some people I have talked to have said they are confused about the need for two ways and which to use.

I wrote this doc that tries to answer those questions.

I plan to get it into a kubernetes.io doc once the work on Aggregated API Servers is released.  I thought I would share it now.  

Anyone with this link has comment permissions.
Feel free to comment!  Thanks.


--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-apps@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/5813499491576008599%40unknownmsgid.

Brian Grant

unread,
Feb 7, 2017, 10:08:41 AM2/7/17
to Brendan Burns, K8s API Machinery SIG, kubernetes-sig-apps, Eric Tune, Xiang Li, Derek Carr
On Mon, Feb 6, 2017 at 8:28 PM, <brendan...@gmail.com> wrote:
I've been thinking about this a bunch:

I think we're all basically in violent agreement about this.

I think that we will drive to a better form of agreement if we recognize that third-party-resources and user-apiservers are actually instances of the same thing.

We've defined the two extreme ends of a spectrum.  They are far enough apart that we believe that they are different things, 
but in reality they are the same, just at different ends of the continuum.

Basically here's my view of the continuum:

Easy End
Existing third-party resources

Medium End
Existing third-party resources plus web hooks for validation, admission control, conversion, etc.
Webhooks are far easier than user apiservers because on all major clouds you can simply host a snippet of javascript as a webhook (e.g. Lambda)

Upper End
Run your own user-apiserver, but delegate back to third-party-resources for storage
Running your own servers not _that_ bad, but storage is tricky, and its helpful if it's all in one place.

Complete End
Full on user-apiserver, run your own server, run your own storage.

I think that it's helpful to think in this way, that actually they are all the same.  Once you realize this, it becomes clear that there should be one API that can span the complete use case from existing TPR, all the way through complete API federation. The challenge that we face is how do we iterate toward that complete solution, and what exactly does the middle ground look like.

I agree there is a spectrum.

Repeating from another thread:

Let's say that we extended TPR with enough hooks, schemas, and other mechanisms that it could implement full Kube-API-like semantics, with spec and status, other subresources, content types, storage encodings, validation, defaulting, conversion, alternate views, admission control, quota, garbage collection, authorization policy, API discovery, etc.

What would it look like to use the two approaches to write such a fully featured API?

1. UAS: Write the API essentially the same way as we build APIs today, with existing tooling, libraries, etc., except that it calls out to a different storage backend, for which a fairly narrow API would suffice.

2. TPR: Write a number of hook endpoints to perform the subparts of the API logic, and use the existing storage backend.

The 2 approaches would be essentially isomorphic, but with control inverted in the latter case. Operationally, they require running the same number of apiserver-like components and controller-like components.

Having used analogous approaches for other systems, I would prefer to write an API using (1). (2) would require implementing a larger number of API endpoints, with many more cross-component calls, unless it degenerated to (1).

The sweet spots for TPRs are where full apiserver functionality isn't required and/or where full extensibility isn't required.
 

As a stake in the ground, I think we should start with the existing third-party object for API registration. It already exists, it already has a controller, and I don't think that there are any blockers to adapt it to all of these different use cases.

I agree it's worth considering unifying the registration mechanisms. However, we'd need to look at the gap between the existing TPR registration approach (e.g., it's naming peculiarities) and the proposed API aggregation mechanism, which relies on the discovery APIs, and we need to consider whether we separately want to register underlying storage resources for federated APIs.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

Brian Grant

unread,
Feb 7, 2017, 10:13:52 AM2/7/17
to Gabe Monroy, kubernetes-sig-apps, Brendan Burns, K8s API Machinery SIG
On Mon, Feb 6, 2017 at 5:53 PM, Gabe Monroy <ga...@deis.com> wrote:

First off I’d like to thank Eric and Anirudh for writing this up.  The sooner we can send a clear signal around how to extend the Kubernetes API, the better.


We at Deis have worked with both TPR and proto-UAS via the service catalog SIG.


Certain benefits of UAS are undeniable: schema and admission control flexibility being top of the list for me.  However, I feel like the operational complexity of UAS is being undersold in this doc.  The storage concerns alone worry me.  Are we asking folks to stand up etcd clusters for each UAS?   That’s a big management burden.

For most of the initial use cases we had in mind, apiservers that required storage (not all would) could share the same etcd cluster. 

  Are we recommending a UAS->TPR passthrough a la: https://github.com/kubernetes-incubator/service-catalog/pull/338.  It’s not clear we have a complete story around UAS just yet.

I haven't looked at that PR, but I would like to use TPR as a storage backed for UAS, yes. 


If Brendan is right and TPRs can be “fixed” with some coordinated effort, is that not a good use of resources right now?  Where does this decision get made?  SIG API Machinery?

SIG API machinery. 

  Regardless, we’re ready to help whichever direction the community decides.


Gabe

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-apps@googlegroups.com.

de...@redhat.com

unread,
Feb 7, 2017, 10:16:53 AM2/7/17
to kubernetes-sig-apps, brendan...@gmail.com, kubernetes-sig...@googlegroups.com, et...@google.com, xian...@coreos.com, derekwa...@gmail.com
I agree it's worth considering unifying the registration mechanisms. However, we'd need to look at the gap between the existing TPR registration approach (e.g., it's naming peculiarities) and the proposed API aggregation mechanism, which relies on the discovery APIs

Naming was just the most obvious example that everyone could easily see makes the two incompatible.  They are different APIs with different handling, each tailored for their purpose at opposite ends of the capability spectrum.  An attempt at unification is likely to make a mess of both implementations.  On the other hand, layering APIs (via a controller perhaps) that treats TPRs as just another kind of GroupVersion would be something reasonably achievable.

Brandon Philips

unread,
Feb 8, 2017, 12:55:35 AM2/8/17
to Daniel Smith, Eric Tune, kubernete...@googlegroups.com, K8s API Machinery SIG
On Mon, Feb 6, 2017 at 2:04 PM Brandon Philips <brandon...@coreos.com> wrote:
Where is this debate happening? I really think we should stay laser focused on simplifying our deployments as much as possible. We are spawning deployment complexity faster than we are providing solutions to users.

As some data points we recently announced deprecation of an early clustering system we built at CoreOS called fleet. Most people in the HN comments are supportive.

However, the dissenting voices all bemoaned how much more complex Kubernetes is to deploy based on moving parts (fleet is one binary + etcd):


Just some data points; but I think they are illustrative.

Brandon

rek...@gmail.com

unread,
Feb 8, 2017, 3:57:51 PM2/8/17
to kubernetes-sig-apps, kubernetes-sig...@googlegroups.com
The writeup of UAS seems very focused around the really vicious nasty work of making users maintain their code on top of the  Kubernetes tree, which seems like an IMO insultingly bad way to treat people trying to work with Kubernetes. In that horrible world, I'd have to suggest teaching everyone to go learn the 'quilt' utility that Debian uses to maintain patches atop other repos. It's not pretty. UAS sounds radically hard to do right now and like an unbelievable set of nightmares for anyone who has to suffer maintaining their code atop Kubernetes upstream, and seems basically certain to invite people to only support a narrow set of releases rather than offering code that can be developed in isolation and just work with Kubernetes.

But why is this a technical limitation? This sounds just like bad old design of UAS, built in the old world of where Go was, and something that can and should be addressed: Go 1.8 allows for shared libraries/plugins. Surely this ability to add libraries, which I hope/believe can be developed externally, would remedy the really tough usability problem of UAS?

For more info on Go 1.8 Plugins:
https://jeremywho.com/go-1.8---plugins/

The current path for UAS sounds horrific. I hope it can be made not-unbearable and not miserable.

Daniel Smith

unread,
Feb 8, 2017, 6:44:26 PM2/8/17
to rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
We're not quite there yet, but we're packaging up the necessary items in their own repo (k8s.io/apiserver). The idea being, UAS authors import this library and configure it. The whole point of allowing UAS is to free users from maintaining forks of kubernetes, so if we end up having the problems you're mentioning, we're doing something very wrong. Once we get this into a usable shape, we will absolutely (have to) be really, really careful about versioning, so we don't ever break consumers of this library. We have some experience with what this will take with the client-go library, where we're not quite meeting the bar yet.

I think it's the interfaces that matter. Whether users compile this in or link against the library, we have the same fundamental problem of maintaining version compatibility and not breaking library consumers. I'm not opposed to using the go 1.8 plugin feature.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/db0f1bd3-6121-4d9c-bab8-6b1f4d2b070f%40googlegroups.com.

Clayton Coleman

unread,
Feb 8, 2017, 7:12:28 PM2/8/17
to Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
Heh, maybe in 1.9, after we fix all the problems of "this segfaulted and someone used buffer overflows to steal the private master keys".
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3bZtF06qU2jvR8fzENJOORYk3AaOc0AziiWf6B6cFXNS0A%40mail.gmail.com.

Brendan Burns

unread,
Feb 9, 2017, 12:35:03 AM2/9/17
to Clayton Coleman, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG

Brian,
You said:

"Operationally, they require running the same number of apiserver-like components and controller-like components."

But I don't think that's actually true for most cloud provider environments.

The beauty of WebHooks and the state of the cloud today is that they can basically be implemented in a Javascript snippet and then launched as a Function in a serverless environment like Azure Functions or Lambda (or even something like fission on a kubernetes cluster)

It's easy to imagine writing a snippet of code that takes in a JSON payload, validates the data, sets some defaults and returns it back to the server as part of a webhook protocol.

The hosting in that model is entirely take care of for you by the cloud provider, versus having to build a binary, package it, run it, etc.

So while the webhook model has the same number of HTTP endpoints, the hosting model for those HTTP endpoints and the complexity of getting them up and running is dramatically reduced.

I think the whole point of identifying the continuum and the trajectory is that we need to go from:

"Gee, I'd really like to just store some data in a JSON API"

to

"Wow, I'd really just love to add validation to my API with a Javascript snippet"

to

"Well, I guess I have to write a full on apiserver, but can you still handle my storage?"

to

"Ok, I guess I have to do it all myself..."

That needs to be a continuum, and the first step up from TPR needs to be only marginally more complicated, not "here's a golang library, link it, build your apiserver, deploy it, and then you have validation"

Anyway, I'm quite optimistic we can achieve this if we can see this as a continuum and work together with a shared set of goals.

--brendan


On Wed, Feb 8, 2017, 4:12 PM Clayton Coleman <ccol...@redhat.com> wrote:
Heh, maybe in 1.9, after we fix all the problems of "this segfaulted and someone used buffer overflows to steal the private master keys".

On Feb 8, 2017, at 6:44 PM, 'Daniel Smith' via K8s API Machinery SIG <kubernetes-sig...@googlegroups.com> wrote:

We're not quite there yet, but we're packaging up the necessary items in their own repo (k8s.io/apiserver). The idea being, UAS authors import this library and configure it. The whole point of allowing UAS is to free users from maintaining forks of kubernetes, so if we end up having the problems you're mentioning, we're doing something very wrong. Once we get this into a usable shape, we will absolutely (have to) be really, really careful about versioning, so we don't ever break consumers of this library. We have some experience with what this will take with the client-go library, where we're not quite meeting the bar yet.

I think it's the interfaces that matter. Whether users compile this in or link against the library, we have the same fundamental problem of maintaining version compatibility and not breaking library consumers. I'm not opposed to using the go 1.8 plugin feature.
On Wed, Feb 8, 2017 at 12:57 PM, <rek...@gmail.com> wrote:
The writeup of UAS seems very focused around the really vicious nasty work of making users maintain their code on top of the  Kubernetes tree, which seems like an IMO insultingly bad way to treat people trying to work with Kubernetes. In that horrible world, I'd have to suggest teaching everyone to go learn the 'quilt' utility that Debian uses to maintain patches atop other repos. It's not pretty. UAS sounds radically hard to do right now and like an unbelievable set of nightmares for anyone who has to suffer maintaining their code atop Kubernetes upstream, and seems basically certain to invite people to only support a narrow set of releases rather than offering code that can be developed in isolation and just work with Kubernetes.

But why is this a technical limitation? This sounds just like bad old design of UAS, built in the old world of where Go was, and something that can and should be addressed: Go 1.8 allows for shared libraries/plugins. Surely this ability to add libraries, which I hope/believe can be developed externally, would remedy the really tough usability problem of UAS?

For more info on Go 1.8 Plugins:
https://jeremywho.com/go-1.8---plugins/

The current path for UAS sounds horrific. I hope it can be made not-unbearable and not miserable.

On Monday, February 6, 2017 at 4:45:45 PM UTC-5, Eric Tune wrote:
We have two options planned for adding new REST endpoints to the Kubernetes API.
Some people I have talked to have said they are confused about the need for two ways and which to use.

I wrote this doc that tries to answer those questions.

I plan to get it into a kubernetes.io doc once the work on Aggregated API Servers is released.  I thought I would share it now.  

Anyone with this link has comment permissions.
Feel free to comment!  Thanks.


--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-sig-api-m...@googlegroups.com.

To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/5162965258764667477%40unknownmsgid.

Xiang Li

unread,
Feb 9, 2017, 12:51:30 AM2/9/17
to Brendan Burns, Clayton Coleman, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
On Wed, Feb 8, 2017 at 9:34 PM, Brendan Burns <brendan...@gmail.com> wrote:

Brian,
You said:

"Operationally, they require running the same number of apiserver-like components and controller-like components."

But I don't think that's actually true for most cloud provider environments.

The beauty of WebHooks and the state of the cloud today is that they can basically be implemented in a Javascript snippet and then launched as a Function in a serverless environment like Azure Functions or Lambda (or even something like fission on a kubernetes cluster)

It's easy to imagine writing a snippet of code that takes in a JSON payload, validates the data, sets some defaults and returns it back to the server as part of a webhook protocol.

The hosting in that model is entirely take care of for you by the cloud provider, versus having to build a binary, package it, run it, etc.

So while the webhook model has the same number of HTTP endpoints, the hosting model for those HTTP endpoints and the complexity of getting them up and running is dramatically reduced.

I think the whole point of identifying the continuum and the trajectory is that we need to go from:

"Gee, I'd really like to just store some data in a JSON API"

to

"Wow, I'd really just love to add validation to my API with a Javascript snippet"

to

"Well, I guess I have to write a full on apiserver, but can you still handle my storage?"

to

"Ok, I guess I have to do it all myself..."

That needs to be a continuum, and the first step up from TPR needs to be only marginally more complicated, not "here's a golang library, link it, build your apiserver, deploy it, and then you have validation"

Anyway, I'm quite optimistic we can achieve this if we can see this as a continuum and work together with a shared set of goals.


As a current TPR user, I strongly support this. I really hope we can work together on the shard set of goals. etcd operator relies on TPR, and will rely on TPR in the future (even we add UAS frontend).

I believe the etcd operator will be a very important for UAS users. It would be the easiest way to bring up a managed reliable kv storage on k8s for a USA. Without a stable TRP, etcd operator will go nowhere. And UAS users will not enjoy the free managed reliable kv storage when they want.
 

--brendan


On Wed, Feb 8, 2017, 4:12 PM Clayton Coleman <ccol...@redhat.com> wrote:
Heh, maybe in 1.9, after we fix all the problems of "this segfaulted and someone used buffer overflows to steal the private master keys".

On Feb 8, 2017, at 6:44 PM, 'Daniel Smith' via K8s API Machinery SIG <kubernetes-sig-api-machinery@googlegroups.com> wrote:

We're not quite there yet, but we're packaging up the necessary items in their own repo (k8s.io/apiserver). The idea being, UAS authors import this library and configure it. The whole point of allowing UAS is to free users from maintaining forks of kubernetes, so if we end up having the problems you're mentioning, we're doing something very wrong. Once we get this into a usable shape, we will absolutely (have to) be really, really careful about versioning, so we don't ever break consumers of this library. We have some experience with what this will take with the client-go library, where we're not quite meeting the bar yet.

I think it's the interfaces that matter. Whether users compile this in or link against the library, we have the same fundamental problem of maintaining version compatibility and not breaking library consumers. I'm not opposed to using the go 1.8 plugin feature.
On Wed, Feb 8, 2017 at 12:57 PM, <rek...@gmail.com> wrote:
The writeup of UAS seems very focused around the really vicious nasty work of making users maintain their code on top of the  Kubernetes tree, which seems like an IMO insultingly bad way to treat people trying to work with Kubernetes. In that horrible world, I'd have to suggest teaching everyone to go learn the 'quilt' utility that Debian uses to maintain patches atop other repos. It's not pretty. UAS sounds radically hard to do right now and like an unbelievable set of nightmares for anyone who has to suffer maintaining their code atop Kubernetes upstream, and seems basically certain to invite people to only support a narrow set of releases rather than offering code that can be developed in isolation and just work with Kubernetes.

But why is this a technical limitation? This sounds just like bad old design of UAS, built in the old world of where Go was, and something that can and should be addressed: Go 1.8 allows for shared libraries/plugins. Surely this ability to add libraries, which I hope/believe can be developed externally, would remedy the really tough usability problem of UAS?

For more info on Go 1.8 Plugins:
https://jeremywho.com/go-1.8---plugins/

The current path for UAS sounds horrific. I hope it can be made not-unbearable and not miserable.

On Monday, February 6, 2017 at 4:45:45 PM UTC-5, Eric Tune wrote:
We have two options planned for adding new REST endpoints to the Kubernetes API.
Some people I have talked to have said they are confused about the need for two ways and which to use.

I wrote this doc that tries to answer those questions.

I plan to get it into a kubernetes.io doc once the work on Aggregated API Servers is released.  I thought I would share it now.  

Anyone with this link has comment permissions.
Feel free to comment!  Thanks.


--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.

Clayton Coleman

unread,
Feb 9, 2017, 9:26:52 AM2/9/17
to Xiang Li, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
General comment has been - why do you need an HA cluster of etcd to run a UAS?  Kubernetes can give you close to 4 nines of availability even as a singleton instance if your cluster appropriately detects down and stuck nodes.

Most UAS should probably be run as a pod of controller, apiserver, and small etcd instance, with appropriate forgiveness.  You would need to do backups correctly... but even that's fairly easy to self contain in that unit, or to lean on an orthogonal PV storage mechanism.

Xiang Li

unread,
Feb 9, 2017, 10:52:11 AM2/9/17
to Clayton Coleman, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
On Thu, Feb 9, 2017 at 6:26 AM, Clayton Coleman <ccol...@redhat.com> wrote:
General comment has been - why do you need an HA cluster of etcd to run a UAS?  Kubernetes can give you close to 4 nines of availability even as a singleton instance if your cluster appropriately detects down and stuck nodes.

How long will it take kubernetes to detect an node/pod failure and recover from it? Can it be faster than application layer detection? Also if you run 1 member cluster then it means 1 failure == unavailability. 

As for backup, if you have decided to run separate etcd, you probably have a large data set. And you wont want to do a backup every seconds. If you also want best perf, you want want to run PV. Then if you lose the node, you lose data. There are tradeoffs users should be able to make.


Most UAS should probably be run as a pod of controller, apiserver, and small etcd instance, with appropriate forgiveness. 

I assume most UAS should probably just run a controller deployment + api server and no etcd. They will use TPR as storage. There will be users want to store large data set and want the max availability. They should run HA etcd probably.
 

--brendan


To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.

To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.

To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.

To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.

Xiang Li

unread,
Feb 9, 2017, 10:57:12 AM2/9/17
to Clayton Coleman, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
On Thu, Feb 9, 2017 at 7:52 AM, Xiang Li <xian...@coreos.com> wrote:


On Thu, Feb 9, 2017 at 6:26 AM, Clayton Coleman <ccol...@redhat.com> wrote:
General comment has been - why do you need an HA cluster of etcd to run a UAS?  Kubernetes can give you close to 4 nines of availability even as a singleton instance if your cluster appropriately detects down and stuck nodes.

How long will it take kubernetes to detect an node/pod failure and recover from it? Can it be faster than application layer detection? Also if you run 1 member cluster then it means 1 failure == unavailability. 

As for backup, if you have decided to run separate etcd, you probably have a large data set.

Even if you run 1 member etcd, you might want to use the operator which handles backup/recover automatically for you.

Clayton Coleman

unread,
Feb 9, 2017, 10:58:20 AM2/9/17
to Xiang Li, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
On Thu, Feb 9, 2017 at 10:52 AM, Xiang Li <xian...@coreos.com> wrote:


On Thu, Feb 9, 2017 at 6:26 AM, Clayton Coleman <ccol...@redhat.com> wrote:
General comment has been - why do you need an HA cluster of etcd to run a UAS?  Kubernetes can give you close to 4 nines of availability even as a singleton instance if your cluster appropriately detects down and stuck nodes.

How long will it take kubernetes to detect an node/pod failure and recover from it? Can it be faster than application layer detection? Also if you run 1 member cluster then it means 1 failure == unavailability. 

Once forgiveness lands, the answer would be yes - you would be able to set a much lower bound (60s of node downtime) before you are rescheduled.  That said, if your PV can't be detached safely then it can be longer than your interval.  In practice, I think we *need* to get to the point where Kube makes that transition smooth and easy and effective, so that we don't have more than 52 minutes of downtime a year.
 

As for backup, if you have decided to run separate etcd, you probably have a large data set. And you wont want to do a backup every seconds. If you also want best perf, you want want to run PV. Then if you lose the node, you lose data. There are tradeoffs users should be able to make.

All the UAS that have been mooted are fairly small - tens of megs of actual values, not gigs.
 


Most UAS should probably be run as a pod of controller, apiserver, and small etcd instance, with appropriate forgiveness. 

I assume most UAS should probably just run a controller deployment + api server and no etcd. They will use TPR as storage. There will be users want to store large data set and want the max availability. They should run HA etcd probably.

There will be - but you have to ask who is doing the cluster configuration.  All the people I know who are going to want to do UAS in the near term are people who a) have serious reliability concerns and b) have existing HA clusters they'll want to reuse.  For them, simply reusing the master's etcd is much more reasonable.  Over time, using TPR and paying the latency / bandwidth cost will become more useful, as would running separate etcd instances.  But I think it's important to highlight that UAS starts as tools for serious, "product" level extension of Kube, and then works backwards to "people who want to do complex things and own the lifecycle of their component", while TPR grows up from "I can easily extend my cluster" to "I want more controls on those resources".
 
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.

To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.

Xiang Li

unread,
Feb 9, 2017, 11:09:39 AM2/9/17
to Clayton Coleman, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
On Thu, Feb 9, 2017 at 7:58 AM, Clayton Coleman <ccol...@redhat.com> wrote:


On Thu, Feb 9, 2017 at 10:52 AM, Xiang Li <xian...@coreos.com> wrote:


On Thu, Feb 9, 2017 at 6:26 AM, Clayton Coleman <ccol...@redhat.com> wrote:
General comment has been - why do you need an HA cluster of etcd to run a UAS?  Kubernetes can give you close to 4 nines of availability even as a singleton instance if your cluster appropriately detects down and stuck nodes.

How long will it take kubernetes to detect an node/pod failure and recover from it? Can it be faster than application layer detection? Also if you run 1 member cluster then it means 1 failure == unavailability. 

Once forgiveness lands, the answer would be yes - you would be able to set a much lower bound (60s of node downtime) before you are rescheduled.  That said, if your PV can't be detached safely then it can be longer than your interval.  In practice, I think we *need* to get to the point where Kube makes that transition smooth and easy and effective, so that we don't have more than 52 minutes of downtime a year.
 

As for backup, if you have decided to run separate etcd, you probably have a large data set. And you wont want to do a backup every seconds. If you also want best perf, you want want to run PV. Then if you lose the node, you lose data. There are tradeoffs users should be able to make.

All the UAS that have been mooted are fairly small - tens of megs of actual values, not gigs.
 


Most UAS should probably be run as a pod of controller, apiserver, and small etcd instance, with appropriate forgiveness. 

I assume most UAS should probably just run a controller deployment + api server and no etcd. They will use TPR as storage. There will be users want to store large data set and want the max availability. They should run HA etcd probably.

There will be - but you have to ask who is doing the cluster configuration.  All the people I know who are going to want to do UAS in the near term are people who a) have serious reliability concerns and b) have existing HA clusters they'll want to reuse.  For them, simply reusing the master's etcd is much more reasonable. 

Is it? Assuming that etcd operator works, what is the advantages of managing etcd yourself over managing it by the etcd operator? Similarly, what is the advantages of using 1 member etcd controlled by Kube over HA managed by the etcd operator?
 

Clayton Coleman

unread,
Feb 9, 2017, 11:31:05 AM2/9/17
to Xiang Li, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
Most of the people running these clusters have 2 years of experience running Kube in production and HA etcd, so the operator is giving them less control and less flexibility.  That doesn't mean they might not use the operator in the future, but they aren't going to replace something known and working (HA cluster etcds with processes already setup) with a new set of human processes just because there's something that could replace it.  It's likely to be a slow and gradual transition.

Xiang Li

unread,
Feb 9, 2017, 12:17:21 PM2/9/17
to Clayton Coleman, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
On Thu, Feb 9, 2017 at 8:31 AM, Clayton Coleman <ccol...@redhat.com> wrote:
Most of the people running these clusters have 2 years of experience running Kube in production and HA etcd, so the operator is giving them less control and less flexibility.  That doesn't mean they might not use the operator in the future, but they aren't going to replace something known and working (HA cluster etcds with processes already setup) with a new set of human processes just because there's something that could replace it.  It's likely to be a slow and gradual transition.

Agree. I believe it will happen eventually, and TPR is important to its maturity. Back to the topic, I still hope to see sig api can have shared goal to make TPR stable quickly.

David Eads

unread,
Feb 15, 2017, 12:45:52 PM2/15/17
to Xiang Li, Clayton Coleman, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
In his doc, Eric noted that breaking changes may be needed to address some of the problems noted in https://github.com/kubernetes/features/issues/95.  Two of the biggest problems we have are inconsistent/poorly defined API semantics compared to kubernetes style APIs and incomplete/potentially conflicting information when defining TPRs themselves.

I've described a way to fix these problems in a doc here: https://docs.google.com/a/redhat.com/document/d/1Gg158jO1cRBq-8RrWRAWA2IRF9avscuRaWFmY2Wb6qw/edit?usp=sharing.  An API can be built that resolves many of the problems in issue 95 and we can have a clean migration path from one to the other, but I don't see how the existing API can be transformed in a backwards compatible way that also resolves the problems that we have.

--
You received this message because you are subscribed to a topic in the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-sig-apps/0gbmMNvZWUo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-apps@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/CAJLA0AB4XUcNXVZSirw-WG20P866qQACWs%3D3WYWt5z4OTVOTjw%40mail.gmail.com.

Eric Chiang

unread,
Feb 15, 2017, 7:11:36 PM2/15/17
to David Eads, Xiang Li, Clayton Coleman, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
Hey David,

There are a lot of heavy users of TPRs in the community that would be
affected by a breaking change (cc'ing a couple on this thread).

I don't think anyone would advocate that TPRs in a perfect place now,
but the conversation for customized resources seems to be focusing
around solutions like custom API servers anyway. Are we sure the pain
of breaking this API pays off in the end?

Eric
>>>>>>>>>> send an email to kubernetes-sig-api-m...@googlegroups.com.
>>>>>>>>>> To post to this group, send email to
>>>>>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/db0f1bd3-6121-4d9c-bab8-6b1f4d2b070f%40googlegroups.com.
>>>>>>>>>>
>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "K8s API Machinery SIG" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to kubernetes-sig-api-m...@googlegroups.com.
>>>>>>>>> To post to this group, send email to
>>>>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>>>>>
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3bZtF06qU2jvR8fzENJOORYk3AaOc0AziiWf6B6cFXNS0A%40mail.gmail.com.
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>>> the Google Groups "K8s API Machinery SIG" group.
>>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>>> https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
>>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>>> kubernetes-sig-api-m...@googlegroups.com.
>>>>>>>>> To post to this group, send email to
>>>>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/5162965258764667477%40unknownmsgid.
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>> the Google Groups "K8s API Machinery SIG" group.
>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>> https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>> kubernetes-sig-api-m...@googlegroups.com.
>>>>>>>> To post to this group, send email to
>>>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAOgwWTtMWgP1zZ-k56nj4MKMe0fhs2505fcqwQWnF2YLjUGYrQ%40mail.gmail.com.
>>>>>>>>
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "K8s API Machinery SIG" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>> an email to kubernetes-sig-api-m...@googlegroups.com.
>>>>>> To post to this group, send email to
>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAJLA0AAVKs%3DVreYAbztGGD7ogmJwwnBR15xRm9jwLVVSJkGOmA%40mail.gmail.com.
>>>>>>
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "kubernetes-sig-apps" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/kubernetes-sig-apps/0gbmMNvZWUo/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> kubernetes-sig-...@googlegroups.com.
>> To post to this group, send email to kubernete...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kubernetes-sig-apps/CAJLA0AB4XUcNXVZSirw-WG20P866qQACWs%3D3WYWt5z4OTVOTjw%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "K8s API Machinery SIG" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-sig-api-m...@googlegroups.com.
> To post to this group, send email to
> kubernetes-sig...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFS1Mj%2BEi78svwchkqc1DtQmN_RrCrYYAZf-aj7AhhLLytrcjg%40mail.gmail.com.

Clayton Coleman

unread,
Feb 15, 2017, 7:25:28 PM2/15/17
to Eric Chiang, David Eads, Xiang Li, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
The concern is that once we set beta on TPR, we support that forever.
We don't get a second chance.

Eric Chiang

unread,
Feb 15, 2017, 7:41:02 PM2/15/17
to Clayton Coleman, David Eads, Xiang Li, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
I'm a little confused, TPRs are under extensions/v1beta1. Even if
they've not been considered beta internally, they seem to be
advertised as such.

Steve Sloka

unread,
Feb 15, 2017, 9:15:33 PM2/15/17
to K8s API Machinery SIG, de...@redhat.com, xian...@coreos.com, ccol...@redhat.com, brendan...@gmail.com, dbs...@google.com, rek...@gmail.com, kubernete...@googlegroups.com
Hey everyone, wanted to add my feedback as well as one of the users of TPR. I've written a Vault integration for secrets as well as an Elasticsearch integration which follow the TPR / Controller model. The simplicity of how TPR work inside Kubernetes is pretty neat coming from a world where I've not been a core maintainer and much of how the pieces internally work are "magic". 

My biggest struggles implementing TPR have been understanding how to talk to the API server correctly in Go using the go-client. For instance, there was an issue if you delete a StatefulSet it doesn't remove the associated pods because kubectl does 2 steps when you delete the resource and the API doesn't (https://github.com/kubernetes/client-go/issues/91#issuecomment-277766967). 

I like the ability to quickly create operators / controllers with a few kubectl commands and I'm up and running. For my on-prem clusters, since they are managed by corporate IT, they are usually reluctant to deploy custom pieces (that UAS might impose), however, my cloud deployments don't have the same constraints. 

If we need some work done to better support TPR, I'm happy to engage and send up PR's.

~ Steve 

On Wednesday, February 15, 2017 at 7:11:36 PM UTC-5, Eric Chiang wrote:
Hey David, my 
>>>>>>>>>> To post to this group, send email to
>>>>>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/db0f1bd3-6121-4d9c-bab8-6b1f4d2b070f%40googlegroups.com.
>>>>>>>>>>
>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "K8s API Machinery SIG" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> To post to this group, send email to
>>>>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>>>>>
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3bZtF06qU2jvR8fzENJOORYk3AaOc0AziiWf6B6cFXNS0A%40mail.gmail.com.
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>>> the Google Groups "K8s API Machinery SIG" group.
>>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>>> https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
>>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>>> To post to this group, send email to
>>>>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/5162965258764667477%40unknownmsgid.
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>> the Google Groups "K8s API Machinery SIG" group.
>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>> https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>> To post to this group, send email to
>>>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAOgwWTtMWgP1zZ-k56nj4MKMe0fhs2505fcqwQWnF2YLjUGYrQ%40mail.gmail.com.
>>>>>>>>
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "K8s API Machinery SIG" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>> To post to this group, send email to
>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAJLA0AAVKs%3DVreYAbztGGD7ogmJwwnBR15xRm9jwLVVSJkGOmA%40mail.gmail.com.
>>>>>>
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "kubernetes-sig-apps" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/kubernetes-sig-apps/0gbmMNvZWUo/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> To post to this group, send email to kubernete...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kubernetes-sig-apps/CAJLA0AB4XUcNXVZSirw-WG20P866qQACWs%3D3WYWt5z4OTVOTjw%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "K8s API Machinery SIG" group.
> To unsubscribe from this group and stop receiving emails from it, send an

Clayton Coleman

unread,
Feb 15, 2017, 9:33:40 PM2/15/17
to Eric Chiang, David Eads, Xiang Li, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
Most of the things in v1beta1 ended up there before we had even really
defined what beta meant (in the same way we would apply the label to a
feature today). And just because it went to beta doesn't mean it's
the right long term design or that it won't be changed before it's
"final" (batch.Job is an example of something we revised
significantly). I don't think that there's a desire to break
existing TPR uses, but since the storage aspects and how it fits with
other pieces also tie into how compatible future changes would be, it
doesn't seem right to ignore the problems.

I don't know if anyone here has tried to migrate TPR users from one
version to another of the same API, but if you have that feedback is
probably pretty useful.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAA8S17HkJoW_NiHeHJOjo6GUeOXw6MTR8OLUhuO4iu_BoRRmpw%40mail.gmail.com.

Jordan Liggitt

unread,
Feb 15, 2017, 9:51:40 PM2/15/17
to Eric Chiang, David Eads, Xiang Li, Clayton Coleman, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
The discussion about the TPR schema is independent of the custom API
server discussion. I think TPRs fill a legitimate need, but need a few
fundamental schema changes if we want to graduate them from the
current alphaish state.

The current TPR API allows conflicting and ambiguous resource
registration, makes you jump through hoops to get a specific
kind/resource (due to auto-camelcasing and pluralization), leaves
multi-version behavior unimplemented and undefined, and doesn't allow
for a spec/status split.

TPRs can be a very useful lightweight extension, but I don't think the
current API is a good foundation. We should fix it to specify the
required information and make usage less surprising (my preference),
or rebuild it as a second resource. The problems with naming and the
need for a spec/status split rule out evolving the current object, in
my opinion.


> On Feb 15, 2017, at 7:11 PM, Eric Chiang <eric....@coreos.com> wrote:
>
> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAA8S17GVO0tD2-PC2CRSTw_BWomP7FjJ%2BfkV5%3DeJHszUEY5H2w%40mail.gmail.com.

Jordan Liggitt

unread,
Feb 15, 2017, 10:05:09 PM2/15/17
to Clayton Coleman, Eric Chiang, David Eads, Xiang Li, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
Agree. TPR started in the API group "experimental/v1beta1". The whole group was renamed to "extensions/v1beta1", but the TPR resource was not intentionally declared beta with forward compatibility guarantees (the original "requirements to graduate to beta"[1] issue is still unresolved, for example). 

It's very unfortunate the current state was not communicated more clearly, but I would like to see TPRs succeed at being a straightforward, lightweight extension mechanism, and I don't think promoting an API with fundamental issues to stable is a good way to accomplish that. 


Brandon Philips

unread,
Feb 16, 2017, 2:41:38 AM2/16/17
to David Eads, Xiang Li, Clayton Coleman, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
On Wed, Feb 15, 2017 at 9:45 AM David Eads <de...@redhat.com> wrote:
I've described a way to fix these problems in a doc here: https://docs.google.com/a/redhat.com/document/d/1Gg158jO1cRBq-8RrWRAWA2IRF9avscuRaWFmY2Wb6qw/edit?usp=sharing.  An API can be built that resolves many of the problems in issue 95 and we can have a clean migration path from one to the other, but I don't see how the existing API can be transformed in a backwards compatible way that also resolves the problems that we have.

I think Brendan and others are helping to find potential clean upgrade path in the doc. I don't think it is sufficient to say the migration path is use this new API and the old one will go away soon.

I have started collecting TPR users on a gist because I think we need more data on how many applications that have been built specifically for Kubernetes would be affected:  https://gist.github.com/philips/a97a143546c87b86b870a82a753db14c

If you know of others please comment.

I think whatever we do we all agree we should ensure these users don't get broken as TPRs are clearly a simple and useful extension point based on the number of users I am finding.

Cheers,

Brandon
 

--brendan


To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-sig-apps/0gbmMNvZWUo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-sig-...@googlegroups.com.

To post to this group, send email to kubernete...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To post to this group, send email to kubernete...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/CAFS1Mj%2BEi78svwchkqc1DtQmN_RrCrYYAZf-aj7AhhLLytrcjg%40mail.gmail.com.

Brian Grant

unread,
Feb 16, 2017, 1:37:48 PM2/16/17
to Clayton Coleman, Eric Chiang, David Eads, Xiang Li, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
On Wed, Feb 15, 2017 at 6:33 PM, Clayton Coleman <ccol...@redhat.com> wrote:
Most of the things in v1beta1 ended up there before we had even really
defined what beta meant (in the same way we would apply the label to a
feature today). 

extensions/v1beta1 is a grab bag of alpha and beta APIs. It was created (with difficulty) before the full API-group mechanism was developed. For the extensions group, "beta" doesn't necessarily mean beta.
 

>>>>>>>>>>>>> To post to this group, send email to

>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/db0f1bd3-6121-4d9c-bab8-6b1f4d2b070f%40googlegroups.com.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>>>>> Groups "K8s API Machinery SIG" group.
>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,

>>>>>>>>>>>> To post to this group, send email to

>>>>>>>>>>>>
>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3bZtF06qU2jvR8fzENJOORYk3AaOc0AziiWf6B6cFXNS0A%40mail.gmail.com.
>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>>>>>> the Google Groups "K8s API Machinery SIG" group.
>>>>>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>>>>>> https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
>>>>>>>>>>>> To unsubscribe from this group and all its topics, send an email to

>>>>>>>>>>>> To post to this group, send email to

>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/5162965258764667477%40unknownmsgid.
>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>>>>> the Google Groups "K8s API Machinery SIG" group.
>>>>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>>>>> https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
>>>>>>>>>>> To unsubscribe from this group and all its topics, send an email to

>>>>>>>>>>> To post to this group, send email to

>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAOgwWTtMWgP1zZ-k56nj4MKMe0fhs2505fcqwQWnF2YLjUGYrQ%40mail.gmail.com.
>>>>>>>>>>>
>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "K8s API Machinery SIG" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>>>>>>> To post to this group, send email to

>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAJLA0AAVKs%3DVreYAbztGGD7ogmJwwnBR15xRm9jwLVVSJkGOmA%40mail.gmail.com.
>>>>>>>>>
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to a topic in the
>>>>> Google Groups "kubernetes-sig-apps" group.
>>>>> To unsubscribe from this topic, visit
>>>>> https://groups.google.com/d/topic/kubernetes-sig-apps/0gbmMNvZWUo/unsubscribe.
>>>>> To unsubscribe from this group and all its topics, send an email to

>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/kubernetes-sig-apps/CAJLA0AB4XUcNXVZSirw-WG20P866qQACWs%3D3WYWt5z4OTVOTjw%40mail.gmail.com.
>>>>>
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "K8s API Machinery SIG" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an

>>>> To post to this group, send email to

>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFS1Mj%2BEi78svwchkqc1DtQmN_RrCrYYAZf-aj7AhhLLytrcjg%40mail.gmail.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
> To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-apps@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/659607419676481359%40unknownmsgid.

Jordan Liggitt

unread,
Feb 16, 2017, 8:39:57 PM2/16/17
to Brandon Philips, David Eads, Xiang Li, Clayton Coleman, Brendan Burns, Daniel Smith, rek...@gmail.com, kubernetes-sig-apps, K8s API Machinery SIG
On Thu, Feb 16, 2017 at 2:41 AM, Brandon Philips <brandon...@coreos.com> wrote:
On Wed, Feb 15, 2017 at 9:45 AM David Eads <de...@redhat.com> wrote:
I've described a way to fix these problems in a doc here: https://docs.google.com/a/redhat.com/document/d/1Gg158jO1cRBq-8RrWRAWA2IRF9avscuRaWFmY2Wb6qw/edit?usp=sharing.  An API can be built that resolves many of the problems in issue 95 and we can have a clean migration path from one to the other, but I don't see how the existing API can be transformed in a backwards compatible way that also resolves the problems that we have.

I think Brendan and others are helping to find potential clean upgrade path in the doc. I don't think it is sufficient to say the migration path is use this new API and the old one will go away soon.


I could envision an "export, upgrade server, import"-style approach working (similar to the PetSet > StatefulSet transition).

There are ambiguities in the existing definitions and storage that make it difficult to be confident that data already in the system is correct.

Example 1: These three TPRs resolve to different kinds, but conflict when registering resource handlers, and step on each other in storage:

Example 2: Existing instances of third party resource objects are not always removed when the TPR object is deleted:
1. If there's a conflicting TPR object (like in example 1), instances are not removed
2. If a TPR is deleted and recreated "quickly", instances are not removed.
3. If a TPR is deleted and an error is encountered cleaning up instances, the cleanup task is not persistent across server starts.

Issue like this are all fixable, but mean that we can't count on current data to be correct. It's possible the fixes could be largely contained to the internals of the storage handler, meaning an export/upgrade/import migration of the TPRs and instance data could be sufficient to put us on more solid footing. We'll know more as the sig-api-machinery folks work out the design.

 
I have started collecting TPR users on a gist because I think we need more data on how many applications that have been built specifically for Kubernetes would be affected:  https://gist.github.com/philips/a97a143546c87b86b870a82a753db14c

If you know of others please comment.

I think whatever we do we all agree we should ensure these users don't get broken as TPRs are clearly a simple and useful extension point based on the number of users I am finding.



That's a great list, but knowing that TPRs will be used for things like credentials and cert management makes me feel even more strongly that the ambiguities in the storage layer have to be fixed before promoting it to stable.


bassam....@quantum.com

unread,
Feb 24, 2017, 7:10:58 PM2/24/17
to K8s API Machinery SIG, de...@redhat.com, xian...@coreos.com, ccol...@redhat.com, brendan...@gmail.com, dbs...@google.com, rek...@gmail.com, kubernete...@googlegroups.com
Hi,

I'm late to the party but I wanted to share our perspective on this important issue. I'm leading the effort around Rook (http://rook.io) and we are using TPRs heavily to integrate our storage layer into K8S.

One aspect of K8S that is really appealing to us is the pattern of defining desired state and having controllers/operators act on the state within the cluster. This model improves the way that admins interact with the cluster and enables a high degree of automation.

While you can achieve this pattern with both AA and TPR, I'd argue that TPR is a better fit. It forces developers to think about a declarative state separately from the imperative/procedure code that implements it. I think it would be easy to diverge from this desired state / controller pattern with an AA implementation.

I understand some of the limitations of TPRs today but I think there is an opportunity to fix these without changing the dramatically. For example, validation should be achievable with web hooks or a simple constraint language decorarting the schema.

Thanks!

>>>>>>>>>> To post to this group, send email to
>>>>>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/db0f1bd3-6121-4d9c-bab8-6b1f4d2b070f%40googlegroups.com.
>>>>>>>>>>
>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "K8s API Machinery SIG" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> To post to this group, send email to
>>>>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>>>>>
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3bZtF06qU2jvR8fzENJOORYk3AaOc0AziiWf6B6cFXNS0A%40mail.gmail.com.
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>>> the Google Groups "K8s API Machinery SIG" group.
>>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>>> https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
>>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>>> To post to this group, send email to
>>>>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/5162965258764667477%40unknownmsgid.
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>> the Google Groups "K8s API Machinery SIG" group.
>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>> https://groups.google.com/d/topic/kubernetes-sig-api-machinery/o2v-HjAHk0k/unsubscribe.
>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>> To post to this group, send email to
>>>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAOgwWTtMWgP1zZ-k56nj4MKMe0fhs2505fcqwQWnF2YLjUGYrQ%40mail.gmail.com.
>>>>>>>>
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "K8s API Machinery SIG" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>> To post to this group, send email to
>>>>>> kubernetes-sig...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAJLA0AAVKs%3DVreYAbztGGD7ogmJwwnBR15xRm9jwLVVSJkGOmA%40mail.gmail.com.
>>>>>>
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "kubernetes-sig-apps" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/kubernetes-sig-apps/0gbmMNvZWUo/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> To post to this group, send email to kubernete...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kubernetes-sig-apps/CAJLA0AB4XUcNXVZSirw-WG20P866qQACWs%3D3WYWt5z4OTVOTjw%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "K8s API Machinery SIG" group.
> To unsubscribe from this group and stop receiving emails from it, send an

Daniel Smith

unread,
Feb 28, 2017, 4:46:05 PM2/28/17
to Eric Tune, kubernetes-sig-apps, K8s API Machinery SIG
As a follow-up to this, we've put together a doc with some more reasoning behind the split: https://docs.google.com/document/d/1lU1SnVtEec2iIfYx5U3L5N0za2YhfEOik0uPen276Ks/edit

On Mon, Feb 6, 2017 at 1:45 PM, 'Eric Tune' via K8s API Machinery SIG <kubernetes-sig...@googlegroups.com> wrote:
We have two options planned for adding new REST endpoints to the Kubernetes API.
Some people I have talked to have said they are confused about the need for two ways and which to use.

I wrote this doc that tries to answer those questions.

I plan to get it into a kubernetes.io doc once the work on Aggregated API Servers is released.  I thought I would share it now.  

Anyone with this link has comment permissions.
Feel free to comment!  Thanks.


--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAASt_VE8gfqdZmW_61Qw54vNZeUUFafgiPUy7h5331mi6-gPHg%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages