One possible solution to that could be to use a dynamic client for the usage inside of k8s.io/kubernetes if it was absolutely needed.On Fri, Aug 17, 2018 at 3:10 PM David Eads <de...@redhat.com> wrote:Wait, I may have spoken too soon. You need to have k8s.io/kubernetes import your repo? We don't allow the vendoring of repos that depend upon k8s.io/apimachinery (or any other staging repo), because it creates a chicken and egg problem when something need refactoring/fixing. I thought this was going out of tree and could be managed on a different cadence.On Fri, Aug 17, 2018 at 2:57 PM Saad Ali <saa...@google.com> wrote:Thanks for the clarification! Will create a new repo under github.com/kubernetes-csi/... to house this and have k8s core import that.On Fri, Aug 17, 2018 at 11:18 AM David Eads <de...@redhat.com> wrote:We have examples of APIs which live in other repos (CRDs come to mind). If you follow that example you can keep your API and client in the repo with the rest of your code. When vgo arrives you can have separate modules in one repo When you wish to make use of those clients, you can import the config types and you can import the client you want to use. The two work together well.On Fri, Aug 17, 2018 at 12:56 PM Saad Ali <saa...@google.com> wrote:Definitely for new components like Snapshots we are putting everything "out of tree".In this case, we have a new objecct we want to introduce "CSIDriver" that we want the core k8s binaries to use. SIG-Arch clarified that in this case it would be fine to have some core binary or mechanism installing the CRD. But it's unclear where the API schema and generated client packages should go. Given that context, any changes in your recommendation?ÂOn Fri, Aug 17, 2018 at 9:51 AM Daniel Smith <dbs...@google.com> wrote:It is true that if you add the types there, regular clients will be generated. If we have done this before then I won't object now. However. It is not a good precedent IMO.* Clients should have an expectation to import multiple typed client packages.* Authors should not feel like their api type needs to be in k8s.io/api to be "real"* k8s.io/api should not function like a global lock. APIs developed out-of-tree needn't evolve at the same rate as in-tree APIs: they could release faster or slower, for example. Putting your api types in the main type registry removes some of the benefits of developing out of tree in the first place.The design of record for client libraries for something like a year has been a go-base with dynamic client, rest client, etc and no generated clients; and separate repos with generated clients, one per source where the apis are generated.It may be expedient to put types in k8s.io/api but it is not good for the longer term because it makes it harder to get the client libraries into the right shape.On Fri, Aug 17, 2018 at 12:07 AM Saad Ali <saa...@google.com> wrote:Tim or Daniel, can you confirm this:> I talked to Stefan Schimanski (sig-apimachinery) and this is not true.> We can put our types.go to pkg/apis/csi-storage/v1alpha1 and hack/*> will generate client and informers for us, both part of the usual> interfaces (e.g. clientset.Interface).For https://github.com/kubernetes/community/pull/2514 can I put my resource schema under "github.com/kubernetes/kubernetes/pkg/apis/{newGroupForStorageCRDs}/v1alpha1"?Or should the schema be put in an external package, e.g. "github.com/kubernetes-csi/{newRepoForStorageCRDSchemas}/...", and that path imported by k8s controllers that need to operate on this type?On Thu, Aug 9, 2018 at 4:53 AM Jan Safranek <jsaf...@redhat.com> wrote:On 02/08/18 18:41, Jan Safranek wrote:
> * I had to pass a new clientset interface for the CRD from cmd/kubelet
> into depths of pkg/kubelet. It's not hard, but I'd expect some
> resistance there.
I talked to Stefan Schimanski (sig-apimachinery) and this is not true.
We can put our types.go to pkg/apis/csi-storage/v1alpha1 and hack/*
will generate client and informers for us, both part of the usual
interfaces (e.g. clientset.Interface).
So adding new types.go for a CRD is actually pretty easy.
> * We need to add some controller that installs the CRD on cluster
> startup. And update it on cluster updates/downgrades.
API server has post-start hook that can be used to add such things. For
example, default RBAC rules for controllers are inserted in a post-start
hook.
We could inject a hook e.g. here:
https://github.com/kubernetes/kubernetes/blob/3cb771a8662ae7d1f79580e0ea9861fd6ab4ecc0/pkg/master/master.go#L369
> To me it seems that CRD adds a lot of new work with very little
> benefits. Most importantly, IMO it won't remove need to go through API
> review, which is the reason we don't want in-tree object.
>
> So, does it make sense to rush the CRD in following 4-5 weeks?
This is still valid. And the counter is at 3-4 weeks now.
--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-storage-wg-csi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-stora...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage-wg-csi/d577d7d1-f612-a36b-ed17-5c5c1de54643%40redhat.com.
For more options, visit https://groups.google.com/d/optout.
Ok, I'll hold offÂ>Â You need to have k8s.io/kubernetes import your repo?Yes, we need core controllers to be able to read these objects and act on them (and possibly create them as well).
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-storage-wg-csi+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-storage-wg-csi@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage-wg-csi/d577d7d1-f612-a36b-ed17-5c5c1de54643%40redhat.com.
For more options, visit https://groups.google.com/d/optout.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3baghqi7GE5V-iGh%2B-XWTedpxakxh3v%2BkMFZHg%2BzpskwAQ%40mail.gmail.com.--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-architecture@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-stora...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage-wg-csi/d577d7d1-f612-a36b-ed17-5c5c1de54643%40redhat.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
Saad and I discussed two options. In both options, "CSIDriver" will be installed as a CRD. kube-apiserver binary does not depend on the "CSIDriver" type.Option 1:
- create a staging repo "csi-api" and let the publish-bot to sync it to "kubernetes-csi/csi-api".
- put the types.go in "csi-api"
- generate CSIDriver's own clients, listers, and informers in "csi-api"
- this is how "apiextension-apiserver" repo hosts the "CustomResouceDefinition" types and clients.
Option 2:
- put the types.go in the existing staging "api" repo, under a new API group.Â
- CSIDriver's clients, listers, and informers will be generated in the same package as the kubernetes ones.
At first glance, option 1 decouples "csi-api" from the existing "api". However, "CSIDriver" users will almost always use CSIDriver clients with the kubernetes PV/PVC clients. For that to work, with today's golang dependency management, the two sets of clients need to vendor the same version of k8s.io/apimachinery, which isn't backwards compatible. Thus separating CSI clients from kubernetes clients only creates barriers for users.Option 2 is more straightforward.
Saad and I discussed two options. In both options, "CSIDriver" will be installed as a CRD. kube-apiserver binary does not depend on the "CSIDriver" type.Option 1:
- create a staging repo "csi-api" and let the publish-bot to sync it to "kubernetes-csi/csi-api".
- put the types.go in "csi-api"
- generate CSIDriver's own clients, listers, and informers in "csi-api"
- this is how "apiextension-apiserver" repo hosts the "CustomResouceDefinition" types and clients.
Option 2:
- put the types.go in the existing staging "api" repo, under a new API group.Â
- CSIDriver's clients, listers, and informers will be generated in the same package as the kubernetes ones.
At first glance, option 1 decouples "csi-api" from the existing "api". However, "CSIDriver" users will almost always use CSIDriver clients with the kubernetes PV/PVC clients. For that to work, with today's golang dependency management, the two sets of clients need to vendor the same version of k8s.io/apimachinery, which isn't backwards compatible. Thus separating CSI clients from kubernetes clients only creates barriers for users.
Option 2 is more straightforward.
On Fri, Aug 17, 2018 at 2:21 PM, Saad Ali <saa...@google.com> wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAO_RewYXBEk2f5OuiDQ0EQb1-1gV%3DFpnQSWWrq%3DHL4yq_JQAhg%40mail.gmail.com.
"k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/runtime/serializer"
Generated clientsets (meaning not dynamic) currently depend on these parts of apimachinery:"k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/runtime/serializer"
Of this short list, what, exactly, do we think needs to be refactored? Isn't this list short enough that we could make an exception for vendoring generated clients?
On Tue, Aug 21, 2018 at 10:29 AM Saad Ali <saa...@google.com> wrote:
I believe the argument against that was: "You need to have k8s.io/kubernetes import your repo? We don't allow the vendoring of repos that depend upon k8s.io/apimachinery (or any other staging repo), because it creates a chicken and egg problem when something need refactoring/fixing. I thought this was going out of tree and could be managed on a different cadence... One possible solution to that could be to use a dynamic client for the usage inside of k8s.io/kubernetes if it was absolutely needed."
On Tue, Aug 21, 2018 at 9:59 AM Eric Tune <et...@google.com> wrote:
Hi all, coming late to this thread.AIUI, the decision was made to host the types,go in k/k rather than the new repos.Is it too late to reconsider this decision?There are a couple of SDKs for authoring CRDs (kubebuilder and operator-sdk). ÂWithout saying that which this project should use, I do think this and all new CRD types should be strongly encouraged to use some CRD SDK - to increase consistency of generated code, clients, tests, and docs.ÂThese SDKs are built around the presumption that a group of related CRDs are defined in a single project, including the types.go.Â
>> >Â Â Â Â Â Â Â Â Â <kubernetes-sig-architecture@googlegroups.com
>> >Â Â Â Â Â Â Â Â Â <mailto:kubernetes-sig-archit...@googlegroups.com>>
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â kubernetes-sig-storage-wg-csi+unsubscribe@googlegroups.com
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â <mailto:kubernetes-sig-storage-wg-csi%2Bunsubscribe@googlegroups.com>.
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â To post to this
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â group, send
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â email to
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â kubernetes-sig-storage-wg-c...@googlegroups.com
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â <mailto:kubernetes-sig-storage-wg-csi@googlegroups.com>.
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â To view this
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â discussion on
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â the web visit
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â https://groups.google.com/d/msgid/kubernetes-sig-storage-wg-csi/d577d7d1-f612-a36b-ed17-5c5c1de54643%40redhat.com.
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â For more
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â options, visit
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â https://groups.google.com/d/optout.
>> >
>> >Â Â Â Â Â Â Â Â Â Â Â --
>> >Â Â Â Â Â Â Â Â Â Â Â You received this message because you are subscribed
>> >Â Â Â Â Â Â Â Â Â Â Â to the Google Groups "kubernetes-sig-architecture"
>> >Â Â Â Â Â Â Â Â Â Â Â group.
>> >Â Â Â Â Â Â Â Â Â Â Â To unsubscribe from this group and stop receiving
>> >Â Â Â Â Â Â Â Â Â Â Â emails from it, send an email to
>> >Â Â Â Â Â Â Â Â Â Â Â kubernetes-sig-architecture+unsub...@googlegroups.com
>> >Â Â Â Â Â Â Â Â Â Â Â <mailto:kubernetes-sig-architecture+unsubscribe@googlegroups.com>.
>> >Â Â Â Â Â Â Â Â Â Â Â To post to this group, send email to
>> >Â Â Â Â Â Â Â Â Â Â Â kubernetes-sig-architecture@googlegroups.com
>> >Â Â Â Â Â Â Â Â Â Â Â <mailto:kubernetes-sig-archit...@googlegroups.com>.
>> >Â Â Â Â Â Â Â Â Â Â Â To view this discussion on the web visit
>> >Â Â Â Â Â Â Â Â Â Â Â https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3baghqi7GE5V-iGh%2B-XWTedpxakxh3v%2BkMFZHg%2BzpskwAQ%40mail.gmail.com
>> >Â Â Â Â Â Â Â Â Â Â Â <https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3baghqi7GE5V-iGh%2B-XWTedpxakxh3v%2BkMFZHg%2BzpskwAQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>> >
>> >Â Â Â Â Â Â Â Â Â Â Â For more options, visit
>> >Â Â Â Â Â Â Â Â Â Â Â https://groups.google.com/d/optout.
>> >
>> >
>> >
>>
--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-architecture+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-architecture@googlegroups.com.
>> >Â Â Â Â Â Â Â Â Â <kubernetes-si...@googlegroups.com
>> >Â Â Â Â Â Â Â Â Â <mailto:kubernetes-si...@googlegroups.com>>
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â kubernetes-sig-stora...@googlegroups.com
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â <mailto:kubernetes-sig-storage-wg-csi%2Bunsu...@googlegroups.com>.
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â To post to this
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â group, send
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â email to
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â kubernetes-sig...@googlegroups.com
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â <mailto:kubernetes-sig...@googlegroups.com>.
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â To view this
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â discussion on
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â the web visit
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â https://groups.google.com/d/msgid/kubernetes-sig-storage-wg-csi/d577d7d1-f612-a36b-ed17-5c5c1de54643%40redhat.com.
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â For more
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â options, visit
>> >Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â https://groups.google.com/d/optout.
>> >
>> >Â Â Â Â Â Â Â Â Â Â Â --
>> >Â Â Â Â Â Â Â Â Â Â Â You received this message because you are subscribed
>> >Â Â Â Â Â Â Â Â Â Â Â to the Google Groups "kubernetes-sig-architecture"
>> >Â Â Â Â Â Â Â Â Â Â Â group.
>> >Â Â Â Â Â Â Â Â Â Â Â To unsubscribe from this group and stop receiving
>> >Â Â Â Â Â Â Â Â Â Â Â emails from it, send an email to
>> >Â Â Â Â Â Â Â Â Â Â Â kubernetes-sig-arch...@googlegroups.com
>> >Â Â Â Â Â Â Â Â Â Â Â <mailto:kubernetes-sig-arch...@googlegroups.com>.
>> >Â Â Â Â Â Â Â Â Â Â Â To post to this group, send email to
>> >Â Â Â Â Â Â Â Â Â Â Â kubernetes-si...@googlegroups.com
>> >Â Â Â Â Â Â Â Â Â Â Â <mailto:kubernetes-si...@googlegroups.com>.
>> >Â Â Â Â Â Â Â Â Â Â Â To view this discussion on the web visit
>> >Â Â Â Â Â Â Â Â Â Â Â https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3baghqi7GE5V-iGh%2B-XWTedpxakxh3v%2BkMFZHg%2BzpskwAQ%40mail.gmail.com
>> >Â Â Â Â Â Â Â Â Â Â Â <https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAB_J3baghqi7GE5V-iGh%2B-XWTedpxakxh3v%2BkMFZHg%2BzpskwAQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>> >
>> >Â Â Â Â Â Â Â Â Â Â Â For more options, visit
>> >Â Â Â Â Â Â Â Â Â Â Â https://groups.google.com/d/optout.
>> >
>> >
>> >
>>
--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CABBBJP0NjLJ8YMHjYLaURTbEmi%3Dyp%2BfvZC67uq%2B_LUTwKCPgoQ%40mail.gmail.com.
> I don't want to derail the conversation, but if these types are so tightly coupled with core Kubernetes components, why is it so important for them to be CRDs? I'm sure I'm missing something, but right now it looks to me like we're creating unnecessary pain...The argument from SIG-Architecture that makes most sense to me is that we want to strip the API Server of Kubernetes specific types so that it becomes a generic, reusable binary that can be used by other projects that want to expose a declarative API. At that point there would be no pre-installed types, everything would be installed as a CRD. To get to that vision we need to "draw the line somewhere" and stop adding more types to the API server.
No, I'm thinking about best practices around how we manage the installation of CRDs and controllers for both administrator and distribution installed extensions that may, or may not, interact with core components and the trade offs of the nessecary RBAC permissions to make it work. This includes isolating controllers to reduce the blast radius of goruotine panics, and when to hide a controller as part of a control plane or to expose it as an administrator owned Pod/StatefulSet/Deployment (assuming the K8s cluster makes a distinction between control plane and user nodes). Our vendoring issues and thier resolution deserve a thorough and separate treatment imo.ÂThanks,   -Ken
Thanks Eric.>> Isn't that what http://github.com/kubernetes/apiserver is?> Good point. Someone from sig-arch might have a more compelling answer.Spoke to Tim about this. He pointed out that we eventually want that and the in-tree Kubernetes API server to converge so we don't have to maintain it exclusively as a fork for k8s.
--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CADtktAXFL2UBVTQ1KA-m6mWhXTAwdo_E6%2BhSBJpXLkccE03YSA%40mail.gmail.com.
On Tue, Aug 21, 2018 at 8:27 PM, Saad Ali <saa...@google.com> wrote:Thanks Eric.>> Isn't that what http://github.com/kubernetes/apiserver is?> Good point. Someone from sig-arch might have a more compelling answer.Spoke to Tim about this. He pointed out that we eventually want that and the in-tree Kubernetes API server to converge so we don't have to maintain it exclusively as a fork for k8s.sounds like there is confusion around the pieces that make up kube-apiserver... the kube-apiserver is built using that library already, not maintained as a fork.These repos are published out of staging, and are the foundation of the kube-apiserver:
- k8s.io/apiserver is the generic apiserver library, which implements consistent routing and REST behavior, given a map of resources and REST storage implementations
- k8s.io/apiextensions-apiserver implements native REST storage for the CRD type, and dynamic unstructured REST storage for registered custom resources
- k8s.io/kube-aggregator implements native REST storage for the APIService type, and routes requests for arbitrary resources to the registered backends