Proposal for Kubernetes CSI Controller

[Luis Pabon]

In the Kubernetes-CSI community, for some time now, we have been trying to determine how to make the integration between CSI drivers and Kubernetes easier. Now with the csi-controller[1] proposal, we could have an opportunity to make things easier when deploying CSI drivers by letting Kubernetes manage the controller instead of the storage vendor.

This could be done by the following:

1. Integrate the csi-controller functionality into Kubernetes. The csi-controller[1] vendors-in each of the CSI side-cars and manages their execution and setup automatically depending on the Kubernetes version and the information from the CSI driver. The implementation could be including the source into existing containers or create a new one. It could just be deployed as it is today, as a deployment of replica 3.
2. Add a system RBAC for the csi-controller.
3. Watch for CSIDriver objects. When a CSIDriver object is found, the csi-controller would spawn new services for the CSI driver. The CSIDriver object may provide specific information to the csi-controller on how to setup the services. For example: spec.snapshotter.nameprefix[2]

With this model, Kubernetes would take over the responsibility of managing the CSI services instead of the storage vendor. This would greatly simplify the deployment of the CSI drivers onto any version of Kubernetes since the csi-controller would know exactly how to manage the interaction between that specific version of Kubernetes and the CSI driver.

Let me know what you think. This proposal may or may not be possible. I'm still not sure about #1 above, but I am sure others can come up with a great solution. We can also discuss this at the SIG-Storage F2F meeting in Kubecon San Diego.

- Luis

[1] https://groups.google.com/d/msg/kubernetes-sig-storage-wg-csi/qZvuOlCME5Y/QWrBb9-bBwAJ
[2] https://github.com/lpabon/csi-controller/blob/master/cmd/csi-controller/main.go#L143

[kfox11...@gmail.com]

On Wednesday, October 30, 2019 at 11:46:29 AM UTC-7, Luis Pabon wrote:

3. Watch for CSIDriver objects. When a CSIDriver object is found, the csi-controller would spawn new services for the CSI driver. The CSIDriver object may provide specific information to the csi-controller on how to setup the services. For example: spec.snapshotter.nameprefix[2]

This is very interesting. csi is supposed to be agnostic to which orchestrator it is installed in, so everything should be in place to support this.

I'm for it in general. But may need to be optional or at least fairly flexible in some cases. Perhaps the csi driver needs configuration that is somewhat hardware specific on particular pools of nodes. Like, two different instances of the driver daemonset with different config options but is the same driver so only one CSIDriver object. The api would have to be flexable enough to handle these sorts of things. But having k8s manage the plumbing itself would be much nicer then having the storage vendors have to I think.

Thanks,
Kevin

[Jan Safranek]

On 30/10/2019 19:46, Luis Pabon wrote:
> In the Kubernetes-CSI community, for some time now, we have been trying
> to determine how to make the integration between CSI drivers and
> Kubernetes easier. Now with the csi-controller[1] proposal, we could
> have an opportunity to make things easier when deploying CSI drivers by
> letting Kubernetes manage the controller instead of the storage vendor.
>
> This could be done by the following:
>
> 1. Integrate the csi-controller functionality into Kubernetes. The
> csi-controller[1] vendors-in each of the CSI side-cars and manages their
> execution and setup automatically depending on the Kubernetes version
> and the information from the CSI driver. The implementation could be
> including the source into existing containers or create a new one. It
> could just be deployed as it is today, as a deployment of replica 3.

I am not sure I understand. Do you propose to create a new external
sidecar, csi-controller, or do you want a new controller in
kube-controller-manager? From csi-controller code it seems it's a new
sidecar.

> 2. Add a system RBAC for the csi-controller.
> 3. Watch for CSIDriver objects. When a CSIDriver object is found, the
> csi-controller would spawn new services for the CSI driver. The
> CSIDriver object may provide specific information to the csi-controller
> on how to setup the services. For example: spec.snapshotter.nameprefix[2]

That's great! Especially if it supported shared informers, so we could
save a lot of memory.

> With this model, Kubernetes would take over the responsibility of
> managing the CSI services instead of the storage vendor. This would
> greatly simplify the deployment of the CSI drivers onto any version of
> Kubernetes since the csi-controller would know exactly how to manage the> interaction between that specific version of Kubernetes and the CSI
driver.

I still must be missing something, because I do not see how it moves
management of the sidecar from storage vendor to Kubernetes vendor -
it's yet another sidecar. How a thing that installs a CSI driver (most
probably shipped by driver vendor) knows which sidecar images/versions a
Kubernetes vendor uses? How Kubernetes vendor tells this
installer/operator that there is a new updated sidecar, possibly with a
CVE fixed, so the installer updates + restarts its deployments /
daemonsets? We've reduced the problem from 5 sidecars to 2 (csi-sidecar
+ node-registrar), but the problem stays the same.

Jan

[Fox, Kevin M]

I think the idea is that the CSIDriver object would be loaded into the cluster, and Kubernetes would notice and fire up the daemonsets/statefulsets with the specified container and sidecars needed to function properly. Upgrades to the driver would be updating the CSIDriver object. So the storage vendor would only provide the container and the CSIDriver object.

Thanks,
Kevin
________________________________________
From: kubernetes-sig...@googlegroups.com [kubernetes-sig...@googlegroups.com] on behalf of Jan Safranek [jsaf...@redhat.com]
Sent: Thursday, October 31, 2019 3:09 AM
To: Luis Pabon; kubernetes-sig-storage; kubernetes-sig-storage-wg-csi
Cc: container-storage-interface-community
Subject: Re: Proposal for Kubernetes CSI Controller

Jan

--
You received this message because you are subscribed to a topic in the Google Groups "kubernetes-sig-storage-wg-csi" group.
To unsubscribe from this topic, visit https://protect2.fireeye.com/v1/url?k=03df73fd-5f6a4c44-03df59e8-0cc47adc5fce-817ef1159ab52040&q=1&e=0c75ed23-147d-4f3a-8f9a-f6fb1ffbd14e&u=https%3A%2F%2Fgroups.google.com%2Fd%2Ftopic%2Fkubernetes-sig-storage-wg-csi%2FHB5_8uLiq74%2Funsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-sig-stora...@googlegroups.com.
To view this discussion on the web visit https://protect2.fireeye.com/v1/url?k=fbb91555-a70c2aec-fbb93f40-0cc47adc5fce-447f6f9777feb443&q=1&e=0c75ed23-147d-4f3a-8f9a-f6fb1ffbd14e&u=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-sig-storage-wg-csi%2F3e96bb7b-df41-28e4-c12a-4ab652c6f209%2540redhat.com.