csi-controller: Simple, single, intelligent controller for CSI for Kubernetes

[Luis Pabon]

Hi all,

  Last Monday I discussed at the Kubernetes-CSI community meeting a PoC I was working on. The goal of the PoC is to create a single CSI controller side car to replace all other side cars on deployment. At Portworx, we realized that we had placed a lot of intelligence in our installer to determine which side-car with a specific version to use at a specific version of Kubernetes. We realized that separating the installer intelligence from the system would make it difficult to install.

  Instead, we propose csi-controller[1], a simple, single, intelligent side-car controller for CSI. It vendors in the sources for the side cars and interrogates Kubernetes and the CSI driver to determine which side car to load. We propose that this model makes it simpler to deploy, since it automatically determines what to do.

Please feel free to try it out in your deployment. Here is an example output from the csi-controller logs:

I1028 23:17:05.204711       1 feature_gate.go:216] feature gates: &{map[]}
I1028 23:17:05.204831       1 main.go:165] Version: v0.1-5-g0df2487
W1028 23:17:05.222373       1 main.go:229] Skipping CSI NODE CRD check
I1028 23:17:05.222407       1 connection.go:151] Connecting to unix:///csi/csi.sock
I1028 23:17:05.224130       1 common.go:111] Probing CSI driver for readiness
I1028 23:17:05.225893       1 main.go:251] Detected CSI driver pxd.portworx.com
I1028 23:17:05.229003       1 attacher.go:59] Kubernetes 1.13.x detected which requires the attacher process even if the CSI driver does not support it
I1028 23:17:05.229217       1 provisioner.go:44] Driver pxd.portworx.com supports provisioning
I1028 23:17:05.231177       1 controller.go:680] Using saving PVs to API server in background
I1028 23:17:05.231215       1 snapshotter.go:44] Driver pxd.portworx.com supports snapshots
I1028 23:17:05.243319       1 resizer.go:38] Driver pxd.portworx.com supports resize
I1028 23:17:05.243347       1 connection.go:151] Connecting to unix:///csi/csi.sock
I1028 23:17:05.244133       1 common.go:111] Probing CSI driver for readiness
I1028 23:17:05.247155       1 main.go:327] Using endpoints for leader election in Kubernetes version 1.13.x
I1028 23:17:05.247363       1 leaderelection.go:241] attempting to acquire leader lease  kube-system/csi-controller-leader-pxd-portworx-com...
I1028 23:17:24.892907       1 leader_election.go:172] new leader detected, current leader: px-csi-ext-77ffb6d4f9-7ldtr

I will be bringing this up at the Kubernetes SIG-Storage F2F in Kubecon San Diego for further discussion, but I would like to see if any of you can try it out on your systems and provide some feedback.

- Luis

[1] https://github.com/lpabon/csi-controller

[Gorka Eguileor]

On 28/10, Luis Pabon wrote:
> Hi all,
> Last Monday I discussed at the Kubernetes-CSI community meeting a PoC I
> was working on. The goal of the PoC is to create a single CSI controller
> side car to replace all other side cars on deployment. At Portworx, we
> realized that we had placed a lot of intelligence in our installer to
> determine which side-car with a specific version to use at a specific
> version of Kubernetes. We realized that separating the installer
> intelligence from the system would make it difficult to install.
> Instead, we propose csi-controller[1], a simple, single, intelligent
> side-car controller for CSI. It vendors in the sources for the side cars
> and interrogates Kubernetes and the CSI driver to determine which side car
> to load. We propose that this model makes it simpler to deploy, since it
> automatically determines what to do.
>

Hi Luis,

I think that's a great idea. We were facing the same situation
in Ember-CSI and it was pailful.

I love the auto detection of features in your sidecar. :-)

>
> I will be bringing this up at the Kubernetes SIG-Storage F2F in Kubecon San
> Diego for further discussion, but I would like to see if any of you can try
> it out on your systems and provide some feedback.
>
> - Luis
>
> [1] https://github.com/lpabon/csi-controller

I will definitely try it and let you know how it goes.

Cheers,
Gorka.

[Xiangqian Yu]

Hi Luis,

Is there any document I can get more color for your POC? I missed last community meeting. thanks.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-storage-wg-csi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-stora...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage-wg-csi/20191029124556.jyhvp7zm2wjd2inq%40localhost.

[Luis Pabon]

Hi Xiangqian,

  I do not have a document, all I have is the README.md in the repo. 

- Luis

[Ben Swartzlander]

On 10/28/19 7:46 PM, Luis Pabon wrote:
> Hi all,
>   Last Monday I discussed at the Kubernetes-CSI community meeting a PoC
> I was working on. The goal of the PoC is to create a single CSI
> controller side car to replace all other side cars on deployment. At
> Portworx, we realized that we had placed a lot of intelligence in our
> installer to determine which side-car with a specific version to use at
> a specific version of Kubernetes. We realized that separating the
> installer intelligence from the system would make it difficult to install.
>   Instead, we propose csi-controller[1], a simple, single, intelligent
> side-car controller for CSI. It vendors in the sources for the side cars
> and interrogates Kubernetes and the CSI driver to determine which side
> car to load. We propose that this model makes it simpler to deploy,
> since it automatically determines what to do.

I can see this saving a little bit of RAM at runtime, and also saving
some hassle for CSI driver authors. I still prefer the model where
Kubernetes handles all of the sidecar stuff and CSI drivers are ignorant
about the existence of sidecars or RBAC or anything else they shouldn't
need to think about.

Does this combine all of the individual sidecar leader elections into
one winner-take-all leader election? That would also be a win as far as
I'm concerned.

> Please feel free to try it out in your deployment. Here is an example
> output from the csi-controller logs:
>
> I1028 23:17:05.204711       1 feature_gate.go:216] feature gates: &{map[]}
> I1028 23:17:05.204831       1 main.go:165] Version: v0.1-5-g0df2487
> W1028 23:17:05.222373       1 main.go:229] Skipping CSI NODE CRD check
> I1028 23:17:05.222407       1 connection.go:151] Connecting to
> unix:///csi/csi.sock
> I1028 23:17:05.224130       1 common.go:111] Probing CSI driver for
> readiness
> I1028 23:17:05.225893       1 main.go:251] Detected CSI driver

> pxd.portworx.com <http://pxd.portworx.com>

> I1028 23:17:05.229003       1 attacher.go:59] Kubernetes 1.13.x detected
> which requires the attacher process even if the CSI driver does not
> support it
> I1028 23:17:05.229217       1 provisioner.go:44] Driver pxd.portworx.com

> <http://pxd.portworx.com> supports provisioning

> I1028 23:17:05.231177       1 controller.go:680] Using saving PVs to API
> server in background
> I1028 23:17:05.231215       1 snapshotter.go:44] Driver pxd.portworx.com

> <http://pxd.portworx.com> supports snapshots

> I1028 23:17:05.243319       1 resizer.go:38] Driver pxd.portworx.com

> <http://pxd.portworx.com> supports resize

> I1028 23:17:05.243347       1 connection.go:151] Connecting to
> unix:///csi/csi.sock
> I1028 23:17:05.244133       1 common.go:111] Probing CSI driver for
> readiness
> I1028 23:17:05.247155       1 main.go:327] Using endpoints for leader
> election in Kubernetes version 1.13.x
> I1028 23:17:05.247363       1 leaderelection.go:241] attempting to
> acquire leader lease  kube-system/csi-controller-leader-pxd-portworx-com...
> I1028 23:17:24.892907       1 leader_election.go:172] new leader
> detected, current leader: px-csi-ext-77ffb6d4f9-7ldtr
>
> I will be bringing this up at the Kubernetes SIG-Storage F2F in Kubecon
> San Diego for further discussion, but I would like to see if any of you
> can try it out on your systems and provide some feedback.
>
> - Luis
>
> [1] https://github.com/lpabon/csi-controller
>

> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-storage-wg-csi" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kubernetes-sig-stora...@googlegroups.com

> <mailto:kubernetes-sig-stora...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/kubernetes-sig-storage-wg-csi/CAEvoFsEmBMyri3in1rk7xHGipj%3DZE%2B-XT0rDQE1FTe8UDeHyQA%40mail.gmail.com
> <https://groups.google.com/d/msgid/kubernetes-sig-storage-wg-csi/CAEvoFsEmBMyri3in1rk7xHGipj%3DZE%2B-XT0rDQE1FTe8UDeHyQA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Luis Pabon]

Ben,

  Yeah, All sidecars objects are now in a single leader election. Which is one of the reasons we wanted to do this because for Kubernetes 1.13.x some side cars had one way of doing it and some side cars had other ways. Then in 1.14 all could use leases style.

Long story short: Yes, they are all together now, nice and happy ;-)

- Luis

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-stora...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage-wg-csi/84c7657f-60a7-d479-9323-b3e160518870%40swartzlander.org.

[Luis Pabon]

Oh, and that's true Ben. With this csi-controller, it could be possible to have it automatically setup everything if it had an operator or something like that to deploy it on detection of a CSIDriver. Something we can discuss when we meet at the F2F.

- Luis

[Michelle Au]

I think this is a good step in the right direction towards simplifying the deployment model for vendors.  How do things like K8s cluster upgrades work?

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage-wg-csi/CAEvoFsEFAHbci7WbmcX4wB%3Dd%3DZqVK7Htssx6OHhgi6HiwM__-w%40mail.gmail.com.

[Luis Pabon]

With the csi-controller managed by the storage vendors, I think it would work as they do today, where the csi-controller side-car may or may not need to be updated.

I do have another email/proposal following what Ben said, around having this done by Kubernetes, which is a conversation we have been discussing for a while now. I will send another email for that discussion, which is separate, but related to this one.

- Luis

[Luis Pabon]

The second proposal for a better integration between Kuberentes and CSI drivers is here: 

https://groups.google.com/d/msg/kubernetes-sig-storage-wg-csi/HB5_8uLiq74/UYzN0jEhAAAJ

That way we can separate both proposals.

- Luis

[Michelle Au]

Hi Luis,

Thinking about this some more, I think consolidating the sidecars into one has a lot of nice benefits:

* aligning all the leaders to the same container

* sharing informers across all the sidecars

* simplifying the sharing of common code and test utilities (being able to eliminate csi-lib-utils and csi-release-tools)

However, I don't think our group currently has the bandwidth right now to maintain and test this in our CI, along with the work of deprecating and refactoring all our sidecars and test/release infrastructure to use this, while we're also still making substantial changes to some of the newer sidecars for new features. We already have many repos that have been languishing in kubernetes-csi because we don't have owners committed to maintaining them to our standards in the long term.

I also have some doubts on how the automatic K8s version detection will work when dealing with K8s alpha->beta->GA APIs and importing multiple versions of the same sidecar.

For the time being, I'm in favor of keeping the repo where it currently is, seeing if the model works well longer term, and we can consider adopting it into the SIG once it gains adoption, community support and infrastructure, and community commitment to restructuring our current sidecar infrastructure to support this.

-Michelle

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage-wg-csi/CAEvoFsG1TwYYBd4kw2ACgJwdgEk4d71-vPeVMnTD0dyr9ztxpQ%40mail.gmail.com.

[Luis Pabon]

Hi Michelle,

  That is fine with me. We can continue working on it on my repo in the meantime. Also, I'm not sure importing multiple versions of the same sidecar would work either. 

We'll mature the csi-controller on my repo and provide feedback again, hopefully at the next kubecon.

- Luis