Removing Heketi/GlusterFS from kubernetes/kubernetes ASAP!

[Davanum Srinivas]

Folks,

Please note that Heketi is in "Deep Maintenance" [1], please see [2] for the latest push back from the Heketi team on changes we would need to keep vendoring heketi into kubernetes/kubernetes.

We cannot keep heketi in our code base as heketi itself is literally going away.

I'd like to propose immediate deprecation and removal of heketi from our code base by invoking the exception clause[3]. Preferably deprecation in 1.25 and removal in 1.26 (or) even immediately removing heketi from 1.25 itself (which would be my preference).

Thanks,

Dims

[1] https://github.com/heketi/heketi#maintenance-status

[2] https://github.com/heketi/heketi/pull/1904#issuecomment-1197100513

[3] https://kubernetes.io/docs/reference/using-api/deprecation-policy/#exceptions

--

Davanum Srinivas :: https://twitter.com/dims

[Saad Ali]

Any thoughts from SIG-Arch on https://groups.google.com/g/kubernetes-sig-storage/c/smk6M2tH_ek
GlusterFs is the primary consumer of heketi.
Maybe we announce deprecation of GlusterFS in 1.25 and removal of GlusterFS in 1.26. 

Otherwise GlusterFS will need to be refactored to remove/replace this dependency -- which is a non-trivial amount of work.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-st...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/CANw6fcHubNqObhvaQBd88W%2B5tf%2Bhtt1BARuPcC8CzMhG2jvQRA%40mail.gmail.com.

[Davanum Srinivas]

Saad,

my personal hat on : +1 to "Maybe we announce deprecation of GlusterFS in 1.25 and removal of GlusterFS in 1.26" as it is the simplest way without creating more work for ourselves.

Let's wait for others to chime in as well.

-- Dims

[Saad Ali]

Sounds good, thanks Dims! 

[Michelle Au]

The challenge is there is not an obvious CSI driver replacement for gluster, and there doesn't seem to be any interest from any gluster maintenaners to pursue CSI migration.

On Wed, Jul 27, 2022 at 5:47 PM Kevin Fox <kfox...@gmail.com> wrote:

Isn't the plan to replace the internal driver with the external csi driver anyway?

Thanks,

Kevin

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/CABBBJP35sNLnNaAZ29Gzz0%3DmtEKiM0Maw2Hq%3D8hQGZp357P6QQ%40mail.gmail.com.

[Saad Ali]

Thanks for the input Humble! 

Dims, any objections to Humble's proposal of announcing GlusterFS deprecation in 1.25, but delaying removal to 1.27?

On Wed, Jul 27, 2022 at 8:35 PM Humble Chirammal <hchi...@redhat.com> wrote:

Adding sig-arch and other CCs to my previous thread.

---------- Forwarded message ---------
From: Humble Chirammal <hchi...@redhat.com>
Date: Thursday, July 28, 2022 at 8:50:46 AM UTC+5:30
Subject: Re: Removing Heketi/GlusterFS from kubernetes/kubernetes ASAP!
To: kubernetes-sig-storage <kubernetes-...@googlegroups.com>

We discussed possibilities of migration via https://github.com/kubernetes/kubernetes/issues/100897 . The main challenge is lack of CSI driver

which is maintained at present, we had one ( https://github.com/gluster/gluster-csi-driver/releases ), but unfortunately not maintained any more.

I agree with Dims that, it is difficult to continue with an unmaintained dependency in place like Heketi.

Being author and maintainer for glusterfs dynamic provisioner I ACK for deprecation of GlusterFS in 1.25,  however regarding complete removal, I would

like to propose 1.27 release if possible . The reason being glusterfs is one of the very first dynamic provisioner which got into kube v1.4 and I still get pings

from the users on the same or there are good number of users still using it. imo, Having a couple of releases available for them to migrate ..etc would be a

good take from our community pov and also a better user experience which is important especially when we are talking about storage/data.  

Based on the consensus of removal release, I will adjust the deprecation PRs .

[Davanum Srinivas]

Saad,

The problem is that heketi vendoring gets in the way of updating other important things (and cleaning up dependencies that it uses). 

I'd prefer to do the removal earlier than later and don't want to wait till 1.27.

How about we put out a call for actual users as humble suggests that are out there via an email to k-dev. If anyone responds, then we go with 1.27, if no one responds we do 1.26.

thanks,

Dims

[Davanum Srinivas]

Humble,

for 1.25, we can definitely do the deprecation and say "and this feature will be removed soon after in a subsequent release".

In the email to k-dev, please ask folks for info on: 

- what version of kubernetes they are on

- how often they upgrade

- what vendor or distro they are using (if they are not using what the community releases)

This will help with the decision making process.

thanks,

Dims

On Thu, Jul 28, 2022 at 7:15 AM Humble Chirammal <hchi...@redhat.com> wrote:

Sounds good to me. 

Lets send the notification of deprecation to k-dev then based on the response, decide on the removal version.

Do we have to close the survey before code freeze of 1.25 , which is a couple days away ?

--

Cheers,
Humble

[Davanum Srinivas]

Amar,

Understood. Please note that this request from me was not about CSI-friendliness specifically. It was about our use of the heketi (vendored) code in kubernetes/kubernetes. 

Given that the team that supports the code does not want to deal with our use of said code, we are painted into a corner and we don't want to be in this situation.

thanks,

Dims

On Thu, Jul 28, 2022 at 7:28 AM Amar Tumballi <ama...@gmail.com> wrote:

On Thu, 28 Jul, 2022, 6:31 am 'Michelle Au' via kubernetes-sig-storage, <kubernetes-...@googlegroups.com> wrote:

The challenge is there is not an obvious CSI driver replacement for gluster, and there doesn't seem to be any interest from any gluster maintenaners to pursue CSI migration.

Kadalu [1] project provides CSI and an operator to manage the gluster resources.

But providing an online migration for users is not a focus for the team for now. All heketi users can use kadalu with a minor downtime as they need to restart the server process with kadalu and start PVC mount through CSI.

As a glusterfs maintainer, there is no easy way to get it's management layer changed to be CSI friendly soon. And hence I do vote for depreciation/ removal of glusterfs from kubernetes codebase by 1.26, with announcement in 1.25.

1- https://github.com/kadalu/kadalu

Regards,

Amar (amarts)

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/CAAG1FwheHTrYWCXo7HGYRzQmwjyV4F9%2BJn54Uh7En4Nm7QXR9Q%40mail.gmail.com.

[Saad Ali]

Thanks for the input and guidance here. SIG Storage met today, see notes.

We agreed that we will plan to deprecate GlusterFS in 1.25 and remove code in 1.26 with the option of adjusting plan based on k-dev feedback.

Humble will lead this effort.

[Davanum Srinivas]

Thanks to all of you!