Hi all,First of all, thank you for this project, it is very useful for anything stateful on k8s. I'm also glad to run into folks from good old oVirt again :)
I wanted to ask, since I didn't see an obvious answer in the docs, whether there is anything done about volumeattachments beyond actually deleting and recreating the nodes. IF the volumeattachment is removed early, that would allow for avoiding the dreaded multi-attach error which prevents workloads that consume RWO volumes from being rescheduled.
Things work as they are, but the rescheduling takes much less time if a VA is removed early on in the fencing flow.
Cheers,Dan
--Cheers,Dan
You received this message because you are subscribed to the Google Groups "medik8s" group.
To unsubscribe from this group and stop receiving emails from it, send an email to medik8s+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/medik8s/ae819061-6e14-4f13-b412-5394de509728n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
On Thu, Sep 9, 2021 at 5:19 AM Andrew Beekhof <abee...@redhat.com> wrote:On Wednesday, 8 September 2021 at 11:53:20 pm UTC+10 dya...@gmail.com wrote:Hi all,First of all, thank you for this project, it is very useful for anything stateful on k8s. I'm also glad to run into folks from good old oVirt again :)Glad you're finding it useful :)
I wanted to ask, since I didn't see an obvious answer in the docs, whether there is anything done about volumeattachments beyond actually deleting and recreating the nodes. IF the volumeattachment is removed early, that would allow for avoiding the dreaded multi-attach error which prevents workloads that consume RWO volumes from being rescheduled.
Things work as they are, but the rescheduling takes much less time if a VA is removed early on in the fencing flow.Correct me if I'm wrong, but my understanding is that once the Node CR is removed, then k8s ignores any existing volume attachments in the same way that it understands that any Pods associated with the Node are also not running.The Node CR is removed as soon as the node is put into a safe state, so I'd not expect there to be an earlier opportunity.Hi. It's a bit more complicated: pods stay in status "Terminating" for some time after the node was deleted, and during that time volumes are not released yet.We released a new Poison Pill operator version just recently, which at least waits long enough before re-creating the node, so that all pods terminate and volumes are released.
--Cheers,Marc--Cheers,Dan
You received this message because you are subscribed to the Google Groups "medik8s" group.
To unsubscribe from this group and stop receiving emails from it, send an email to medik8s+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/medik8s/ae819061-6e14-4f13-b412-5394de509728n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to a topic in the Google Groups "medik8s" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/medik8s/_esi9rj9mTU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to medik8s+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/medik8s/CAH5Q-kW0y0qBzKQUjM%2Bi3_FGSPFPkntN%2B911NSjcX2%3DdsCrkXA%40mail.gmail.com.
On Thu, Sep 9, 2021 at 3:28 AM Marc Sluiter <mslu...@redhat.com> wrote:On Thu, Sep 9, 2021 at 5:19 AM Andrew Beekhof <abee...@redhat.com> wrote:On Wednesday, 8 September 2021 at 11:53:20 pm UTC+10 dya...@gmail.com wrote:Hi all,First of all, thank you for this project, it is very useful for anything stateful on k8s. I'm also glad to run into folks from good old oVirt again :)Glad you're finding it useful :)
I wanted to ask, since I didn't see an obvious answer in the docs, whether there is anything done about volumeattachments beyond actually deleting and recreating the nodes. IF the volumeattachment is removed early, that would allow for avoiding the dreaded multi-attach error which prevents workloads that consume RWO volumes from being rescheduled.
Things work as they are, but the rescheduling takes much less time if a VA is removed early on in the fencing flow.Correct me if I'm wrong, but my understanding is that once the Node CR is removed, then k8s ignores any existing volume attachments in the same way that it understands that any Pods associated with the Node are also not running.The Node CR is removed as soon as the node is put into a safe state, so I'd not expect there to be an earlier opportunity.Hi. It's a bit more complicated: pods stay in status "Terminating" for some time after the node was deleted, and during that time volumes are not released yet.We released a new Poison Pill operator version just recently, which at least waits long enough before re-creating the node, so that all pods terminate and volumes are released.This looks very good, but my point is about something else here. When I use a network attached RWO volume, the rescheduling of a deployment happens very quickly, when a node is detected as NotReady. This is where the new workload gets stuck unable to start because of the dreaded Multi-Attach error, which persists until the VA is removed. This takes a pretty long time. From my days at oVirt I remember us making sure a typical VM downtime wouldn't exceed 2 minutes, and even that was considered borderline excessive for a proper HA system. Here we are talking about triple that time and easily more.
I've tried a very simple test - a script that monitors the nodes status, and when a node is NotReady for over 1 minute, it deletes the volumeattachments on it. With this in place, my workloads get rescheduled nicely in under 90 seconds, keeping my A reasonably H.
So I'm arguing here that it shouldn't be enough to just delete and recreate the node, locking mechanisms such as volumeattachment need to be explicitly removed as soon as possible, in order to unblock stateful workload rescheduling quickly. When a node is deleted, the VA eventually gets removed, but it takes a very long time.
Hope it all makes sense to you guys?
----Cheers,Marc--Cheers,Dan
You received this message because you are subscribed to the Google Groups "medik8s" group.
To unsubscribe from this group and stop receiving emails from it, send an email to medik8s+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/medik8s/ae819061-6e14-4f13-b412-5394de509728n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to a topic in the Google Groups "medik8s" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/medik8s/_esi9rj9mTU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to medik8s+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/medik8s/CAH5Q-kW0y0qBzKQUjM%2Bi3_FGSPFPkntN%2B911NSjcX2%3DdsCrkXA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "medik8s" group.
To unsubscribe from this group and stop receiving emails from it, send an email to medik8s+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/medik8s/CALLXwb6zfZYgQFOBSgZUKW_8S8tK6dR0-H3ox1tTtFBf8kX6Mw%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/medik8s/CAH5Q-kWC_3Yp8v8gYT47Hhpc7M35XeA7wJCkkg%3DV9O%2B4XGpWng%40mail.gmail.com.