Re: "cancel/undo/rollback" PV delete

[Tim Hockin]

Yikes!  Boy, we should really make that safer.  Maybe the API and/or CLI should fail somehow if namespace is specified for non-namespaced operations.

That said, there isn't really an undo.  It's not part of the object lifecycle, as currently defined.  Your only guaranteed option, probably, is to make new PVs as clones of the current ones (did you say which cloud?). This is probably something you have to do manually, though it may get easier in the future.

The problem is that, while you have a finalizer blocking progress, we don't know generically that no other finalizers have already run.  Something else might have been first and done some non-reversible operation.  So we can only go forward or stop, but not backwards.

I am sorry to say this.  It seems like an easy landmine to step on.  I didn't even know we supported bulk-delete like this - maybe we just shouldn't?

Tim

On Sun, Jul 29, 2018, 9:25 PM Pippin Wallace <nip...@gmail.com> wrote:

Hello Tim.

First, thank you very much for replying.  You are one of the top k8s leaders and I grateful for your contributions!

What I did was try to shoot my foot off by thinking that PV's where namespace scoped like PVC so I ran a # kubectl delete pv -n <namespace> and it tried to nuke them all.

So at this point I have all my PV's in a "Status: Terminating" but since they still have bound PVC's they are not being deleted.

What I would love to do is get the PV's back to a sane state with minimal data lose if possible.

Kind regards,

Pippin

On Sunday, July 29, 2018 at 10:17:40 PM UTC-6, Tim Hockin wrote:

Kubernetes does not have a concept of cancelling a delete.  Can you explain what you are trying to do?

On Sun, Jul 29, 2018, 8:35 PM Pippin Wallace <nip...@gmail.com> wrote:

I apologize in advance for asking here, but where should I seek assistance for how to  "cancel/undo/rollback" a PV delete?

I have asked on stackoverflow and entered a case with GCP but with over 60 PV's in a "Status: Terminating (lasts 4h)" the suggestions are not practical.

https://stackoverflow.com/questions/51585649/cancel-or-undo-deletion-of-persistent-volumes-in-kubernetes-cluster

Regards and apologizes again,

Pippin 

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-st...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/9677fd3a-90ac-4f1e-ba91-fb11f3c974f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-st...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/a581c6f3-8b62-4de7-b433-200a6d6ab2d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mo Khan]

https://github.com/kubernetes/kubernetes/pull/62167 seems relevant?  I thought we already had a check for this type of error though...

You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAO_RewYkkOrbJ%3Dxydj%2BJj50pRSbqzR25ysALCOZo5VhmBgXyxQ%40mail.gmail.com.

[Clayton]

kubectl should absolutely not allow “kubectl delete -n ‘’” to work.  We have —all-namespaces —all for that.  No one would typo the latter.

Please file a high severity issue and we’ll fix this for 1.12.

You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAO_RewYkkOrbJ%3Dxydj%2BJj50pRSbqzR25ysALCOZo5VhmBgXyxQ%40mail.gmail.com.

[David Eads]

I'll argue Jordan's side until he gets in.  When you are creating many items from multiple manifests that contain both namespaced and non-namespaced resources (imagine something creating a CRD), you may very well do `kubectl apply -f /dir/ -n my-namespace`.  When you do this, you want the namespaced things to be namespaced and non-namespaced ones created.  If we make that command start failing instead, the user will be very frustrated.  Similarly, to clean up I want to do `kubectl delete -f /dir/ -n my-namespace`.

In addition to the you "fixed" this and broke kubectl backwards compatibility, the experience and consistency of kubectl would be broken.  I feel for the user (I've been in this situation before), but when you're managing resources with cluster-admin rights you need to be very careful about what you do.

You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/3DA6A16B-EA72-4307-B568-98FC784E58B8%40gmail.com.

[Clayton Coleman]

It’s probably also time we consider requiring delete of bulk cluster scoped resource to require an additional flag to confirm the action, possibly just in interactive mode.  There’s just so few scenarios that it’s  done that the extra check is ok.

You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/3DA6A16B-EA72-4307-B568-98FC784E58B8%40gmail.com.

[Clayton Coleman]

I was talking about explicitly breaking how deleting cluster scoped resources work to be safer.

A human at a keyboard should never accidentally delete all pv.  No matter what.  There needs to be a safety harness.  If the user then gets in the habit of bypassing the safety harness, that’s fine.  But not offering the safety harness isn’t really an acceptable outcome - you deserve to be sued.

I’m not disagreeing to your point about not running as cluster admin and using sudo impersonate (that’s safer), or that we might make this something that only happens in interactive shells.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFS1MjK%3DqK2wJxuh8Ccay%3D5FdEC6F%2BesH_HJks%2B_YPm_hejmcw%40mail.gmail.com.

[Tim Hockin]

I can buy the argument that having a dir of mixed namespaced and non-namespaced is supported today, so breaking it would be bad.

I'm not arguing that any form of `kubectl <something> -f <dir>` should change.

I am agreeing with Clayton that any form of bulk delete be more safe.  User suggested that `kubectl delete pv -n <namespace>` is what caused this, but I can't seem to repro that:  `error: resource(s) were provided, but no name, label selector, or --all flag specified`

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/CAH16Sh%2BCPv0BRo_wYgbW0OwPdveEESxVppT3jf4FHGZDLv2xuw%40mail.gmail.com.

[Clayton]

You’d have this with

kubectl delete —all -n foo pv

Arguably specifying both -n and -all for cluster scoped resources should error.  

kubectl delete —all -n foo pv

error: PersistentVolumes are cluster scoped

kubectl delete —all -n foo pv,pvc

<we assume you know what you’re doing>

[Fox, Kevin M]

Yeah. Is it ever a good idea to allow such an easy bulk delete of all pv's? I'd say, no.

you can always list them out and do a for loop over them to delete if you really really wanted to nuke the universe.

Perhaps a 'do you really want to delete these items Y/N' should be added to all the delete all type operations too, unless the user specifies a -y.

Thanks,
Kevin

---

From: 'Tim Hockin' via kubernetes-sig-storage [kubernetes-...@googlegroups.com]
Sent: Sunday, July 29, 2018 9:41 PM
To: Pippin Wallace
Cc: kubernetes-sig-storage; K8s API Machinery SIG; Daniel Smith; kubernete...@googlegroups.com; Saad Ali; Jing Xu; kubernetes-sig-architecture
Subject: Re: "cancel/undo/rollback" PV delete

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/CAO_RewYkkOrbJ%3Dxydj%2BJj50pRSbqzR25ysALCOZo5VhmBgXyxQ%40mail.gmail.com.

[Tim Hockin]

We have to be careful not to break scripts that people have already written.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-storage/1A3C52DFCD06494D8528644858247BF01C15E563%40EX10MBOX03.pnnl.gov.

[Mehdy Bohlool]

Maybe something like this:

$ kubectl delete -all pv  # regardless of -n flag

*** This action marked as dangerous ***

You are about to delete 36 Cluster scoped Persistent Volumes. 

If that is not your intention, you have 30 seconds to stop this by pressing Ctrl+C ...

and count down the seconds in the message. We may want to do that for all cluster-scoped types or apply some rule on the type and number of items to be deleted. We can even make this configurable (what types are dangerous to delete, etc.) by cluster admin. This will not break any script and the message can be silenced by a flag (--sure?).

Mehdy Bohlool |

Software Engineer |

me...@google.com |

mbohlool@github

You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAO_RewZN%3DOHga1hTs4Zs%2BVbN9SyafN%3DevwjZSHH_HDvQo9D2kA%40mail.gmail.com.

[Clayton]

That will just slow scripts down, without actually making them safer.

Interactive mode vs not as the cue to prompt is pretty normal in this case (require confirmation), which doesn’t affect scripts.

I’m not sure I’ve seen a CLI that does something quite like you describe, and the time limit could be just as dangerous if someone looks away.

Our goal is identify and warn users predictably.  Confirmation before broad bulk action is usually a flag, a prompt, or similar interaction.  I’d suggest we don’t stray too far from common practice. 

You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CACd0WeE-nQKFS-J5k004W8qs0CraPZsxgku%3DOb051yh_Q6QxuQ%40mail.gmail.com.