Reconsiler unable to unmount

[Ioannis Papathanasiou]

Hi everyone,

we have implemented a CSI driver that performs LUKS encryption.

In general, upon workload Pod deletion, our Node plugin after unmounting from the staging path, it performs also the LuksClose operation.

Lately, we faced a problem during a workload Pod removal. The problem was that although, during NodeUnpublishVolume, we got the response that target path was unmounted, looking on the system logs, it seems that path could not be unmounted. For this reason, LuksClose could not be performed and volume could not be reused.

My scope is to understand

• why there was this misleading successful unmounting outcome
• why reconsiler could not unmount the volume
• if it is possible to recover from this situation in order to be able to stage the volume again

Below I include both logs from Node plugin and system in the sequence they occurred:

2023-05-08T17:08:39.198Z {"message":"NodeUnpublishVolume: trying to unmount potential mount point /var/lib/kubelet/pods/5ef4be3b-22b0-44e1-89c6-d756a5615d1d/volumes/kubernetes.io~csi/pvc-097133be-a959-46ef-a056-1c0a689e1f8d/mount","severity":"info","timestamp":"2023-05-08T17:08:39.198Z","version":"1.2.0"}

2023-05-08T19:08:39.209533+02:00 seliics02470 kubelet[25059]: I0508 19:08:39.197354   25059 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/csi-88483a6f.csi.com^fd495a15-edb4-11ed-ab36-ce2e1b705dc7\") pod \"5ef4be3b-22b0-44e1-89c6-d756a5615d1d\" (UID: \"5ef4be3b-22b0-44e1-89c6-d756a5615d1d\") "

2023-05-08T17:08:39.302Z {"message":"NodeUnpublishVolume: Unmounted target path: /var/lib/kubelet/pods/5ef4be3b-22b0-44e1-89c6-d756a5615d1d/volumes/kubernetes.io~csi/pvc-097133be-a959-46ef-a056-1c0a689e1f8d/mount","severity":"info","timestamp":"2023-05-08T17:08:39.302Z","version":"1.2.0"}

2023-05-08T17:08:39.303Z {"message":"NodeUnpublishVolume: Removed target path directory: /var/lib/kubelet/pods/5ef4be3b-22b0-44e1-89c6-d756a5615d1d/volumes/kubernetes.io~csi/pvc-097133be-a959-46ef-a056-1c0a689e1f8d/mount","severity":"info","timestamp":"2023-05-08T17:08:39.303Z","version":"1.2.0"}

2023-05-08T19:08:39.305442+02:00 seliics02470 kubelet[25059]: I0508 19:08:39.305339   25059 operation_generator.go:856] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/csi.com^fd495a15-edb4-11ed-ab36-ce2e1b705dc7" (OuterVolumeSpecName: "datadir") pod "5ef4be3b-22b0-44e1-89c6-d756a5615d1d" (UID: "5ef4be3b-22b0-44e1-89c6-d756a5615d1d"). InnerVolumeSpecName "pvc-097133be-a959-46ef-a056-1c0a689e1f8d". PluginName "kubernetes.io/csi", VolumeGidValue ""

2023-05-08T19:08:39.306935+02:00 seliics02470 kubelet[25059]: E0508 19:08:39.306847   25059 reconciler.go:189] "operationExecutor.UnmountVolume failed (controllerAttachDetachEnabled true) for volume \"datadir\" (UniqueName: \"kubernetes.io/csi/csi.com^fd495a15-edb4-11ed-ab36-ce2e1b705dc7\") pod \"5ef4be3b-22b0-44e1-89c6-d756a5615d1d\" (UID: \"5ef4be3b-22b0-44e1-89c6-d756a5615d1d\") : UnmountVolume.NewUnmounter failed for volume \"datadir\" (UniqueName: \"kubernetes.io/csi/88483a6f.csi.com^fd495a15-edb4-11ed-ab36-ce2e1b705dc7\") pod \"5ef4be3b-22b0-44e1-89c6-d756a5615d1d\" (UID: \"5ef4be3b-22b0-44e1-89c6-d756a5615d1d\") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/5ef4be3b-22b0-44e1-89c6-d756a5615d1d/volumes/kubernetes.io~csi/pvc-097133be-a959-46ef-a056-1c0a689e1f8d/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/5ef4be3b-22b0-44e1-89c6-d756a5615d1d/volumes/kubernetes.io~csi/pvc-097133be-a959-46ef-a056-1c0a689e1f8d/vol_data.json]: open /var/lib/kubelet/pods/5ef4be3b-22b0-44e1-89c6-d756a5615d1d/volumes/kubernetes.io~csi/pvc-097133be-a959-46ef-a056-1c0a689e1f8d/vol_data.json: no such file or directory" err="UnmountVolume.NewUnmounter failed for volume \"datadir\" (UniqueName: \"kubernetes.io/csi/88483a6f.csi.com^fd495a15-edb4-11ed-ab36-ce2e1b705dc7\") pod \"5ef4be3b-22b0-44e1-89c6-d756a5615d1d\" (UID: \"5ef4be3b-22b0-44e1-89c6-d756a5615d1d\") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/5ef4be3b-22b0-44e1-89c6-d756a5615d1d/volumes/kubernetes.io~csi/pvc-097133be-a959-46ef-a056-1c0a689e1f8d/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/5ef4be3b-22b0-44e1-89c6-d756a5615d1d/volumes/kubernetes.io~csi/pvc-097133be-a959-46ef-a056-1c0a689e1f8d/vol_data.json]: open /var/lib/kubelet/pods/5ef4be3b-22b0-44e1-89c6-d756a5615d1d/volumes/kubernetes.io~csi/pvc-097133be-a959-46ef-a056-1c0a689e1f8d/vol_data.json: no such file or directory"

many thanks,

Ioannis