Detaching cinder volumes

[rgonc...@gmail.com]

Hi all,
I've been using kubernetes 1.5.2 and docker 1.12.6 without any major problems. However, after upgrading to kubernetes version 1.7.2, volumes remain attached to nodes when they became unavailable (powered off), preventing the volume from being attached to another node.

According to the logs on master nodes, kubernetes is trying to detach the volume, but an error occurs (Error: Volume not attached according to node status for volume ...). Relevant log message follows:

Operation for "\"kubernetes.io/cinder/5cfe60d7-0b82-4933-9e69-aeb2cbd2521d\"" failed. No retries permitted until 2017-08-01 17:53:01.154312563 +0100 WEST (durationBeforeRetry 32s). Error: Volume not attached according to node status for volume "pvc-21b92fac-76d5-11e7-90b6-fa163e4b4ca8" (UniqueName: "kubernetes.io/cinder/5cfe60d7-0b82-4933-9e69-aeb2cbd2521d") pod "smartiot-kafka-0" (UID: "bac7a5a8-76d9-11e7-90b6-fa163e4b4ca8")
Aug 01 17:52:35 smartiot-node1.c.ptin.corppt.com kubelet[1146]: W0801

The volume remains attached to the unavaiable node, because kubernetes fails to mount the volume in another node, with the error: (Multi-Attach error for volume <volume-id>....). Accesing openstack console I can also confirm the the given volume remains attached to the unavailable host.

Any idea what might be causing this?

Thanks for any help you could provide.

Rui

[rgonc...@gmail.com]

Just adding more info...

Detaching the volume using openstack API, does not solve the problem. Kubernates remains unable to attach. Here the log:

FailedAttachVolume Multi-Attach error for volume "pvc-21b92fac-76d5-11e7-90b6-fa163e4b4ca8" Volume is already exclusively attached to one node and can't be attached to another

[Michelle Au]

Hi Rui,

Could you open an issue on github, and cc sig-storage on it.

Thanks,

Michelle

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[rgonc...@gmail.com]

Thanks Michelle.

Done: https://github.com/kubernetes/kubernetes/issues/50004