Handling multiple access modes

[pmpa...@gmail.com]

Hello,

I'm implementing a CSI driver and have a problem with access modes. For example in GKE, I can define a Volume Claim (PVC) supporting both MULTI_NODE_READER_ONLY (ReadOnlyMany) and SINGLE_NODE_WRITER (ReadWriteOnce). If I create a single Workload (pod, or ReplicaSet with just one replica), the Volume is mounted RW, but when there are more Workloads (replicas spread on multiple nodes), it is RO. I thought it is the CO that decides which Access Mode to use, however, when I try to do the same with my driver, I always get the first mode specified in the yaml I used to provision the Volume (PVC). Do I miss something?

If I add PUBLISH_UNPUBLISH_VOLUME Controller Capability, I also get an error from External Attacher: "AttachVolume.Attach failed for volume [...] : CSI does not support ReadOnlyMany and ReadWriteOnce on the same PersistentVolume". It suggests that CSI does not support multiple access modes at all, but, as far as I know, GKE also uses CSI (GCE-PD)...

Currently I'm using v0.3.0 spec and on-premises Kubernetes 1.13 (RKE).

Regards,

Piotr

[Saad Ali]

+kubernetes-sig-storage-wg-csi since this is a Kubernetes related question.

• At volume provision time, the Kubernetes external-provisioner, will create a volume capability for each access mode specified in PVC and pass them to CSI driver in CreateVolume as a list of volume capabilities (see code).
• The driver at this point must ensure that the volume to provision can be consumed in any of the ways specified.
• At volume attach time, the Kubernetes external-attacher, will look at all access modes specified in the PV object, and select one to pass to the CSI driver in ControllerpublishVolume call as a volume capability.
• The driver at this point must ensure that the volume is attached in the way specified. 
• If a PV specifies both, ReadWriteMany always beats ReadOnlyMany (see code).
• To get around this, you can use the "ControllerPublishVolumeRequest.Readonly" field instead to figure out if the volume should be attached readonly or not (your driver has to advertise the PUBLISH_READONLY controller capability if it does). 
• That field is always set to PV.CSIVolumeSource.Readonly by the external-attacher (see code).
• But problem is that the external-provisioner doesn't currently set PV.CSIVolumeSource.Readonly.
• kubernetes/issues/70505 has a proposal to fix that last mile problem. Once this is fixed you should be able to use PV.CSIVolumeSource.Readonly as a signal for whether to attach readonly or not.

If you are willing to help fix it, we are looking for someone to work on it.

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interf...@googlegroups.com.
To post to this group, send email to container-storage-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/container-storage-interface-community/93897e8d-8f9a-4f85-85d2-9d82a1c759d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pmpa...@gmail.com]

Hello again,

thanks for your answer. However, either I still don't understand something, or there is something wrong somewhere.

My goal is not to specify "ReadOnly" explicitly, but to let CSI/Kubernetes/external-attacher decide which mode is the right one. My problem is strictly related to ReadOnlyMany and ReadWriteOnce - in case of both specified, the decision should be made on the basis of the number of workloads using the volume, like in GKE. You say that this decision is made by the Kubernetes external-attacher, so why does it say that 'CSI does not support ReadOnlyMany and ReadWriteOnce on the same PersistentVolume' (https://github.com/kubernetes-csi/external-attacher/blob/3156c24cc625b6af2d0940f70e1213a1235b7e04/pkg/controller/util.go#L182)? Is this the issue with the external-attacher then? How does it work in GKE?

W dniu środa, 24 kwietnia 2019 04:12:54 UTC+2 użytkownik Saad Ali napisał:

+kubernetes-sig-storage-wg-csi since this is a Kubernetes related question.

• At volume provision time, the Kubernetes external-provisioner, will create a volume capability for each access mode specified in PVC and pass them to CSI driver in CreateVolume as a list of volume capabilities (see code).
• The driver at this point must ensure that the volume to provision can be consumed in any of the ways specified.
• At volume attach time, the Kubernetes external-attacher, will look at all access modes specified in the PV object, and select one to pass to the CSI driver in ControllerpublishVolume call as a volume capability.
• The driver at this point must ensure that the volume is attached in the way specified. 
• If a PV specifies both, ReadWriteMany always beats ReadOnlyMany (see code).
• To get around this, you can use the "ControllerPublishVolumeRequest.Readonly" field instead to figure out if the volume should be attached readonly or not (your driver has to advertise the PUBLISH_READONLY controller capability if it does). 
• That field is always set to PV.CSIVolumeSource.Readonly by the external-attacher (see code).
• But problem is that the external-provisioner doesn't currently set PV.CSIVolumeSource.Readonly.
• kubernetes/issues/70505 has a proposal to fix that last mile problem. Once this is fixed you should be able to use PV.CSIVolumeSource.Readonly as a signal for whether to attach readonly or not.

If you are willing to help fix it, we are looking for someone to work on it.

On Tue, Apr 23, 2019 at 3:33 AM <pmpa...@gmail.com> wrote:

Hello,

I'm implementing a CSI driver and have a problem with access modes. For example in GKE, I can define a Volume Claim (PVC) supporting both MULTI_NODE_READER_ONLY (ReadOnlyMany) and SINGLE_NODE_WRITER (ReadWriteOnce). If I create a single Workload (pod, or ReplicaSet with just one replica), the Volume is mounted RW, but when there are more Workloads (replicas spread on multiple nodes), it is RO. I thought it is the CO that decides which Access Mode to use, however, when I try to do the same with my driver, I always get the first mode specified in the yaml I used to provision the Volume (PVC). Do I miss something?

If I add PUBLISH_UNPUBLISH_VOLUME Controller Capability, I also get an error from External Attacher: "AttachVolume.Attach failed for volume [...] : CSI does not support ReadOnlyMany and ReadWriteOnce on the same PersistentVolume". It suggests that CSI does not support multiple access modes at all, but, as far as I know, GKE also uses CSI (GCE-PD)...

Currently I'm using v0.3.0 spec and on-premises Kubernetes 1.13 (RKE).

Regards,

Piotr

--
You received this message because you are subscribed to the Google Groups "container-storage-interface-community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to container-storage-interface-community+unsubscribe@googlegroups.com.
To post to this group, send email to container-storage-interface-comm...@googlegroups.com.