Issue mounting claims to a persistent volume within a replication controller

[Aaron Taylor]

I’m working on setting up a replication controller where each node shares a readOnly claim to a gcePersistentDisk persistent volume. I’m running into an issue where when I create the replication controller, only one node successfully is created, and the other are stuck in `ContainerCreating` status because the first node mounted the volume in Read/Write mode, prevent any of the others from doing the same. I’ve specified `readOnly: true` in the PersistentVolume spec, in the volumes for the replication controller, and in the mount for the container spec, as well as the ReadOnlyMany access mode in both the PersistentVolume and the PersistentVolumeClaim. Nothing I've tried has helped with this situation.

When I run `kubectl describe` on one of the failing pods, I get the following error messages:

Reason: FailedMount

Message: Unable to mount volumes for pod "app-486g4_default": Could not attach GCE PD "storage-1". Timeout waiting for mount paths to be created.

Reason: FailedSync

Message: Error syncing pod, skipping: Could not attach GCE PD "athena-backups-1". Timeout waiting for mount paths to be created.

The volumes from the same describe call are as follows:

Volumes:

  data:

    Type: EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium:

  backups:

    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

    ClaimName: storage-claim

    ReadOnly: true

From looking at the code here: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/gce_pd/gce_pd.go#L84-L91 it appears that I have both cases covered, but I'm apparently missing something.

Any idea what could be going wrong here?

[Mark Turansky]

Hi Aaron,

You set readOnly on the PV.Spec?  If so, that's why it didn't work.

When you use a GCEPD directly in a pod, it's readOnly bool is the correct one to use.  But when you use a volume through a PVClaim, it's the readOnly bool on the persistentVolumeClaim that is used.  You, the user, use the claim as a proxy to the volume.  The admin (who creates the PV) doesn't know when you want read/write vs. read-only.  

Try setting the bool on pod.Spec.Volumes[n].persistentVolumeClaim.ReadOnly and let me know if that works.

Mark

[Kilian Ries]

Hi Mark,

i'm having exact the same problem: i'm trying to setup a replication controller with replica=3 on two minions and my pods should access the same readonly GCEPD.

My configs are:

kind: ReplicationController

...

       volumeMounts:

        - name: nginx-data-disk

          mountPath: /etc/nginx/conf.d

      volumes:

        - name: nginx-data-disk

          persistentVolumeClaim:

             claimName: claim-nginx-data-disk       

             readOnly: true

       

kind: PersistentVolume

metadata:

  name: volume-nginx-data-disk

spec:

  capacity:

    storage: 1Gi

  accessModes:

    - ReadOnlyMany

  gcePersistentDisk:

    pdName: nginx-data-disk

    fsType: ext4

kind: PersistentVolumeClaim

metadata:

  name: claim-nginx-data-disk

spec:

  accessModes:

    - ReadOnlyMany

  resources:

    requests:

      storage: 1Gi

kubelet.log:

gce.go:416] GCE operation failed: googleapi: Error 400: The disk resource 'nginx-data-disk' is already being used by 'cluster1-default-pool-4091b8f3-wys3'

gce_util.go:187] Error attaching PD "nginx-data-disk": googleapi: Error 400: The disk resource 'nginx-data-disk' is already being used by 'cluster1-default-pool-4091b8f3-wys3'

The Replication Controller is able to setup the first pod, the second fails.

Greets

Kilian

[Ilya Kozlov]

Hi!

Exactly the same for me, Kubernetes v.1.2.0