Dynamic volume provisioning not working with AWS EBS

1,095 views
Skip to first unread message

jamelse...@gmail.com

unread,
Aug 24, 2017, 9:21:21 PM8/24/17
to Kubernetes user discussion and Q&A
I am trying to create a dynamic storage claim. After deploying my StorageClass and PersistentVolumeClaim, the EBS volume is actually created and appears in my AWS management console, but for some reason, the PersistentVolumeClaim is always stuck in "Pending" state. It never actually binds to the EBS volume that was created (in the same Availability Zone). I tried to deploy a Pod that uses the PersistentVolumeClaim, but I just get the following error: "SchedulerPredicates failed due to PersistentVolumeClaim is not bound: 'xxx', which is unexpected." A PersistentVolume is never created in the namespace of the cluster. Oddly enough, the EBS volume in AWS is removed when I delete the PersistentVolumeClaim, which implies to me that it does have some connection even though the PVC state never becomes "Bound". Any help will be greatly appreciated.

Here is my StorageClass:

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
name: standard
namespace: default
annotations:
storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
zone: us-west-2a

Here is my PersistentVolumeClaim:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: hadoop-data-01
namespace: default
annotations:
volume.beta.kubernetes.io/storage-class: "standard"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi

And here is the Pod I attempt to deploy to use the PVC:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: hadoop-datanode-01
spec:
replicas: 1
template:
metadata:
labels:
app: hadoop-datanode-01
spec:
hostname: hadoop-datanode-01
volumes:
- name: hadoop-data-01
persistentVolumeClaim:
claimName: hadoop-data-01
containers:
- name: hadoop-datanode-01
image: uhopper/hadoop-datanode:2.7.2
env:
- name: CORE_CONF_fs_defaultFS
value: hdfs://hadoop-namenode:8020
ports:
- containerPort: 50075
name: fs
volumeMounts:
- mountPath: /hadoop/dfs/data
name: hadoop-data-01

Michelle Au

unread,
Aug 24, 2017, 9:37:06 PM8/24/17
to Kubernetes user discussion and Q&A
What does 'kubectl describe' on the PVC show?


--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

jamelse...@gmail.com

unread,
Aug 24, 2017, 10:11:23 PM8/24/17
to Kubernetes user discussion and Q&A
Thanks! I completely spaced on checking that. Found the reason in the output and fixed issue. Works now.

Really appreciate your help!

Rodrigo Campos

unread,
Aug 25, 2017, 9:36:22 AM8/25/17
to kubernet...@googlegroups.com
Oh, cool. Just so we know, what was the issue? :)



Thanks!

On Thursday, August 24, 2017, <jamelse...@gmail.com> wrote:
Thanks! I completely spaced on checking that. Found the reason in the output and fixed issue. Works now.

Really appreciate your help!

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

jamelse...@gmail.com

unread,
Aug 25, 2017, 4:23:38 PM8/25/17
to Kubernetes user discussion and Q&A
Sure. Turns out the node was trying to tag the provisioned EBS volume but was lacking the corresponding AWS role permissions to do that. Once I updated the role with the right permissions, everything worked.

Rodrigo Campos

unread,
Aug 25, 2017, 4:40:02 PM8/25/17
to kubernet...@googlegroups.com
Oh, great. Thanks!


On Friday, August 25, 2017, <jamelse...@gmail.com> wrote:
Sure. Turns out the node was trying to tag the provisioned EBS volume but was lacking the corresponding AWS role permissions to do that. Once I updated the role with the right permissions, everything worked.

Reply all
Reply to author
Forward
0 new messages