kube-registry stuck in ContainerCreating state
152 views
Skip to first unread message
Ajith Joseph
Mar 26, 2018, 7:41:05 PM3/26/18
to rook-dev
I have setup a rook cluster following the tutorial. Everything seems to be running
root@rook-tools:/# ceph status
cluster:
id: ba50ab43-163b-40a3-925f-f0adce289c12
health: HEALTH_OK
services:
mon: 3 daemons, quorum rook-ceph-mon0,rook-ceph-mon2,rook-ceph-mon1
mgr: rook-ceph-mgr0(active)
osd: 4 osds: 4 up, 4 in
data:
pools: 2 pools, 200 pgs
objects: 0 objects, 0 bytes
usage: 61717 MB used, 419 GB / 479 GB avail
pgs: 200 active+clean
root@rook-tools:/# ceph osd status
+----+---------------------+-------+-------+--------+---------+--------+---------+-----------+
| id | host | used | avail | wr ops | wr data | rd ops | rd data | state |
+----+---------------------+-------+-------+--------+---------+--------+---------+-----------+
| 0 | rook-ceph-osd-d2qf5 | 12.5G | 107G | 0 | 0 | 0 | 0 | exists,up |
| 1 | rook-ceph-osd-gv75h | 18.4G | 101G | 0 | 0 | 0 | 0 | exists,up |
| 2 | rook-ceph-osd-gxhpp | 19.1G | 100G | 0 | 0 | 0 | 0 | exists,up |
| 3 | rook-ceph-osd-mhgn4 | 10.0G | 109G | 0 | 0 | 0 | 0 | exists,up |
+----+---------------------+-------+-------+--------+---------+--------+---------+-----------+
root@rook-tools:/# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
479G 419G 61717M 12.57
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
myfs-metadata 1 0 0 126G 0
myfs-data0 2 0 0 252G 0
root@rook-tools:/# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR
myfs-data0 0 0 0 0 0 0 0 0 0 0 0
myfs-metadata 0 0 0 0 0 0 0 0 0 0 0
total_objects 0
total_used 61717M
total_avail 419G
total_space 479G
However, I am not able to run kube-registry
kube-registry-v0-gn5bw 0/1 ContainerCreating 0 16h
kube-registry-v0-hrdmf 0/1 ContainerCreating 0 16h
kube-registry-v0-sctj9 0/1 ContainerCreating 0 16h
Travis Nielsen
Mar 26, 2018, 8:08:18 PM3/26/18
to Ajith Joseph, rook-dev
There is likely an issue mounting the filesystem. Try looking at `kubectl describe <pod>` for the registry pods and see what message there might be about the mount. And for a faster communication, you might consider joining the Rook Slack. There is a link from rook.io.
--
You received this message because you are subscribed to the Google Groups "rook-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rook-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rook-dev/c6541315-6952-43ad-a402-250b1c927261%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Message has been deleted
Ajith Joseph
Mar 26, 2018, 10:17:21 PM3/26/18
to rook-dev
$ kubectl -n kube-system describe pod kube-registry-v0-gn5bw
Name: kube-registry-v0-gn5bw
Namespace: kube-system
Node: ip-172-20-44-53.us-west-2.compute.internal/172.20.44.53
Start Time: Mon, 26 Mar 2018 00:36:11 -0700
Labels: k8s-app=kube-registry
version=v0
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"kube-system","name":"kube-registry-v0","uid":"5b24a999-30c8-11e8-8f55-...
Status: Pending
IP:
Controlled By: ReplicationController/kube-registry-v0
Containers:
registry:
Container ID:
Image: registry:2
Image ID:
Port: 5000/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
cpu: 100m
memory: 100Mi
Requests:
cpu: 100m
memory: 100Mi
Environment:
REGISTRY_HTTP_ADDR: :5000
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
Mounts:
/var/lib/registry from image-store (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5zwfl (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
image-store:
Type: FlexVolume (a generic volume resource that is provisioned/attached using an exec based plugin)
Driver: Options: %v
FSType: rook.io/rook
SecretRef: ceph
ReadOnly: <nil>
%!(EXTRA bool=false, map[string]string=map[fsName:myfs clusterName:rook]) default-token-5zwfl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5zwfl
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.alpha.kubernetes.io/notReady:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 14m (x280 over 16h) kubelet, ip-172-20-44-53.us-west-2.compute.internal (combined from similar events): MountVolume.SetUp failed for volume "image-store" : mount command failed, status: Failure, reason: failed to mount filesystem myfs to /var/lib/kubelet/pods/5b28dbe4-30c8-11e8-8f55-021d7381bd8c/volumes/rook.io~rook/image-store with monitor 100.71.26.187:6790,100.70.130.155:6790,100.64.95.112:6790:/ and options [name=admin secret=**********************************]: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/5b28dbe4-30c8-11e8-8f55-021d7381bd8c/volumes/rook.io~rook/image-store --scope -- mount -t ceph -o name=admin,secret=************************************* 100.71.26.187:6790,100.70.130.155:6790,100.64.95.112:6790:/ /var/lib/kubelet/pods/5b28dbe4-30c8-11e8-8f55-021d7381bd8c/volumes/rook.io~rook/image-store
Output: Running as unit run-335.scope.
mount: 100.71.26.187:6790,100.70.130.155:6790,100.64.95.112:6790:/: can't read superblock
Warning FailedMount 8m (x422 over 17h) kubelet, ip-172-20-44-53.us-west-2.compute.internal Unable to mount volumes for pod "kube-registry-v0-gn5bw_kube-system(5b28dbe4-30c8-11e8-8f55-021d7381bd8c)": timeout expired waiting for volumes to attach/mount for pod "kube-system"/"kube-registry-v0-gn5bw". list of unattached/unmounted volumes=[image-store]
Warning FailedSync 3m (x457 over 17h) kubelet, ip-172-20-44-53.us-west-2.compute.internal Error syncing pod
Reply all
Reply to author
Forward
0 new messages