Error: cannot find volume "rook-binaries" to mount into container "copy-bins"
204 views
Skip to first unread message
lei...@gmail.com
Jun 29, 2020, 12:28:34 PM6/29/20
to rook-dev
Hello,
CentOS Linux release 7.8.2003 (Core)
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:49:29Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
I have KVM to created 3 master and 4 node
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-provisioner-6bcb7cdd75-rzjwn 5/5 Running 1 30m
csi-cephfsplugin-provisioner-6bcb7cdd75-zf86z 5/5 Running 5 30m
csi-cephfsplugin-qn58s 3/3 Running 0 30m
csi-cephfsplugin-rqffd 3/3 Running 0 30m
csi-cephfsplugin-xfxnf 3/3 Running 3 30m
csi-rbdplugin-8j94s 3/3 Running 3 30m
csi-rbdplugin-njx4b 3/3 Running 0 30m
csi-rbdplugin-provisioner-5bcf44d764-rdss5 6/6 Running 4 30m
csi-rbdplugin-provisioner-5bcf44d764-s2j9g 6/6 Running 7 30m
csi-rbdplugin-vj6b8 3/3 Running 0 30m
rook-ceph-crashcollector-sealos04-688d6c444-smlkh 1/1 Running 2 11h
rook-ceph-crashcollector-sealos05-54f9dc4c59-7nl26 1/1 Running 1 11h
rook-ceph-crashcollector-sealos06-8469bf6dd8-x7wlc 1/1 Running 1 11h
rook-ceph-mgr-a-6df59bb654-c2m2n 1/1 Running 2 11h
rook-ceph-mon-a-cc4d98f47-pbh9d 1/1 Running 2 11h
rook-ceph-mon-b-7784dcb994-qzbms 1/1 Running 1 11h
rook-ceph-mon-c-6947d7b77f-7bhxb 1/1 Running 1 11h
rook-ceph-operator-567d7945d6-kv96p 1/1 Running 0 30m
rook-ceph-osd-prepare-sealos04-kmkwv 0/1 Completed 0 30m
rook-ceph-osd-prepare-sealos05-ww649 0/1 Completed 0 30m
rook-ceph-osd-prepare-sealos06-6grrr 0/1 Completed 0 30m
rook-ceph-tools-6d659f5579-vmtp4 1/1 Running 2 5h30m
rook-discover-6n94j 1/1 Running 0 30m
rook-discover-jtmfj 1/1 Running 0 30m
rook-discover-x7757 1/1 Running 1 30m
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned rook-ceph/rook-ceph-osd-prepare-sealos04-kmkwv to sealos04
Normal Created 30m kubelet, sealos04 Created container copy-bins
Normal Started 30m kubelet, sealos04 Started container copy-bins
Normal Pulled 30m kubelet, sealos04 Container image "ceph/ceph:v15.2.3" already present on machine
Normal Created 30m kubelet, sealos04 Created container provision
Normal Started 30m kubelet, sealos04 Started container provision
Normal SandboxChanged 30m kubelet, sealos04 Pod sandbox changed, it will be killed and re-created.
Normal Pulled 30m (x2 over 30m) kubelet, sealos04 Container image "rook/ceph:master" already present on machine
Warning Failed 30m kubelet, sealos04 Error: cannot find volume "rook-binaries" to mount into container "copy-bins"
cluster:
id: e2ebd699-bcc2-4948-9d83-46da9021dcdc
health: HEALTH_WARN
Reduced data availability: 1 pg inactive
OSD count 0 < osd_pool_default_size 3
services:
mon: 3 daemons, quorum a,b,c (age 2m)
mgr: a(active, since 2m)
osd: 0 osds: 0 up, 0 in
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs: 100.000% pgs unknown
1 unknown
node 4
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 200G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 49G 0 part
├─centos_sealos01-root 253:0 0 45.1G 0 lvm /
└─centos_sealos01-swap 253:1 0 3.9G 0 lvm
vdb 252:16 0 193K 0 disk
Node 5
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 200G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 49G 0 part
├─centos_sealos01-root 253:0 0 45.1G 0 lvm /
└─centos_sealos01-swap 253:1 0 3.9G 0 lvm
vdb 252:16 0 193K 0 disk
Node 6
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 200G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 49G 0 part
├─centos_sealos01-root 253:0 0 45.1G 0 lvm /
└─centos_sealos01-swap 253:1 0 3.9G 0 lvm
vdb 252:16 0 193K 0 disk
Still don't know why sometime vdb will disappeared, I have to use KVM to reattach it, may be this issue caused ?
Thanks
lei...@gmail.com
Jun 29, 2020, 1:58:29 PM6/29/20
to rook-dev
Hello,
After recreated vdb on all nodes, I make sure reboot not disappear, but have another problem:
You can see three rook-ceph-crashcollector on Init:0/2 over 30mins.
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
csi-cephfsplugin-j4l9q 3/3 Running 0 47m 10.0.10.136 sealos06 <none> <none>
csi-cephfsplugin-provisioner-6bcb7cdd75-kwx44 5/5 Running 0 47m 100.100.17.237 sealos06 <none> <none>
csi-cephfsplugin-provisioner-6bcb7cdd75-vgjmb 5/5 Running 0 47m 100.100.43.27 sealos05 <none> <none>
csi-cephfsplugin-q4dcj 3/3 Running 0 47m 10.0.10.134 sealos04 <none> <none>
csi-cephfsplugin-rwxjx 3/3 Running 0 47m 10.0.10.135 sealos05 <none> <none>
csi-rbdplugin-9hzgz 3/3 Running 0 47m 10.0.10.135 sealos05 <none> <none>
csi-rbdplugin-bwkww 3/3 Running 0 47m 10.0.10.136 sealos06 <none> <none>
csi-rbdplugin-jhvsp 3/3 Running 0 47m 10.0.10.134 sealos04 <none> <none>
csi-rbdplugin-provisioner-5bcf44d764-gr6hg 6/6 Running 0 47m 100.96.132.214 sealos04 <none> <none>
csi-rbdplugin-provisioner-5bcf44d764-jqktl 6/6 Running 0 47m 100.100.17.239 sealos06 <none> <none>
rook-ceph-crashcollector-sealos04-5ddcd95d66-x2drp 0/1 Init:0/2 0 47m <none> sealos04 <none> <none>
rook-ceph-crashcollector-sealos05-54f9dc4c59-rxjlz 0/1 Init:0/2 0 35m <none> sealos05 <none> <none>
rook-ceph-crashcollector-sealos06-8469bf6dd8-tb6hz 0/1 Init:0/2 0 24m <none> sealos06 <none> <none>
rook-ceph-mon-a-6b59f495dc-hmg4p 1/1 Running 0 47m 100.96.132.217 sealos04 <none> <none>
rook-ceph-mon-d-689cc86b94-vzhvr 1/1 Running 0 35m 100.100.43.24 sealos05 <none> <none>
rook-ceph-mon-f-7846f769fc-fwfjm 1/1 Running 0 24m 100.100.17.238 sealos06 <none> <none>
rook-ceph-operator-567d7945d6-csx4t 1/1 Running 0 47m 100.100.17.227 sealos06 <none> <none>
rook-discover-542ft 1/1 Running 0 47m 100.100.17.234 sealos06 <none> <none>
rook-discover-gbw6d 1/1 Running 0 47m 100.96.132.210 sealos04 <none> <none>
rook-discover-qrtfh 1/1 Running 0 47m 100.100.43.18 sealos05 <none> <none>
What is rook-ceph-crash-collector-keyring ? thanks
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 49m default-scheduler Successfully assigned rook-ceph/rook-ceph-crashcollector-sealos04-5ddcd95d66-x2drp to sealos04
Warning FailedMount 31m (x4 over 42m) kubelet, sealos04 Unable to attach or mount volumes: unmounted volumes=[rook-ceph-crash-collector-keyring], unattached volumes=[rook-ceph-crash default-token-9zdkm rook-ceph-crash-collector-keyring rook-config-override rook-ceph-log]: timed out waiting for the condition
Warning FailedMount 29m (x2 over 45m) kubelet, sealos04 Unable to attach or mount volumes: unmounted volumes=[rook-ceph-crash-collector-keyring], unattached volumes=[rook-ceph-log rook-ceph-crash default-token-9zdkm rook-ceph-crash-collector-keyring rook-config-override]: timed out waiting for the condition
Warning FailedMount 26m (x2 over 47m) kubelet, sealos04 Unable to attach or mount volumes: unmounted volumes=[rook-ceph-crash-collector-keyring], unattached volumes=[default-token-9zdkm rook-ceph-crash-collector-keyring rook-config-override rook-ceph-log rook-ceph-crash]: timed out waiting for the condition
Warning FailedMount 18m (x23 over 49m) kubelet, sealos04 MountVolume.SetUp failed for volume "rook-ceph-crash-collector-keyring" : secret "rook-ceph-crash-collector-keyring" not found
Warning FailedMount 8m42s (x7 over 40m) kubelet, sealos04 Unable to attach or mount volumes: unmounted volumes=[rook-ceph-crash-collector-keyring], unattached volumes=[rook-config-override rook-ceph-log rook-ceph-crash default-token-9zdkm rook-ceph-crash-collector-keyring]: timed out waiting for the condition
Warning FailedMount 4m8s kubelet, sealos04 Unable to attach or mount volumes: unmounted volumes=[rook-ceph-crash-collector-keyring], unattached volumes=[rook-ceph-crash-collector-keyring rook-config-override rook-ceph-log rook-ceph-crash default-token-9zdkm]: timed out waiting for the condition
Reply all
Reply to author
Forward
0 new messages