osd node restart problem

[Laszlo Budai]

Hi,

we have a ceph (v14.2.6) cluster deployed by rook. We have drained and restarted a node, and the ceph-osd pod is failing to start.

Actually the activate-osd init container is complaining that it it cannot find the LVM device belonging to the OSD.

+ ceph-volume lvm activate --no-systemd --bluestore 16 26592a7b-641f-4356-985e-6fb301e39aac
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-16
Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-7fb2c3cc-a88c-4b1f-b5a4-b4b597981a4c/osd-data-ab3c8982-e2f4-4310-90ca-cb5b87db22b4 --path /var/lib/ceph/osd/ceph-16 --no-mon-config
 stderr: failed to read label for /dev/ceph-7fb2c3cc-a88c-4b1f-b5a4-b4b597981a4c/osd-data-ab3c8982-e2f4-4310-90ca-cb5b87db22b4: (2) No such file or directory

I have checked the host, and I can see the following blkid output:

/dev/nvme1n1: UUID="ImUMx2-j9je-34BB-GjEC-dlzl-E96K-nQdudt" TYPE="LVM2_member"

On the host I cannot issue LVM related commands because it doesn't has the lvm2 package installed. Is that required?

I see the init container would mount the /dev from the host. Is it assuming then that the host is activating the lvm volume?

Thank you,

Laszlo