Ceph osd disk not working for iscsi storage disk

[James Hunter]

I have some disk that don't have a file system on them. I can see that manger finds the disk and installs a ceph lvm partion on it. Then at the bottom of the log it says it can't use the disk because it has a lmv2 partition member. Can you use iscsi storage disk?

[James Hunter]

I even tried mounting in in kvm as a disk and it still returned an error. 

Disk /dev/sda: 500 GiB, 536870928384 bytes, 1048576032 sectors

Disk model: QEMU HARDDISK   

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

2021-01-26 01:19:45.094367 I | op-mon: parsing mon endpoints: a=10.98.245.237:6789

2021-01-26 01:19:45.117005 I | op-osd: CRUSH location=root=default host=kubernetes-master

2021-01-26 01:19:45.117023 I | cephcmd: crush location of osd: root=default host=kubernetes-master

2021-01-26 01:19:45.117042 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt -- /usr/sbin/lvm --help

2021-01-26 01:19:45.138232 I | cephosd: successfully called nsenter

2021-01-26 01:19:45.138292 I | cephosd: binary "/usr/sbin/lvm" found on the host, proceeding with osd preparation

2021-01-26 01:19:45.138308 D | exec: Running command: dmsetup version

2021-01-26 01:19:45.143009 I | cephosd: Library version:   1.02.171-RHEL8 (2020-05-28)

Driver version:    4.43.0

2021-01-26 01:19:45.164026 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config

2021-01-26 01:19:45.166074 I | cephclient: generated admin config in /var/lib/rook/rook-ceph

2021-01-26 01:19:45.166237 D | cephosd: config file @ /etc/ceph/ceph.conf: [global]

fsid                           = 849a1f71-3eb6-4461-99b1-951db0791407

mon initial members            = a

mon host                       = [v2:10.98.245.237:3300,v1:10.98.245.237:6789]

public addr                    = 192.168.237.9

cluster addr                   = 192.168.237.9

osd_pool_default_size          = 1

mon_warn_on_pool_no_redundancy = false

[client.admin]

keyring = /var/lib/rook/rook-ceph/client.admin.keyring

2021-01-26 01:19:45.166244 I | cephosd: discovering hardware

2021-01-26 01:19:45.166251 D | exec: Running command: lsblk --all --noheadings --list --output KNAME

2021-01-26 01:19:45.212476 D | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.227089 D | exec: Running command: sgdisk --print /dev/sda

2021-01-26 01:19:45.234101 D | exec: Running command: udevadm info --query=property /dev/sda

2021-01-26 01:19:45.246241 D | exec: Running command: lsblk --noheadings --pairs /dev/sda

2021-01-26 01:19:45.263993 D | exec: Running command: lsblk /dev/sr0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.267192 W | inventory: skipping device "sr0". unsupported diskType rom

2021-01-26 01:19:45.267291 D | exec: Running command: lsblk /dev/nbd0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.269559 W | inventory: skipping device "nbd0". diskType is empty

2021-01-26 01:19:45.269578 D | exec: Running command: lsblk /dev/nbd1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.271596 W | inventory: skipping device "nbd1". diskType is empty

2021-01-26 01:19:45.271615 D | exec: Running command: lsblk /dev/nbd2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.273646 W | inventory: skipping device "nbd2". diskType is empty

2021-01-26 01:19:45.273720 D | exec: Running command: lsblk /dev/nbd3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.278408 W | inventory: skipping device "nbd3". diskType is empty

2021-01-26 01:19:45.278431 D | exec: Running command: lsblk /dev/nbd4 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.280878 W | inventory: skipping device "nbd4". diskType is empty

2021-01-26 01:19:45.280900 D | exec: Running command: lsblk /dev/nbd5 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.284672 W | inventory: skipping device "nbd5". diskType is empty

2021-01-26 01:19:45.284693 D | exec: Running command: lsblk /dev/nbd6 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.291831 W | inventory: skipping device "nbd6". diskType is empty

2021-01-26 01:19:45.291865 D | exec: Running command: lsblk /dev/nbd7 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.296017 W | inventory: skipping device "nbd7". diskType is empty

2021-01-26 01:19:45.296048 D | exec: Running command: lsblk /dev/vda --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.306717 D | exec: Running command: sgdisk --print /dev/vda

2021-01-26 01:19:45.310037 D | exec: Running command: udevadm info --query=property /dev/vda

2021-01-26 01:19:45.320444 D | exec: Running command: lsblk --noheadings --pairs /dev/vda

2021-01-26 01:19:45.330321 I | inventory: skipping device "vda" because it has child, considering the child instead.

2021-01-26 01:19:45.332265 D | exec: Running command: lsblk /dev/vda1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.343751 D | exec: Running command: udevadm info --query=property /dev/vda1

2021-01-26 01:19:45.371390 D | exec: Running command: lsblk /dev/vda2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.373557 D | exec: Running command: udevadm info --query=property /dev/vda2

2021-01-26 01:19:45.382017 D | exec: Running command: lsblk /dev/nbd8 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.385209 W | inventory: skipping device "nbd8". diskType is empty

2021-01-26 01:19:45.385239 D | exec: Running command: lsblk /dev/nbd9 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.397425 W | inventory: skipping device "nbd9". diskType is empty

2021-01-26 01:19:45.397465 D | exec: Running command: lsblk /dev/nbd10 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.399418 W | inventory: skipping device "nbd10". diskType is empty

2021-01-26 01:19:45.399445 D | exec: Running command: lsblk /dev/nbd11 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.401161 W | inventory: skipping device "nbd11". diskType is empty

2021-01-26 01:19:45.401179 D | exec: Running command: lsblk /dev/nbd12 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.403065 W | inventory: skipping device "nbd12". diskType is empty

2021-01-26 01:19:45.403084 D | exec: Running command: lsblk /dev/nbd13 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.404994 W | inventory: skipping device "nbd13". diskType is empty

2021-01-26 01:19:45.405024 D | exec: Running command: lsblk /dev/nbd14 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.406800 W | inventory: skipping device "nbd14". diskType is empty

2021-01-26 01:19:45.406820 D | exec: Running command: lsblk /dev/nbd15 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME

2021-01-26 01:19:45.408419 W | inventory: skipping device "nbd15". diskType is empty

2021-01-26 01:19:45.408445 D | inventory: discovered disks are [0xc00036e480 0xc0004197a0 0xc000419c20]

2021-01-26 01:19:45.408456 I | cephosd: creating and starting the osds

2021-01-26 01:19:45.414774 D | cephosd: No Drive Groups configured.

2021-01-26 01:19:45.414807 D | cephosd: desiredDevices are [{Name:all OSDsPerDevice:1 MetadataDevice: DatabaseSizeMB:0 DeviceClass: IsFilter:true IsDevicePathFilter:false}]

2021-01-26 01:19:45.414812 D | cephosd: context.Devices are [0xc00036e480 0xc0004197a0 0xc000419c20]

2021-01-26 01:19:45.414817 I | cephosd: skipping device "sda" because it contains a filesystem "LVM2_member"

2021-01-26 01:19:45.414821 I | cephosd: skipping device "vda1" because it contains a filesystem "xfs"

2021-01-26 01:19:45.414823 I | cephosd: skipping device "vda2" because it contains a filesystem "xfs"

2021-01-26 01:19:45.414886 I | cephosd: configuring osd devices: {"Entries":{}}

2021-01-26 01:19:45.414890 I | cephosd: no new devices to configure. returning devices already configured with ceph-volume.

2021-01-26 01:19:45.415333 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm list  --format json

2021-01-26 01:19:45.973793 D | cephosd: {}

2021-01-26 01:19:45.974245 I | cephosd: 0 ceph-volume lvm osd devices configured on this node

2021-01-26 01:19:45.974255 W | cephosd: skipping OSD configuration as no devices matched the storage settings for this node "kubernetes-master"

[James Hunter]

Volume group ceph-5aee976f-7a4b-44c9-a14d-26ca6b1107e3

RenameDelete

UUID

O5sPjd-Qwcm-JZsm-elDK-1muo-cGYh-242e8W

Capacity

500 GiB, 537 GB, 536866717696 bytes

[Rohan Joseph]

> Please run `dmsetup info <device path>` and `lsblk -o +fstype`

 

I can see that manger finds the disk and installs a ceph lvm partion on it. Then at the bottom of the log it says it can't use the disk because it has a lmv2 partition member. Can you use iscsi storage disk?

Where do you see the manager installing a ceph lvm partition on the disk?

--
You received this message because you are subscribed to the Google Groups "rook-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rook-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rook-dev/8276e4ae-8290-4eb6-aff3-64ad73642ab3n%40googlegroups.com.