coreos-install with /dev/disk/by-path/... race condition

[athit.tha...@gmail.com]

Hello !

coreos-install using a by-path device is not working for me. I tried with the short name /dev/sdc instead and it works.

I added some debug to the critical part in the coreos-install script and found out that i need to wait a few seconds for the block id to appear:

First the bug. without sleeping the blkid does not find the OEM label and errors at mount time:

bash-4.3# ./coreos-install -d '/dev/disk/by-path/pci-0000:00:1f.2-ata-2' -C stable -V current -i ignition.json

Current version of CoreOS Container Linux stable is 1409.8.0

Downloading the signature for https://stable.release.core-os.net/amd64-usr/1409.8.0/coreos_production_image.bin.bz2...

2017-08-12 07:52:38 URL:https://stable.release.core-os.net/amd64-usr/1409.8.0/coreos_production_image.bin.bz2.sig [543/543] -> "/tmp/coreos-install.BDp4tfvBX0/coreos_production_image.bin.bz2.sig" [1]

Downloading, writing and verifying coreos_production_image.bin.bz2...

2017-08-12 07:53:15 URL:https://stable.release.core-os.net/amd64-usr/1409.8.0/coreos_production_image.bin.bz2 [288053790/288053790] -> "-" [1]

gpg: Signature made Thu Aug 10 01:16:06 2017 UTC

gpg: using RSA key 07FA9ED31CB5FA26

gpg: key 50E0885593D2DCB4 marked as ultimately trusted

gpg: checking the trustdb

gpg: marginals needed: 3 completes needed: 1 trust model: pgp

gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u

gpg: Good signature from "CoreOS Buildbot (Offical Builds) <buil...@coreos.com>" [ultimate]

+ ls /dev/disk/by-path/

pci-0000:00:1f.2-ata-1 pci-0000:00:1f.2-ata-2 pci-0000:03:00.0-nvme-1

pci-0000:00:1f.2-ata-1-part1 pci-0000:00:1f.2-ata-4 pci-0000:04:00.0-scsi-0:2:0:0

+ ls /dev/sda /dev/sdb /dev/sdb1 /dev/sdc /dev/sdc1 /dev/sdc2 /dev/sdc3 /dev/sdc4 /dev/sdc6 /dev/sdc7 /dev/sdc9 /dev/sdd

/dev/sda /dev/sdb /dev/sdb1 /dev/sdc /dev/sdc1 /dev/sdc2 /dev/sdc3 /dev/sdc4 /dev/sdc6 /dev/sdc7 /dev/sdc9 /dev/sdd

+ ls /dev/disk/by-path/

pci-0000:00:1f.2-ata-1 pci-0000:00:1f.2-ata-2 pci-0000:03:00.0-nvme-1

pci-0000:00:1f.2-ata-1-part1 pci-0000:00:1f.2-ata-4 pci-0000:04:00.0-scsi-0:2:0:0

++ blkid -t LABEL=OEM -o device /dev/disk/by-path/pci-0000:00:1f.2-ata-2

+ local OEM_DEV=

+ set +x

mount: wrong fs type, bad option, bad superblock on ,

 missing codepage or helper program, or other error




 In some cases useful info is found in syslog - try

 dmesg | tail or so.

Error: return code 32 from mount "${OEM_DEV}" "${WORKDIR}/oemfs"

/dev/disk/by-path/pci-0000:00:1f.2-ata-2: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54

/dev/disk/by-path/pci-0000:00:1f.2-ata-2: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa

/dev/disk/by-path/pci-0000:00:1f.2-ata-2: calling ioctl to re-read partition table: Success

adding the sleep:

bash-4.3# ./coreos-install -d '/dev/disk/by-path/pci-0000:00:1f.2-ata-2' -C stable -V current -i ignition.json

 

Current version of CoreOS Container Linux stable is 1409.8.0

Downloading the signature for https://stable.release.core-os.net/amd64-usr/1409.8.0/coreos_production_image.bin.bz2...

2017-08-12 07:42:47 URL:https://stable.release.core-os.net/amd64-usr/1409.8.0/coreos_production_image.bin.bz2.sig [543/543] -> "/tmp/coreos-install.I5TcCQ1SAf/coreos_production_image.bin.bz2.sig" [1]

Downloading, writing and verifying coreos_production_image.bin.bz2...

2017-08-12 07:43:26 URL:https://stable.release.core-os.net/amd64-usr/1409.8.0/coreos_production_image.bin.bz2 [288053790/288053790] -> "-" [1]

gpg: Signature made Thu Aug 10 01:16:06 2017 UTC

gpg:                using RSA key 07FA9ED31CB5FA26

gpg: key 50E0885593D2DCB4 marked as ultimately trusted

gpg: checking the trustdb

gpg: marginals needed: 3  completes needed: 1  trust model: pgp

gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u

gpg: Good signature from "CoreOS Buildbot (Offical Builds) <buil...@coreos.com>" [ultimate]

+ ls /dev/disk/by-path/

pci-0000:00:1f.2-ata-1       pci-0000:00:1f.2-ata-2  pci-0000:03:00.0-nvme-1

pci-0000:00:1f.2-ata-1-part1  pci-0000:00:1f.2-ata-4  pci-0000:04:00.0-scsi-0:2:0:0

+ ls /dev/sda /dev/sdb /dev/sdb1 /dev/sdc /dev/sdc1 /dev/sdc2 /dev/sdc3 /dev/sdc4 /dev/sdc6 /dev/sdc7 /dev/sdc9 /dev/sdd

/dev/sda  /dev/sdb  /dev/sdb1  /dev/sdc  /dev/sdc1  /dev/sdc2  /dev/sdc3  /dev/sdc4  /dev/sdc6 /dev/sdc7  /dev/sdc9  /dev/sdd

+ sleep 5

+ ls /dev/disk/by-path/

pci-0000:00:1f.2-ata-1       pci-0000:00:1f.2-ata-2-part2  pci-0000:00:1f.2-ata-2-part7  pci-0000:04:00.0-scsi-0:2:0:0

pci-0000:00:1f.2-ata-1-part1  pci-0000:00:1f.2-ata-2-part3  pci-0000:00:1f.2-ata-2-part9

pci-0000:00:1f.2-ata-2       pci-0000:00:1f.2-ata-2-part4  pci-0000:00:1f.2-ata-4

pci-0000:00:1f.2-ata-2-part1  pci-0000:00:1f.2-ata-2-part6  pci-0000:03:00.0-nvme-1

++ blkid -t LABEL=OEM -o device /dev/disk/by-path/pci-0000:00:1f.2-ata-2 /dev/disk/by-path/pci-0000:00:1f.2-ata-2-part1 /dev/disk/by-path/pci-0000:00:1f.2-ata-2-part2 /dev/disk/by-path/pci-0000:00:1f.2-ata-2-part3 /dev/disk/by-path/pci-0000:00:1f.2-ata-2-part4 /dev/disk/by-path/pci-0000:00:1f.2-ata-2-part6 /dev/disk/by-path/pci-0000:00:1f.2-ata-2-part7 /dev/disk/by-path/pci-0000:00:1f.2-ata-2-part9

+ local OEM_DEV=/dev/disk/by-path/pci-0000:00:1f.2-ata-2-part6

+ set +x

Installing Ignition config ignition.json...

Success! CoreOS Container Linux stable 1409.8.0 is installed on /dev/disk/by-path/pci-0000:00:1f.2-ata-2

Does anyone knows why partition take so much time to appear in the /dev/disk/by-path directory (the disk is wipeout at the start of the script) ?

--- Athit

[Benjamin Gilbert]

On Sat, Aug 12, 2017 at 1:05 AM,  <athit.tha...@gmail.com> wrote:
> coreos-install using a by-path device is not working for me. I tried with
> the short name /dev/sdc instead and it works.

Thanks for the report and traces!  It turns out that coreos-install wasn't waiting for udev to create /dev/disk symlinks before trying to mount the OEM partition through them.  The only reason it worked with /dev/sdc is that the partition device nodes were being created by devtmpfs long before udev got to it.

The fix is here.

--Benjamin Gilbert