kiwi9 breaks OEM image mount points

[Bahi, David]

Hello KIWI folks,

KIWI 7 did the right thing with LV group and volumes - /dev/<group>/<volume> was the "device" used for mounting in /etc/fstab.

Now in KIWI 9 we have the default by-uuid.

UUID= is not useful because after it creates the fstab and we use a KIWI hook to make the UUIDs unique.

using 'devicepersistency="by-label"' is not a fix because it drops swap (see no _label_ after LABEL= below).

pre-mount:/# cat /mnt/etc/fstab
LABEL=BOOT /boot xfs defaults 1 2
LABEL=ROOT / xfs defaults 1 1
LABEL= swap swap defaults 0 0


What do folks suggest to fix this?

db

[Marcus Schäfer]

Hi,

> KIWI 7 did the right thing with LV group and volumes - /dev/<group>/<volume> was the "device" used for mounting in /etc/fstab.

kiwi-ng does this as well, I did a short test build with home as volume
and you get in fstab:

/dev/systemVG/home /home ext4 defaults 1 2

> Now in KIWI 9 we have the default by-uuid.

right, for non lvm filesystems in my example this looks like this:

UUID=07f11e54-09b0-4578-9f9c-fda0a8c1a9a7 / ext4 defaults 1 1
UUID=d8e5b85b-f674-4fed-af8a-3999eba9218d /boot ext4 defaults 0 0
UUID=1326-7831 /boot/efi vfat defaults 0 0

> UUID= is not useful because after it creates the fstab and we use a KIWI hook to make the UUIDs unique.

You change the UUIDs ? I do not understand why but no matter if that
change applies to a label or a uuid it would affect the correctness
of fstab which makes me think it's not a kiwi issue ;)

> using 'devicepersistency="by-label"' is not a fix because it drops swap (see no _label_ after LABEL= below).

That sounds like a bug to me, but I can't reproduce it.
In my test I got

LABEL=SWAP swap swap defaults 0 0

I checked the code in the dracut module and found:

https://github.com/SUSE/kiwi/blob/master/dracut/modules.d/99kiwi-lib/kiwi-filesystem-lib.sh#L81

So swap is always created with a label and if that fails it would
be a fatal exception leading to an exception at boot time. The
result of an entry with no label can't happen in that case

That brings me to the question; Do you still use the oemboot code with
the custom kiwi initrd on first boot and not dracut ? Meaning do you
have <type ... boot="oemboot/..."> in your image description ?

I need to understand which boot code is in use to get the
right code path for a fix

Thanks

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer (Res. & Dev.) SUSE Linux GmbH
Tel: 0911-740 53 0 Maxfeldstrasse 5
FAX: 0911-740 53 479 D-90409 Nürnberg
HRB: 21284 (AG Nürnberg) Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton
http://www.suse.de
-------------------------------------------------------

[Bahi, David]

Hello Marcus,

On Sep 13, 2018, at 05:59, Marcus Schäfer <m...@suse.de> wrote:

Hi,

KIWI 7 did the right thing with LV group and volumes - /dev/<group>/<volume> was the "device" used for mounting in /etc/fstab.

kiwi-ng does this as well, I did a short test build with home as volume
and you get in fstab:

/dev/systemVG/home /home ext4 defaults 1 2

How do you get this behavior? 

We're seeing UUID as the default and when I looked I only saw by-uuid as default 

and the option to define <type ... 'devicepersistency="by-label"' handling for fstab

We have 

    <preferences profiles="target">

        <type primary="true"

              image="oem"

              boot="oemboot/suse-SLES12"

              devicepersistency="by-label"

              vga="0x317"

              bootpartsize="600"

              filesystem="xfs"

              installprovidefailsafe="false"

              installiso="true"

              installpxe="true"

              bootloader="grub2"

              boottimeout="100"

              checkprebuilt="false"

              kernelcmdline="pci=bfsort kiwidebug=1 net.ifnames=0 crashkernel=1024M,high console=tty1 console=ttyS0,115200n8 earlyprintk=ttyS0,115200n8"

        >

            <oemconfig>

                <oem-ataraid-scan>false</oem-ataraid-scan>

                <oem-boot-title>Dell EMC ECS</oem-boot-title>

                <oem-reboot>true</oem-reboot>

                <oem-recovery>false</oem-recovery>

                <oem-silent-install>false</oem-silent-install>

                <oem-silent-verify>false</oem-silent-verify>

                <oem-skip-verify>true</oem-skip-verify>

                <oem-swap>true</oem-swap>

                <oem-swapsize>1024</oem-swapsize>

                <oem-systemsize>12288</oem-systemsize>

                <oem-unattended>true</oem-unattended>

            </oemconfig>

            <systemdisk name="ECS">

                <volume name="@root" size="4096"/>

            </systemdisk>

        </type>

Now in KIWI 9 we have the default by-uuid.

right, for non lvm filesystems

Well, we're definitely LVM and no longer see the /dev/<vgname>/<lvname mounts:

admin@orem-red:~> sudo pvs

  PV         VG   Fmt  Attr PSize   PFree

  /dev/sda   ECS  lvm2 a--  372.53g    0 

admin@orem-red:~> sudo vgs

  VG   #PV #LV #SN Attr   VSize   VFree

  ECS    1   2   0 wz--n- 372.53g    0 

admin@orem-red:~> sudo lvs

  LV     VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert

  LVRoot ECS  -wi-ao---- 364.53g                                                    

  LVSwap ECS  -wi-ao----   8.00g                                                    

in my example this looks like this:

UUID=07f11e54-09b0-4578-9f9c-fda0a8c1a9a7 / ext4 defaults 1 1
UUID=d8e5b85b-f674-4fed-af8a-3999eba9218d /boot ext4 defaults 0 0
UUID=1326-7831 /boot/efi vfat defaults 0 0

UUID= is not useful because after it creates the fstab and we use a KIWI hook to make the UUIDs unique.

You change the UUIDs ? I do not understand why but no matter if that
change applies to a label or a uuid it would affect the correctness
of fstab which makes me think it's not a kiwi issue ;)

Yes, generating unique UUIDs for the system apparently was never a KIWI requirement for imaging.

We do this so each deployed image has unique fs uuids, this way if we do cluster info gathering this helps distinguish systems.

using 'devicepersistency="by-label"' is not a fix because it drops swap (see no _label_ after LABEL= below).

That sounds like a bug to me, but I can't reproduce it.
In my test I got

LABEL=SWAP swap swap defaults 0 0

I checked the code in the dracut module and found:

https://github.com/SUSE/kiwi/blob/master/dracut/modules.d/99kiwi-lib/kiwi-filesystem-lib.sh#L81

So swap is always created with a label and if that fails it would
be a fatal exception leading to an exception at boot time. The
result of an entry with no label can't happen in that case

From /var/log/boot.kiwi after "fininalizePartitionTable" 

Information: You may need to update /etc/fstab.

+ [[ msdos =~ gpt ]]

+ return 0

+ '[' '' = yes ']'

+ waitForStorageDevice /dev/ECS/LVRoot

+ '[' -z '' ']'

+ '[' 1024 -gt 0 ']'

+ waitForStorageDevice /dev/ECS/LVSwap

+ partitionSize /dev/ECS/LVSwap

+ Echo 'Creating swap space on /dev/ECS/LVSwap'

+ local 'IFS=

'

+ '[' 0 = 0 ']'

+ set +x

+ echo '[  210.194819] Creating swap space on /dev/ECS/LVSwap'

+ '[' 0 = 0 ']'

+ set +x

+ mkswap /dev/ECS/LVSwap

Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)

no label, UUID=b39479a2-916b-4446-9608-acabdddb6001

+ '[' -z '' ']'

+ '[' '!' -z '' ']'

+ '[' -z '' ']'

+ '[' '!' -z '' ']'

+ runHook postrepart

+ local 'IFS=

'

+ '[' '!' -z '' ']'

+ export eval KIWI_ALLOW_HOOK_CMD_postrepart=0

So, that's OEMRepart in repart under /usr/share/kiwi/custom_boot/oemboot

    #======================================

    # Activate swap space

    #--------------------------------------

    if [ -z "$DONT_PARTITION" ] && [ $swapsize -gt 0 ]; then

        waitForStorageDevice $imageSwapDevice &>/dev/null

        if partitionSize $imageSwapDevice &>/dev/null;then

            Echo "Creating swap space on $imageSwapDevice"

            if ! mkswap $imageSwapDevice 1>&2;then

                systemException "Failed to create swap signature" "reboot"

            fi

        fi

    fi

I'll create a local patch to fix this ... looks like it should work fine with --label

admin@orem-red:~> sudo mkswap /dev/ECS/LVSwap --label SWAP

mkswap: /dev/ECS/LVSwap: warning: wiping old swap signature.

Setting up swapspace version 1, size = 8 GiB (8589930496 bytes)

LABEL=SWAP, UUID=db152b96-0934-4ee8-9840-be9dcd1c14bd

kiwi-boot-descriptions-1.1.0-20.1.x86_64

That brings me to the question; Do you still use the oemboot code with
the custom kiwi initrd on first boot and not dracut ? Meaning do you
have <type ... boot="oemboot/..."> in your image description? 

Yes, sorry we're not familiar with how to work with dracut yet.  

https://opensource.suse.com/kiwi/overview/workflow.html

We have lots of kiwi-hook code and fear the move.

Thanks again, Marcus

--

db

p.s. 

I'm also going to re-add support for kiwiserverpath (for use with kiwiservertype and kiwiserver) to override the 'image' default dir.

We have more urgent need for this now because the image file name is not build version unique anymore:  name.xz name.md5 ... what happened to the version strings?

[Bahi, David]

swap, grub, and kiwiserverpath /image patches... my git foo is not powerful enough for github, but maybe it's time I learned.

[Bahi, David]

SWAP label still not available on boot

What imaging did:

admin@provo-red:~> grep -C5 SWAP /var/log/boot.kiwi 

+++ FILTERBIG=1
+++ FILTERSMALL=0
+++ HYBRIDSYSPARTSIZE=300
+++ HYBRIDVGNAME=ECStarget
+++ HYBRIDVGPARK=ECSpark
+++ TGTSWAPSIZEMB=8192
+++ HOSTUNKNOWN=unknown
+++ HOSTPHOENIX=phoenix
+++ HOSTRINJIN=rinjin
+++ HOSTPYTHON=python
+++ HOSTHPSL4540G8=sl4540g8
--
+++ FILTERBIG=1
+++ FILTERSMALL=0
+++ HYBRIDSYSPARTSIZE=300
+++ HYBRIDVGNAME=ECStarget
+++ HYBRIDVGPARK=ECSpark
+++ TGTSWAPSIZEMB=8192
+++ HOSTUNKNOWN=unknown
+++ HOSTPHOENIX=phoenix
+++ HOSTRINJIN=rinjin
+++ HOSTPYTHON=python
+++ HOSTHPSL4540G8=sl4540g8
--
+++ FILTERBIG=1
+++ FILTERSMALL=0
+++ HYBRIDSYSPARTSIZE=300
+++ HYBRIDVGNAME=ECStarget
+++ HYBRIDVGPARK=ECSpark
+++ TGTSWAPSIZEMB=8192
+++ HOSTUNKNOWN=unknown
+++ HOSTPHOENIX=phoenix
+++ HOSTRINJIN=rinjin
+++ HOSTPYTHON=python
+++ HOSTHPSL4540G8=sl4540g8
--

+ '[' 0 = 0 ']'
+ set +x

+ echo '[  176.187941] Creating swap space on /dev/ECS/LVSwap'

+ '[' 0 = 0 ']'
+ set +x

+ mkswap /dev/ECS/LVSwap --label SWAP

Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)

LABEL=SWAP, UUID=3e7ad165-f0e7-4746-b5f7-e74d1f90eab5

+ '[' -z '' ']'
+ '[' '!' -z '' ']'
+ '[' -z '' ']'
+ '[' '!' -z '' ']'
+ runHook postrepart

--
+ local devicepersistency=by-uuid
+ '[' '!' -z by-label ']'
+ devicepersistency=by-label
+ '[' by-label = by-label ']'
++ blkid /dev/ECS/LVSwap -s LABEL -o value
+ local device=LABEL=SWAP
+ echo 'LABEL=SWAP swap swap defaults 0 0'
+ Echo 'Creating boot loader configuration'

+ local 'IFS= 
'
+ '[' 0 = 0 ']'
+ set +x

--
+++ local dev=../../dm-0
++++ basename ../../dm-0
+++ dev=/dev/dm-0
+++ '[' /dev/dm-0 = /dev/sda ']'
+++ for i in '/dev/disk/$prefix/*'
+++ '[' -z /dev/disk/by-label/SWAP ']'
+++ echo /dev/disk/by-label/SWAP
+++ grep -q edd-
++++ readlink /dev/disk/by-label/SWAP
+++ local dev=../../dm-1
++++ basename ../../dm-1
+++ dev=/dev/dm-1
+++ '[' /dev/dm-1 = /dev/sda ']'
+++ echo /dev/sda
--
+++ FILTERBIG=1
+++ FILTERSMALL=0
+++ HYBRIDSYSPARTSIZE=300
+++ HYBRIDVGNAME=ECStarget
+++ HYBRIDVGPARK=ECSpark
+++ TGTSWAPSIZEMB=8192
+++ HOSTUNKNOWN=unknown
+++ HOSTPHOENIX=phoenix
+++ HOSTRINJIN=rinjin
+++ HOSTPYTHON=python
+++ HOSTHPSL4540G8=sl4540g8

What happens at boot:

Sep 14 21:01:58 __HOSTNAME__ kernel:  sdw: sdw1

Sep 14 21:01:58 __HOSTNAME__ kernel: sd 0:0:22:0: [sdw] Attached SCSI disk

Sep 14 21:01:59 __HOSTNAME__ dracut-initqueue[443]: Scanning devices sda  for LVM logical volumes ECS/LVRoot

Sep 14 21:01:59 __HOSTNAME__ dracut-initqueue[443]: inactive '/dev/ECS/LVRoot' [364.53 GiB] inherit

Sep 14 21:01:59 __HOSTNAME__ dracut-initqueue[443]: inactive '/dev/ECS/LVSwap' [8.00 GiB] inherit

Sep 14 21:01:59 __HOSTNAME__ systemd[1]: Found device /dev/mapper/ECS-LVRoot.

Sep 14 21:01:59 __HOSTNAME__ kernel: scsi 0:0:23:0: Direct-Access     ATA      HUS726060ALA640  T1EY PQ: 0 ANSI: 6

Sep 14 21:01:59 __HOSTNAME__ kernel: scsi 0:0:23:0: SATA: handle(0x0023), sas_addr(0x50060480e81d8f04), phy(4), device_name(0x0000000000000000)

...

Sep 14 21:02:02 __HOSTNAME__ systemd[1]: Started Update UTMP about System Boot/Shutdown.

Sep 14 21:03:30 __HOSTNAME__ systemd[1]: dev-disk-by\x2dlabel-SWAP.device: Job dev-disk-by\x2dlabel-SWAP.device/start timed out.

Sep 14 21:03:30 __HOSTNAME__ systemd[1]: Timed out waiting for device dev-disk-by\x2dlabel-SWAP.device.

Sep 14 21:03:30 __HOSTNAME__ systemd[1]: Dependency failed for /dev/disk/by-label/SWAP.

Sep 14 21:03:30 __HOSTNAME__ systemd[1]: Dependency failed for Swap.

Sep 14 21:03:30 __HOSTNAME__ systemd[1]: swap.target: Job swap.target/start failed with result 'dependency'.

Sep 14 21:03:30 __HOSTNAME__ systemd[1]: dev-disk-by\x2dlabel-SWAP.swap: Job dev-disk-by\x2dlabel-SWAP.swap/start failed with result 'dependency'.

Sep 14 21:03:30 __HOSTNAME__ systemd[1]: dev-disk-by\x2dlabel-SWAP.device: Job dev-disk-by\x2dlabel-SWAP.device/start failed with result 'timeout'.

Sep 14 21:03:30 __HOSTNAME__ systemd[1]: Reached target System Initialization.

What by-labels are "available":

admin@provo-red:~> ls -l /dev/disk/by-label
total 0
lrwxrwxrwx 1 root root 10 Sep 14 21:02 BOOT -> ../../sdb1
lrwxrwxrwx 1 root root 10 Sep 14 21:02 ROOT -> ../../dm-0
admin@provo-red:~> 

On Sep 14, 2018, at 15:16, Bahi, David <david...@emc.com> wrote:

swap, grub, and kiwiserverpath /image patches... my git foo is not powerful enough for github, but maybe it's time I learned.

<swap-label.patch>

<grub-os-prober-disable-true.patch>

<non-image-serverpath.patch>

[Marcus Schäfer]

Hi,

> swap, grub, and kiwiserverpath /image patches... my git foo is not
> powerful enough for github, but maybe it's time I learned.

Thanks much for the patches. It would be really great if you can
consider to use/learn how the github process works because that
will then open the gate for review and history tracking.

So I took a look at the swap patch first and turned that into a
github pull request. I realized you put the --label option after
the device spec of mkswap but if the man page tells us the truth
the position matters. mkswap [options] device [size]. I adapted
the patch and also use a function to call mkswap.

Please take the time to review:

https://github.com/SUSE/kiwi-descriptions/pull/27

Next I took a look on the other two patches:

1. grub-os-prober-disable-true

I think I can't go with that because the os prober on native
systems should work with grub. I have seen problems with the
os-prober only on unusual root mounts e.g overlayed or use
of unknown read-only filesystem for root. The way this is
implemented globally switches off the os-prober which I can't
merge to master, sorry

2. non-image-serverpath.patch

This looks like an enhancement in a way that the hardcoded
/srv/tftpboot/image/ path will become configurable. I'm ok
with this idea but the way this is implemented changes the
caller semantic of the fetchImageMD5 method and the netboot
linuxrc requires $SERVERPATH. If you can adapt the change
such that it's really optional and that /srv/tftpboot/image/
stays as the default and can be overwritten by $SERVERPATH
I would be ok.

Usually this type of discussions happens as part of a github
issue such that the things we discussed can directly be used
in the implementation and does not stay in an e-mail thread

Don't hesitate to make use of github, we are happy to
welcome new contributors :)