Error in kiwi log "potential fdisk errors were ignored" (kiwi ng)

Subhash Babu M

unread,

Nov 27, 2020, 10:07:18 AM11/27/20

to kiwi

Hi Marcus,

I am trying to move to kiwi ng from legacy kiwi and noticed some changes in the xml description file. For example in the legacy kiwi we had a "boot" option which could be set to "vmxboot" (vmxboot pointed to /usr/share/kiwi/image/vmxboot/suse-SLES12/) and these boot templates were part of the kiwi-desc-vmxboot package. With kiwi ng I see that boot option is no longer supported for type vmx. Hence I removed the "boot=vmxboot" from my config.xml. Also the kiwi-desc-vmxboot package is no longer available in SLES15.

After that change I tried to build the VMX and I see some errors in the logs.

DEBUG: 15:34:35 | EXEC: [bash -c cat /tmp/tmps7jvsfki | fdisk /dev/loop0]

DEBUG: 15:34:35 | EXEC: Failed with stderr: Re-reading the partition table failed.: Invalid argument

, stdout:

Welcome to fdisk (util-linux 2.33.1).

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

Device does not contain a recognized partition table.

Created a new DOS disklabel with disk identifier 0x40db9cef.

Command (m for help): Partition type

p primary (0 primary, 0 extended, 4 free)

e extended (container for logical partitions)

Select (default p): Partition number (1-4, default 1): First sector (2048-2186260479, default 2048): Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-2186260479, default 2186260479):

Created a new partition 1 of type 'Linux' and of size 300 MiB.

Command (m for help): The partition table has been altered.

Calling ioctl() to re-read partition table.

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

DEBUG: 15:34:35 | potential fdisk errors were ignored

DEBUG: 15:34:35 | EXEC: [sfdisk -c /dev/loop0 1 83]

INFO: 15:34:36 | --> creating LVM root partition

DEBUG: 15:34:36 | p.lxlvm: fdisk: n p 2 cur_position +all_freeM w q

DEBUG: 15:34:36 | EXEC: [bash -c cat /tmp/tmpv6svrokc | fdisk /dev/loop0]

DEBUG: 15:34:36 | EXEC: Failed with stderr: Re-reading the partition table failed.: Invalid argument

, stdout:

the OVA build from this image when deployed fails to find the partitions and enters maintenance mode.

Also when I look at the kiw ng documentation, under <preferences><type> i don't see image=vmx

This is how my profile is written:

<vmconfig-entry>numvcpus = "2"</vmconfig-entry>

</machine>

</systemdisk>

</type>

<packagemanager>zypper</packagemanager>

<rpm-check-signatures>false</rpm-check-signatures>

<rpm-force>true</rpm-force>

<timezone>US/Eastern</timezone>

</preferences>

What am I doing worng?

Marcus Schäfer

unread,

Nov 27, 2020, 10:42:17 AM11/27/20

to kiwi-...@googlegroups.com

Hi,

> I am trying to move to kiwi ng from legacy kiwi and noticed some
> changes in the xml description file. For example in the legacy kiwi we
> had a "boot" option which could be set to "vmxboot"

right, this is all legacy stuff from times when kiwi created its own
initrd. Since kiwi uses dracut none of the boot= stuff is needed anymore.

> With kiwi ng I see that
> boot option is no longer supported for type vmx. Hence I removed the
> "boot=vmxboot" from my config.xml. Also the kiwi-desc-vmxboot package
> is no longer available in SLES15.

correct, just delete boot=

> After that change I tried to build the VMX and I see some errors in the
> logs.

well this looks like an fdisk error from the past. it sends the
ioctl to reread the partition table but holds a lock by itself.
This "error" does hot harm the build as we run in loop devices.
kiwi should have been able to complete the build ?

> the OVA build from this image when deployed fails to find the
> partitions and enters maintenance mode.

I'm afraid this is not enough information for me to find the root cause
of the problem. I see you use lvm, can you confirm you have
installed the package lvm2 ?

<package name="lvm2"/>

This is mandatory such that dracut has a chance to compile in the
tools into the initrd such that the volume group can be activated
at boot time. It's one of the common mistakes people made when
they move from boot=... to dracut. In the old kiwi initrd system
we always included lvm2 by default, but with dracut the component
must be part of the system which actually also makes more sense

> Also when I look at the kiw ng documentation, under <preferences><type>
> i don't see image=vmx

This is correct. There is only one disk type "oem" but kiwi provides
XSLT stylesheets to automatically convert older schema definitions
like the one you use. So you can still keep "vmx" as your disk
setup. It's however advisable to maintain the latest schema layout.
You can print that with:

kiwi-ng image info --description /path/to/description --print-xml

or you can also use yaml if you prefer that

kiwi-ng image info --description /path/to/description --print-yaml

> What am I doing worng?

I guess it's lvm2 package that is missing in the image description.
If this is not the case please provide me a link to your image such
that I can download and test myself (please the .vmdk of the ova, not
the binary ova as I don't play with VMWare)

Thanks

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer (Res. & Dev.) SUSE Software Solutions Germany GmbH
Tel: 0911-740 53 0 Maxfeldstrasse 5
FAX: 0911-740 53 479 D-90409 Nürnberg
HRB: 21284 (AG Nürnberg) Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton
http://www.suse.de
-------------------------------------------------------

signature.asc

Subhash Babu M

unread,

Nov 30, 2020, 6:29:36 PM11/30/20

to kiwi

Hi Marcus,

Yes after adding the lvm2 package the issue is resolved. Now when I deploy the OVA the VM boots up fine. So the errors in the logs related to the fdisk were not the cause of the issue like you pointed out. However, I would like to confirm that in the logs the errors related to the fdisk are still present.

Thanks for your support and time in helping to fix the issue I was facing.

One other thing I noticed in the logs is, at the end of the kiwi build the clean up of the loop devices fail. This problem was not there in the legacy kiwi. Here is a snippet from the end of the image-root.log

INFO: 17:43:33 | Cleaning up BootImageDracut instance

INFO: 17:43:33 | Cleaning up LoopDevice instance

DEBUG: 17:43:33 | EXEC: [losetup -d /dev/loop0]

INFO: 17:43:33 | Cleaning up Disk instance

DEBUG: 17:43:33 | EXEC: [dmsetup remove /dev/mapper/loop0p1]

DEBUG: 17:43:33 | EXEC: [dmsetup remove /dev/mapper/loop0p2]

DEBUG: 17:43:33 | EXEC: Failed with stderr: device-mapper: remove ioctl on loop0p2 failed: Device or resource busy

Command failed.

, stdout: (no output on stdout)

WARNING: 17:43:33 | cleanup of partition device maps failed, /dev/loop0 still busy

INFO: 17:43:33 | Cleaning up VolumeManagerLVM instance

DEBUG: 17:43:33 | EXEC: [mountpoint -q /tmp/kiwi_volumes.ala83f4p/var]

DEBUG: 17:43:33 | EXEC: [mountpoint -q /tmp/kiwi_volumes.ala83f4p/swap]

DEBUG: 17:43:33 | EXEC: [mountpoint -q /tmp/kiwi_volumes.ala83f4p]

DEBUG: 17:43:33 | EXEC: [rm -r -f /tmp/kiwi_volumes.ala83f4p]

DEBUG: 17:43:33 | EXEC: [vgchange -an VGglider]

And here are the devices which are left over after the successful build.

jenkins@glider-build09:~/glider/ssmo_fto/kiwi> sudo pvdisplay

--- Physical volume ---

PV Name /dev/mapper/loop0p2

VG Name VGglider

PV Size 1.02 TiB / not usable 4.00 MiB

Allocatable yes

PE Size 4.00 MiB

Total PE 266804

Free PE 26

Allocated PE 266778

PV UUID 6K5TMl-e0bv-79Ab-VEw4-LlAo-55rc-hIyi2I

jenkins@glider-build09:~/glider/ssmo_fto/kiwi> sudo vgdisplay

--- Volume group ---

VG Name VGglider

System ID

Format lvm2

Metadata Areas 1

Metadata Sequence No 4

VG Access read/write

VG Status resizable

MAX LV 0

Cur LV 3

Open LV 0

Max PV 0

Cur PV 1

Act PV 1

VG Size 1.02 TiB

PE Size 4.00 MiB

Total PE 266804

Alloc PE / Size 266778 / 1.02 TiB

Free PE / Size 26 / 104.00 MiB

VG UUID AEXKUl-fmls-vTDt-4ak9-fvZZ-C1MJ-Nvvb9y

At this point if I try to re-run the kiwi build it will fail with duplicate loop devices error. I even tried to manually delete these devices and then try to run the kiwi build again, but ended up with the same errors. So as of now the only workaround I have figured out is to take a reboot of the build system after every build. Based on this information at the following link https://github.com/OSInside/kiwi/issues/631 looks like there was a fix put in for the loop devices issue. But what I am seeing is different as it does not happen on the first run and it happens on the second run and its due to the left over loop devices from the first run. I like to confirm that I al running the kiwi build with root privileges (using sudo) as suggested.

Are there any changes in kiwi-ng due to which I might have to change something in the config.xml file and I could have missed that and getting this issue?

I can upload the complete image-root.log and the config.xml file if needed.

Once again thanks for the support.

Regards

Subhash

Marcus Schäfer

unread,

Dec 1, 2020, 4:17:48 AM12/1/20

to kiwi-...@googlegroups.com

Hi,

> Yes after adding the lvm2 package the issue is resolved.

glad to hear that

> However, I would like to confirm that in the logs the errors related to
> the fdisk are still present.

yes, they will not go away unless fdisk gets fixed. I think there
is a fix to fdisk available but not all distributions seems to have
it.

> And here are the devices which are left over after the successful
> build.

This is because you built a lvm image and on your host there is
an active lvmetad which holds a lock to any new volume group (also
the ones just temporarly created in loops). Because of that kiwi's
cleanup code has no chance to complete. I suggest you disable
lvemtad on the host which is used to build lvm2 images.
See here:

https://documentation.commvault.com/commvault/v11/article?p=104318_1.htm

On a side note you might want to take a look at kiwi's boxbuild
feature which let you allow to build in a self contained environment
and that sill solve these sort of issues and others:

https://osinside.github.io/kiwi/self_contained.html

signature.asc

Subhash Babu M

unread,

Dec 3, 2020, 3:49:29 PM12/3/20

to kiwi

Thanks for the support. I don't think I can use the buildbox as one of the repos I use in the config.xml points to a folder on the build system. (like this path="file:///home/jenkins/glider/ssmo_fto/kiwi/repo"/) So I will skip that for now.

I just have one last question. Is there a way we can order the package installation in the config.xml file? because I noticed kiwi installing my packages even before installing most of the base packages like systemd, which is causing some issues.

Regards

Subhash

Dell EMC

Marcus Schäfer

unread,

Dec 4, 2020, 3:32:12 AM12/4/20

to kiwi-...@googlegroups.com

On Thu, Dec 03, Subhash Babu M wrote:

> Thanks for the support. I don't think I can use the buildbox as one of
> the repos I use in the config.xml points to a folder on the build
> system. (like
> this path="file:///home/jenkins/glider/ssmo_fto/kiwi/repo"/) So I will
> skip that for now.

You would still be able to use the box. If you run boxbuild the
directory given as --target-dir as well as the directory given as
--description are shared with the box.

I haven't tested this but afaik you could use bind mounts

mount -o bind /home/jenkins/glider/ssmo_fto/kiwi/repo /target-dir/my_repo

you could then reference the repo defintion as follows:

<repository>
<source path="/result/my_repo"/>
</repository>

--target-dir is available below /result in the box
--description is available below /description in the box

There is also an open issue to directly map a given local host path
to the box such that no change in the image description regarding
the path name would be needed.

https://github.com/OSInside/kiwi-boxed-plugin/issues/12

I did not had time to work on this so far. We also want to allow
the new virtiofs way of sharing host data. Currently 9p is used.

> I just have one last question. Is there a way we can order the package
> installation in the config.xml file? because I noticed kiwi installing
> my packages even before installing most of the base packages like
> systemd, which is causing some issues.

The install order of packages is based on the dependency resolution
done by the package manager. This can't be influenced by kiwi. If
your package has to be installed e.g after systemd you need to fix
this in your packages by correct requirement settings such that
the package manager has a chance to correctly resolve the list.

kiwi works in two stages when it installs a new root tree:

1) bootstrap <packages type="bootstrap"/>
2) system <packages type="image"/>

The first pass uses the host package manager to fill an empty root
tree with enough data to chroot into it, aka bootstrap

The second pass runs chrooted and installs the rest with the package
manager inside the new tree

This two stage system gives you a little chance to "order" packages.
So if you place packages in type="bootstrap" you can be sure they come before
the ones you place in type="image"

But this would just be a nasty workaround for an issue that should
be solved inside of the packages :)

Best Regards,

signature.asc

Subhash Babu M

unread,

Dec 7, 2020, 2:05:23 PM12/7/20

to kiwi

Hi Marcus,

Regarding the package ordering, I agree with you. I will setup dependencies in my rpms to set the order right.

Its just that I though a package like systemd, which more or less is part of the OS boot process, would be getting

installed before any other packages. Especially at least before any of the packages which are build by user and not

part of the SLES repos.

Also I only noticed this in kiwi-ng where my packages are being installed even before system packages. Never saw

this behavior with legacy kiwi. Hence thought I will bring it to your notice, incase this was not an expected behavior.

Thanks for the tips you have provided for accessing local repos within buildbox. I will try them out and and let you know how it goes.

Regards

Subhash

Marcus Schäfer

unread,

Dec 8, 2020, 4:37:29 AM12/8/20

to kiwi-...@googlegroups.com

Hi Subhash,

> Regarding the package ordering, I agree with you. I will setup
> dependencies in my rpms to set the order right.

Perfect

> Its just that I though a package like systemd, which more or less is
> part of the OS boot process, would be getting
>
> installed before any other packages. Especially at least before any of
> the packages which are build by user and not

It's correct that systemd is required by many packages but not by all.
A typical bootstrap section for a suse based kiwi build has this
packages:

<packages type="bootstrap">
<package name="udev"/>
<package name="filesystem"/>
<package name="glibc-locale"/>
<package name="cracklib-dict-full"/>
<package name="ca-certificates"/>
<package name="openSUSE-release"/>
</packages>

None of them will pull in systemd. This means the kiwi bootstrap
phase with this settings will already resolve into a relatively big list
of packages excluding systemd.

So the assumption systemd must be a candidate before any others
is not correct.

The point here is that from a kiwi perspective we really can't
control the order of package installations as we trust the package
manager to do it right. The package manager can do it right only
if the packages are correct. In any other case some nasty workarounds
are needed, like the trick I mentioned in my former e-mail

> Also I only noticed this in kiwi-ng where my packages are being
> installed even before system packages. Never saw
>
> this behavior with legacy kiwi. Hence thought I will bring it to your
> notice, incase this was not an expected behavior.

Thanks for bringing this up, much appreciated so we can clarify
on the issue and maybe help others too. The compare with the legacy
kiwi at this point is however not easy to judge. From a logic
perspective kiwi-ng and the legacy kiwi did the same. Both tools
runs the package manager and that call performs the install in
some order.

For a comparison however we should see the log file from both
attempts and check if the zypper command is the same. Next also
the repos must be the same because as you know the package metadata
is key for the ordering done by the package manager. If only one
package is different in the e.g requirements setting the complete
install order can change. Last but not least also the zypper
version must be identical for a comparison. This is because the
code that resolves packages and creates the install transaction
in zypper also changes.

Given all this is the same I doubt kiwi-ng and legacy kiwi would
come to different results. This is because in kiwi there is no
code that influences the work done by the package manager.

If your package has been installed late enough in the past it
could be just by pure luck :) Sorry if I say that but the only
safe way to make sure a package installs at the right time is
by setting its requirements correctly

> Thanks for the tips you have provided for accessing local repos within
> buildbox. I will try them out and and let you know how it goes.

That would be great. The boxbuild plugin is relatively new and I
would like to make this as user friendly as ever possible. Any feedback
is therefore very much appreciated.

Thanks

signature.asc

Subhash Babu M

unread,

Dec 15, 2020, 4:45:29 PM12/15/20

to kiwi

Hi Marcus,

I have a new query. In the legacy kiwi we used to have a config-yast-autoyast.xml file in which we used to set the system hostname and also used to run some customer scripts. Like firewall rules, adding some entries to the fstab etc.

Do we have to place this file with a different name now? I ask this because the setting like hostname and not getting applied. Also in the image-root.log I did not find any specific entries related to this file. Should we rename it to something

like firstboot?

Regards

Subhash

Marcus Schäfer

unread,

Dec 17, 2020, 6:48:48 AM12/17/20

to kiwi-...@googlegroups.com

Hi,

> I have a new query. In the legacy kiwi we used to have
> a config-yast-autoyast.xml file in which we used to set the system
> hostname and also used to run some customer scripts. Like firewall
> rules, adding some entries to the fstab etc.
>
> Do we have to place this file with a different name now? I ask this
> because the setting like hostname and not getting applied. Also in the
> image-root.log I did not find any specific entries related to this
> file. Should we rename it to something
>
> like firstboot?

yeah yast has changed and provides the yast autoyast firstboot
service. Therefore add the following in your packages setup

<package name="yast2-firstboot"/>

Next you need to provide the firstboot.xml file to the system as
overlay file

root/etc/YaST2/firstboot.xml

This file is slightly different from your former config-yast-autoyast.xml
and I can only ask you to get in contact with the YaST team for
details how to create/update the contents correctly. I'm attaching
a version that worked for me some time in the past

Next you need to activate the service which requires the following
file in the overlay tree

root/var/lib/YaST2/tdi_ignore
root/var/lib/YaST2/reconfig_system

These files are just empty. YaST only checks for their presence
and does something. reconfig_system causes yast to be started
and reading your firstboot.xml

If you need custom scripts to be called you can provide them
in root/usr/share/firstboot/scripts/... For those please also
get in contact with the YaST team for further details

Hope this helps