Attempt to format partition from live system: "partition is apparently in use by the system"

[S.]

Hi there, as of the current Tumbleweed and Leap 15.0 live ISOs running in VirtualBox:

- openSUSE-Leap-15.0-KDE-Live-x86_64-Snapshot17.8-Media.iso
- openSUSE-Tumbleweed-KDE-Live-x86_64-Snapshot20180525-Media.iso

Disk partitions can not be formatted. `mkfs.ext4 /dev/sda1` fails with the error:
"/dev/sda1 is apparently in use by the system; will not make a filesystem here!"
However, `lsblk` does not appear to show that it is in use by the system, or at least it's not mounted.

I'm also running into this same error on Leap 15.0 and Tumbleweed images that I just generated myself with Kiwi.

This error didn't used to happen. The last time I tested it was about three months ago, and this problem didn't exist.

I'm attaching a screenshot of my attempts. 

Thanks!

[Marcus Schäfer]

Hi,

> Disk partitions can not be formatted. `mkfs.ext4 /dev/sda1` fails with the error

Hmm, can't reproduce this. Here are the steps I did:

git clone https://github.com/SUSE/kiwi-descriptions.git

kiwi-ng-3 --type iso system build \
--description kiwi-descriptions/suse/x86_64/suse-leap-15.0-JeOS/ \
--target-dir /tmp/mytest/

qemu-img create mydisk 20g

dd if=/tmp/mytest/LimeJeOS-Leap-15.0.x86_64-1.15.0.iso \
of=mydisk conv=notrunc

qemu-kvm mydisk

System boots and reports in df -h output

Filesystem Size Used Avail Use% Mounted on
...
LiveOS_rootfs 20G 53M 19G 1% /


Buid with:

kiwi-ng --version

KIWI (next generation) version 9.15.2

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer (Res. & Dev.) SUSE Linux GmbH
Tel: 0911-740 53 0 Maxfeldstrasse 5
FAX: 0911-740 53 479 D-90409 Nürnberg
HRB: 21284 (AG Nürnberg) Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton
http://www.suse.de
-------------------------------------------------------

[Marcus Schäfer]

Hi,

Thanks for sharing this. I haven't seen this issue so far. My last tests with
the persistent write partition were successful but also some time ago. I'll
give it a new round of testing

[S.]

Thanks a lot for the looking into this! I wonder if it is a problem specific to VirtualBox that doesn't appear in qemu? Also I generated the iso with kiwi , not kiwi-ng-3 , but kiwi --version still shows version 9.15.2 , so I don't think that makes any difference?

[Marcus Schäfer]

Hi,

> Thanks a lot for the looking into this! I wonder if it is a problem
> specific to VirtualBox that doesn't appear in qemu?

I have a hard time believing it could be a VirtualBox problem, but yes
I don't see this in qemu. Maybe a timing issue. I'll run a test on
real hardware and let you know what I'll find

> Also I generated
> the iso with kiwi , not kiwi-ng-3 , but kiwi --version still shows
> version 9.15.2 , so I don't think that makes any difference?

Nope 9.15.2 is almost the latest variant which is ok

[Marcus Schäfer]

Hi,

> I have a hard time believing it could be a VirtualBox problem, but yes
> I don't see this in qemu. Maybe a timing issue. I'll run a test on
> real hardware and let you know what I'll find

also worked for me on real hardware (Fast USB stick on Dell laptop)
So I'm afraid I can't reproduce this

[S.]

Hmm, this would be the first time I've ever had a bug specific to VirtualBox. I also tested it in an older VirtualBox version on an older and slower system, and it behaved the same way.

It must be a specific build option I'm using. Specifically, I wonder if the problem is here:

<type image="iso" primary="true" flags="overlay" hybrid="true" firmware="uefi" kernelcmdline="quiet splash apparmor=0" hybridpersistent_filesystem="ext4" hybridpersistent="true" volid="GeckoLinux_STATIC_Cinnamon" mediacheck="true"/>

Here's my entire Kiwi description directory, if this helps:

https://drive.google.com/open?id=1alHOxBXNXekxFSgB2K9AWxR1fkl4EAHT

Thanks again for your help.

[Marcus Schäfer]

Hi,

> Hmm, this would be the first time I've ever had a bug specific to
> VirtualBox.

yes hard to believe

> <type image="iso" primary="true" flags="overlay" hybrid="true"
> firmware="uefi" kernelcmdline="quiet splash apparmor=0"
> hybridpersistent_filesystem="ext4" hybridpersistent="true"

Don't see how that could cause the "busy partition" problem

> volid="GeckoLinux_STATIC_Cinnamon" mediacheck="true"/>
> Here's my entire Kiwi description directory, if this helps:
> https://drive.google.com/open?id=1alHOxBXNXekxFSgB2K9AWxR1fkl4EAHT
> Thanks again for your help.

I donwloaded it and the image is currently building. I have done
zero modifications to your source. Let's see if I can reproduce it

[David Cassany]

Hi,

> Don't see how that could cause the "busy partition" problem

I believe that multipath could be the cause... Just realized it is something
similar to what I just saw yestarday. I updated to Leap 15 yesterday evening
and I struggled with this confusing multipath issues... When multipath service
is active one should use the device map links to access devices, otherwise
they appear to be blocked or busy. Multipath is a service likely to be
silently active, since installing multipath-tools enables the service and it
is a dependency of many other packages.

Just my two cents...

Regards,
David

[Marcus Schäfer]

Not this time. I could reproduce the problem when building the image
Still investigating what happens but it does not look like the multipath
issue we had in the past

[Marcus Schäfer]

Hi,

> > I believe that multipath could be the cause... Just realized it is something
> > similar to what I just saw yestarday. I updated to Leap 15 yesterday evening
> > and I struggled with this confusing multipath issues... When multipath service
> > is active one should use the device map links to access devices, otherwise
> > they appear to be blocked or busy. Multipath is a service likely to be
> > silently active, since installing multipath-tools enables the service and it
> > is a dependency of many other packages.
>
> Not this time. I could reproduce the problem when building the image
> Still investigating what happens but it does not look like the multipath
> issue we had in the past

*Argh* you are right, it is the multipath issue. I thought it would be
different but on a closer look it's again the device mapper which makes
them busy

In our test images no multipath gets installed, thus we don't see the
issue there

I'll look if that can be fixed in a clean way

As a workaround add the following in your image description


<packages type="uninstall">
<package name="multipath-tools"/>
</packages>

[Marcus Schäfer]

Hi,

> I'll look if that can be fixed in a clean way

https://github.com/SUSE/kiwi/pull/748

I'll rebuild the image and see if that fixes the issue

[David Cassany]

Hi,

> *Argh* you are right, it is the multipath issue. I thought it would be
> different but on a closer look it's again the device mapper which makes
> them busy

Yes, this is so annoying, yesterday eve I got stressed trying to mount my /
home partition from the Leap 15 Live DVD... This live image also has
multipath-tools and it took me damn long to realize what was happening :P

> I'll look if that can be fixed in a clean way

I did not understand where is it actually failing, is it failing in one of our
dracut scripts? I thought it was just failing on some user commands or user
scripts, if so then it is just needed to update them to use the appropriate
device names.

Cheers,
David

[David Cassany]

> I did not understand where is it actually failing, is it failing in one of
> our dracut scripts? I thought it was just failing on some user commands or
> user scripts, if so then it is just needed to update them to use the
> appropriate device names.

Oh, just saw it effects the overlay cow partition for persistent live images.

[Marcus Schäfer]

Hi,

> > I'll look if that can be fixed in a clean way
>
> https://github.com/SUSE/kiwi/pull/748
>
> I'll rebuild the image and see if that fixes the issue

yep worked :)

dd if=GeckoLinux_STATIC_Cinnamon.x86_64-150.180526.0.iso of=mydisk conv=notrunc
qemu-kvm mydisk

... and I have plenty of space left :)

[S.]

Wow, nice work Marcus! I NEVER would have figured that one out.

So I assume this will be incorporated into the next Kiwi release? In that case, I think it would be good to push an update for Leap 15.0, since it can be considered a bug fix. It's really nice that Leap 15.0 now has a modern version of kiwi-ng, and without this fix I would have to add on the Virtualization:/Appliances:/Builder repo to my build system and possibly my image description as well.

Thanks again for all your hard work on this.

[Marcus Schäfer]

Hi,

> So I assume [1]this will be incorporated into the next Kiwi release?

yes, it will first appear in Virtualization:/Appliances:/Builder
from there I'll submit to Factory which is then Tumbleweed.

Those two are regularly updated. The next release window is on Friday

> In
> that case, I think it would be good to push an update for Leap 15.0,

yes we should also create a submission to Leap 15.0

However I have no influence when the request will be processed
by the team around Leap

> since it can be considered a bug fix. It's really nice that Leap 15.0
> now has a modern version of kiwi-ng, and without this fix I would have
> to add on the Virtualization:/Appliances:/Builder repo to my build
> system

true

> and possibly my image description as well.

In this case yes because the fix touched the dracut-kiwi-live package
which needs to be installed

> Thanks again for all your hard work on this.

You are welcome and helped to fix this :) Thanks