Performance tuning boxed plugin

[Till Dörges]

Dear all,

we're using Kiwi and the boxed plugin with a local version of the Leap box
(https://download.opensuse.org/repositories/Virtualization:/Appliances:/SelfContained:/leap/images/)
to build an openSUSE Leap ISO image on a Debian 12 host.

Before that we were using chroot builds on openSUSE Leap.


Going from chroot to boxed build increased ISO creation times by a factor of 4 to 5
on comparable hardware. Nested virtualization is supported.


From looking at the logs I'd summarize that we have two main pain points:

- I/O in the box is slow (e.g. for calls to mksquashfs)
- I/O when sharing files between box and host is slow (e.g. calls to rsync)


Since nobody likes to wait ;-), we've tried to optimize ISO creation times with the
boxed build plugin, but have come up empty so far.


What we've tried:

- different options for the Kiwi box:
- more CPUs and/or more RAM for the box
- sharing FS between box and host
(--9p-sharing, --virtiofs-sharing, --sshfs-sharing)
- with and without "--no-snapshot"
- providing a persistent RPM cache on the host
to be re-used inside a box (via --shared-path)


- different options for QEMU-KVM's caching in
"-drive file=...,driver=qcow2,cache=XXX"
(none, writeback, unsafe, off)


- different boxes (universal)



None of the above really made a difference, build times were always at least longer
by a factor of 4.

I did expect building to take longer when going from chroot to QEMU-KVM, but not like
this.


Are we holding it wrong or are figures like this to be expected?


What (other) options do we have for reducing ISO creation times?


As usual any hints and insights are appreciated!



Thanks and regards -- Till
--
Dipl.-Inform. Till Dörges doe...@pre-sense.de

www.pre-sense.de/fcknzs

PRESENSE Technologies GmbH Nagelsweg 41, D-20097 HH
Geschäftsführer/Managing Director AG Hamburg, HRB 107844
Till Dörges USt-IdNr.: DE263765024

[Marcus Schäfer]

Hi Till,

> we're using Kiwi and the boxed plugin...

> Before that we were using chroot builds on openSUSE Leap.

ok

> Going from chroot to boxed build increased ISO creation times by a factor of
> 4 to 5 on comparable hardware. Nested virtualization is supported.

So you are running on a VM with boxbuild as a nested KVM guest, correct ?

> From looking at the logs I'd summarize that we have two main pain points:
>
> - I/O in the box is slow (e.g. for calls to mksquashfs)
> - I/O when sharing files between box and host is slow (e.g. calls to rsync)

I can confirm that these two parts are the bottlenecks when
using boxbuild. If you build cross arch there is a third aspect which
on the architecture emulation

> Since nobody likes to wait ;-), we've tried to optimize ISO creation times
> with the boxed build plugin, but have come up empty so far.

To be honest I haven't done such a deep comparison as you did.
Thus there is not much I can add to your findings. However, I also
have experienced performance issues and you should be aware that
you will never be as fast as if you just build on the same host
in a chroot. Simply because there is no guest in between and you
never have to shuffle date from the the guest to the host.
I believe a factor of two will always persist.

Here is what I did to increase on the performance:

1. Make sure snapshot files gets written to a tmpfs. By default /tmp
is used for snapshots by qemu. You can change this via:
export TMPDIR=/some/tmpfs/dir

2. Operations like compression and also squashfs are multithreaded.
kiwi calls them such that the tools selects themselves how many
processes they spawn. This depends on the number of CPUs. The box
by default doesn't use smp. You can set the number of CPUs to use
via:

--box-smp-cpus=<number>

3. Use virtiofs for data transfer between guest and host.
You told me that --virtiofs-sharing has no effect on the performance
but in my tests this significantly improved the performance
compared to 9p

4. Increase guest memory. By default 8096M is used which should
be more than enough. But depending on how big your iso is it could
be that more main memory helps. You can influence this number via

--box-memory

Other than that it would be very useful to identify the calls
that suffered the most. So do you have a log file from a chroot
build and a log file of the same image build with boxbuild. Please
run with "kiwi-ng --debug ... system boxbuild ..." This increases
the verbosity of the log file and we can see the time each EXEC
call takes.

> Are we holding it wrong or are figures like this to be expected?

You are doing all correct. In my tests the build times usually
increases by a factor of two but not four or five. I'm not running
on a VM though and I'm mostly building disk images not iso's.
Thus as I said I have not really comparable data like you have.

> What (other) options do we have for reducing ISO creation times?

The ones I had in mind I shared with you. You could also consider
to use erofs instead of squashfs for building the ISOs. mkfs.erofs
is faster than mksquashfs. But you would be moving to another ro
filesystem

If nothing of the above helps, then I must say I have no more
ideas :) People like Alexander Graf who are deep into qemu and
virtualization might have more ideas what you can try. Running
an I/O centric guest on a guest is however not an ideal setup.

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------

[Till Dörges]

Hi Marcus,

thanks for your answer. :-)

On 11.02.25 10:31, Marcus Schäfer wrote:

>> we're using Kiwi and the boxed plugin...
>> Before that we were using chroot builds on openSUSE Leap.
>
> ok
>
>> Going from chroot to boxed build increased ISO creation times by a factor of
>> 4 to 5 on comparable hardware. Nested virtualization is supported.
>
> So you are running on a VM with boxbuild as a nested KVM guest, correct ?

Yes. In both setups Kiwi is running in QEMU-KVM, and boxbuild runs in a nested
QEMU-KVM guest.

> 1. Make sure snapshot files gets written to a tmpfs. By default /tmp
> is used for snapshots by qemu. You can change this via:
> export TMPDIR=/some/tmpfs/dir

That sounds like something we should try.

> 3. Use virtiofs for data transfer between guest and host.
> You told me that --virtiofs-sharing has no effect on the performance
> but in my tests this significantly improved the performance
> compared to 9p

Apologies. My original mail was a bit unclear. That's the only thing we couldn't test

on a Debian 12 host.

Debian 12 only ships an old (not-working) version of virtiofsd, and there seem to be
no native packages for a newer version available (e.g. in backports). Compiling also
ran into some issues IIRC.

>> What (other) options do we have for reducing ISO creation times?
>
> The ones I had in mind I shared with you. You could also consider
> to use erofs instead of squashfs for building the ISOs. mkfs.erofs
> is faster than mksquashfs. But you would be moving to another ro
> filesystem
>
> If nothing of the above helps, then I must say I have no more
> ideas :) People like Alexander Graf who are deep into qemu and
> virtualization might have more ideas what you can try. Running
> an I/O centric guest on a guest is however not an ideal setup.

I know... But the big advantage of boxbuild is encapsulation. chroot only seems to
work well, when using the same Linux distribution for build host and generated image.

Containers (instead of VMs) would also work for our purposes, but IIUC the stackbuild
plugin (https://osinside.github.io/kiwi/plugins/stackbuild.html) cannot be used to
simply create images from RPM sources, can it?

[Marcus Schäfer]

Hi,

> I know... But the big advantage of boxbuild is encapsulation. chroot only
> seems to work well, when using the same Linux distribution for build host
> and generated image.

right

> Containers (instead of VMs) would also work for our purposes,

You mean containers as build host instead of a VM as build host ?
Please be aware that the container always shares the kernel with
the host. This dependency can impact the image build inside of
the container but it dependes very much on what type of image
you build inside of the container.

> but IIUC the
> stackbuild plugin (https://osinside.github.io/kiwi/plugins/stackbuild.html)
> cannot be used to simply create images from RPM sources, can it?

I can't follow you how stackbuild could help ? stackbuild allows
to build images based on containers. If there is software missing
in the used container stack to build the image e.g a bootloader
you can combine the stack of containers with package repositories
and kiwi will install the requested and needed packages to build
the image. A typical example is; I want to build a VM image from
a container. So the workflow for stackbuild is:

container -> set_as_root -> kiwi -> install_missing_components -> image

and that concept you can use in many different variants.
How is this related to your build-host isolation topic ?

Thanks

[Till Dörges]

On 11.02.25 12:33, Marcus Schäfer wrote:

>> I know... But the big advantage of boxbuild is encapsulation. chroot only
>> seems to work well, when using the same Linux distribution for build host
>> and generated image.
>
> right
>
>> Containers (instead of VMs) would also work for our purposes,
>
> You mean containers as build host instead of a VM as build host ?

Containers to encapsulate the actual build.

The kiwi boxbuild plugin currently starts a VM. If it could just start something more
lightweight.

> Please be aware that the container always shares the kernel with
> the host. This dependency can impact the image build inside of
> the container but it dependes very much on what type of image
> you build inside of the container.

But I understand the choice for using a VM for encapsulation.

>> but IIUC the
>> stackbuild plugin (https://osinside.github.io/kiwi/plugins/stackbuild.html)
>> cannot be used to simply create images from RPM sources, can it?

[...]

> a container. So the workflow for stackbuild is:
>
> container -> set_as_root -> kiwi -> install_missing_components -> image
>
> and that concept you can use in many different variants.
> How is this related to your build-host isolation topic ?

Thanks for the explanation. And no, it is not related.


Regards -- Till

[Marcus Schäfer]

Hi,

> Containers to encapsulate the actual build.

Yeah, this is on my plan. I will build the box VMs also as containers
and then there will be a patch to boxbuild to specify something
like '--box universal --container'. I have this change on my todo
list for some time but didn't had the time to code it

I should make a business out of all the requests to kiwi :-)