Upgrade mode for OEM install?

43 views
Skip to first unread message

Gavin Lambert

unread,
Oct 8, 2024, 10:17:11 PM10/8/24
to kiwi
With type="oem" that uses installiso="true", is it possible to configure it (or otherwise customize the scripts) so that it can act like follows:
  • For an empty disk (or a non-empty one if explicitly instructed) it performs a "clean install" including disk partitioning and formatting, as now.
  • For a non-empty disk (perhaps meeting specific compatibility checks, e.g. partition layout) it performs an "in-place upgrade", where it rewrites the boot/efi/root partition contents but leaves another partition untouched (i.e. skips partitioning the whole disk too).
I'm thinking of two possible use cases for this:
  • With spare_part_is_last and something similar to oem_resize so that the root partition is a fixed size and the auto-expanded spare partition is the one left untouched on upgrade.
  • With overlayroot_write_partition (and a fixed squashfs partition size) it could overwrite the boot and squashfs partitions but leave the persistent overlay untouched.
Is it possible to use installiso="true" with a custom dracut module instead of dracut-kiwi-oem-dump?  (It complains when I try that.)  Or is there a better approach?

One completely different approach I'm also considering is to use type="iso" (bootable USB) instead, which mounts either an extra partition or a separate local disk as writable storage.  This is perhaps simpler but has some different caveats.

Marcus Schäfer

unread,
Oct 14, 2024, 10:07:14 AM10/14/24
to kiwi-...@googlegroups.com
Hi,

> With type="oem" that uses installiso="true", is it possible to
> configure it (or otherwise customize the scripts) so that it can act
> like follows:
>
> * For an empty disk (or a non-empty one if explicitly instructed) it
> performs a "clean install" including disk partitioning and
> formatting, as now.
> * For a non-empty disk (perhaps meeting specific compatibility
> checks, e.g. partition layout) it performs an "in-place upgrade",
> where it rewrites the boot/efi/root partition contents but leaves
> another partition untouched (i.e. skips partitioning the whole disk
> too).

If you set installiso="true" this leads to an ISO image that
contains specific initrd modules called 90kiwi-dump and 99kiwi-dump-reboot
which implements the deployment code.

This deployment only allows for your empty disk use case, or for
disks that you can safely wipe. We have no implementation for an
"in-place upgrade"

You can provide your own module for driving an "in-place upgrade"

> Is it possible to use installiso="true" with a custom dracut module
> instead of dracut-kiwi-oem-dump? (It complains when I try that.) Or
> is there a better approach?

In the kiwi code the name of the initrd module to use for the
deployment is hardcoded:

./builder/install.py: self.boot_image_task.include_module('kiwi-dump')
./builder/install.py: self.boot_image_task.include_module('kiwi-dump-reboot')

If you write your own dracut module you need to name it:

/usr/lib/dracut/modules.d/90kiwi-dump

That will cause an intentional conflict with our "dracut-kiwi-oem-dump"
package. The respective runtime check can be disabled via:

/etc/kiwi.yml

runtime_checks:
- disable:
- check_dracut_module_for_oem_install_in_package_list

So your workflow to your own code could be:

1. Write a dracut module with a different deployment code and
make sure to place it as /usr/lib/dracut/modules.d/90kiwi-dump

2. Package this module, choose some name for the package

3. Disable the above mentioned runtime check in a kiwi setup file
e.g /etc/kiwi.yml or ~/.config/kiwi/config.yml

4. Add your dracut package to the package list in the kiwi image file

<package name="yours"/>

When you build this the install iso will contain your code instead
of yours and we know it according to your package list, just in
case there are errors we know whom to contact ;)

> One completely different approach I'm also considering is to use
> type="iso" (bootable USB) instead, which mounts either an extra
> partition or a separate local disk as writable storage. This is
> perhaps simpler but has some different caveats.

It could actually be the better approach because our way of deployment
is super easy, we dump an image to a target, period. An in-place-upgrade
process offers way more side tracks, might require questions and interaction
with the user how to treat the data and so on. Doing that in an initrd
is not a good place. If you build a live ISO as the driver to an
upgrade process you can run everything from a real system which gives
you a much much better runtime.

Hope this helps

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
signature.asc

Gavin Lambert

unread,
Oct 16, 2024, 2:49:42 AM10/16/24
to kiwi
Not sure if this is a bug or working as intended 😅

I had a config.xml with these parts:

    <preferences profiles="MyProfile">
        <type image="oem" primary="true" ...>...</type>
        <type image="iso" .../>
    </preferences>
    <packages type="oem">
        <package name="dracut-kiwi-oem-repart"/>
        <package name="dracut-kiwi-oem-dump"/>
    </package>
    <packages type="iso">
        <package name="dracut-kiwi-live"/>
    </packages>

And I was building with a two-step command:

    kiwi-ng --type oem --profile=MyProfile system build --description . --target-dir build-oem
    kiwi-ng --type iso --profile=MyProfile system create --root build-oem/build/image-root --target-dir build-iso

The idea being to produce both outputs without doing the (quite lengthy in my case) prepare step over again.  Unfortunately (and perhaps not surprising in hindsight), this resulted in the iso image being unbootable because it hadn't actually installed the dracut-kiwi-live package -- but also didn't show any errors that it was missing.  This confused me for a while.

I guess I was expecting that since both types were named in the same profile without further conditions (in the XML) then it would have included the packages for both when generating the rootfs even if a specific type was named for the create sub-step -- but clearly it thought differently.

It looks like what I *should* have been doing was to run it as a three-step command instead:

    kiwi-ng --profile=MyProfile system generate --description . --root build/image-root
    kiwi-ng --type oem --profile=MyProfile system create --root build/image-root --target-dir build-oem
    kiwi-ng --type iso --profile=MyProfile system create --root build/image-root --target-dir build-iso

Which does install both sets of packages and then creates the images, which is why I don't think it's a bug per se -- but perhaps the runtime checks could detect this case too?

(Though after this the iso still doesn't actually boot, which I'm still investigating -- the oem works fine though.)

Side note: it does seem a bit awkward at the moment to produce multiple outputs; perhaps --type for system build could accept multiple values like --profile does, and allow building to the same target dir?  Though maybe that's tricky.

Another side note: is there a way to do separate generate/create steps like this with boxbuild?  It didn't seem so from the docs.  (Though I can't actually get boxbuild working atm for reasons discussed elsewhere anyway.  I'd like to get it working because I think kiwi is leaving dangling processes on my host.)

Gavin Lambert

unread,
Oct 17, 2024, 1:55:53 AM10/17/24
to kiwi
On Wednesday, October 16, 2024 at 7:49:42 PM UTC+13 I wrote:
    kiwi-ng --profile=MyProfile system prepare --description . --root build/image-root
    kiwi-ng --type oem --profile=MyProfile system create --root build/image-root --target-dir build-oem
    kiwi-ng --type iso --profile=MyProfile system create --root build/image-root --target-dir build-iso

Which does install both sets of packages and then creates the images, which is why I don't think it's a bug per se -- but perhaps the runtime checks could detect this case too?

(Though after this the iso still doesn't actually boot, which I'm still investigating -- the oem works fine though.)

Ok, if I do a "--type iso system build" or a "--type iso system prepare / --type iso system create" then the ISO boots, but if I do the commands above then it fails to start because it can't find the rootfs UUID.  It does look like it mounted the squashfs and overlay as expected, though; I suspect it's getting things mixed up with the oem rootfs -- in particular the oem build seems to create an /etc/fstab file that upsets the iso build.

For now I've modified the above sequence to the following, which appears to work (at the cost of more disk space):

    kiwi-ng --profile=MyProfile system prepare --description . --root build/image-root
    rsync -a build/image-root/ build/oem-root/
    rsync -a build/image-root/ build/iso-root/
    kiwi-ng --type oem --profile=MyProfile system create --root build/oem-root --target-dir build-oem
    kiwi-ng --type iso --profile=MyProfile system create --root build/iso-root --target-dir build-iso

Is there a better way to build multiple types from one prepare?  Or is this approach doomed?  (I'm installing a lot of stuff via packages and config.sh, so that step is quite slow.)

I might eventually end up picking only one of these to build, but I'm also thinking about adding a container image as well, so I think multiple outputs will likely stay a thing.

Is there a way to get the hybrid persistence file working with Ventoy?  It does work when I burn the iso directly to a USB stick, but when I tried using live-grub-stick it wasn't bootable (that doesn't seem to support EFI?), and with Ventoy it is bootable but isn't persistent.

Marcus Schäfer

unread,
Oct 18, 2024, 10:18:12 AM10/18/24
to kiwi-...@googlegroups.com
Hi,

> Not sure if this is a bug or working as intended 😅

Whenever I get this question I answer with; It's working as intended :-))

> And I was building with a two-step command:
>
> kiwi-ng --type oem --profile=MyProfile system build --description .
> --target-dir build-oem
>
> kiwi-ng --type iso --profile=MyProfile system create --root
> build-oem/build/image-root --target-dir build-iso
>
> The idea being to produce both outputs without doing the (quite lengthy
> in my case) prepare step over again. Unfortunately (and perhaps not
> surprising in hindsight), this resulted in the iso image being
> unbootable because it hadn't actually installed the dracut-kiwi-live
> package -- but also didn't show any errors that it was missing. This
> confused me for a while.

This is correct if you run "system build" you run in fact
system prepare + system create. Both steps are done with type set to oem
and profile set to MyProfile. The produced root tree from that prepare
step is tight to "oem+MyProfile"

Any step you take now from this state that does not create the same
"oem+MyProfile" can work but doesn't have to.

The reason why you don't get an error message here is because your
second "system create" call just uses the given --root tree as it
is. kiwi has no code that looks at this tree and checks if it is
suitable to be used for the "system create" task. We could spent
some effort to do sanity checks as there is some metadata in the
tree that would allow for some checks but I doubt we could prevent
any imaginable error condition.

So yes this burden is put on the user side

> I guess I was expecting that since both types were named in the same
> profile without further conditions (in the XML) then it would have
> included the packages for both when generating the rootfs even if a
> specific type was named for the create sub-step -- but clearly it
> thought differently.

Nope, if you are using type specific packages sections e.g
'<packages type="iso">' those are tightly coupled to the image
type that you are building. A root tree produced for one type
is actually only valid for this type and for nothing else

> It looks like what I *should* have been doing was to run it as a
> three-step command instead:
>
> kiwi-ng --profile=MyProfile system prepare --description . --root
> build/image-root

Yes if you prepare the tree without a type specification first then
kiwi takes all types defined in the image description into account
and installs all packages for the defined types

> kiwi-ng --type oem --profile=MyProfile system create --root
> build/image-root --target-dir build-oem
> kiwi-ng --type iso --profile=MyProfile system create --root
> build/image-root --target-dir build-iso
>
> Which does install both sets of packages and then creates the images,
> which is why I don't think it's a bug per se -- but perhaps the runtime
> checks could detect this case too?

I agree it could be considered unexpected behavior too, because
you explicitly expressed I want package A for type iso and package B for
type oem, why do I get both now. A runtime check could have warned
you about this situation, correct.

> Side note: it does seem a bit awkward at the moment to produce multiple
> outputs; perhaps --type for system build could accept multiple values
> like --profile does, and allow building to the same target dir? Though
> maybe that's tricky.

It's tricky because the way you described the image requires sort of a
rollback of the tree when building for type A and for type B. kiwi does
not support that. If you are keen on filesystem technologies you could
consider a workflow that does the following:

* create btrfs snapshot for image build A
kiwi-ng system build --type ... --profile ...into-snapshot

* create btrfs snapshot for image build B
kiwi-ng system build --type ... --profile ...into-snapshot

bring up snapshot A
kiwi-ng system build ... --allow-existing-root

bring up snapshot B
kiwi-ng system build ... --allow-existing-root

That way you can manage multiple versions of your root tree and
rebuild your image by re-using the correct tree. The root tree data diff
is managed by btrfs in an effective way.

> Another side note: is there a way to do separate prepare/create steps
> like this with boxbuild? It didn't seem so from the docs. (Though I
> can't actually get boxbuild working atm for reasons discussed elsewhere
> anyway. I'd like to get it working because I think kiwi is leaving
> dangling processes on my host.)

What do you mean by separate ? separate --target-dirs yes. parallel
builds should be possible too. If kiwi leaves dangling processes on
your host that would be really a bad thing and we should debug why

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------
signature.asc

Marcus Schäfer

unread,
Oct 18, 2024, 10:34:59 AM10/18/24
to kiwi-...@googlegroups.com
Hi,

> kiwi-ng --profile=MyProfile system prepare --description . --root
> build/image-root
>
> kiwi-ng --type oem --profile=MyProfile system create --root
> build/image-root --target-dir build-oem
> kiwi-ng --type iso --profile=MyProfile system create --root
> build/image-root --target-dir build-iso
>
> Which does install both sets of packages and then creates the images,
> which is why I don't think it's a bug per se -- but perhaps the runtime
> checks could detect this case too?
>
> (Though after this the iso still doesn't actually boot, which I'm still
> investigating -- the oem works fine though.)

It's because the oem build modified the tree for the disk boot
setup and that confused the subsequent iso build. You can try to
delete the created etc/fstab file in an 'images.sh' script.
images.sh is called at the beginning of "system create" and can be
used to clean artifacts from former builds

> For now I've modified the above sequence to the following, which
> appears to work (at the cost of more disk space):
> kiwi-ng --profile=MyProfile system prepare --description . --root
> build/image-root
> rsync -a build/image-root/ build/oem-root/
> rsync -a build/image-root/ build/iso-root/
> kiwi-ng --type oem --profile=MyProfile system create --root
> build/oem-root --target-dir build-oem
> kiwi-ng --type iso --profile=MyProfile system create --root
> build/iso-root --target-dir build-iso
> Is there a better way to build multiple types from one prepare? Or is
> this approach doomed?

preserving the tree twice is an option, or you try with filesystem
capabilities e.g btrfs snapshots

> Is there a way to get the hybrid persistence file working with Ventoy?

I've never used the Ventoy tool, sorry

> It does work when I burn the iso directly to a USB stick, but when I
> tried using live-grub-stick it wasn't bootable (that doesn't seem to
> support EFI?), and with Ventoy it is bootable but isn't persistent.

live-grub-stick afaik only installs an MBR to the stick and does
not create a EFI layout. But to be sure you should contact the
maintainer of live-grub-stick, Lars Rupp lr...@suse.de
signature.asc

Gavin Lambert

unread,
Oct 21, 2024, 7:24:51 PM10/21/24
to kiwi
On Saturday, October 19, 2024 at 3:18:12 AM UTC+13 Marcus wrote:
> Another side note: is there a way to do separate prepare/create steps
> like this with boxbuild? It didn't seem so from the docs. (Though I
> can't actually get boxbuild working atm for reasons discussed elsewhere
> anyway. I'd like to get it working because I think kiwi is leaving
> dangling processes on my host.)

What do you mean by separate ? separate --target-dirs yes. parallel
builds should be possible too.

I meant that boxbuild appears to only do the equivalent of "build", but that means I can't do the "prepare", "create", "create" workflow described above.  So if I wanted to use boxbuild then I'd end up doing separate prepare steps, which is what I was trying to avoid since that's quite lengthy in my case and the result should be nearly identical in both cases.

As an alternative, is it somehow possible to make a single "oem" build that produces a single installer ISO that offers a live boot menu option as well?
 
If kiwi leaves dangling processes on your host that would be really a bad thing and we should debug why
 
I think this was mostly a red herring; I did have some stray init processes that I thought were from docker (I'm running kiwi in a docker container in lieu of boxbuild) but on closer inspection actually they were from something else.  Having said that, after doing many kiwi builds it seems to eventually trigger sgdisk to hang (non-interruptible wait state), and then I have to reboot.  It's quite a large number of builds though so it's irritating but not insurmountable.  I suspect it's not really kiwi or even sgdisk's fault but some quirk of WSL's kernel. 😅

Marcus Schäfer

unread,
Oct 22, 2024, 4:25:30 AM10/22/24
to kiwi-...@googlegroups.com
Hi,

> I meant that boxbuild appears to only do the equivalent of "build", but
> that means I can't do the "prepare", "create", "create" workflow
> described above.

Yes boxbuild calls 'system build' in the box.

> So if I wanted to use boxbuild then I'd end up doing
> separate prepare steps, which is what I was trying to avoid since
> that's quite lengthy in my case and the result should be nearly
> identical in both cases.

Hmm, true

> As an alternative, is it somehow possible to make a single "oem" build
> that produces a single installer ISO that offers a live boot menu
> option as well?

When building an installer ISO (installiso="true") you can specify
an alternative template for the bootloader. See: https://osinside.github.io/kiwi/image_description/elements.html and search for 'grub_template'

This could be used to add a new boot entry and maybe with some
tricks can be "abused" to boot the installer ISO up to a login
such that it looks like a live system. To be honest this is
pretty hacky, I would not do it.

From the other information you provided it seems like you are
building the image in a container. Thus I assume you do something
similar to 'podman run ...' which then starts a kiwi process.
Each container instance manages its own storage space, often as
an overlay. So you could also consider to target each kiwi build
to one dedicated container instance. Thoughts ?
signature.asc

Gavin Lambert

unread,
Oct 23, 2024, 6:53:28 PM10/23/24
to kiwi
On Tuesday, October 22, 2024 at 9:25:30 PM UTC+13 Marcus wrote:
When building an installer ISO (installiso="true") you can specify
an alternative template for the bootloader. See: https://osinside.github.io/kiwi/image_description/elements.html and search for 'grub_template'

This could be used to add a new boot entry and maybe with some
tricks can be "abused" to boot the installer ISO up to a login
such that it looks like a live system. To be honest this is
pretty hacky, I would not do it.

Is there a copy of the standard template or another example around somewhere?  The docs only have a list of variables.

But you're probably right, that sounds more complicated than I wanted; it was just a "would be nice" to avoid making two nearly-identical images, but that in itself is probably just a short-term thing while I'm testing ideas.
 
From the other information you provided it seems like you are
building the image in a container. Thus I assume you do something
similar to 'podman run ...' which then starts a kiwi process.
Each container instance manages its own storage space, often as
an overlay. So you could also consider to target each kiwi build
to one dedicated container instance. Thoughts ?

I don't think that would help; I'm bind-mounting the source tree into the container, not copying it, so that the output results also end up in the same tree.  If I did the build in the container filesystem instead then it would change which things needed to be copied but I don't think it would reduce the copying at all.  It might be easier to do the OEM and ISO "system create" steps in parallel that way, but that's not really the slow bit (and the slowest part of that is the XZ compression, which would only get worse if done in parallel).
Reply all
Reply to author
Forward
0 new messages