Kernel file in config partitions and swupdate

[Marco Braga]

Hi,

I am currently testing a system based on yocto dunfell + swupdate + EFIBootguard and I am concerned by the way the kernel is stored in the same partition of the configuration.

I am using 2 config partitions for EFIBootguard, and also a double copy strategy for rootfs. At start, partition 0 contains the kernel associated with the installed rootfs.

If I use swupdate to update some parts of the system (not related to rootfs or kernel), this creates a new configuration on partition 1, but the kernel file position is still on partition 0.

Should then I need to update both the kernel and the rootfs, this would switch back to having a new active configuration on partition 0, but the new kernel should be written on configuration partition 1 to preserve the currently running kernel (resident on partition 0) as a fail safe in case of a rollback.

This brings to a situation where cfg in use is in a partition,  but the kernel is on another one.

It also makes updated more complex: due to the double copy strategy, not only I have to detect the inactive rootfs partition, but also the inactive configuration partition where to store the kernel. I am really uneasy about this.

Sadly EFIBootguard cannot boot the kernel stored on ext4 /boot rootfs directlry, so I am 

looking for an alterlative strategy. At the moment this involves 2 partitions dedicated to storing the kernel file, without saving it in the configuration partitons.

What do you think?

Thanks!

[Jan Kiszka]

On 13.06.22 22:49, Marco Braga wrote:
> Hi,
> I am currently testing a system based on yocto dunfell + swupdate +
> EFIBootguard and I am concerned by the way the kernel is stored in the
> same partition of the configuration.
> I am using 2 config partitions for EFIBootguard, and also a double copy
> strategy for rootfs. At start, partition 0 contains the kernel
> associated with the installed rootfs.
> If I use swupdate to update some parts of the system (not related to
> rootfs or kernel), this creates a new configuration on partition 1, but
> the kernel file position is still on partition 0.

Do not touch the EFI Boot Guard config unless you updated something that
is related to it. You can perfectly use swupdate to change artifacts on
your installed system without touching the boot paths, e.g. deploy a new
application into a common (not a/b-updated) partition.

However, if you update something that affects the updatability of the
resulting boot path (kernel, rootfs and possibly more), make sure to
create a consistent v2 of all those bits. Then switch. You may optimize
the update, avoiding to transfer also unchanged artifacts by either
using a delta update or by copying unchanged bits over from v1 on the
target using swupdate scripting.

> Should then I need to update both the kernel and the rootfs, this would
> switch back to having a new active configuration on partition 0, but the
> new kernel should be written on configuration partition 1 to preserve
> the currently running kernel (resident on partition 0) as a fail safe in
> case of a rollback.
> This brings to a situation where cfg in use is in a partition,  but the
> kernel is on another one.

And that is a bug in your configuration. Kernel and rootfs should be
considered one logical artifact, even if they are distributed in
separate pieces. So should the related config switch look like. Anything
else will just break in practice.

> It also makes updated more complex: due to the double copy strategy, not
> only I have to detect the inactive rootfs partition, but also the
> inactive configuration partition where to store the kernel. I am really
> uneasy about this.
> Sadly EFIBootguard cannot boot the kernel stored on ext4 /boot rootfs
> directlry, so I am 

That is a related to UEFI not supporting ext4 or other filesystems by
default. Could be addressed with extra UEFI drivers, but I'm not even
aware if those exist. It's easier and probably also more secure (if you
are after secure boot) to confine the functionality of UEFI.

> looking for an alterlative strategy. At the moment this involves 2
> partitions dedicated to storing the kernel file, without saving it in
> the configuration partitons.
> What do you think?

There is no need to store the kernel and the corresponding EBGENV.DAT in
different partitions if you follow the switching pattern above.

Jan

--
Siemens AG, Technology
Competence Center Embedded Linux

[Marco Braga]

Hi Jan,

thanks for answering.

Il giorno mar 14 giu 2022 alle ore 12:17 Jan Kiszka <jan.k...@siemens.com> ha scritto:

Do not touch the EFI Boot Guard config unless you updated something that
is related to it. You can perfectly use swupdate to change artifacts on
your installed system without touching the boot paths, e.g. deploy a new
application into a common (not a/b-updated) partition.

The problem is that swupdate uses the bootloader configuration to store update status when used with hawkbit server.

Quick check (https://github.com/sbabic/swupdate/blob/a10214e88eba43ae0382c84d9f0d10cf5d623e2a/suricatta/server_hawkbit.c#L926 and several other points).

By default swupdate creates a "transaction" every time it makes an update even if the file does not touch boot related artifacts (kernel or rootfs). This behavior can be forcibly disabled, but then swupdate cannot read back the update status.

 

And that is a bug in your configuration. Kernel and rootfs should be
considered one logical artifact, even if they are distributed in
separate pieces. So should the related config switch look like. Anything
else will just break in practice.

I totally understand and agree, I am trying to make things work with this setup:

EFIBootguard + swupdate + hawkibit.

The situation is this:

1. swupdate + hawkbit expects to send update status to hawkbit if it finds TESTING usate in current EBG configuration

https://github.com/sbabic/swupdate/blob/a10214e88eba43ae0382c84d9f0d10cf5d623e2a/suricatta/server_hawkbit.c#L926

2. swupdate normally creates a "transaction" on each update, and current implementation of EBG clones the current configuration to a new one:

https://github.com/sbabic/swupdate/blob/a10214e88eba43ae0382c84d9f0d10cf5d623e2a/bootloader/ebg.c#L57

3. Due to above, it may happens that:

a. A system update stores kernel in C:BOOT0/bzImage, using CFG #0

b. swupdate applies an update (which may be something totally unrelated to boot artifacts - kernel or rootfs etc)

c. swupdate clones CFG #0 to CFG #1 due to its implementation of transactions, but the kernel path is still C:BOOT0/bzImage

d. on reboot, swupdate finds CFG#1 in ustate TESTING ans commits CFG #1 setting ustate to OK, then notifies hawkbit that the update went well.

Result is that CFG#1 is used for boot, and its kernel path is on partition used by CFG #0

The update has nothing to to with boot, so it didn't touch kernel path in CFG #1 which still points to C:BOOT0/bzImage.

I am not sure how to handle this.

 

> looking for an alterlative strategy. At the moment this involves 2
> partitions dedicated to storing the kernel file, without saving it in
> the configuration partitons.
> What do you think?

There is no need to store the kernel and the corresponding EBGENV.DAT in
different partitions if you follow the switching pattern above.

 Ok, assuming I find a workaround for the problem above, what is the suggested way of updating the kernel for next boot?

If I understand correctly I should create a new CFG (ex: bg_setenv -u), then mount the partition used by the new config and write the kernel there, umount, set the kernel path as required and ustate to INSTALLED and reboot.

Is it correct?

What if the FAT partition is corrupted when writing the kernel there? (Still a possibility on embedded devices).

Thanks,

Marco

[Christian Storm]

Hi,

> > Do not touch the EFI Boot Guard config unless you updated something that
> > is related to it. You can perfectly use swupdate to change artifacts on
> > your installed system without touching the boot paths, e.g. deploy a new
> > application into a common (not a/b-updated) partition.
> >
>
> The problem is that swupdate uses the bootloader configuration to store
> update status when used with hawkbit server.

Not only when used with the hawkBit suricatta module but also for other
"ingress" channels. It seems that your "problem" is actually different
*types* of artifacts: one is firmware for which you probably will want to
persistently store state and the other is some (application) software for
which you don't want such behavior, right?


> Quick check (

> https://github.com/sbabic/swupdate/blob/a10214e88eba43ae0382c84d9f0d10cf5d623e2a/suricatta/server_hawkbit.c#L926
> and several other points).
> By default swupdate creates a "transaction" every time it makes an update
> even if the file does not touch boot related artifacts (kernel or rootfs).
> This behavior can be forcibly disabled, but then swupdate cannot read back
> the update status.

There are actually *two* markers SWUpdate persistently stores in the
bootloader environment:
(1) bootloader_transaction_marker → recovery_status
(2) bootloader_state_marker → ustate (per default)

> > And that is a bug in your configuration. Kernel and rootfs should be
> > considered one logical artifact, even if they are distributed in
> > separate pieces. So should the related config switch look like. Anything
> > else will just break in practice.
> >
>
> I totally understand and agree, I am trying to make things work with this
> setup:
> EFIBootguard + swupdate + hawkibit.
> The situation is this:
> 1. swupdate + hawkbit expects to send update status to hawkbit if it finds
> TESTING usate in current EBG configuration
> https://github.com/sbabic/swupdate/blob/a10214e88eba43ae0382c84d9f0d10cf5d623e2a/suricatta/server_hawkbit.c#L926
>
> 2. swupdate normally creates a "transaction" on each update, and current
> implementation of EBG clones the current configuration to a new one:
> https://github.com/sbabic/swupdate/blob/a10214e88eba43ae0382c84d9f0d10cf5d623e2a/bootloader/ebg.c#L57

You can disable one or both of the above with a snippet like the
following in the sw-description file *per update artifact* ―
though also globally which you don't want to if I got it right:

software =
{
bootloader_transaction_marker = false;
bootloader_state_marker = false;

...

So, setting bootloader_transaction_marker = false solves your problem?



Kind regards,
Christian

--
Dr. Christian Storm
Siemens AG, Technology, T CED SES-DE
Otto-Hahn-Ring 6, 81739 München, Germany

[Marco Braga]

Hi Christian,

thanks a lot for helping out!

Il giorno mar 14 giu 2022 alle ore 17:29 Christian Storm <christi...@siemens.com> ha scritto:

It seems that your "problem" is actually different
*types* of artifacts: one is firmware for which you probably will want to
persistently store state and the other is some (application) software for
which you don't want such behavior, right?

Totally correct, you nailed the problem. We need to update an application (running on node) without touching the rootfs or kernel. Still, we'd also like to update the kernel+rootfs combination if required.

There are actually *two* markers SWUpdate persistently stores in the
bootloader environment:
(1) bootloader_transaction_marker → recovery_status
(2) bootloader_state_marker → ustate (per default)

Correct. Both are handled differently depending on the actual bootloader. If using grub or u-boot, it uses variables. When using EFIBootguard instead (1) creates a new configuration cloning the old one.

Due to the "clone" behavior and with 2 configuration partitions, after the first update it happens that CFG #1 has the highest revision number. kernelparams still point to the correct rootfs device (unchanged by the update) and kernelpath still points to the previous kernel path, which is the CFG #0 partition.

Now, should we decide to update the kernel, we'll be in the dire situation of having to touch both the partitions:CFG #0 contains the failback kernel, so we'll have to write the new one in the partition of CFG #1, while the update will actually change the configuration of partition CFG #0 (which is not in use).

It seems a mess to me.

At the moment I am working with a different setup: I have double copy partitions of boot rootfs and kernel. Kernels are not stored in the same partitions as EFI configs, but configs point to the correct kernel path using labels. It seems a lot more safe to me, even without considering swupdate.

You can disable one or both of the above with a snippet like the
following in the sw-description file *per update artifact* ―
though also globally which you don't want to if I got it right:

software =
{
    bootloader_transaction_marker = false;
    bootloader_state_marker = false;

    ...

So, setting bootloader_transaction_marker = false solves your problem?

Thanks for your suggestion, this may actually work. swupdate will not be able to report update status back to hawkbit automatically though, so this will require specific handling (restarting swupdate with "-c 2" parameter?).

I am not sure if this is the best way to go, but I'll try it.

In any case, updating the kernel file on a separate partition seems way safer to me in case of a power failure, so if there are no drawbacks I may consider to continue with separate kernel partitions and keeping only configs in EBG partitions.

As a last question, what is the suggested kernel update procedure with EFIBootguard?

Do you have any considerations or suggestions?

Thanks,

Marco