[meta-isar] Proposal to improve initial device bootstrapping

[Heinisch, Alexander]

# Device Bootstrapping

This is a proposal to improve initial device bootstrapping with meta-isar by making the `isar-image-installer` a more versatile and general tool.

## Background

Currently, the `isar-image-installer` contains the target image to be deployed (copied) to the target device in it's root filesystem. The installer image has to be copied to a usb stick and executed on the target device.

In our current manufacturing setup we are targeting prebuilt devices without any OS precommissioned. Flashing images directly to disk is not possible at that stage easily.
That's why we are using the `isar-image-installer` to deploy the target images to the device via usb.

## Motivation

This approach works fine when working with a single device on desk, having keyboard and screen attached, but does not scale for large rollouts for multiple devices during manufacturing.

To scale that process I suggest not only supporting a usb stick scenario, but also a variant to boot via pxe boot (or ipxe-boot) into an live os (which could (and probably will) be `isar-image-installer`).

> Note: Currently, we are targeting x86 based architectures providing UEFI.

## Identified Problems

1.  **Problem**: The installer script has to provide an unattended mode.

    **Possible Solution**: Add setting for unattended mode either via well known config file or via kernel cmdline.

2.  **Problem**: When embedding the target image into the installer rootfs a rebuild of the installer image is required everytime we change the target image.

    **Possible Solution**: Installer image could download target image from http/ftp/s3 server at runtime and install it from memory. (Therefore, we have to ensure enough memory is provided, or probably support some kind of streaming functionality)

3.  **Problem**: Since pxe transferrs only the kernel and the initramfs via TFTP (rather slow) When using pxe we have to provide the rootfs of the installer via nfs.

    **Possible Solution**: Having an online installer downloading the target images from some external source, enables us to put all installer logic in the installers initramfs. Thus, no need for an installer-rootfs.

    > Note: This not always works. Since we also want to support the usb use case, loading the target image from rootfs is still a desireable option we have to maintain!

4.  **Problem**: Enrolling secure boot keys has to be done manually now. Currently we are using scripts to do so which get executed after the installer ran.
This is needed, since the installer is not signed.

    **Possible Solution**: Sign installer.

5.  **Problem**: Still, enrolling the keys manually upfront is cumbersome and error prone, and buying devices with preenrolled keys, oftentimes is not wanted due to additional cost and additional trust. Enrolling the keys after installation can be done, but again, is a manual task which should be automated.

    **Possible Solution**: Enroll secureboot keys as an additional step during installation.

    > Note: Since `installation` is not an appropriate term anymore, when not only the image get's installed but additional steps like key-enrollment takes place, I will call that workflow `target-bootstrapping` in the remainder of this text.

6.  **Problem**: Disc encryption is currently done on first boot of the device (detects if disk is already encrypted, and if not, encrypts it.) We saw that this process sometimes takes several minutes and is one of the crucial parts when initially starting up. In our scenario after a device got precommissioned it is put aside and stored (without initial boot of the target os). Once manufacturing needs to pick up a new device it is taken from there and assembled to the main asset shipped to the customer during asset production. Since that step has to be as easy and as fast as possible, waiting several minutes (due to initial encryption) to check basic device information or worse, failing at that stage is inacceptible.

    **Possible Solution**: Encrypt target device disks as an additional step during `target-bootstrapping`.

7.  **Problem**: After the initial procomissioning of the device status information of the device (e.g. serial number, hardware info) has to be transferred to our central mgmt. system.

    **Possible Solution**: Run custom scripts as part of the `target-bootstrapping`

8.  **Problem**: During `target-bootstrapping` the progress of the bootstrapping has to be visualized. When talking about bootstrapping multiple devices attaching a screen is not desired. Thus we plan to give some status indication via LED drivers as well, and also report status to our central mgmt. system.

    **Possible Solution**: Run custom scripts for status reporting. This means, that customizeable scripts shall be invoked before and after every single bootstrapping phase, and ideally also reporting an overall progress.

## Draft

Instead of excecuting the deploy image script as a systemd service we propose to implement a configurable target-bootstrapper, which takes prepackaged scripts as an input and invokes them in a generic way.

```
TARGET_BOOTSTRAPPER_ADDITIONAL_PACKAGES += " deploy-image"
TARGET_BOOTSTRAPPER_TASK_deploy-image[script] = "deploy-image-wic.sh"
TARGET_BOOTSTRAPPER_TASK_deploy-image[workdir] = "/usr/bin"
TARGET_BOOTSTRAPPER_TASK_deploy-image[effort] = "2"
```

This configuration enables us to reuse existing upstream (e.g. deploy-image [1]) as well as downstream scripts (e.g. encrypt partition [2] from cip-core or enroll secure boot keys from other downstream repo) without code-duplication.

To allow such bootstrapper to report progress between execution of each of the prepackaged scripts, customized status reporting utilities can be configured and will be invoked. Such utilities include e.g. led drivers, status reporting via an REST service, aso.

Each script-configuration can not only specify a dedicated workdir and entrypoint, but also an effort-estimate to weight the work performed within a single script more accurately.

Besides coming up with an initial draft of such target-bootstrapping (will send a patchseries in the upcoming days) one of the first steps will be to refactor the existing deploy-image-wic.sh to allow for `unattended-mode` (based on this patche-series [3] from Jan Kiszka) and extend the script to support downloading the target images from an http server.


[1] https://github.com/ilbers/isar/blob/master/meta-isar/recipes-installer/deploy-image/files/deploy-image-wic.sh
[2] https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/master/recipes-initramfs/initramfs-crypt-hook/files/encrypt_partition.script?ref_type=heads
[3] https://patchwork.isar-build.org/project/isar/patch/6279c4d497ade9a55cad9c0f2f21834ae...@siemens.com/

Looking forward for your inputs,
Thank you!
Alexander

[Gylstorff Quirin]

On 7/16/24 3:30 PM, Heinisch, Alexander (T CED SES-AT) wrote:
> # Device Bootstrapping
>
> This is a proposal to improve initial device bootstrapping with meta-isar by making the `isar-image-installer` a more versatile and general tool.
>
> ## Background
>
> Currently, the `isar-image-installer` contains the target image to be deployed (copied) to the target device in it's root filesystem. The installer image has to be copied to a usb stick and executed on the target device.
>
> In our current manufacturing setup we are targeting prebuilt devices without any OS precommissioned. Flashing images directly to disk is not possible at that stage easily.
> That's why we are using the `isar-image-installer` to deploy the target images to the device via usb.
>
> ## Motivation
>
> This approach works fine when working with a single device on desk, having keyboard and screen attached, but does not scale for large rollouts for multiple devices during manufacturing.
>
> To scale that process I suggest not only supporting a usb stick scenario, but also a variant to boot via pxe boot (or ipxe-boot) into an live os (which could (and probably will) be `isar-image-installer`).
>
>> Note: Currently, we are targeting x86 based architectures providing UEFI.
>
> ## Identified Problems
>
> 1.  **Problem**: The installer script has to provide an unattended mode.
>
>     **Possible Solution**: Add setting for unattended mode either via well known config file or via kernel cmdline.
>
> 2.  **Problem**: When embedding the target image into the installer rootfs a rebuild of the installer image is required everytime we change the target image.
>
>     **Possible Solution**: Installer image could download target image from http/ftp/s3 server at runtime and install it from memory. (Therefore, we have to ensure enough memory is provided, or probably support some kind of streaming functionality)

This sounds like what SWUpdate or rauc already provide. Which could be
used to describe the system and also provide a installation state.

Quirin

[Heinisch, Alexander]

> > 2. **Problem**: When embedding the target image into the installer rootfs a rebuild of the installer image is required everytime we change the target image.
> >
> > **Possible Solution**: Installer image could download target
> > image from http/ftp/s3 server at runtime and install it from memory.
> > (Therefore, we have to ensure enough memory is provided, or probably
> > support some kind of streaming functionality)
>
> This sounds like what SWUpdate or rauc already provide. Which could be used to describe the system and also provide a installation state.
>
> Quirin

(I will focus on SWUpdate, since we already ship it in our target images.)

That means instead of deploying a wic image like now, we could ship
e.g. SWUpdate and use an swu file to execute the tasks described?
Therefore, a reuse of partitioning and image deployment from such utility
was possible, putting the steps described for encryption,
secure-boot deployment, aso. as post scripts.

Although, I see the value when it comes to image download/streaming
and verification of such, I'm not sure if the gained value of having SWUpdate
outweights the effort of shipping SWUpdate for an installer,
compared to a relatively simple script.
bmaptool, which is used right now to deploy the image, already supports
downloads from http(s)://, ftp://, ssh:// and file:// sources, checksums
and signature verification.

+ Reuse of existing logic (SWUpdate)
- Harder (although not that hard at all) to understand compared to
(imo relatively) simple script logic
- Build and ship additional component/dependency SWUpdate
~ Build additional image for swu
(probably that's done anyways if SWUpdate is in use for OS/Firmware updates,
but downstream projects could use other stacks like rauc, mender, ... already)
+ Same workflows for encryption / sb-enrollment could also be used on upgrades

What was the reason not to use SWUpdate for the current installer?

BR Alexander

[Gylstorff Quirin]

On 7/18/24 1:34 PM, Heinisch, Alexander (T CED SES-AT) wrote:
>>> 2. **Problem**: When embedding the target image into the installer rootfs a rebuild of the installer image is required everytime we change the target image.
>>>
>>> **Possible Solution**: Installer image could download target
>>> image from http/ftp/s3 server at runtime and install it from memory.
>>> (Therefore, we have to ensure enough memory is provided, or probably
>>> support some kind of streaming functionality)
>>
>> This sounds like what SWUpdate or rauc already provide. Which could be used to describe the system and also provide a installation state.
>>
>> Quirin
>
> (I will focus on SWUpdate, since we already ship it in our target images.)
>
> That means instead of deploying a wic image like now, we could ship
> e.g. SWUpdate and use an swu file to execute the tasks described?
> Therefore, a reuse of partitioning and image deployment from such utility
> was possible, putting the steps described for encryption,
> secure-boot deployment, aso. as post scripts.
>
> Although, I see the value when it comes to image download/streaming
> and verification of such, I'm not sure if the gained value of having SWUpdate
> outweights the effort of shipping SWUpdate for an installer,
> compared to a relatively simple script.
> bmaptool, which is used right now to deploy the image, already supports
> downloads from http(s)://, ftp://, ssh:// and file:// sources, checksums
> and signature verification.

Signatures are pgp based so an additional effort to generate and
maintain them.

>
> + Reuse of existing logic (SWUpdate)
> - Harder (although not that hard at all) to understand compared to
> (imo relatively) simple script logic
> - Build and ship additional component/dependency SWUpdate
> ~ Build additional image for swu
> (probably that's done anyways if SWUpdate is in use for OS/Firmware updates,
> but downstream projects could use other stacks like rauc, mender, ... already)
> + Same workflows for encryption / sb-enrollment could also be used on upgrades
>
> What was the reason not to use SWUpdate for the current installer?

The initial use case of the current installer was to have a simple way
to deploy test image onto the disk of a target device. This first
implementation was using dd instead of bmaptool.


Quirin
>
> BR Alexander

[Niedermayr, BENEDIKT]

Yes, sounds like swupdate and friends...

Nevertheless, I find the idea of using swupdate rather than the current script not that bad at all.
I has some nice side effects. For example any custom pre/post processing could be simply done by
writing scripts and add them to the swu container. So there would be no need to customize the
current update script. That decouples downstream customizations very nicely...

In conjunction with wfx [1] as update server I could imagine that very complex setups were possible.

I'm not sure wether or not isar is the right place for this feature. This could potentially fit
better into isar-cip-core which implements these type common patterns?!


[1] https://github.com/siemens/wfx

>
> 3.  **Problem**: Since pxe transferrs only the kernel and the initramfs via TFTP (rather slow)
> When using pxe we have to provide the rootfs of the installer via nfs.

You could checkout iPXE. It uses http/https for downloading.

>
>     **Possible Solution**: Having an online installer downloading the target images from some
> external source, enables us to put all installer logic in the installers initramfs. Thus, no need
> for an installer-rootfs.

Using an initramfs may be fine, but as soon as the installer needs internet you will encounter
issues due to missing support of different libraries. In other words, the more features we support
the more needs to be added into the initramfs and installation into initramfs can be very cumbersome
and time consuming (files have to be installed directly since no package-manager available).