versioning beaglebone images with OSTree

[John Allwine]

I'd like to start a discussion about creating complete Beaglebone images that leverage OSTree to be able to atomically update the system as a whole. The scripts in https://github.com/beagleboard/image-builder generate complete images for the Beaglebone that include specific kernel, apt packages, boot settings, git repositories, etc. Updating a deployed Beaglebone without reflashing a new image involves piecemeal updating of those various components. Improperly updating can leave the system in a broken state and can be difficult to get back into a good state. It would be great to be able to leverage those image-builder scripts to construct the rootfs, add that tree as a commit to an OSTree repository and properly configured Beaglebones could download that commit and atomically switch to it to update the whole system while preserving portions of the system such as home directories and other key directories (/etc, /var?). If something did break, rolling back is easy as well.

 

Configuring a Beaglebone this way would make most of the system read-only so using apt-get to install new packages wouldn't work without altering its implementation, but that seems like a worthy trade off. This would be for someone who has a Beaglebone with an out-of-the-box image and some scripts/servers set up in their home directory who doesn't want to worry too much about the system as a whole, but wants to be able to easily update it without reflashing or doing piecemeal updates. People who develop software for Beaglebones in their customers' devices could host their own OSTree repository and make their own modifications to the image-builder scripts if they have their own set of system dependencies (this is what I'd like to do).

Does anyone else think this would be useful? Is there anyone with the expertise to know what details would need to be taken into account to make this work properly?

OSTree documentation is here: https://ostreedev.github.io/ostree/

It lists a number of examples of it being used in various Linux distributions.

[Robert Nelson]

I remember seeing one of Peter Robinson's demo of Fedora IoT a few
years back at ELC, that used OSTree+btrfs. It worked pretty well. At
that time, I made sure btrfs worked well for us, to possibly look down
that road. My biggest issue, the 4gb eMMC, was too limited for the
out of box images to do something like that. For an iot/console image
the idea would still work well.. While working on bullseye images
this week, i noticed we still have the "--no-merged-usr" flag for
debootstrap, we should try with that removed in 'bullseye', as ostree
needs that..

We did have ostree installed on the lxqt images:

https://github.com/beagleboard/image-builder/blob/master/configs/bb.org-debian-buster-lxqt-v5.4.conf#L138

--no-merged-usr (due to bugs in stretch/buster..)
https://github.com/beagleboard/image-builder/blob/master/scripts/debootstrap.sh#L138

Regards,

--
Robert Nelson
https://rcn-ee.com/

[John Allwine]

What is the biggest image currently?

OSTree is smart about space, so if updated incrementally and old images get pruned, it could work. But I agree it’s a big limitation. Using a microSD card in that case would be better.

> On Apr 16, 2021, at 6:04 PM, Robert Nelson <robert...@gmail.com> wrote:

> --
> For more options, visit http://beagleboard.org/discuss
> ---
> You received this message because you are subscribed to the Google Groups "BeagleBoard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beagleboard...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/beagleboard/CAOCHtYgHJiESN2df7jVbUErSwL5mcoPva-woQXR91t%3D0nCJJDQ%40mail.gmail.com.

[John Allwine]

I noticed that adding ostree to deb_includes causes problems. What’s different about adding it to deb_additional_pkgs?

> On Apr 17, 2021, at 7:27 AM, John Allwine <jo...@pocketnc.com> wrote:
>
> What is the biggest image currently?

[Robert Nelson]

On Sat, Apr 17, 2021 at 8:35 AM John Allwine <jo...@pocketnc.com> wrote:
>
> I noticed that adding ostree to deb_includes causes problems. What’s different about adding it to deb_additional_pkgs?

So, deb_includes get directly passed to debootstrap in the first phase
of debootstrap, so if it's requirements are little too complex it will
fail, the deb_additional_pkgs get added during the second pass, where
most of the debian initial base is already setup..

So, when adding a new package, i always add it to deb_includes first,
then test.. If it fails i bump it to deb_additional_pkgs..

[Robert Nelson]

bullseye and later now has --no-merged-usr disabled:

https://github.com/RobertCNelson/omap-image-builder/commit/2d7bf137e3447038142a83751ed4fca76faca5fe

[John Allwine]

Thanks Robert! Do you have a sense of what the major hurdles of implementing this are? From what I can tell:

* Setting up U-Boot to look in the proper ostree deployment

* Setting up bind mounts to the directories in /

* Ensuring the system doesn't write to any directories other than /etc and /var

* Handling merges of /etc on update

* Some kind of automatic rollback handling in case something goes wrong with an update and system won't boot

Does this sound right? Any thoughts on any of those? Am I missing anything major (my guess is likely!)?

[John Allwine]

Where is the source code for what u-boot runs by default on the Beaglebone images? I see some references to patches that I assume are applied to u-boot on the Beaglebone:

https://github.com/beagleboard/image-builder/blob/master/target/chroot/beagleboard.org-buster.sh#L276-L291

Is there somewhere I can see the applied versions of those or do I have to manually apply them? I'm trying to understand what exactly the environment variables in uEnv.txt do.

[Robert Nelson]

On Tue, May 4, 2021 at 5:28 PM John Allwine <jo...@pocketnc.com> wrote:
>
> Where is the source code for what u-boot runs by default on the Beaglebone images? I see some references to patches that I assume are applied to u-boot on the Beaglebone:
> https://github.com/beagleboard/image-builder/blob/master/target/chroot/beagleboard.org-buster.sh#L276-L291
>
> Is there somewhere I can see the applied versions of those or do I have to manually apply them? I'm trying to understand what exactly the environment variables in uEnv.txt do.

You can find that tree here:

https://github.com/beagleboard/u-boot/tree/v2019.04-bbb.io-am335x

it's built/provided via the debian package: bb-u-boot-am335x-evm

https://github.com/rcn-ee/repos/tree/master/bb-u-boot-am335x-evm

[John Allwine]

Thanks Robert!

What builds the initrd file? One strategy I'm seeing for using OSTree is do some of the filesystem mounting in initrd, but I'm not sure how to modify what is currently used.

-John