Building a read-only Tumbleweed live image with an encrypted rootfs - any way to make this work?

[Ioan Matei]

Hello everyone,

I'm looking for a way to build a Tumbleweed KIWI live image with some variant of encrypted, password-unlockable rootfs that will then be flashed to what essentially is a glorified USB drive. 

The current version of this image is a rather minimal dmsquash-based read-only KIWI live image where the build results in an ISO that is then flashed to the drive. 

This result has worked well so far, including painless Secure Boot support through OpenSuse's shim loader. However I am now confronted with the requirement of encrypting the contents of the root filesystem for further releases of this image.

On a system that's not built automatically I'd use LVM/LUKS for this goal and let GRUB display the password prompt needed to unlock the rootfs. I've also seen in the KIWI docs that there is an option for LUKS for several image types, this doesn't seem to apply to live images though. 

Is there any way to entice KIWI to build a read-only live image that is LUKS-encrypted? Ideally this approach would feature a Secure Boot-compliant GRUB with the standard password prompt known from LVM/LUKS, and produce the same kind of easily flashable ISO that is the output of a standard KIWI live build process. 

(I'm aware that this request is an edge case of an edge case but I'd still be very thankful for any possible solutions to this quagmire).

Thank you!

Ioan

[Marcus Schäfer]

Hi,

> Is there any way to entice KIWI to build a read-only live image that is
> LUKS-encrypted? Ideally this approach would feature a Secure
> Boot-compliant GRUB with the standard password prompt known from
> LVM/LUKS, and produce the same kind of easily flashable ISO that is the
> output of a standard KIWI live build process.

The problem here is that you want this to build as a live ISO image.
The current live image builder has no support for encryption as you
already found out. I did an experimental patch to the live builder
such that it builds me an encrypted ISO. See here:

https://github.com/OSInside/kiwi/tree/luks_for_live

and the diff:

https://github.com/OSInside/kiwi/compare/luks_for_live?expand=1

It did work for me but I'm not sure if this is a useful setup
for the following reasons:

* We loose all compression advantage because of randomized encryption.
As such the ISO is not small

* The initrd and kernel are readable from the ISO unencrypted which
can be considered a security flaw. grub reading from eltorito, see
the xorriso call when you build... maybe this can be done differently
such that cryptomount from grub can be used. For iso's this would
require more kiwi changes though which I haven't done.

* Booting requires interaction because you need to enter the
luks passphrase unless set empty (luks=""). If set empty
no security at all exists only the layer and it can't be
turned to be secure because it's all read-only (ISO)
for testing and simplicity I used an empty passphrase

* For security re-encryption should be done because if not the
LUKS header could be read from the image binary. kiwi supports
that but it cannot be applied on a read-only media, again (ISO)

As such I wonder about the use case :)

I hope the provided patch and information helps

Best regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer

[Marcus Schäfer]

Hi

> It did work for me but I'm not sure if this is a useful setup
> for the following reasons:

after boot it looks like this for me:

localhost:~ # lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
fd0 2:0 1 4K 0 disk
loop0 7:0 0 2.3G 1 loop /run/overlay/squashfs_container
loop1 7:1 0 2.3G 1 loop
└─luks 254:0 0 2.3G 1 crypt /run/overlay/rootfsbase
sr0 11:0 1 2.4G 0 rom /run/overlay/live

so the iso9660 mount, followed by the squashfs mount is unencrypted.
The subsequent luks is opened against the squashfs content, LUKS
encrypted rootfs.

There are for sure different layouts possible and once I
understand the security concept when re-encrypt is not
possible due to a read-only media, I'm happy to add more
adjustments to this feature.

Thanks

Regards,

[Ioan Matei]

Hi Marcus,

honestly thank you for taking the time to create a new KIWI branch to support this outlandish request.

I've just set up a fresh Tumbleweed VM and tried your approach by using your modified kiwi-live-lib.sh, module-setup.sh (both in /usr/lib/dracut/modules.d), as well as the modified live.py (copied to /lib/python3.13/site-packages/kiwi/builder).

When satisfying the additional dependencies of cryptodisk, dolly, curl and dracut-kiwi-live, the Tumbleweed test image from kiwi-descriptions successfully builds and the logs show that the LUKS volume is created successfully.

Booting it, however, leads to dracut throwing the error "unknown filesystem type 'crypto_LUKS' (please see screenshot attached).

I've also attached screenshots of the config.xml in question. Might I have overlooked some dependency? This happens with a LUKS passphrase being set, and also with an empty passphrase.

  

Regarding the use case of this image: I'm not allowed to go into too much detail but basically the bootable image is part of a field service kit for some specialized food machinery. Even when unencrypted its contents would be quite useless without several other components but management wants to add another layer of encryption on top so that:

- people without the LUKS passphrase would not be able to easily view the contents of the rootfs

- people with access to the passphrase would be able to work with the tools provided in the image but be prevented from permanently altering the contents of the rootfs

- the result of of the build needs to be able to be flashed as an ISO image or, if necessary, via some dd-esque solution, resulting in a bootable USB or possibly NVMe drive.

I'm aware of the danger of not re-encrypting LUKS headers but the overarching opinion at the company is that a strong enough passphrase would buy us enough time to push an OTA update to the machinery to render the decrypted contents useless.

I'm also open to any other type of build that would allow more secure re-encryption but I haven't yet found any other type of image that ensures that even in the event of a root credential leak files can't permanently be altered without re-flashing the whole image. 

Thank you

Ioan

[Marcus Schäfer]

Hi Ioan,

> I've just set up a fresh Tumbleweed VM and tried your approach by using
> your modified kiwi-live-lib.sh, module-setup.sh (both in
> /usr/lib/dracut/modules.d), as well as the modified live.py (copied to
> /lib/python3.13/site-packages/kiwi/builder).
>
> When satisfying the additional dependencies of cryptodisk, dolly, curl
> and dracut-kiwi-live, the Tumbleweed test image from kiwi-descriptions
> successfully builds and the logs show that the LUKS volume is created
> successfully.
>
> Booting it, however, leads to dracut throwing the error "unknown
> filesystem type 'crypto_LUKS' (please see screenshot attached).

Yeah I'm pretty sure the changes in the dracut code has not made
it to the initrd of your image. I'm sorry I should have provided
more details about this one.

So when you have checked out the git with the luks_for_live branch
active, you need to apply a manual copy step of the dracut code to
become effective. The process of building should look like this:

1. build the image with the kiwi changes of the branch as you did

kiwi-ng system build ... --target-dir /tmp/mytest

2. Once this first build process completed copy in the actual
dracut code changes from the kiwi branch to your image root
tree.

sudo cp -a dracut/modules.d/* /tmp/mytest/build/image-root/usr/lib/dracut/modules.d/

3. Repeat the build process from step 1)

kiwi-ng system build ... --target-dir /tmp/mytest --allow-existing-root

The reason why this is needed is because the dracut code is provided
by a package named dracut-kiwi-live which gets installed in step 1)
This code was changed in the git branch and to make it effectively used
in the image build you need to copy it into the tree after the first
build and rebuild again with --allow-existing-root. This is a neat trick
to change root content :)

Hope this helps

[Ioan Matei]

Hi Marcus,

thank you for the hint with copying the dracut modules to the root tree and then rebuilding the image! I honestly would not have thought of that approach.

The image now works as expected when built with an empty LUKS passphrase.

When setting a passphrase and building the image, however, I have no chance of entering the passphrase at boot. Instead, dracut-initqueue tells me "No key available with this passphrase" - consequently the boot fails as /run/overlay/rootfsbase can't be mounted. Might there be some additional grub settings necessary to enable the prompt for the passphrase?

Thank you

Ioan

[Marcus Schäfer]

Hi,

> The image now works as expected when built with an empty LUKS
> passphrase.

happy you got to the same state than I am at the moment :)

> When setting a passphrase and building the image, however, I have no
> chance of entering the passphrase at boot.

Right, this is because there is no input console which allows
you to type anything. The preferred way to solve this is by
calling:

systemctl start systemd-cryptsetup@luks

I haven't gotten to this point and just wanted to give you
something at hand to work with.

I'm still struggling a bit about the usefulness of what
we are producing :) It's for sure not secure and I will also
not be able to get approvers for this change upstream.
At least I believe so according to the current set of
information we have.

If my time allows I'm going to add a commit such that
password protected luks works. But from there I think you
would need to carry that patch on your own.

Thoughts ?

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer

-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------

[Marcus Schäfer]

Hi,

> Right, this is because there is no input console which allows
> you to type anything. The preferred way to solve this is by
> calling:
>
> systemctl start systemd-cryptsetup@luks
>
> I haven't gotten to this point and just wanted to give you
> something at hand to work with.

Fixed now, see my last commit to the luks_for_live branch

Let me know if this is really something you plan to go forward
with. Since re-encryption is not possible this turns the information
in the image description into a secret that needs to be protected.

Thanks

[Marcus Schäfer]

Hi Ioan,

JFI

we had a conversation on the topic on the matrix channel
and it seems there are some use cases for which an encrypted
ISO would be useful. I think I'm going to polish the branch
we have and turn it into a pull request for upstream review

It would still be great if you can give me feedback if
the current encryption model for the live ISO meets your
needs.

Thanks

[Ioan Matei]

Hi Marcus, 

thank you so much for the fix and the explanation of the inner workings of luks-cryptsetup.

This approach is definitely one I will go forward with and I'd totally understand if your time goes towards other projects - this starting point alone has already been invaluable!

I've tried to build the test image with the current version of the luks_for_live branch, including a rebuild with the new dracut files.

Unfortunately the boot fails yet again. /dev/mapper/luks seems to not even exist to dracut, even though I can clearly see that the kiwi prepare step creates a luks volume. Do you have any pointers as to what could cause such behavior?

Thank you,

Ioan

[Ioan Matei]

Hi Marcus,

That's great news! I really appreciate your effort (and am sorry that this seems to bring even more work now to you).

Of course I'm available for any kind of testing and am already stoked to get everything working!

Best,

Ioan

[Marcus Schäfer]

Hi Ioan,

> That's great news! I really appreciate your effort (and am sorry that
> this seems to bring even more work now to you).

No problem, I'm happy to implement stuff that is subject for a real
use case. That having said I finished the implementation to be ready
for review and testing. Please find the open PR here:

https://github.com/OSInside/kiwi/pull/2905

In the PR you also see the link to the integration test image.
Feel free to download this ISO for testing:

wget https://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images/iso/kiwi-test-image-live.x86_64-Encrypted.iso

A kiwi version with the functionality in place can be found in the
Staging area here:

https://build.opensuse.org/project/show/Virtualization:Appliances:Staging

> Of course I'm available for any kind of testing and am already stoked
> to get everything working!

Testing from your side with the information above would be fantastic.
Feel free to comment on the open PR.

Thanks much

[Ioan Matei]

Hi Marcus,

I'm happy to report that the boot process of my test image built with the staging version of kiwi successfully allows me to get to the passphrase entry screen and successfully unlock the created LUKS volume with the password set in the config.xml. As such it mirrors the behavior of the test iso provided by you (yay!)

A few things to note:

- Even though I updated the offline repo for my test image with the dracut-kiwi-live version provided by your staging build (verified by looking at the tumbleweed.packages output of the build and finding dracut-kiwi-live 10.2.33), I still had to perform the two-step build with copying the dracut modules inside the root tree again and rebuilding with allow-existing-root to be able to unlock the LUKS volume. Is this expected behavior or did I miss something yet again?

- For this test I've set hybridpersistent to true and used erofs, just like in the image descriptions in your build. I'll test tomorrow whether I'll also be successful with hybridpersistent set to false.

- I'll also try to get a graphical plymouth splash working tomorrow for the boot and password entry. So far I've found the rd.kiwi.allow_plymouth boot parameter which I'll also try to include in my test image tomorrow. 

I'll report back with further results soon!

Best,

Ioan

[Marcus Schäfer]

Hi,

> I'm happy to report that the boot process of my test image built with
> the staging version of kiwi successfully allows me to get to the
> passphrase entry screen and successfully unlock the created LUKS volume
> with the password set in the config.xml. As such it mirrors the
> behavior of the test iso provided by you (yay!)

nice :)

> A few things to note:
>
> - Even though I updated the offline repo for my test image with the
> dracut-kiwi-live version provided by your staging build (verified by
> looking at the tumbleweed.packages output of the build and finding
> dracut-kiwi-live 10.2.33), I still had to perform the two-step build
> with copying the dracut modules inside the root tree again and
> rebuilding with allow-existing-root to be able to unlock the LUKS
> volume. Is this expected behavior or did I miss something yet again?

Hmm, no this should not be required. The two pass build is only needed
during development when the actual dracut code changes haven't found
its way to the package. Are you sure the right dracut-kiwi-xxx package
was used in your build and fetched from Staging ?

If you look at my integration test build you see something like this:

osc buildinfo images x86_64 -M Encrypted | grep dracut-kiwi

<bdep name="dracut-kiwi-live" noinstall="1" version="10.2.33" release="19.1" arch="x86_64" hdrmd5="db66e133d1c7277152b5b86d21f2d82f" project="Virtualization:Appliances:Staging" repository="openSUSE_Tumbleweed"/>

So your build has to fetch dracut-kiwi-live from Staging in release 19.1 or
higher

> - For this test I've set hybridpersistent to true and used erofs, just
> like in the image descriptions in your build. I'll test tomorrow
> whether I'll also be successful with hybridpersistent set to false.
>
> - I'll also try to get a graphical plymouth splash working tomorrow for
> the boot and password entry. So far I've found the
> rd.kiwi.allow_plymouth boot parameter which I'll also try to include in
> my test image tomorrow.
>
> I'll report back with further results soon!

Looking forward

[Ioan Matei]

Hi Marcus,

when I added the staging repo to my config.xml I was able to successfully build the image without manually doing the two pass build. Thanks for pointing me towards the correct solution!

I'm also happy to report that building the image with hybridpersistent=false works as expected - there is no COW partition present anymore and still the squashfs.img gets encrypted correctly. 

And showing a graphical plymouth screen was also easier than expected: adding "splash=silent" and "quiet" to the kernel boot parameters enabled Plymouth and allowed me to enter the LUKS passphrase with a graphical prompt. 

Thanks again for your continued support and the changes in the KIWI code that made encrypted ISO images possible!

Best,

Ioan

[Marcus Schäfer]

Hi Ioan,

> when I added the staging repo to my config.xml I was able to
> successfully build the image without manually doing the two pass build.
> Thanks for pointing me towards the correct solution!

yep, perfect

> I'm also happy to report that building the image with
> hybridpersistent=false works as expected - there is no COW partition
> present anymore and still the squashfs.img gets encrypted correctly.

Great

> And showing a graphical plymouth screen was also easier than expected:
> adding "splash=silent" and "quiet" to the kernel boot parameters
> enabled Plymouth and allowed me to enter the LUKS passphrase with a
> graphical prompt.

Ah good to know that this works in graphics mode as well.
I haven't tested that part. Thanks for the feedback

> Thanks again for your continued support and the changes in the KIWI
> code that made encrypted ISO images possible!

You are welcome. I'm going to submit a new release in the next
days. There are few open PRs that I want to be part of the
next release.

Best regards,