Heads firmware talk at CCC

[Trammell Hudson]

On Tuesday I presented my work on the Heads firmware at 33C3 and
gave Qubes OS (and Joanna's 32C3 tallk) a shout-out. You can watch
the video, titled "Bootstrapping a slightly more secure laptop":

https://media.ccc.de/v/33c3-8314-bootstraping_a_slightly_more_secure_laptop

Or read my notes:

https://trmm.net/Heads_33c3

If anyone is attending CCC and wants to meet up to discuss how I've
modified Qubes to work with Heads and coreboot, I'll be around today
and tomorrow, then in Berlin for New Years. I'd really like to figure
out how to pass the secret key from the Heads bootloader to Qubes'
initrd in a supported fashion.

--
Trammell

[Rusty Bird]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Trammell Hudson:

> On Tuesday I presented my work on the Heads firmware at 33C3 and
> gave Qubes OS (and Joanna's 32C3 tallk) a shout-out. You can watch
> the video, titled "Bootstrapping a slightly more secure laptop":

This looks really nice, I hope to finally be able to try it soonish.

With Qubes already moving towards open firmware (which probably means
coreboot in the foreseeable future?), Heads seems like it could be a
kind of natural successor to AEM and get rid of all the tboot/SINIT
related headaches.

Has there been any progress in upstreaming the hypervisor patch, now
that you have a rock solid use case?

> I'd really like to figure
> out how to pass the secret key from the Heads bootloader to Qubes'
> initrd in a supported fashion.

If I understand it right, [rd.]luks.key= isn't working as it should?
I've played around with that a tiny bit and systemd-cryptsetup-generator
was indeed behaving weirdly, some "out of memory" nonsense.

Rusty
-----BEGIN PGP SIGNATURE-----

iQJ8BAEBCgBmBQJYZXnKXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4NEI1OUJDRkM2MkIxMjlGRTFCMDZEMDQ0
NjlENzhGNDdBQUYyQURGAAoJEEadePR6ryrf1/YP/R7PqGlsQIE5vrcnGpg/6vm5
udi9Hb629B0sTi1IPK0rHQqJxRJiVvVUQkciMab+KSbzwfxMTFD7A0eufa73ujiE
dYEOKajP0Avwa8nUMYx9TRpj5S0KqmUmPLBGEGnQbvt3Qg0itsbaAqJsZN9SOM/f
zDPeYzudR3Bm0H/ExzQ2wKGS0tQ4DWNVi7tOu/j4uW8UBZS2ifF1BX7GjRTA/CWR
mYI9bx0QcC9XmMeb+gJHznPGR8N9XXjOh1RjqRZu/Db3e5kB7Jet10NQaaM1hO+2
Ou3PW4/P35RX232l0EqPq/PwXwoVwx7JIRIVBctX755siTDV1reP4piHYdmn6O9Z
m8WpfDaPTDSDp73+7NosZE9oydb30iNIBeGaKf4/c7MWfKgh1Y8M0GPMQYWIAxQL
QxFzIH/gkJ/vWlmtARrgiSrMmMNsOFe1TiXksJc1vnVzXS5nlKV8RQAcREURCb43
tSii74nWW11qxPsDa211wl6nH5xV5zS+D4z3/t5l0VjoIR/JB/LmjPuRGvuMWrsW
2g/Mtn1qYUS4KiK0zsT/qsebstTz0EXqiokaNhAX50e+BG4nDljftTo8MZzOpndl
vL9CJ15NwG415KW0tv8gclx6HEUMwnXOEM9mXbNiDjNTKIJ49ZtQSliG6WWhymhO
P9VL8vcktNT+xDJX6TAZ
=iDY+
-----END PGP SIGNATURE-----

[Rusty Bird]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Rusty Bird:

> Trammell Hudson:
> > On Tuesday I presented my work on the Heads firmware at 33C3 and
> > gave Qubes OS (and Joanna's 32C3 tallk) a shout-out. You can watch
> > the video, titled "Bootstrapping a slightly more secure laptop":
>
> This looks really nice, I hope to finally be able to try it soonish.
>
> With Qubes already moving towards open firmware (which probably means
> coreboot in the foreseeable future?), Heads seems like it could be a
> kind of natural successor to AEM and get rid of all the tboot/SINIT
> related headaches.
>
> Has there been any progress in upstreaming the hypervisor patch, now
> that you have a rock solid use case?
>
> > I'd really like to figure
> > out how to pass the secret key from the Heads bootloader to Qubes'
> > initrd in a supported fashion.
>
> If I understand it right, [rd.]luks.key= isn't working as it should?
> I've played around with that a tiny bit and systemd-cryptsetup-generator
> was indeed behaving weirdly, some "out of memory" nonsense.

Which might be fixed by
https://github.com/systemd/systemd/commit/c802a7306bdc3e82378a87acd9402bbabe9f6b28

Rusty
-----BEGIN PGP SIGNATURE-----

iQJ8BAEBCgBmBQJYZX4oXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4NEI1OUJDRkM2MkIxMjlGRTFCMDZEMDQ0
NjlENzhGNDdBQUYyQURGAAoJEEadePR6ryrfbZYQAIn/8xXRR6hR+y59wtmJ1rkW
YopRLe83XpyDp3+LJ1qsMygy0EsmKCk964ccUlMrm8VyrDuHV40qpvRMqLew/jX9
H9h4wqoVgNcp6rtfzWMzEOfRCLsid8w6emcqOn9fZFaM7bISWNFXwZFmt3j14RWM
+py9oVLTRSjKYIfOX2VcSFZykQjIGl/fXHYPQskLEfL6Y0x2ngWM/ODpMv5Yd5kI
VXCUy6CgCoyo7jl7AD59Bcozxvm4+ne9g2NiL6nzg/t0w6l8JwieetVSzEsN0mvY
XvoaLBOPyXZNXyGIarJjLyAjEO85KEUz04bslfzF6gJ6O125dhaEGj+rbpi3THAo
T6b4kbr9DZRSZ5ggjVvLuHykz9Am4YyAoAYHv/2DkAPAjNYoT3PVc52buUoCnowp
ocKPqJLQ6SLvLQFjza8qEo+5Ka2Tgy4mRDJBKsjFfS2cCG1eeuH9+aWJtZ4Gkral
yX31ry1e0YSk+DLTNlo2mldjMU1XQj6fauSaWcDh4Gi1CtVPEri/OphTco+lkTfC
Zye3ZSvsInmSblBHCVRCSTc1hoAIS/Qf8hWBdg4XbJEkdfbLIZukh10vicuz8UKU
7bvJQrQGhDYXFvJgLN0zAZkdrwHismbejl0cxr74tDW8s8AUTCmc9TUnTFTmDn0D
AV9b17WxQP3Yxt7wQa1Q
=AlFX
-----END PGP SIGNATURE-----

[Tai...@gmx.com]

Qubes isn't "moving to open firmware", new x86-64 intel and (now) AMD
(ZEN+FM2) processors make real libre firmware impossible with more and
more "security" measures such as FSP, boot guard, ME/PSP, etc.
And before anyone says "but purism", they're lying and entirely full of
it as they will never be able to convince intel to open source anything
at all, if google can't pull it off nobody can.
Blobbed firmware isn't open firmware.

You have impressive skills, but x86 is dead - even if you manage to
somehow turn a recent x86 device in to a libre firmware platform
intel/amd will simply issue a "fix" with the next hardware release.
My next laptop will be a custom build, stick a non-laptop motherboard in
a DIY case as there aren't any other real options for mobile workstations.


The only legitimate option for open source firmware machines at this
point is POWER and (some) ARM, just ultra high end servers and low power
devices (like the novena)

IBM sells servers only, presumably you could insert a graphics card
however there is also the TALOS project which aims to produce a fully
open (including hardware) POWER8 workstation motherboard (I assume you
already know, but here it is for everyone else)
https://www.crowdsupply.com/raptorcs/talos

[Trammell Hudson]

On Thu, Dec 29, 2016 at 09:20:41PM +0000, Rusty Bird wrote:
> Rusty Bird: [...]

> > Has there been any progress in upstreaming the hypervisor patch, now
> > that you have a rock solid use case?

I haven't revistied that particular patch; they said they were interested
in supporting "legacy free" systems, although my patch was really hackish.
The right way to do it is to move early command line parsing to before
the EBDA is examined.

The Chromebook with a VBT works with fewer patches, so I also need to
revisit what is different between it and the thinkpads.


> > Trammell Hudson:
> > [...]

> > > I'd really like to figure
> > > out how to pass the secret key from the Heads bootloader to Qubes'
> > > initrd in a supported fashion.
> >
> > If I understand it right, [rd.]luks.key= isn't working as it should?
> > I've played around with that a tiny bit and systemd-cryptsetup-generator
> > was indeed behaving weirdly, some "out of memory" nonsense.

I don't think that I get that error; it seems to just be ignored and
Qube's initrd prompts for a disk password.

> Which might be fixed by
> https://github.com/systemd/systemd/commit/c802a7306bdc3e82378a87acd9402bbabe9f6b28

Hmm. Yeah, that would make a difference...

The one drawback to the rd.luks.key approach is that only a single key
can be passed in. For some use cases separate /, /boot and /home keys
are worth having, which involve editing the /etc/crypttab file in
the initrd before starting Xen.

--
Trammell

[Rusty Bird]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Trammell Hudson:

> The one drawback to the rd.luks.key approach is that only a single key
> can be passed in.

You can also use "[rd.]luks.key=LUKSUUID=KEYFILE", no idea if that works
any better in practise...

Rusty
-----BEGIN PGP SIGNATURE-----

iQJ8BAEBCgBmBQJYZoXVXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4NEI1OUJDRkM2MkIxMjlGRTFCMDZEMDQ0
NjlENzhGNDdBQUYyQURGAAoJEEadePR6ryrfFZ0P/3muJXRvRR9gfUF/AJV2yM9y
Iu/JjyHf5OzNcTHHAY4EbFZMtBmlugPZP+1PDds0X6ajd7RYxrFCHvkcMuDKXj1K
+oDtsbud9SD7P4Y+5cm3V/oKkmaJFI5CzBbtDwZ9REnEFsmr1XTwlCXLAxSY/IqH
RmMa5Dm4ytmbZay0iZUk/Ak/GXqDXSfqE6eVtifmgzfwhdb4bWVQ8ewUMRck3MDp
GJcAR7CPPsGPceAyF5O7sRlZ8gmoafLbmFgKExmyqfZwuyKQm3HE8AEaCKoS4sLZ
HDDIBmP+ixXx4gR4pv5PUQrvhhG5sjUOXpzCY0WtbpU4g7q8iOZGVHX7zmjXqXkI
ISAv8lH/LSNhvUQRxOkYWfbtAr19Q5kXJUJIwqQFdLheSepptKwL+M76l6oFqj4W
v0uL4wbv5zovbNwAa1qqSNFFMg+/fh1fHb0fDgVHydesoa5F8Ln8ZaWAk1qhiH1U
PjrZKYV48vTYE2BVClbNIsGkFog3slNkeGajZAVqm2taMQNMhI0lSwSeMjzAk2As
GOnlBZ5e/NFj0jbmtzMQrNhylYFSPclJaJxXm+G3eVr4OZc55AZbGxE1AgNGzXQH
bKT2q7UXofuwiSmT1hQHT8lrmdpdnlWqUAdXd38/sAVYkIzyL3CoDCLFJHppzb7/
BY2L3xxprRa6VSn4MFbQ
=HFQ1
-----END PGP SIGNATURE-----

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Dec 30, 2016 at 06:01:02AM -0700, Trammell Hudson wrote:
> On Thu, Dec 29, 2016 at 09:20:41PM +0000, Rusty Bird wrote:
> > Rusty Bird: [...]
> > > Has there been any progress in upstreaming the hypervisor patch, now
> > > that you have a rock solid use case?
>
> I haven't revistied that particular patch; they said they were interested
> in supporting "legacy free" systems, although my patch was really hackish.
> The right way to do it is to move early command line parsing to before
> the EBDA is examined.
>
> The Chromebook with a VBT works with fewer patches, so I also need to
> revisit what is different between it and the thinkpads.

:)

Also, thanks for great work and great talk!

Can you elaborate on Qubes modifications? Have you achieved read-only
rootfs with dm-verity? What workflow do you have for upgrades? Do
templates are part of read-only fs, or read-write?

> > > Trammell Hudson:
> > > [...]
> > > > I'd really like to figure
> > > > out how to pass the secret key from the Heads bootloader to Qubes'
> > > > initrd in a supported fashion.
> > >
> > > If I understand it right, [rd.]luks.key= isn't working as it should?
> > > I've played around with that a tiny bit and systemd-cryptsetup-generator
> > > was indeed behaving weirdly, some "out of memory" nonsense.
>
> I don't think that I get that error; it seems to just be ignored and
> Qube's initrd prompts for a disk password.
>
> > Which might be fixed by
> > https://github.com/systemd/systemd/commit/c802a7306bdc3e82378a87acd9402bbabe9f6b28
>
> Hmm. Yeah, that would make a difference...
>
> The one drawback to the rd.luks.key approach is that only a single key
> can be passed in. For some use cases separate /, /boot and /home keys
> are worth having, which involve editing the /etc/crypttab file in
> the initrd before starting Xen.

As Rusty already mentioned, multiple keys should work. Alternatively you
can use dracut with host-only mode, which will include /etc/cryptab from
your host. Not sure if keyfile itself is included, but in theory should
be.

PS We've tried to catch you yesterday, but failed...

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJYZvNaAAoJENuP0xzK19csU8cH/jBmR/u3gIhA4xb7fjiL9c+C
9PAUCohu1V00s0QwDDxNM9Ku40mi77kuPfmFKvpgCQRiuxQWgqsrS0yS45QKMpP2
d9xMlek+ciQB9e84nzrPS4QDUKmjn4RHfnubqodpfu425b/iMah0EMq+dfrCUJvT
U50XsNmyN0VYaYCMjvUHyuuMDPZI4fhxN3SdA3J/Gx3DlFh3MpVw+tXKlQAU5x6M
ck4I1wH3cwBGrhVPoploxyvXgtJwfHqy4dgrXrC/BauW4eG6EhANSC1A6hG4DGtr
P8lARByfSFzyuL/njAYAZJa0/DoY/XLyDgfd4D3PP6hZMtcOR/Wqj8JlON30kao=
=iiss
-----END PGP SIGNATURE-----

[Trammell Hudson]

On Fri, Dec 30, 2016 at 06:01:02AM -0700, Trammell Hudson wrote:

> On Thu, Dec 29, 2016 at 09:20:41PM +0000, Rusty Bird wrote:
> > If I understand it right, [rd.]luks.key= isn't working as it should?
> > I've played around with that a tiny bit and systemd-cryptsetup-generator
> > was indeed behaving weirdly, some "out of memory" nonsense.
>
> I don't think that I get that error; it seems to just be ignored and
> Qube's initrd prompts for a disk password.

When I actually read the text on the screen during the boot, I see the
same "out of memory" error. This does seem to indicate that it is the
systemd bug that you had identified:

> > Which might be fixed by
> > https://github.com/systemd/systemd/commit/c802a7306bdc3e82378a87acd9402bbabe9f6b28

Have you been able to test this change? Figuring out how to deploy one
particular patch into the qubes initrd seems complicated; one big
advantage of shell scripts for these things is that they can be
edited on the fly, rather than requiring recompiling all of systemd.

--
Trammell

[Rusty Bird]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Trammell Hudson:

> When I actually read the text on the screen during the boot, I see the
> same "out of memory" error. This does seem to indicate that it is the
> systemd bug that you had identified:
>
> > > Which might be fixed by
> > > https://github.com/systemd/systemd/commit/c802a7306bdc3e82378a87acd9402bbabe9f6b28

Hey so at least there's only _one_ bug. I hope.

> Have you been able to test this change?

No, sorry.

> Figuring out how to deploy one
> particular patch into the qubes initrd seems complicated; one big
> advantage of shell scripts for these things is that they can be
> edited on the fly, rather than requiring recompiling all of systemd.

Reading the code, it looks like this bug only affects the general case,
where no specific UUID has been given. Maybe enumerating all UUIDs and
passing the same key file for each of them would work for you?

Rusty
-----BEGIN PGP SIGNATURE-----

iQJ8BAEBCgBmBQJYbq2LXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4NEI1OUJDRkM2MkIxMjlGRTFCMDZEMDQ0
NjlENzhGNDdBQUYyQURGAAoJEEadePR6ryrfbPEP/jGpbIX/CfPAAyktdu5SqgVK
O69zpGIp8wEYrEUcBQN4phDj2LGz8iHK/c8mL6xXIZIuAW5g8h7JEINaEvE10mJv
LrXuEWnB4BOnPq26jH2J9Kve/C24xU9nGSm9vmto2YQvr2mlNM43d/z/M0T7NK7H
KX8WMkwB5vAxiLGATMDhOiJQp32qdI+zDXE3ihn2QSutzzgMAwuFixx55m5dm5lC
6ANzmaIcNh1xSce1tRH7C+WPeNSU8MUe//IktmxphQqA93cB7R2uABncjHvBNDVO
BT9t+CDYOo+XM9dkqeoljISduIZlln3MQ8YZAdVP8TMxDRd3VYs9aluRoWLPIQ86
bIFHUkUPvI7MywiNLdcSymGJZn4YWhqU3+dW0EfSGPCjswY0Wycf+f6hr6OMwkqw
+U2mikJNDtWYSzGV/ZXHESZRfHn3y5G0qNbalosq4A1zTlKcCFauEto+fOUIINJr
QebKw1qyLk5Up/Z6gD00RbDeekWYYs2wyuBf2j6ruXhx83+HPIrfvA2UkoFPGb5c
jE1buyStnNRGJvLfqcwWKeulJSwF6236txDZa56SUZwdbAn6nxArgmqieXFrrwjf
oZ0CsJR2YlCTl5x+LDGLTsN1huijoinsadlXUfclpEDUl0xwELEl9cYyHC8DG6C/
Sl1qk4RbglDL/81lP7Ml
=HOZh
-----END PGP SIGNATURE-----

[Radoslaw Szkodzinski]

For security purposes, it would be enough if we could get open
sources, the blob and a reproducible build, while being able to verify
if the original blob came from such reproducible build.
This way we can audit the blob and perhaps trust it. Qubes per se does
not care about libre or replacing, it cares about security.

However Intel has never been conducive to opening the blobs. AMD might
be easier to convince, their main problem is maintenance it seems, not
lack of will.
They used to contribute to Coreboot etc a bunch of years past.
Their FSP thing is also based on modern ARM instead of weird custom
ARC/SPARC chip, making it slightly easier to work with.

Also who knows what kind of backdoors lurk in there. If that is the
true reason, then we're plain out of luck.

> You have impressive skills, but x86 is dead - even if you manage to somehow
> turn a recent x86 device in to a libre firmware platform intel/amd will
> simply issue a "fix" with the next hardware release.
> My next laptop will be a custom build, stick a non-laptop motherboard in a
> DIY case as there aren't any other real options for mobile workstations.

That doesn't change anything about the blobs. The latest blob-free
hardware you could get is 2012 era for desktops.

> The only legitimate option for open source firmware machines at this point
> is POWER and (some) ARM, just ultra high end servers and low power devices
> (like the novena)

Novena in fact is available now, but sadly nobody is making a libre
ultrabook like that yet. It's not quite strong enough for certain
usages, but those usages rarely require full assurance.
The problem is lack of full hardware virtualization yet. The current
TrustZone etc. is too limited for what Qubes requires.
Paravirtualization would work though.

> IBM sells servers only, presumably you could insert a graphics card however
> there is also the TALOS project which aims to produce a fully open
> (including hardware) POWER8 workstation motherboard (I assume you already
> know, but here it is for everyone else)
> https://www.crowdsupply.com/raptorcs/talos

POWER is way expensive, niche and unsurprisingly much less supported
everywhere than ARM.
I think it does support hardware virtualization but never had time to
play with it. IBM likes their semi-walled garden and loves to sell
their own "solutions".

--
Radosław Szkodziński

[Tai...@gmx.com]

You can't have security without libre firmware, you can do all the
auditing in the world but you'll never figure out what FSP actually does.

>
> However Intel has never been conducive to opening the blobs. AMD might
> be easier to convince, their main problem is maintenance it seems, not
> lack of will.
> They used to contribute to Coreboot etc a bunch of years past.
> Their FSP thing is also based on modern ARM instead of weird custom
> ARC/SPARC chip, making it slightly easier to work with.
>
> Also who knows what kind of backdoors lurk in there. If that is the
> true reason, then we're plain out of luck.
>
>> You have impressive skills, but x86 is dead - even if you manage to somehow
>> turn a recent x86 device in to a libre firmware platform intel/amd will
>> simply issue a "fix" with the next hardware release.
>> My next laptop will be a custom build, stick a non-laptop motherboard in a
>> DIY case as there aren't any other real options for mobile workstations.
> That doesn't change anything about the blobs. The latest blob-free
> hardware you could get is 2012 era for desktops.

Yes it does as blob free workstation hardware is more recent (I play the
latest games on a 2012 opteron with less than 20% cpu usage)

>
>> The only legitimate option for open source firmware machines at this point
>> is POWER and (some) ARM, just ultra high end servers and low power devices
>> (like the novena)
> Novena in fact is available now, but sadly nobody is making a libre
> ultrabook like that yet. It's not quite strong enough for certain
> usages, but those usages rarely require full assurance.
> The problem is lack of full hardware virtualization yet. The current
> TrustZone etc. is too limited for what Qubes requires.
> Paravirtualization would work though.
>
>> IBM sells servers only, presumably you could insert a graphics card however
>> there is also the TALOS project which aims to produce a fully open
>> (including hardware) POWER8 workstation motherboard (I assume you already
>> know, but here it is for everyone else)
>> https://www.crowdsupply.com/raptorcs/talos
> POWER is way expensive, niche and unsurprisingly much less supported
> everywhere than ARM.
> I think it does support hardware virtualization but never had time to
> play with it. IBM likes their semi-walled garden and loves to sell
> their own "solutions".

KVM works on POWER and most linux appliations will cross compile without
significant effort.