Root on ZFS on GELI: Passphrase is asked twice

[Marin Bernard]

Hi,

There seems to be a regression in HardenedBSD regarding booting from a
GELI-encrypted ZFS pool. As far as I know, this problem is present in
HardenedBSD 11.1-STABLE and 12-CURRENT. I was unable to reproduce it
with an up-to-date install media of FreeBSD 11.1-RELEASE. This is what
actually points towards a regression.

What actually happens is that the box is unable to boot unless the user
inputs the GELI passphrase twice: once at loader stage to decrypt the
boot pool, then again in the middle of the kernel boot process to
decrypt the root pool.

Several error messages are shown on the console dealing with GELI. I
know for sure that the issue is present in a BIOS context. I may have
noticed it also with UEFI setups, but I'm not so sure.

Steps to reproduce:

1. Install HardenedBSD 11-STABLE or 12-CURRENT from the current ISO.

2. Use the bsdinstall wizard to set up root on ZFS with GELI encryption

3. Boot from the new installation.

4. Enter the passphrase when the loaded asks for it. It will decrypt
GELI volumes, and report the following non-fatal error:

failed to read pad2 area from primary vdev

This message is not shown when one boots from a similar, fresh install
of FreeBSD 11.1-RELEASE.

6. The boot menu appears. Just wait to let the box boot with default
options.

7. Kernel is launched. A module loading error appears right on top of
dmesg. Again, this error does not show on a similar FreeBSD 11.1-
RELEASE install. Here are the first lines of dmesg:

[1] Copyright (c) 1992-2018 The FreeBSD Project.
[1] Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
1994
[1] The Regents of the University of California. All rights
reserved.
[1] FreeBSD is a registered trademark of The FreeBSD Foundation.
[1] FreeBSD 11.1-STABLE-HBSD #0 [STABLE:HardenedBSD-11-STABLE-
v1100054.3]: Sun Jan 14 02:43:29 UTC 2018
[1] ro...@nyi-01.build.hardenedbsd.org:/usr/obj/usr/src/sys/HARDENED
BSD amd64
[1] FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based
on LLVM 5.0.1)
[1] VT(vga): text 80x25
[1] HardenedBSD: initialize and check features (__HardenedBSD_version
1100054 __FreeBSD_version 1101506).
[1] can't re-use a leaf (geom_eli)!
[1] module_register: cannot register g_eli from kernel; already loaded
from geom_eli.ko
[1] Module g_eli failed to register: 17

8. Boot process carries on with device inventory, then freezes as the
kernel is unable to mount the root fs, asking for the GELI passphrase
again:

[6] Trying to mount root from zfs:zroot/ROOT/default []...
[8] Enter passphrase for ada0p3:

9. If you enter the passphrase a second time, the boot process
completes normally.

It's been quite some time (a few weeks, maybe months) since I first
noticed this problem. Since I do not deal with root encryption every
day, I just forgot about it until today.

Is there anything more I can do to help fixing it?

Thanks,

Marin.

[Shawn Webb]

Hey Marin,

My response is largely inline. Could you please show us how your zfs
pool is sconfigured by using `zpool status`? Do you have any other
geli-encrypted disks?

On Sun, Feb 25, 2018 at 10:23:00PM +0100, Marin Bernard wrote:
> Hi,
>
> There seems to be a regression in HardenedBSD regarding booting from a
> GELI-encrypted ZFS pool. As far as I know, this problem is present in
> HardenedBSD 11.1-STABLE and 12-CURRENT. I was unable to reproduce it
> with an up-to-date install media of FreeBSD 11.1-RELEASE. This is what
> actually points towards a regression.
>
> What actually happens is that the box is unable to boot unless the user
> inputs the GELI passphrase twice: once at loader stage to decrypt the
> boot pool, then again in the middle of the kernel boot process to
> decrypt the root pool.
>
> Several error messages are shown on the console dealing with GELI. I
> know for sure that the issue is present in a BIOS context. I may have
> noticed it also with UEFI setups, but I'm not so sure.
>
> Steps to reproduce:
>
> 1. Install HardenedBSD 11-STABLE or 12-CURRENT from the current ISO.

Note that full-disk encryption on 11-STABLE isn't truly full-disk
encryption. Prior to 12-CURRENT, FreeBSD requires an unencrypted boot
pool that contains the kernel and its modules.

>
> 2. Use the bsdinstall wizard to set up root on ZFS with GELI encryption
>
> 3. Boot from the new installation.
>
> 4. Enter the passphrase when the loaded asks for it. It will decrypt
> GELI volumes, and report the following non-fatal error:
>
> failed to read pad2 area from primary vdev

I've gotten this error, too. It seems safe to ignore.

>
> This message is not shown when one boots from a similar, fresh install
> of FreeBSD 11.1-RELEASE.

Have you tried FreeBSD 12-CURRENT?

>
> 6. The boot menu appears. Just wait to let the box boot with default
> options.
>
> 7. Kernel is launched. A module loading error appears right on top of
> dmesg. Again, this error does not show on a similar FreeBSD 11.1-
> RELEASE install. Here are the first lines of dmesg:
>
> [1] Copyright (c) 1992-2018 The FreeBSD Project.
> [1] Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
> 1994
> [1] The Regents of the University of California. All rights
> reserved.
> [1] FreeBSD is a registered trademark of The FreeBSD Foundation.
> [1] FreeBSD 11.1-STABLE-HBSD #0 [STABLE:HardenedBSD-11-STABLE-
> v1100054.3]: Sun Jan 14 02:43:29 UTC 2018
> [1] ro...@nyi-01.build.hardenedbsd.org:/usr/obj/usr/src/sys/HARDENED
> BSD amd64
> [1] FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based
> on LLVM 5.0.1)
> [1] VT(vga): text 80x25
> [1] HardenedBSD: initialize and check features (__HardenedBSD_version
> 1100054 __FreeBSD_version 1101506).
> [1] can't re-use a leaf (geom_eli)!
> [1] module_register: cannot register g_eli from kernel; already loaded
> from geom_eli.ko
> [1] Module g_eli failed to register: 17

I've seen that before, too. It also seems safe to ignore.

>
> 8. Boot process carries on with device inventory, then freezes as the
> kernel is unable to mount the root fs, asking for the GELI passphrase
> again:
>
> [6] Trying to mount root from zfs:zroot/ROOT/default []...
> [8] Enter passphrase for ada0p3:

I've had this issue pop up on older systems. Recent 11-STABLE and
12-CURRENT systems don't have that issue for me, even on those older
systems (older as in physical hardware).

>
> 9. If you enter the passphrase a second time, the boot process
> completes normally.

This can happen depending on the zfs pool configuration, hence why I
asked for it above. If you have a second disk as a mirror vdev, for
example, the system will prompt for the password for the second disk
(even though the password is the same as the first disk).

>
> It's been quite some time (a few weeks, maybe months) since I first
> noticed this problem. Since I do not deal with root encryption every
> day, I just forgot about it until today.
>
> Is there anything more I can do to help fixing it?

I need concrete reproduction steps. I haven't had the double-prompt
issue for well over a year, even on older hardware. I'll dig into the
errors/warnings I said seemed safe to ignore. However, I'm not an
expert in that code. I try not to do much with cryptography since I'm
not an expert in cryptography and the slightest mistakes in crypto can
have extremely large ramifications.

I'll get back to you when I know more.

Thanks,

--
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal: +1 443-546-8752
GPG Key ID: 0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE

[Oliver Pinter]

Just remove geometry_eli.ko from your loader.conf. We always compiles them in the kernel. When the loader overwrites this instance, then it clears the cached password, thus the double asking of password. 

[ma...@olivarim.com]

Hi Oliver,

 

Thanks for answering. I’ll try that. Is it expected behaviour on a fresh HardenedBSD install? Shouldn’t loader.conf come with sane defaults?

 

Thanks,

 

Marin.

 

De : Oliver Pinter
Envoyé le :dimanche 25 février 2018 23:18
À : Marin Bernard
Cc : us...@hardenedbsd.org
Objet :Re: Root on ZFS on GELI: Passphrase is asked twice

--
You received this message because you are subscribed to the Google Groups "HardenedBSD Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@hardenedbsd.org.

 

[Oliver Pinter]

The bug is in bsdinstall. It unconditionally adds the GELI line at install time.

Just simple -ENOTIME to fix it in a good way, and I not want to just work it around. 

To unsubscribe from this group and stop receiving emails from it, send an email to users+unsubscribe@hardenedbsd.org.

 

[Marin Bernard]

Hi Shawn,

Sorry for the delay. Here are some inline answers to your inline
questions. In short: removing 'geom_eli_load="YES"' from loader.conf
fixes the double prompt issue.

> My response is largely inline. Could you please show us how your zfs
> pool is sconfigured by using `zpool status`? Do you have any other
> geli-encrypted disks?

Here is the output of `zpool status`. The ZFS pool hosting the root fs
comprises 2 mirrored drives. These are the only drives of the system.

pool: zroot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p3.eli ONLINE 0 0 0
ada1p3.eli ONLINE 0 0 0

errors: No known data errors

> Note that full-disk encryption on 11-STABLE isn't truly full-disk
> encryption. Prior to 12-CURRENT, FreeBSD requires an unencrypted boot
> pool that contains the kernel and its modules.

Yes, the separate boot pool was mandatory in previous versions of
FreeBSD, but are you sure that 11-STABLE is still relying on it? I ran
all encryption tests on fresh instances of HardenedBSD 11-STABLE, and
the installer only created a single encrypted pool, as shown by the
output of the `zpool status`, above. There is no separate boot pool;
only a geom mirror for mirrored swap.

> > 2. Use the bsdinstall wizard to set up root on ZFS with GELI
> > encryption
> >
> > 3. Boot from the new installation.
> >
> > 4. Enter the passphrase when the loaded asks for it. It will
> > decrypt
> > GELI volumes, and report the following non-fatal error:
> >
> > failed to read pad2 area from primary vdev
>
> I've gotten this error, too. It seems safe to ignore.

Yes, this is also my conclusion; it seems related to the new
zfsbootcfg(8) utility. The error is not shown on FreeBSD 11.1-RELEASE
or 12-CURRENT, though.

>
> >
> > This message is not shown when one boots from a similar, fresh
> > install
> > of FreeBSD 11.1-RELEASE.
>
> Have you tried FreeBSD 12-CURRENT?

Yes, a few minutes ago. I just installed FreeBSD 12-CURRENT (r328383)
on an identical setup (root on ZFS mirror + GELI). It boots normally,
i.e.:

- GELI Passphrase is asked once by the boot loader (before showing
the boot menu)

- No 'pad2 area' error

- No module loading error from the kernel

- No need to enter the passphrase twice

> > [1] can't re-use a leaf (geom_eli)!
> > [1] module_register: cannot register g_eli from kernel; already
> > loaded
> > from geom_eli.ko
> > [1] Module g_eli failed to register: 17
>
> I've seen that before, too. It also seems safe to ignore.

Yes, that's what Oliver wrote in a previous message. This is because
GELI is included by default in the HARDENEDBSD kernel, thus trying to
load it a second time as a module leads to failure. Removing the line
from loader.conf resolves the issue, and also the second passphrase
prompt.

> >
> >
> > 9. If you enter the passphrase a second time, the boot process
> > completes normally.
>
> This can happen depending on the zfs pool configuration, hence why I
> asked for it above. If you have a second disk as a mirror vdev, for
> example, the system will prompt for the password for the second disk
> (even though the password is the same as the first disk).

Removing 'geom_eli_load="YES"' from loader.conf fixes the issue.

Now that the main issue is fixed, I have two more questions:

1. If GELI is included by default in the HardenedBSD kernel instead of
being provided as a separate module, shouldn't the version of
bsdinstall(8) shipped with HardenedBSD be amended to avoid adding a
'geom_eli_load="YES"' entry to loader.conf?

2. Tring to load GELI as a module when it is hardwired in the kernel
throws an error. So far, so good. However, it also leads the kernel to
dismiss any passphrase supplied earlier in the boot process, hence the
double prompt. Do you know if this is a bug or a security feature?

Thanks!

Marin

[Shawn Webb]

The double prompt you're seeing, then, is the kernel asking for the
passphrase for the second drive.

Agreed. I'll take care of that this week.

>
> 2. Tring to load GELI as a module when it is hardwired in the kernel
> throws an error. So far, so good. However, it also leads the kernel to
> dismiss any passphrase supplied earlier in the boot process, hence the
> double prompt. Do you know if this is a bug or a security feature?

See above. The double passphrase prompt isn't due to reloading of the
kernel module (in fact, the kernel module can't be loaded multiple
times).

[Oliver Pinter]

You are wrong here. This depends on module information. If there are different modinfo for the compiled in module and different for external, then you are able to load the same named module twice. This is how for eg Intel e1000 module works...

[Shawn Webb]

That's true. I believe in this case, though, that geom_eli cannot be
loaded multiple times. In fact, when I run `kldload geom_eli`, I get
an error that it's already loaded.

Thus, the double prompting issue is due to using two geli providers in
the mirror vdev. The bootloader takes care of the first geli provider.
However, the kernel still needs the passphrase (which likely is the
same as the provider decrypted by the bootloader) for the second
provider.

[Marin Bernard]

> > > >
> > > > 1. If GELI is included by default in the HardenedBSD kernel
> > > > instead of
> > > > being provided as a separate module, shouldn't the version of
> > > > bsdinstall(8) shipped with HardenedBSD be amended to avoid
> > > > adding a
> > > > 'geom_eli_load="YES"' entry to loader.conf?
> > >
> > > Agreed. I'll take care of that this week.

Thanks!

In that case how do you explain that removing the 'geom_eli_load="YES"'
line from loader.conf fixes the issue? The kernel still has to decrypt
the second disk, yet the box only asks for the passphrase once.

[Shawn Webb]

In looking at the code, I can see that my assumptions were (mostly)
wrong. The geom_eli module does not use the MODULE_VERSION macro,
which would prevent double loading of the module from bootloader to
later on in the boot process.

Oliver's correct in his analysis that when the module gets loaded the
second time, the cached passphrase is cleared.

I'd like to spend some time researching if setting MODULE_VERSION
would also prevent the double load issue. Obviously, removing the
offending line from bsdinstall is a good first step and would help
with new installs. However, I'd like to see about fixing existing
installs, and perhaps MODULE_VERSION would do that, perhaps not.

Either way, more research is needed. I'll get back to you once I have
more info.

[Oliver Pinter]

Addig a check for bsdinstall is enough at first time. Like filter the result  of kldstat -v, and see that the kernel contains the geom_eli.ko, and if so, skip the generation of the load line. This will be the workaround. 

The proper fix would be the addition of module version line to kernel module, but this would be fine to push to upstream first.

[Shawn Webb]

On Thu, Mar 01, 2018 at 08:45:30AM +0100, Oliver Pinter wrote:
> Addig a check for bsdinstall is enough at first time. Like filter the
> result of kldstat -v, and see that the kernel contains the geom_eli.ko,
> and if so, skip the generation of the load line. This will be the
> workaround.
> The proper fix would be the addition of module version line to kernel
> module, but this would be fine to push to upstream first.

Patch for review submitted upstream:
https://reviews.freebsd.org/D14553