Problem with systemd and cryptsetup - how to solve it the systemd way?

[Anders Andersson]

I am trying to set up an encrypted btrfs system on a Debian Jessie
system but I am running into a lot of issues, and now I need some help
debugging the basics. It seems like I'm running in to a new bug for
everything I do.

To get to the bottom of this I set up a virtual Debian Jessie with two
drives, the first with an unencrypted OS, and the second with one
partition and and empty luks device:

root@jessie:~# cryptsetup luksFormat /dev/sdb1
root@jessie:~# cryptsetup luksOpen /dev/sdb1 crypt
root@jessie:~# dd if=/dev/zero of=/dev/mapper/crypt
root@jessie:~# cryptsetup luksClose crypt


Then I add this to /etc/crypttab:
---- /etc/crypttab ---
# <target name> <source device> <key file> <options>
crypt /dev/sdb1 none luks,noauto
---- /etc/crypttab ---

I ask systemd to generate the relevant unit files for this:
root@jessie:~# systemctl daemon-reload

..which gives me some files in /var/run/systemd/generator
root@jessie:~# tree --charset=ascii /var/run/systemd/generator
|-- dev-mapper-crypt.device.requires
| `-- systemd-c...@crypt.service -> ../systemd-c...@crypt.service
`-- systemd-c...@crypt.service


---- /var/run/systemd/generator/systemd-c...@crypt.service ----
# Automatically generated by systemd-cryptsetup-generator

[Unit]
Description=Cryptography Setup for %I
Documentation=man:crypttab(5) man:systemd-cryptsetup-generator(8)
man:systemd-cryptsetup@.service(8)
SourcePath=/etc/crypttab
DefaultDependencies=no
Conflicts=umount.target
BindsTo=dev-mapper-%i.device
IgnoreOnIsolate=true
After=systemd-readahead-collect.service
systemd-readahead-replay.service cryptsetup-pre.target
Before=cryptsetup.target
BindsTo=dev-sdb1.device
After=dev-sdb1.device
Before=umount.target

[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutSec=0
ExecStart=/lib/systemd/systemd-cryptsetup attach 'crypt' '/dev/sdb1'
'none' 'luks,noauto'
ExecStop=/lib/systemd/systemd-cryptsetup detach 'crypt'
---- /var/run/systemd/generator/systemd-c...@crypt.service ----






After this I try starting the new device:
root@jessie:~# systemctl start dev-mapper-crypt.device
Please enter passphrase for disk VBOX_HARDDISK (crypt)! ****
... here it stalls for what is likely 90 seconds ...
Job for dev-mapper-crypt.device timed out.
root@jessie:~#


Ok. It kinda failed but not really. The device has been unlocked and
exists in /dev/mapper/ as expected. Looking at the journal gives me
the following relevant lines:

Dec 10 14:18:33 jessie systemctl[4064]: Failed to stop
systemd-ask-password-plymouth.path: Unit
systemd-ask-password-plymouth.path not loaded.
Dec 10 14:18:33 jessie systemctl[4064]: Failed to stop
systemd-ask-password-plymouth.service: Unit
systemd-ask-password-plymouth.service not loaded.
Dec 10 14:19:08 jessie systemd-cryptsetup[4062]: Set cipher aes, mode
xts-plain64, key size 256 bits for device /dev/sdb1.
Dec 10 14:20:03 jessie systemd[1]: Job dev-mapper-crypt.device/start timed out.
Dec 10 14:20:03 jessie systemd[1]: Timed out waiting for device
dev-mapper-crypt.device.
Dec 10 14:20:03 jessie systemd[1]:
systemd-ask-password-console.service stop-sigterm timed out. Killing.
Dec 10 14:20:03 jessie systemd[1]:
systemd-ask-password-console.service: main process exited,
code=killed, status=9/KILL
Dec 10 14:20:03 jessie systemd[1]: Unit
systemd-ask-password-console.service entered failed state.
Dec 10 14:20:03 jessie systemd[1]: systemd-ask-password-wall.service
start-pre operation timed out. Terminating.
Dec 10 14:20:03 jessie systemd[1]: Failed to start Forward Password
Requests to Wall.
Dec 10 14:20:03 jessie systemd[1]: Unit
systemd-ask-password-wall.service entered failed state.



Rebooting the machine and starting the device with "cryptdisks_start
crypt" works, and does not give any immediate errors in the journal,
but because it does not go through systemd I get in trouble later when
I start adding filesystems that requires multiple encrypted devices,
so I really want to get this very basic thing working first!

My question is thus: How am I supposed to solve this the "systemd
way"? I want to be able to start an encrypted block device using a
normal systemd service/device so that I can later have systemd units
depend on this.

[David Christensen]

On 12/10/2015 05:38 AM, Anders Andersson wrote:
> I am trying to set up an encrypted btrfs system on a Debian Jessie

...

> root@jessie:~# cryptsetup luksFormat /dev/sdb1
> root@jessie:~# cryptsetup luksOpen /dev/sdb1 crypt
> root@jessie:~# dd if=/dev/zero of=/dev/mapper/crypt
> root@jessie:~# cryptsetup luksClose crypt

...

> Then I add this to /etc/crypttab:
> ---- /etc/crypttab ---
> # <target name> <source device> <key file> <options>
> crypt /dev/sdb1 none luks,noauto
> ---- /etc/crypttab ---

When I want a LUKS data partition to be mounted at boot, I create a LUKS
key:

# dd if=/dev/random of=/root/.luks-keyfile bs=1 count=32

# chmod 0400 /root/.luks-keyfile


Add the key to the LUKS partition:

# cryptsetup luksAddKey /dev/sdb1 /root/.luks-keyfile


(Note: it is preferable to use /dev/disk/by-id/* paths for disk
administration commands rather than /dev/sd* paths, as the letter
assignments can and do get re-arranged on reboot.)


Add an entry to /etc/crypttab:

sdb1_crypt /dev/sdb1 /root/.luks-keyfile luks


Create a mount point:

# mkdir /var/luks/sdb1_data


Add an entry to /etc/fstab:

/dev/mapper/sdb1_crypt /var/luks/sdb1_data auto defaults,nofail 0 2


Reboot, verify, and/or troubleshoot.


David

[Jonathan Dowland]

On Thu, Dec 10, 2015 at 02:38:02PM +0100, Anders Andersson wrote:
> ---- /var/run/systemd/generator/systemd-c...@crypt.service ----

Do you know whether that is persistent? I used the generator to create
the service file, but I then put the service file into /etc/systemd/user
and removed the crypttab file altogether.

> After this I try starting the new device:
> root@jessie:~# systemctl start dev-mapper-crypt.device
> Please enter passphrase for disk VBOX_HARDDISK (crypt)! ****
> ... here it stalls for what is likely 90 seconds ...
> Job for dev-mapper-crypt.device timed out.
> root@jessie:~#

Did you type the passphrase when prompted? I find the prompt a little
awkward, as it seems to be to every PTY and TTY by default.

> Ok. It kinda failed but not really. The device has been unlocked and
> exists in /dev/mapper/ as expected.

^ This would imply that you did type the passphrase correctly, but

> systemd-ask-password-console.service stop-sigterm timed out. Killing.

^ this would suggest that the passphrase-issuing didn't complete.

Is it possible you left it open from your manual playing when you tried
starting the systemd job, by accident, and systemd had not created the
/dev/mapper/crypt device?

> My question is thus: How am I supposed to solve this the "systemd
> way"? I want to be able to start an encrypted block device using a
> normal systemd service/device so that I can later have systemd units
> depend on this.

So in theory what you've done should work for the first part, and you
would then need to create .mount units that depend on the crypt device,
like this (examples cribbed from my system doing exactly this):

> [Unit]
> RequiresOverridable=systemd-fsck@dev-mapper-docker\x2dfront.service
> After=systemd-fsck@dev-mapper-docker\x2dfront.service
> [Mount]
> What=/dev/mapper/docker-front
> Where=/var/lib/docker
> Type=btrfs
> Options=defaults,noauto

and then you would manually start *this unit* to automatically trigger
the dependency and systemd would prompt you for the passphrase:

> # systemctl start /var/lib/docker
> Please enter passphrase for disk qusp_vg-docker--back (docker-front) on /var/lib/docker! ***

(two points: one, my real passphrase is more than 3 characters. Second,
notice that I can use the FS path as the unit name - systemd can recognise
it and convert it into the right name var-lib-docker.mount. I find this more
convenient than remembering systemd's esoteric escaping rules.)

> # systemctl status /var/lib/docker
> ● var-lib-docker.mount - /var/lib/docker
> Loaded: loaded (/etc/fstab)
> Active: active (mounted) since Fri 2015-12-11 09:57:43 GMT; 9s ago
> Where: /var/lib/docker
> What: /dev/mapper/docker-front
> Process: 21422 ExecMount=/bin/mount -n /dev/mapper/docker-front /var/lib/docker -t btrfs -o defaults,noauto (code=exited, status=0/SUCCESS)

> # systemctl status 'systemd-fsck@dev-mapper-docker\x2dfront.service'
> ● systemd-fsck@dev-mapper-docker\x2dfront.service - File System Check on /dev/mapper/docker-front
> Loaded: loaded (/lib/systemd/system/systemd-fsck@.service; static)
> Active: active (exited) since Fri 2015-12-11 09:57:43 GMT; 1min 2s ago
> Process: 21416 ExecStart=/lib/systemd/systemd-fsck %f (code=exited, status=0/SUCCESS)
> Main PID: 21416 (code=exited, status=0/SUCCESS)

I've only just started doing this myself. I have additional dependencies set up
so my docker daemon depends on the mount (and isn't started on boot), but I haven't
gone further with more mounts etc. yet.

(notice that my crypttab service file has a really awkward name. That's because
I used a dash in the source and destination device names: docker-back, docker-front,
which systemd escapes. It's a bit unwieldy on the shell, I have to quote every use
so the shell doesn't expand \x2d itself, so I'd recommend avoiding using dashes in
device names with systemd.)


--
Jonathan Dowland

[Anders Andersson]

On Fri, Dec 11, 2015 at 11:04 AM, Jonathan Dowland <jm...@debian.org> wrote:
> On Thu, Dec 10, 2015 at 02:38:02PM +0100, Anders Andersson wrote:
>> My question is thus: How am I supposed to solve this the "systemd
>> way"? I want to be able to start an encrypted block device using a
>> normal systemd service/device so that I can later have systemd units
>> depend on this.
>
> So in theory what you've done should work for the first part, and you
> would then need to create .mount units that depend on the crypt device,
> like this (examples cribbed from my system doing exactly this):

Thank you for your reply! After having spent time digging in this some
more I agree with your conclusion that it should work, and I think I
have now found why it does not work in my case.

Unlocking the disk fails half-way if there is no "valid" UUID on the
LUKS *target*. A valid UUID is one that exists and is unique. This
means that it fails in these somewhat common scenarios:

*) Your LUKS device contains random numbers for a key, is zeroed out,
or contains an unknown filesystem.
*) It is one part of a multi-device btrfs filesystem (this is the killer for me)
*) You are mounting an LVM snapshot to another mountpoint (I have not
tested this)

In a multi-device btrfs filesystem, each device has the same UUID. I'm
not going to debate the sanity of that, this is how it is. The first
to be unlocked will work, the rest will be technically unlocked, but
then something happens because the UUID already exists from the first
device, so they time out.

I have created a clear test case for this using the latest release of
Fedora and sent to the systemd mailinglist for comments, but
apparently it is too obscure for someone to comment on, so I guess I
have to find other people to bug.

In the meantime, my workaround is to modify the service files to
remove all references to /dev/mapper/xxx and instead use
"Requires=crypt1.service crypt2.service" etc. This seems to work.

Thank you for your time!

// Anders