Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#924560: cryptsetup luksOpen requires 1GB of RAM in the default configuration
183 views
Skip to first unread message
Dimitri John Ledkov
Mar 14, 2019, 8:40:03 AM3/14/19
to
Package: cryptsetup
Version: 2:2.1.0-1
Severity: important
Dear Maintainer,
Currently the new cryptsetup defaults to LUKS2 format with the
following parameters:
Default PBKDF for LUKS2: argon2i
Iteration time: 2000, Memory required: 1048576kB, Parallel threads: 4
Meaning that 1GB of RAM is required at luksOpen. This is a significant
RAM increase compared to the previous defaults used in LUKS1. Meaning
that many devices will no longer be able to installs afresh, using
full-disk encryption.
For example many IoT and Pi devices have 1GB of ram in total, and thus
would OOM kill when trying to luksOpen.
Please consider reducing the default memory requirement of the argon2i
in luks2 by default, or switching to pbkdf2 for LUKS2 as well.
If there are multiple encrypted datavolumes, unlocked automatically
with crypttab, under systemd, they would be unlocked in parallel,
meaning peak memory requirement would be 1GB*N on boot for those
systems.
I think it is unfortunate to not support default encryption on 1GB big
devices and VMs.
I have filed a similar bug report in Ubuntu as well just now:
https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/1820049
Regards,
Dimitri.
Version: 2:2.1.0-1
Severity: important
Dear Maintainer,
Currently the new cryptsetup defaults to LUKS2 format with the
following parameters:
Default PBKDF for LUKS2: argon2i
Iteration time: 2000, Memory required: 1048576kB, Parallel threads: 4
Meaning that 1GB of RAM is required at luksOpen. This is a significant
RAM increase compared to the previous defaults used in LUKS1. Meaning
that many devices will no longer be able to installs afresh, using
full-disk encryption.
For example many IoT and Pi devices have 1GB of ram in total, and thus
would OOM kill when trying to luksOpen.
Please consider reducing the default memory requirement of the argon2i
in luks2 by default, or switching to pbkdf2 for LUKS2 as well.
If there are multiple encrypted datavolumes, unlocked automatically
with crypttab, under systemd, they would be unlocked in parallel,
meaning peak memory requirement would be 1GB*N on boot for those
systems.
I think it is unfortunate to not support default encryption on 1GB big
devices and VMs.
I have filed a similar bug report in Ubuntu as well just now:
https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/1820049
Regards,
Dimitri.
Christoph Anton Mitterer
Mar 14, 2019, 12:30:03 PM3/14/19
to
On Thu, 2019-03-14 at 12:36 +0000, Dimitri John Ledkov wrote:
> Meaning that 1GB of RAM is required at luksOpen. This is a
> significant
> RAM increase compared to the previous defaults used in LUKS1.
Well that's by design of Argon2, to make brute force hashing much
> Meaning that 1GB of RAM is required at luksOpen. This is a
> significant
> RAM increase compared to the previous defaults used in LUKS1.
harder for an attacker.
> For example many IoT and Pi devices have 1GB of ram in total, and
> thus
> would OOM kill when trying to luksOpen.
ones),... so why should the majority of users suffer greatly in their
security just to fulfill the anyway questionable needs of a small
minority?
> Please consider reducing the default memory requirement of the
> argon2i
> in luks2 by default, or switching to pbkdf2 for LUKS2 as well.
faster MD5, or replace AES by even faster Caesar cipher...
If someone really thinks he needs the just "encrypted-label" attached
to their thing but doesn't care about actual security, the defaults can
always manually be lowered... but why should everyone suffer from this?
Debian should not further weaken the sane (and actually quite
conservative) defaults of upstream.
There's apparently already the decision made (against the explicit
recommendation from upstream) to overwrite the no-discard default.
Let's please try to not create even yet another debacle like Debian-
OpenSSL-RNG in dm-crypt/cryptsetup.
Cheers,
Chris.
Guilhem Moulin
Mar 14, 2019, 1:00:03 PM3/14/19
to
Control: tag -1 moreinfo
Hi Dimitri,
On Thu, 14 Mar 2019 at 12:36:13 +0000, Dimitri John Ledkov wrote:
> Currently the new cryptsetup defaults to LUKS2 format with the
> following parameters:
>
> Default PBKDF for LUKS2: argon2i
> Iteration time: 2000, Memory required: 1048576kB, Parallel threads: 4
The actual parameters used for the KDF are determined by benchmark upon
luksFormat (or luksAddKey). See cryptsetup(8):
--pbkdf <PBKDF spec>
[…]
Note that increasing memory cost also increases time, so the final
parameter values are measured by a benchmark. The benchmark tries to
find iteration time (--iter-time) with required memory cost
--pbkdf-memory. If it is not possible, the memory cost is decreased
as well. The parallel cost --pbkdf-parallel is constant, is is
checked against available CPU cores (if not available, it is
decreased) and the maximum parallel cost is 4.
> For example many IoT and Pi devices have 1GB of ram in total, and thus
> would OOM kill when trying to luksOpen.
Is that something you experienced? I just deployed a fresh a Debian sid
VM with 2vCPUs, 1GiB RAM a default encryption stack (LVM2 on top of
LUKS/dm-crypt). Works just fine :-) The actual memory cost is about
half of the physical memory.
$ cryptsetup luksDump /dev/vda5 | grep -Fe "Time cost:" -e "Memory:" -e "Threads:"
Time cost: 4
Memory: 505372
Threads: 2
$ dd if=/dev/zero of=/tmp/disk.img bs=1M count=64
$ printf test | cryptsetup luksFormat --debug --key-file=- /tmp/disk.img
[…]
# Running argon2i() benchmark.
# PBKDF benchmark: memory cost = 32, iterations = 4, threads = 2 (took 2 ms)
# PBKDF benchmark: memory cost = 512, iterations = 4, threads = 2 (took 1 ms)
# PBKDF benchmark: memory cost = 8192, iterations = 4, threads = 2 (took 18 ms)
# PBKDF benchmark: memory cost = 113777, iterations = 4, threads = 2 (took 233 ms)
# PBKDF benchmark: memory cost = 122078, iterations = 4, threads = 2 (took 234 ms)
# PBKDF benchmark: memory cost = 130425, iterations = 4, threads = 2 (took 249 ms)
# PBKDF benchmark: memory cost = 130948, iterations = 4, threads = 2 (took 251 ms)
# PBKDF benchmark: memory cost = 505422, iterations = 8, threads = 2 (took 2015 ms)
# Benchmark returns argon2i() 8 iterations, 505422 memory, 2 threads (for 512-bits key).
[…]
Command successful.
$ printf test | cryptsetup luksOpen --debug --key-file=- /tmp/disk.img crypt_disk
[…]
# Only 2 active CPUs detected, PBKDF threads decreased from 4 to 2.
# Not enough physical memory detected, PBKDF max memory decreased from 1048576kB to 505422kB.
# PBKDF argon2i, hash sha256, time_ms 2000 (iterations 0), max_memory_kb 505422, parallel_threads 2.
[…]
Command successful.
Works equally fine on a VM with 512MiB RAM. There the memory cost is
set to ~250MiB.
> Please consider reducing the default memory requirement of the argon2i
> in luks2 by default, or switching to pbkdf2 for LUKS2 as well.
I think diverging from upstream (and other distros) with respect to
default algorithms requires careful consideration. And in that case,
compared to PBKDF2 Argon2 has interesting properties (such as resistance
to GPU cracking) which would be a shame not to benefit from out of the
box.
> I think it is unfortunate to not support default encryption on 1GB big
> devices and VMs.
AFAICT it does. What I guess doesn't is if the machine's resources are
significantly reduced between luksFormat/luksAddKey and luksOpen.
OTOH parallel unlocking might be an issue indeed, especially if the
devices are unlocked in an unattended fashion (for instance via key
files). That said on my 1GiB VM I could reliably boot with
$ cat /etc/crypttab
root_crypt /dev/vda4 none luks
swap_crypt /dev/vda2 /etc/keys/swap.key luks,swap
home_crypt /dev/vda3 /etc/keys/home.key luks
(root_crypt being unlocked an initramfs stage, the other two later by
systemd) and
$ cat /etc/fstab
/dev/vda1 /boot ext2 defaults 0 2
/dev/mapper/swap_crypt none swap sw 0 0
/dev/mapper/root_crypt / ext4 defaults 0 1
/dev/mapper/home_crypt /home ext4 defaults 0 2
However the OOM killer does trigger when I add a third volume. I guess
the boot process is racy and might, or might not, abort depending on
what's being started at the same time. But I'd argue that setup is far
from a typical installation, and the same goes for low-memory devices
with plenty of encrypted disks attached. (Moreover Debian's default
initramfs implementation comes from initramfs-tools, where unlocking is
sequential. So root-device encryption is out of the picture.)
Having a separate encrypted /home partition (unlocked by systemd)
doesn't seem to cause any problem when formatted with the default
parameters, which I understood is what this bug report is about.
Cheers,
--
Guilhem
Hi Dimitri,
On Thu, 14 Mar 2019 at 12:36:13 +0000, Dimitri John Ledkov wrote:
> Currently the new cryptsetup defaults to LUKS2 format with the
> following parameters:
>
> Default PBKDF for LUKS2: argon2i
> Iteration time: 2000, Memory required: 1048576kB, Parallel threads: 4
luksFormat (or luksAddKey). See cryptsetup(8):
--pbkdf <PBKDF spec>
[…]
Note that increasing memory cost also increases time, so the final
parameter values are measured by a benchmark. The benchmark tries to
find iteration time (--iter-time) with required memory cost
--pbkdf-memory. If it is not possible, the memory cost is decreased
as well. The parallel cost --pbkdf-parallel is constant, is is
checked against available CPU cores (if not available, it is
decreased) and the maximum parallel cost is 4.
> For example many IoT and Pi devices have 1GB of ram in total, and thus
> would OOM kill when trying to luksOpen.
VM with 2vCPUs, 1GiB RAM a default encryption stack (LVM2 on top of
LUKS/dm-crypt). Works just fine :-) The actual memory cost is about
half of the physical memory.
$ cryptsetup luksDump /dev/vda5 | grep -Fe "Time cost:" -e "Memory:" -e "Threads:"
Time cost: 4
Memory: 505372
Threads: 2
$ dd if=/dev/zero of=/tmp/disk.img bs=1M count=64
$ printf test | cryptsetup luksFormat --debug --key-file=- /tmp/disk.img
[…]
# Running argon2i() benchmark.
# PBKDF benchmark: memory cost = 32, iterations = 4, threads = 2 (took 2 ms)
# PBKDF benchmark: memory cost = 512, iterations = 4, threads = 2 (took 1 ms)
# PBKDF benchmark: memory cost = 8192, iterations = 4, threads = 2 (took 18 ms)
# PBKDF benchmark: memory cost = 113777, iterations = 4, threads = 2 (took 233 ms)
# PBKDF benchmark: memory cost = 122078, iterations = 4, threads = 2 (took 234 ms)
# PBKDF benchmark: memory cost = 130425, iterations = 4, threads = 2 (took 249 ms)
# PBKDF benchmark: memory cost = 130948, iterations = 4, threads = 2 (took 251 ms)
# PBKDF benchmark: memory cost = 505422, iterations = 8, threads = 2 (took 2015 ms)
# Benchmark returns argon2i() 8 iterations, 505422 memory, 2 threads (for 512-bits key).
[…]
Command successful.
$ printf test | cryptsetup luksOpen --debug --key-file=- /tmp/disk.img crypt_disk
[…]
# Only 2 active CPUs detected, PBKDF threads decreased from 4 to 2.
# Not enough physical memory detected, PBKDF max memory decreased from 1048576kB to 505422kB.
# PBKDF argon2i, hash sha256, time_ms 2000 (iterations 0), max_memory_kb 505422, parallel_threads 2.
[…]
Command successful.
Works equally fine on a VM with 512MiB RAM. There the memory cost is
set to ~250MiB.
> Please consider reducing the default memory requirement of the argon2i
> in luks2 by default, or switching to pbkdf2 for LUKS2 as well.
default algorithms requires careful consideration. And in that case,
compared to PBKDF2 Argon2 has interesting properties (such as resistance
to GPU cracking) which would be a shame not to benefit from out of the
box.
> I think it is unfortunate to not support default encryption on 1GB big
> devices and VMs.
significantly reduced between luksFormat/luksAddKey and luksOpen.
OTOH parallel unlocking might be an issue indeed, especially if the
devices are unlocked in an unattended fashion (for instance via key
files). That said on my 1GiB VM I could reliably boot with
$ cat /etc/crypttab
root_crypt /dev/vda4 none luks
swap_crypt /dev/vda2 /etc/keys/swap.key luks,swap
home_crypt /dev/vda3 /etc/keys/home.key luks
(root_crypt being unlocked an initramfs stage, the other two later by
systemd) and
$ cat /etc/fstab
/dev/vda1 /boot ext2 defaults 0 2
/dev/mapper/swap_crypt none swap sw 0 0
/dev/mapper/root_crypt / ext4 defaults 0 1
/dev/mapper/home_crypt /home ext4 defaults 0 2
However the OOM killer does trigger when I add a third volume. I guess
the boot process is racy and might, or might not, abort depending on
what's being started at the same time. But I'd argue that setup is far
from a typical installation, and the same goes for low-memory devices
with plenty of encrypted disks attached. (Moreover Debian's default
initramfs implementation comes from initramfs-tools, where unlocking is
sequential. So root-device encryption is out of the picture.)
Having a separate encrypted /home partition (unlocked by systemd)
doesn't seem to cause any problem when formatted with the default
parameters, which I understood is what this bug report is about.
Cheers,
--
Guilhem
Dimitri John Ledkov
Mar 14, 2019, 1:40:03 PM3/14/19
to
On Thu, 14 Mar 2019 at 16:55, Guilhem Moulin <gui...@debian.org> wrote:
>
> > For example many IoT and Pi devices have 1GB of ram in total, and thus
> > would OOM kill when trying to luksOpen.
>
> Is that something you experienced? I just deployed a fresh a Debian sid
> VM with 2vCPUs, 1GiB RAM a default encryption stack (LVM2 on top of
> LUKS/dm-crypt). Works just fine :-) The actual memory cost is about
> half of the physical memory.
>
>
> > For example many IoT and Pi devices have 1GB of ram in total, and thus
> > would OOM kill when trying to luksOpen.
>
> Is that something you experienced? I just deployed a fresh a Debian sid
> VM with 2vCPUs, 1GiB RAM a default encryption stack (LVM2 on top of
> LUKS/dm-crypt). Works just fine :-) The actual memory cost is about
> half of the physical memory.
>
> Works equally fine on a VM with 512MiB RAM. There the memory cost is
> set to ~250MiB.
>
Nice.
> set to ~250MiB.
>
> > Please consider reducing the default memory requirement of the argon2i
> > in luks2 by default, or switching to pbkdf2 for LUKS2 as well.
>
> I think diverging from upstream (and other distros) with respect to
> default algorithms requires careful consideration. And in that case,
> compared to PBKDF2 Argon2 has interesting properties (such as resistance
> to GPU cracking) which would be a shame not to benefit from out of the
> box.
>
> > I think it is unfortunate to not support default encryption on 1GB big
> > devices and VMs.
>
> AFAICT it does. What I guess doesn't is if the machine's resources are
> significantly reduced between luksFormat/luksAddKey and luksOpen.
>
might prepare rootfs / SDcard / golden image / etc on a bigger
developer machine, prior to flashing it onto SDcard to then deploy on
the target device.
So the solution there is to run the benchmark on the device, once,
record parameters and use those when creating the golden image for
memory, threads and possibly iterations too? I.e. I think I managed to
use
printf test | sudo cryptsetup luksFormat --debug --key-file=-
--pbkdf-memory=505372 --pbkdf-parallel=2 --pbkdf-force-iterations=4
/dev/nbd0
To create an "easy" encrypted filesystem with matching algo parameters
for a "small" device, on my "big" desktop.
> OTOH parallel unlocking might be an issue indeed, especially if the
> devices are unlocked in an unattended fashion (for instance via key
> files). That said on my 1GiB VM I could reliably boot with
>
> $ cat /etc/crypttab
> root_crypt /dev/vda4 none luks
> swap_crypt /dev/vda2 /etc/keys/swap.key luks,swap
> home_crypt /dev/vda3 /etc/keys/home.key luks
>
> (root_crypt being unlocked an initramfs stage, the other two later by
> systemd) and
>
> $ cat /etc/fstab
> /dev/vda1 /boot ext2 defaults 0 2
> /dev/mapper/swap_crypt none swap sw 0 0
> /dev/mapper/root_crypt / ext4 defaults 0 1
> /dev/mapper/home_crypt /home ext4 defaults 0 2
>
> However the OOM killer does trigger when I add a third volume. I guess
> the boot process is racy and might, or might not, abort depending on
> what's being started at the same time. But I'd argue that setup is far
> from a typical installation, and the same goes for low-memory devices
> with plenty of encrypted disks attached. (Moreover Debian's default
> initramfs implementation comes from initramfs-tools, where unlocking is
> sequential. So root-device encryption is out of the picture.)
>
> Having a separate encrypted /home partition (unlocked by systemd)
> doesn't seem to cause any problem when formatted with the default
> parameters, which I understood is what this bug report is about.
>
than initramfs-tools. I wonder if I should open a bug report in
systemd, to potentially execute luks2 unlock with some locking /
sequentially.
Similarly, I shall probably back out hardcoding of --pbkdf pbkdf out
of s390-tools zkey.
I guess this bug should be tagged wontfixing and lowered priority to
normal, or something.
--
Regards,
Dimitri.
Guilhem Moulin
Mar 14, 2019, 2:40:03 PM3/14/19
to
Control: tag -1 + wontfix
Control: tag -1 - moreinfo
Control: severity -1 normal
On Thu, 14 Mar 2019 at 17:31:05 +0000, Dimitri John Ledkov wrote:
> On Thu, 14 Mar 2019 at 16:55, Guilhem Moulin <gui...@debian.org> wrote:
>> AFAICT it does. What I guess doesn't is if the machine's resources are
>> significantly reduced between luksFormat/luksAddKey and luksOpen.
>
> I guess that is the reason here. Majority of IoT / pi / etc devices
> might prepare rootfs / SDcard / golden image / etc on a bigger
> developer machine, prior to flashing it onto SDcard to then deploy on
> the target device.
I see.
> So the solution there is to run the benchmark on the device, once,
> record parameters and use those when creating the golden image for
> memory, threads and possibly iterations too?
Yup, you can re-use the parameters from a previous benchmark run on the
target system (where the volume will be unlocked) and use it to
luksFormat elsewhere.
This applies to PBKDF2 too by the way, though skipping this step doesn't
have the same severity; for PBKDF2 the benchmark only affect the
iteration time, so if the volume is formatted on a fast system, unlocking
on a slower one will slower than it should be (but won't OOM).
> I wonder if I should open a bug report in systemd, to potentially
> execute luks2 unlock with some locking / sequentially.
Seems sensible.
> I guess this bug should be tagged wontfixing and lowered priority to
> normal, or something.
Alright :-)
--
Guilhem.
Control: tag -1 - moreinfo
Control: severity -1 normal
On Thu, 14 Mar 2019 at 17:31:05 +0000, Dimitri John Ledkov wrote:
> On Thu, 14 Mar 2019 at 16:55, Guilhem Moulin <gui...@debian.org> wrote:
>> AFAICT it does. What I guess doesn't is if the machine's resources are
>> significantly reduced between luksFormat/luksAddKey and luksOpen.
>
> I guess that is the reason here. Majority of IoT / pi / etc devices
> might prepare rootfs / SDcard / golden image / etc on a bigger
> developer machine, prior to flashing it onto SDcard to then deploy on
> the target device.
> So the solution there is to run the benchmark on the device, once,
> record parameters and use those when creating the golden image for
> memory, threads and possibly iterations too?
target system (where the volume will be unlocked) and use it to
luksFormat elsewhere.
This applies to PBKDF2 too by the way, though skipping this step doesn't
have the same severity; for PBKDF2 the benchmark only affect the
iteration time, so if the volume is formatted on a fast system, unlocking
on a slower one will slower than it should be (but won't OOM).
> I wonder if I should open a bug report in systemd, to potentially
> execute luks2 unlock with some locking / sequentially.
> I guess this bug should be tagged wontfixing and lowered priority to
> normal, or something.
--
Guilhem.
Guilhem Moulin
Mar 14, 2019, 2:50:03 PM3/14/19
to
Hi Milan,
On Thu, 14 Mar 2019 at 19:22:42 +0100, Milan Broz wrote:
>>> I think diverging from upstream (and other distros) with respect to
>>> default algorithms requires careful consideration. And in that case,
>>> compared to PBKDF2 Argon2 has interesting properties (such as resistance
>>> to GPU cracking) which would be a shame not to benefit from out of the
>>> box.
>
> For this case you need to specify PBKDF parameters directly and skip benchmark
> (these PBKDF options were added exactly for this use case).
>
> This problem is there even with PBKDF2 for the iterations time - on some
> IoT devices with LUKS device (formatted on developer's machine) the unlocking
> time increases to many minutes. (With Argon PBKDF it is just worse because memory
> can be unavailable.)
Aha, you beat me to it :-)
>> I guess dracut with systemd in the initrd might be affected worse,
>> than initramfs-tools. I wonder if I should open a bug report in
>> systemd, to potentially execute luks2 unlock with some locking /
>> sequentially.
>
> FYI we know about that parallel unlocking problem already and we are trying
> to find (with systemd people) some solution (perhaps based on cgroups memory limits
> and some locking).
Cool, do you have a link to refer to? Couldn't find anything from a
quick glance at systemd's issue tracker.
--
Guilhem.
On Thu, 14 Mar 2019 at 19:22:42 +0100, Milan Broz wrote:
>>> I think diverging from upstream (and other distros) with respect to
>>> default algorithms requires careful consideration. And in that case,
>>> compared to PBKDF2 Argon2 has interesting properties (such as resistance
>>> to GPU cracking) which would be a shame not to benefit from out of the
>>> box.
>
> (these PBKDF options were added exactly for this use case).
>
> This problem is there even with PBKDF2 for the iterations time - on some
> IoT devices with LUKS device (formatted on developer's machine) the unlocking
> time increases to many minutes. (With Argon PBKDF it is just worse because memory
> can be unavailable.)
Aha, you beat me to it :-)
>> I guess dracut with systemd in the initrd might be affected worse,
>> than initramfs-tools. I wonder if I should open a bug report in
>> systemd, to potentially execute luks2 unlock with some locking /
>> sequentially.
>
> to find (with systemd people) some solution (perhaps based on cgroups memory limits
> and some locking).
Cool, do you have a link to refer to? Couldn't find anything from a
quick glance at systemd's issue tracker.
--
Guilhem.
Ondrej Kozina
Mar 26, 2019, 5:50:03 AM3/26/19
to
On Thu, 14 Mar 2019 19:43:26 +0100 Guilhem Moulin <gui...@debian.org>
wrote:
(https://gitlab.com/cryptsetup/cryptsetup/issues/446).
There should be some systemd issue/PR soon as well. I'll link it to the
cryptsetup issue when I see it.
Regards
Ondra
>
> --
> Guilhem.
wrote:
> Hi Milan,
>
> On Thu, 14 Mar 2019 at 19:22:42 +0100, Milan Broz wrote:
> > (...)
>
> On Thu, 14 Mar 2019 at 19:22:42 +0100, Milan Broz wrote:
> > FYI we know about that parallel unlocking problem already and we are trying
> > to find (with systemd people) some solution (perhaps based on cgroups memory limits
> > and some locking).
>
> Cool, do you have a link to refer to? Couldn't find anything from a
> quick glance at systemd's issue tracker.
FYI, I've opened cryptsetup issue for parallel Argon2 invocation case
> > to find (with systemd people) some solution (perhaps based on cgroups memory limits
> > and some locking).
>
> Cool, do you have a link to refer to? Couldn't find anything from a
> quick glance at systemd's issue tracker.
(https://gitlab.com/cryptsetup/cryptsetup/issues/446).
There should be some systemd issue/PR soon as well. I'll link it to the
cryptsetup issue when I see it.
Regards
Ondra
>
> --
> Guilhem.
Ondrej Kozina
Nov 27, 2019, 10:30:04 AM11/27/19
to
Jérôme Charaoui
Mar 11, 2023, 8:40:04 AM3/11/23
to
Package: cryptsetup
Version: 2:2.6.1-1
Severity: critical
Dear maintainer,
Today I upgraded a small KVM machine with a LUKS2 encrypted root and
1GiB of RAM to bookworm, and was very surprised to be confronted with an
OOM immediately upon entering my LUKS password in the initramfs prompt:
Loading Linux 6.1.0-5-amd64 ...
Loading initial ramdisk ...
Please unlock disk vda5_crypt:
[ 7.982435] Out of memory: Killed process 243 (cryptsetup)
total-vm:799000kB, anon-rss:678296kB, file-rss:6440kB, shmem-rss:0kB,
UID:0 pgtables:1384kB oom_score_adj:0
Killed
cryptsetup: ERROR: vda5_crypt: cryptsetup failed, bad password or
options?
This machine was booting just fine before upgrading, under bullseye.
The problem appears to be perhaps related to #924560, but in this
instance, the issue causing an unbootable system post-upgrade.
Thanks,
-- Jérôme
Version: 2:2.6.1-1
Severity: critical
Dear maintainer,
Today I upgraded a small KVM machine with a LUKS2 encrypted root and
1GiB of RAM to bookworm, and was very surprised to be confronted with an
OOM immediately upon entering my LUKS password in the initramfs prompt:
Loading Linux 6.1.0-5-amd64 ...
Loading initial ramdisk ...
Please unlock disk vda5_crypt:
[ 7.982435] Out of memory: Killed process 243 (cryptsetup)
total-vm:799000kB, anon-rss:678296kB, file-rss:6440kB, shmem-rss:0kB,
UID:0 pgtables:1384kB oom_score_adj:0
Killed
cryptsetup: ERROR: vda5_crypt: cryptsetup failed, bad password or
options?
This machine was booting just fine before upgrading, under bullseye.
The problem appears to be perhaps related to #924560, but in this
instance, the issue causing an unbootable system post-upgrade.
Thanks,
-- Jérôme
Cyril Brulebois
Mar 11, 2023, 9:34:33 AM3/11/23
to
Control: severity -1 serious
Hi Guilhem,
Guilhem Moulin <gui...@debian.org> (2023-03-11):
> Don't think we can do anything in src:cryptsetup for existing volumes
> unfortunately. You might need to manually lower the parameters of your
> PBKDF.
Existing systems failing to boot after an upgrade doesn't seem to be
“only” important to me…
> Lowering the severity, because this shouldn't block the transition of -2
> into bookworm (which fixes an unrelated and arguably much more severe RC
> bug).
That's not really how RC bugs work: bugs aren't less RC because it makes
sense for a specific version to migrate…
Either the bug appeared specifically in the version it was filed against,
and it makes sense to block the migration since that's a new RC bug in
that particular version, and the RC-ness stays.
Or the bug was already there in the version currently in testing, and that
means that's not a regression, and the RC-ness stays. You only need to
record the bug as also being found in the previous version (possibly
plural) to make sure britney knows it's not a regression.
> See also the d-i errata for Bookworm Alpha 2. I'm also not certain that
> #-1 is RC, after all Debian's memory requirements have consistently
> increased since the start of the project, so IMHO one has to accept that
> hardware might need to be dusted up on upgrade. Compare
>
> https://www.debian.org/releases/potato/i386/ch-hardware-req.en.html §2.3 with
> https://www.debian.org/releases/bullseye/amd64/ch03s04.en.html §3.4
Sure, we can discuss the severity of the bug I filed. But #1032734 really
can't be “just” important.
Cheers,
--
Cyril Brulebois (ki...@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Hi Guilhem,
Guilhem Moulin <gui...@debian.org> (2023-03-11):
> On Sat, 11 Mar 2023 at 08:26:27 -0500, Jérôme Charaoui wrote:
> > Today I upgraded a small KVM machine with a LUKS2 encrypted root and 1GiB of
> > RAM to bookworm, and was very surprised to be confronted with an OOM
> > immediately upon entering my LUKS password in the initramfs prompt:
> > […]
> > Today I upgraded a small KVM machine with a LUKS2 encrypted root and 1GiB of
> > RAM to bookworm, and was very surprised to be confronted with an OOM
> > immediately upon entering my LUKS password in the initramfs prompt:
> > The problem appears to be perhaps related to #924560, but in this instance,
> > the issue causing an unbootable system post-upgrade.
>
> No, this is related to #1028250 and https://gitlab.com/cryptsetup/cryptsetup/-/issues/802#note_1287298872 .
> > the issue causing an unbootable system post-upgrade.
>
> Don't think we can do anything in src:cryptsetup for existing volumes
> unfortunately. You might need to manually lower the parameters of your
> PBKDF.
Existing systems failing to boot after an upgrade doesn't seem to be
“only” important to me…
> Lowering the severity, because this shouldn't block the transition of -2
> into bookworm (which fixes an unrelated and arguably much more severe RC
> bug).
That's not really how RC bugs work: bugs aren't less RC because it makes
sense for a specific version to migrate…
Either the bug appeared specifically in the version it was filed against,
and it makes sense to block the migration since that's a new RC bug in
that particular version, and the RC-ness stays.
Or the bug was already there in the version currently in testing, and that
means that's not a regression, and the RC-ness stays. You only need to
record the bug as also being found in the previous version (possibly
plural) to make sure britney knows it's not a regression.
> See also the d-i errata for Bookworm Alpha 2. I'm also not certain that
> #-1 is RC, after all Debian's memory requirements have consistently
> increased since the start of the project, so IMHO one has to accept that
> hardware might need to be dusted up on upgrade. Compare
>
> https://www.debian.org/releases/potato/i386/ch-hardware-req.en.html §2.3 with
> https://www.debian.org/releases/bullseye/amd64/ch03s04.en.html §3.4
Sure, we can discuss the severity of the bug I filed. But #1032734 really
can't be “just” important.
Cheers,
--
Cyril Brulebois (ki...@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Jérôme Charaoui
Mar 11, 2023, 3:00:03 PM3/11/23
to
> Tagging ‘moreinfo’ then. I can definitely see how one can reproduce
> this theoretically (and possibly in the future when the kernel's memory
> requirement increase high enough), and mentioned that in the upstream
> bug, but I'm unable to find a reproducer after dist-upgrading bullseye
> systems to bookworm (all created from d-i's debian-11.6.0-amd64-netinst.iso,
> and “Encrypted LVM” partition scheme, on VMs with 1024M RAM).
>
> Jérôme, what memory cost is the keyslot using? (Paste the output of
> `cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:`.) Would also be
> interested to see by how much the amount of memory available to
> cryptsetup has changed before and after the uprade. Please edit
> /usr/share/initramfs-tools/scripts/local-top/cryptroot and add `free` at
> the begining of the setup_mapping() function (patch attached). My own
> findings are as follows (again on a minimal netinst system without
> changing any default). cryptsetup isn't even close to memory
> exhaustion.
Memory cost:
----- 8< -----
# cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:
PBKDF: argon2i
Time cost: 4
Memory: 787907
Threads: 1
----- >8 -----
Free memory:
----- 8< -----
available
Mem: 993756 74720 725012 60 194024
660412
Swap: 0 0 0
----- >8 -----
I've also tested on another of my virtual machines that has 1GiB of RAM
and got another result, closer to what you got in your experimentation:
----- 8< -----
# cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:
PBKDF: argon2i
Time cost: 6
Memory: 499912
Threads: 1
----- >8 -----
So, I think what's happening is that the first VM may have been created
with a different (larger) memory configuration, and was reduced at a
later point in time. I don't have absolute certainty of this, but it
would very well explain the discrepancy in memory cost.
Also, I think I agree with your assessment that in the memory usage
increase of the kernel may be involved: between the two releases,
according to your numbers it appears to have increased nearly 25% (!).
So it could also explain why it (probably very nearly) worked under
bullseye.
I there any way we could make the cryptsetup-initramfs hook aware of
this, and emit a warning if it finds that the encrypted root lacks a
keyslot with appropriate (low-enough) memory cost?
Thanks,
-- Jérôme
> this theoretically (and possibly in the future when the kernel's memory
> requirement increase high enough), and mentioned that in the upstream
> bug, but I'm unable to find a reproducer after dist-upgrading bullseye
> systems to bookworm (all created from d-i's debian-11.6.0-amd64-netinst.iso,
> and “Encrypted LVM” partition scheme, on VMs with 1024M RAM).
>
> Jérôme, what memory cost is the keyslot using? (Paste the output of
> `cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:`.) Would also be
> interested to see by how much the amount of memory available to
> cryptsetup has changed before and after the uprade. Please edit
> /usr/share/initramfs-tools/scripts/local-top/cryptroot and add `free` at
> the begining of the setup_mapping() function (patch attached). My own
> findings are as follows (again on a minimal netinst system without
> changing any default). cryptsetup isn't even close to memory
> exhaustion.
Memory cost:
----- 8< -----
# cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:
PBKDF: argon2i
Time cost: 4
Memory: 787907
Threads: 1
----- >8 -----
Free memory:
----- 8< -----
Loading Linux 6.1.0-5-amd64 ...
Loading initial ramdisk ...
total used free shared buff/cache
Loading initial ramdisk ...
available
Mem: 993756 74720 725012 60 194024
660412
Swap: 0 0 0
----- >8 -----
I've also tested on another of my virtual machines that has 1GiB of RAM
and got another result, closer to what you got in your experimentation:
----- 8< -----
# cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:
PBKDF: argon2i
Time cost: 6
Memory: 499912
Threads: 1
----- >8 -----
So, I think what's happening is that the first VM may have been created
with a different (larger) memory configuration, and was reduced at a
later point in time. I don't have absolute certainty of this, but it
would very well explain the discrepancy in memory cost.
Also, I think I agree with your assessment that in the memory usage
increase of the kernel may be involved: between the two releases,
according to your numbers it appears to have increased nearly 25% (!).
So it could also explain why it (probably very nearly) worked under
bullseye.
I there any way we could make the cryptsetup-initramfs hook aware of
this, and emit a warning if it finds that the encrypted root lacks a
keyslot with appropriate (low-enough) memory cost?
Thanks,
-- Jérôme
Guilhem Moulin
Mar 11, 2023, 4:30:05 PM3/11/23
to
Control: tag -1 - moreinfo
Control: severity -1 important
Control: retitle -1 Argon2 memory cost is not future proof and might OOM on dist-upgrade on memory-constrained systems
On Sat, 11 Mar 2023 at 14:53:37 -0500, Jérôme Charaoui wrote:
>> Jérôme, what memory cost is the keyslot using? (Paste the output of
>> `cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:`.)
>
> ----- 8< -----
> # cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:
> PBKDF: argon2i
> Time cost: 4
> Memory: 787907
> Threads: 1
> ----- >8 -----
> […]
> # cryptsetup luksDump /dev/vda5 | grep -A3 PBKDF:
> PBKDF: argon2i
> Time cost: 4
> Memory: 787907
> Threads: 1
> ----- >8 -----
> So, I think what's happening is that the first VM may have been created with
> a different (larger) memory configuration, and was reduced at a later point
> in time. I don't have absolute certainty of this, but it would very well
> explain the discrepancy in memory cost.
I'm very sure it's the case: buster, bullseye, bookworm (and everything
> a different (larger) memory configuration, and was reduced at a later point
> in time. I don't have absolute certainty of this, but it would very well
> explain the discrepancy in memory cost.
in between) never set the memory cost to more than half the physical
memory. So it's just not possible to end up with such a high memory
cost on a machine with only 1GiB RAM. Memory-hard KDF parameters are
non portable and this is a feature not a bug :-) Upon hardware change
one needs to run the benchmark again via cryptsetup-luksChangeKey(8) or
similar to tune the parameters to the new system; and downgrading
hardware needs to be done with care as folks who bootstrap images for
RPI-like boards are surely aware.
It's a coincidence that you triggered the OOM-killer only after the post
dist-upgrade reboot, you were already likely close to memory exhaustion
after reducing memory.
> I there any way we could make the cryptsetup-initramfs hook aware of this,
> and emit a warning if it finds that the encrypted root lacks a keyslot with
> appropriate (low-enough) memory cost?
access to the header where this information resides, 2/ it doesn't know
beforehand which keyslot will be used, and 3/ the issue is not specific
to cryptsetup-initramfs. There is an upstream commit on the main branch
that adds a warning to libcryptsetup, might cherry-pick that to bookworm
instead.
Anyway, lowering this to sub-RC now that it's demystified. In your case
the root issue (KDF parameter portability) is wontfix/notabug, but I'm
hijacking this to point out that KDF parameters are not future proof.
(This is what the forwarded upstream bug points to, and what I initially
thought you might be experiencing.) Things work fine out of the box on
minimal systems (also with <1GiB RAM), but several releases down the
road we might ship a kernel or early boot daemons requiring a lot more
memory, and the KDF *will* exhaust memory at that point. The upstream
fix in !490 (neither in bookworm nor sid yet) improves things a bit but
really is only buying time. Milan suggested that systems with little
RAM are probably better off using a non-memory-hard KDF. Perhaps
upstream could be convinced to have different defaults depending on the
amount of physical memory, if not then perhaps it could be done in
partman-crypto (I personally wouldn't feel comfortable carrying such a
patch in src:cryptsetup or have defaults that are unaligned with
upstream or other distros). Won't help with existing keyslots though.
--
Guilhem.
0 new messages