Moving volatile.img to another location

[Vít Šesták]

Hello,

I am trying to move my volatile.img files to another location. There are two (three) reasons for it:

1. Some additional secrecy; the filesystem is encrypted by a password from /dev/random, so no one can read the volatile.img files after reboot.
2. Performance (+ reduces wear when used with SSD): The filesystem for is tuned for performance at the cost of consistency when computer is forcibly turned off. This is undesirable for main FS, but OK for a dedicated filesystem for this purpose.

I have prepared such a temporary filesystem to be mounted under /largetmp. (Well, I might change it just to /tmp.) There is nothing I have problem with, so I'll share the details below.

Now, I would like to move all the volatile imgs there. I have grepped for volatile.img in Qubes sources. The easiest (though somewhat hacky) way to do this seems to be adding the following line before truncate command to /usr/lib/qubes/prepare-volatile-image.sh:
ln -s "/largetmp/volatile-$(printf %s "$FILENAME" | base64 -w0 | sed 's#/#:#g')" "$FILENAME"

HOWEVER, there seems to be a potential security issue with that. When one looks at /usr/libexec/qubes/udev-block-add-change, it seems to blacklist those volatile.img files in order to prevent parsing the file by dom0's kernel (partition table etc.). I obviously could modify that file in order to exclude all /largetmp/volatile-* files (and maybe I could even test that), but I am not sure if this can be done securely without being merged by Qubes. If this is not merged by Qubes and some dom0 update is applied, it may revert the udev script change without reverting the /qubes/prepare-volatile-image.sh file, which is obviously some risk.

Is there a better (secure) way to do this? Is there some chance of implementing this by Qubes in future?



Now, I'll include my config for /largetmp. It is basically an implementation of http://shnatsel.blogspot.cz/2012/05/tmpfs-for-real-life-write-buffering.html with encryption by a random key. The partition is ext4 on dm-crypt (plain) on LVM.

1. LVM: The LVM is optional, but I recommend this for MBR partitioning table. The LVM basically works as a stable way for referring the partition for dm-crypt. On GPT, they AFAIK have stable UUIDs in the partition table. On MBR, they may have a stable UUID inside the partition only, which is not usable in this case (for plain dm-crypt). Having a stable identifier is very important for safety: You destroy all data of the referred partition on every boot. If you use unstable identifier like /dev/sdaX and the identifier is changed (e.g. you modify your partitions or swap HDD<->SSD*), it may accidentally match another partition and destroy all its data. So, this is my reason for using LVM. You may use GPT instead.**

My LVM partition is pretty simple. It is a volume group (VG) that consists of one physical volume (PV). It may contain just one or two logical volumes (LVs). In my case, the second LV is used for swap.

2. dm-crypt with ext4: I added following line to /etc/crypttab:
dom0-largetmp      /dev/<PV name>/<encrypted largetmp LV name>     /dev/random tmp,cipher=aes-cbc-essiv:sha256

This causes an empty ext4 filesystem (notice the tmp option) to be created on /dev/<PV name>/<encrypted largetmp LV name> encrypted by a random key. I've verified that default options for the filesystem are pretty decent for this purpose (e.g. no metadata journal and no superblock backups). It seems it does not initialize the inode table lazily, which might be desirable for reducing boot time.

After adding the line to crypttab, you will probably want to regenerate daemons and restart the cryptsetup:

$ sudo /usr/lib/systemd/system-generators/systemd-cryptsetup-generator
$ sudo systemctl daemon-reload
$ sudo systemctl restart cryptsetup.target

3. fstab: This is pretty easy, but you might want to tune some parameters for performance. For example, commit=36000 (i.e. 10 hours) will make it a bit like tmpfs with HDD fallback. Some options might be redundant, but they should do no harm. There is my /etc/fstab entry:

/dev/mapper/dom0-largetmp    /largetmp    ext4  defaults,barrier=0,delalloc,data=writeback,commit=36000,noatime,nodiratime,norecovery,noacl,nouser_xattr,noauto_da_alloc,noinit_itable    0 0

You can also see some cryttab/fstab lines (with bonus – cryptswap) there: https://gist.github.com/v6ak/d5d49375d59cfae8e455

Regards,
Vít Šesták 'v6ak'

*) Swapping SSD <-> HDD might sound unlikely, but even that might have a good reason: My BIOS does not allow booting from SSD when the SSD is a second drive. (No, choosing CD boot does not the job.) However, removing the first drive (originally HDD) requires much more disassembling of my laptop than removing the second drive (originally optical drive, later SSD). So, I didn't have the swap done until recently. (I recently disassembled the laptop for another reason.)
**) My BIOS prevented me from using GPT, because it is such a crap and can't boot from that. This was my only reason for choosing LVM over GPT.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sat, Nov 14, 2015 at 11:24:23AM -0800, Vít Šesták wrote:
> Hello,
>
> I am trying to move my volatile.img files to another location. There are
> two (three) reasons for it:
>
> 1. Some additional secrecy; the filesystem is encrypted by a password from
> /dev/random, so no one can read the volatile.img files after reboot.
> 2. Performance (+ reduces wear when used with SSD): The filesystem for is
> tuned for performance at the cost of consistency when computer is forcibly
> turned off. This is undesirable for main FS, but OK for a dedicated
> filesystem for this purpose.
>
> I have prepared such a temporary filesystem to be mounted under /largetmp.
> (Well, I might change it just to /tmp.) There is nothing I have problem
> with, so I'll share the details below.
>
> Now, I would like to move all the volatile imgs there. I have grepped for
> volatile.img in Qubes sources. The easiest (though somewhat hacky) way to
> do this seems to be adding the following line before truncate command to
> /usr/lib/qubes/prepare-volatile-image.sh:
> ln -s "/largetmp/volatile-$(printf %s "$FILENAME" | base64 -w0 | sed
> 's#/#:#g')" "$FILENAME"
>
> HOWEVER, there seems to be a potential security issue with that. When one
> looks at /usr/libexec/qubes/udev-block-add-change, it seems to blacklist
> those volatile.img files in order to prevent parsing the file by dom0's
> kernel (partition table etc.).

No, that exclude is just to hide the devices from qvm-block.

It isn't possible to exclude the device from partition table parsing in
general, but for loop (and device-mapper) devices it needs to be
explicitly requested (it is, in prepare-volatile-image.sh, but it happens
before giving the VM access to the device - actually even before
starting the VM).

The other thing is excluding the device from scanning by udev - which
is much more important, because udev happen to throw as many parsers at
the device as it can (blkid etc). This thing is done in
/etc/udev/rules.d/00-qubes-ignore-devices.rules and
/etc/udev/rules.d/60-persistent-storage.rules - all the loop* devices
are excluded, regardless from location.

So, this part of Qubes security shouldn't be compromised because of
relocated volatile.img.

> I obviously could modify that file in order
> to exclude all /largetmp/volatile-* files (and maybe I could even test
> that), but I am not sure if this can be done securely without being merged
> by Qubes. If this is not merged by Qubes and some dom0 update is applied,
> it may revert the udev script change without reverting the
> /qubes/prepare-volatile-image.sh file, which is obviously some risk.
>

> *Is there a better (secure) way to do this? Is there some chance of
> implementing this by Qubes in future?*

I can't say we wont do that ever, but surely not before Qubes 4.0, where
some part of storage handling will be reworked.

> **) My BIOS prevented me from using GPT, because it is such a crap and
> can't boot from that. This was my only reason for choosing LVM over GPT.

That's strange, BIOS (in legacy mode) should have nothing to do with
partition table - it should just load the code from the beginning of
disk... So probably someone had "better" idea what BIOS should do.


- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJWR6B0AAoJENuP0xzK19csrroH/iiSGJd/0bv6dbKtfco9HnWt
9DsAVzDmvNpdLAWUkQD6xfarPRN+0YFM34mpe0i6ol6Y6bPLxPwLPPtAxfgyc1PZ
GLbLvOqQgfRE5BW7JI+ma9fOa51xGze49WXFBNnyFRydY/xE+Jq4cOcP2Ujl+Aie
V9quiatUaW7qP8ChaK6i1ahTXZR6PvsEyih9b3+KcKTprGjjg2DuK/Ac6Maw73Gs
+XzNpw9bvCXo4CY5YivutbhAfUke7/8wrF7mBFNBEHsZVH01yBDWzL/SUjRDgro0
hEeQB4Vkptc895qyj6t7pJMlmO40hc3wzVUhyDznIzuxjsEb9HbAQDub7lBRBrk=
=xwtq
-----END PGP SIGNATURE-----

[Vít Šesták]

Thank you for clarification. So, I started using the mentioned hack. I'll put there some notes if I will find some issues. I hope they will be useful for someone.

First issue found. This issue is specific to this hack and would likely not occur in some regular code rewrite. The original patch attempted to resuse old volatile files. This is potentially issue for multiple reasons (including potential security issues). The fix is simple: just truncate the file. So, the, the patch will grow to two lines:

ln -s "/largetmp/volatile-$(printf %s "$FILENAME" | base64 -w0 | sed 's#/#:#g')" "$FILENAME"

truncate -s 0 "$FILENAME"

Ad my BIOS: Yes, it is strange, and the BIOS is probably the crappiest part of my laptop. (Other parts are decent.) I haven't looked deep, but I remember being unable to boot freshly installed Ubuntu with GPT and no standard attempt (like grub2-install /dev/sda) helped. After spending much more time than I wanted to, I gave it up. If you are interested in that, I am not the only man with BIOS vs. GPT problems: http://www.rodsbooks.com/gdisk/bios.html

Regards,
Vít Šesták 'v6ak'

[Vít Šesták]

I'll add one fix and one tune.

The fix: We need to remove the potentially dangling symlink. Without that, it would fail starting after reboot. There is the updated patch:

rm -f "$FILENAME" # deletes the potentially dandling symlink
ln -s "/largetmp/volatile-$(printf %s "$FILENAME" | base64 -w0 | sed 's#/#:#g')" "$FILENAME" # redirect it to /largetmp
truncate -s 0 "$FILENAME" # truncate the file in order to handle relicts form previous runs correctly



The tune – speed up the boot: When I use tmp option in cryptsetup, it takes too much time (> 5s) on my setup. However, mkfs.ext4 can be called manually with proper parameters that make it much faster. I've created a systemd service for that: https://gist.github.com/v6ak/d5d49375d59cfae8e455#file-largetmp-fs-service

With a HDD partition of several GiBs, the startup time penalty is now just about one second, which is pretty OK for me. The debugfs -R features command suggests that the filesystem has few unneeded features, but I don't care much now.


Regards,
Vít Šesták 'v6ak'

[Vít Šesták]

One more update:

* Fixes DispVMs. When you remove volatile.img for *-dvm VM, the VM will fail to start. So, those volatile.img files are explicitly excluded from this hack.
* Automatically removes volatile.img after some time. If the file is still used, it does not break anything: It is just unlinked, but the inode is not deallocated until it is closed.
* In order to prevent race condition in automatic removal of the img file, timestamp is added to the file name.

The updated patch for /usr/lib/qubes/prepare-volatile-image.sh:

case "$FILENAME" in
    /var/lib/qubes/appvms/*-dvm/volatile.img)
        # One should not change the behavior for DVM templates!
        ;;
    *)
        # We want to move the volatile.img for other VMs
        REAL_FILENAME="/largetmp/volatile-$(printf %s "$FILENAME" | base64 -w0 | sed 's#/#:#g')--$(date +%s)"
        rm -f "$FILENAME" # deletes the potentially dangling symlink
        ln -s "$REAL_FILENAME" "$FILENAME"
        truncate -s 0 "$FILENAME"
        (sleep 600 && rm "$REAL_FILENAME")&
        ;;
esac