Moving vm data to separate partition

[Trammell Hudson]

I'd like to configure my Qubes installation to have a read-only and
dm-verity protected / and /boot partitions for the hypervisor, dom0,
qubes configurations and templates, with a separate read-write partition
for the user data and volatile portions protected by a sealed TPM key.

In the "VM Settings" - "Advanced" tab the "Paths" do not seem to
be editable. I could edit the file by hand after setting up Qubes,
although if there is an official way to do it during installation or
after-the-fact that would be nicer.

--
Trammell

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Aug 09, 2016 at 08:32:39AM -0600, Trammell Hudson wrote:
> I'd like to configure my Qubes installation to have a read-only and
> dm-verity protected / and /boot partitions for the hypervisor, dom0,
> qubes configurations and templates, with a separate read-write partition
> for the user data and volatile portions protected by a sealed TPM key.

This is very interesting setup! Please tell us if you manage to do it.

> In the "VM Settings" - "Advanced" tab the "Paths" do not seem to
> be editable. I could edit the file by hand after setting up Qubes,
> although if there is an official way to do it during installation or
> after-the-fact that would be nicer.

There is no supported way for changing those path. But you may try to
using `mount --bind` to redirect some of those directories or even
single files (like /var/lib/qubes/appvms).
Also search the list archive for relocating volatile.img files - AFAIR
there was some script for that.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXqiOOAAoJENuP0xzK19cses8H/3t525+mYoV7EE5EzZdTxRbX
OHWzmNNzrd+AKJZ4rMNFBHeP1rGLSuetl+bWgxG4yk3SNptWv/eJk/70KeGU/ZtW
iyTjKqmf7eVO+LwBRBn8mCUKxs/JxbYp+ErKNd5OzDMRzeMPZqFvW6YUL2nXx8RV
7JSKQEd8Fr0w/V32En8BDhTCxDOLP2HJAiJTmq/Scn6YI9Am2J624MK2gPNkWy5j
92Y2FsRey6MxEDnbalkbxiCCleMFPr+txL8Q8E8qpwGU3J1zUv49Rzvis3NAW4XM
6K7Sn/kTzQejgPfdJ0cGr3zxE/kPfgZ8v2L6WU/47LArrwsDYLzqzviev0eQxE4=
=Ngm3
-----END PGP SIGNATURE-----

[Trammell Hudson]

On Tue, Aug 09, 2016 at 08:40:15PM +0200, Marek Marczykowski-Górecki wrote:
> On Tue, Aug 09, 2016 at 08:32:39AM -0600, Trammell Hudson wrote:
> > I'd like to configure my Qubes installation to have a read-only and
> > dm-verity protected / and /boot partitions for the hypervisor, dom0,
> > qubes configurations and templates, with a separate read-write partition
> > for the user data and volatile portions protected by a sealed TPM key.
>
> This is very interesting setup! Please tell us if you manage to do it.

Assuming it all works, I'll be doing a writeup and hopefully a CCC talk
on the configuraiton.

The coreboot parts are working really well now -- the root of trust is
setup in the romstage, which can be protected by the SPI flash chip's WP#
and BP bits to prevent software updates (although still a potential evil
maid risk).

All of the CBFS entries are measured before they are executed or unpacked,
which includes MRC, SMM, and the Linux kernel/initrd. tpmtop runs in the
Linux payload and if the TPM can unseal the TOTP secret, it attests that
the system is a hopefully good state. Also in the ROM is a GPG keyring
with public signing keys.

The /boot partition is protected with dm-verity, the root hash
signature is verified with GPG, the xen.gz, vmlinuz and initrd
can also be signed and verified. The private key is in a Yubikey,
so updates to that partition will require the hardware token to resign
the hashes.

And then Qubes starts up and the hard parts begin...

> > In the "VM Settings" - "Advanced" tab the "Paths" do not seem to
> > be editable. I could edit the file by hand after setting up Qubes,
> > although if there is an official way to do it during installation or
> > after-the-fact that would be nicer.
>
> There is no supported way for changing those path.

Is it possible to change them at all? Even when I edit the
/var/lib/qubes/appvms/personal/personal.conf file,
after I start the VM with qvm-start it appears that the file is
re-written (my changes are gone and the paths are back to the
defaults).

Is there a reason to recreate the file instead of using its contents?

> [...]

> Also search the list archive for relocating volatile.img files - AFAIR
> there was some script for that.

It looks like a symlink for the private.img file will work, but the
volatile.img is always re-created. I see the post that modifies the
/usr/lib/qubes/prepare-volatile-img.sh, so maybe those two are sufficient
to migrate the files from / to /home.

--
Trammell

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Yes, all the VM settings are stored in /var/lib/qubes/qubes.xml. The VM
config is merely for debugging purposes - it is only written and never
read by the tools. The same generated config (based on qubes.xml) is
sent directly to libvirt.

As for file paths - in theory you could edit qubes.xml to move it
somewhere else, but in practice there are multiple places relying on
exact /var/lib/qubes directory - for example udev scripts preventing all
the filesystem guessing code, or hiding those devices from qvm-block.
Also DispVM preparation script rely on those paths...

So, better use mount --bind.

BTW in Qubes 4.0 we already have an API to support arbitrary images
location.

> > [...]
> > Also search the list archive for relocating volatile.img files - AFAIR
> > there was some script for that.
>
> It looks like a symlink for the private.img file will work, but the
> volatile.img is always re-created. I see the post that modifies the
> /usr/lib/qubes/prepare-volatile-img.sh, so maybe those two are sufficient
> to migrate the files from / to /home.

Besides the above, for volatile.img it should be enough to modify this
script.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXrXhcAAoJENuP0xzK19cs5QMH/2YvadsEjvc7K5VGAuXQBBHU
bkp6MF3d+VhS5N4r7X0UDZP57d4FrafUtRH4l9toUMJkx9R3EeYC4dsrH1dT5NBK
phKHFu+Cd+qHTIAgTrwZs88gG4ZUOLcoU3vG/iSiI2cNt24Z5vzJBCfeD+J/0Gdq
eQ6EcGNTaqPpEG5RZW90SHTaqJDp+ZnbG5Raqvnq+g3YxZwOCOzBg6UfvflRjMbq
Yy+TgXt0SeHBU1TKLbWtIieQtE9Rfpl7lbbGfa0ZsLGAxgjWPJ59PdmIeP8QVbeU
GZKo6wY2Tzcu+qldDSpfOTwbqaqrE3AlIzj085OszOo9kRWepzyqU1XT+lNIn4I=
=MeYj
-----END PGP SIGNATURE-----

[Trammell Hudson]

On Fri, Aug 12, 2016 at 09:18:52AM +0200, Marek Marczykowski-Górecki wrote:
> > Is there a reason to recreate the file instead of using its contents?
>
> Yes, all the VM settings are stored in /var/lib/qubes/qubes.xml. The VM
> config is merely for debugging purposes - it is only written and never
> read by the tools. The same generated config (based on qubes.xml) is
> sent directly to libvirt.

Oh, that is even better -- I can just move the entire appvms/personal
directory to /home and leave a single symlink in /var/lib/qubes/appvms.

> [...]

> BTW in Qubes 4.0 we already have an API to support arbitrary images
> location.

Looking forward to it! 3.2-rc1 has been working well for me, but it
sounds like 4 will be great.

--
rammell

[Trammell Hudson]

On Fri, Aug 12, 2016 at 09:18:52AM +0200, Marek Marczykowski-Górecki wrote:

> > > [...]
> > > Also search the list archive for relocating volatile.img files - AFAIR
> > > there was some script for that.
>

> Besides the above, for volatile.img it should be enough to modify this
> script.

It seems that script is insufficient for a read-only / filesystem
since the prepare-volatile-img.sh script calls truncate on the file
and qubes/storage/__init__.py calls os.remove() on the file.

I modified the python code to check for a symlink and remove the
destination of the link instead:

# Re-create only for template based VMs
if source_template is not None and self.volatile_img:
if (os.path.islink(self.volatile_img)):
if os.path.exists(self.volatile_img):
os.remove(os.readlink(self.volatile_img))
elif os.path.exists(self.volatile_img):
os.remove(self.volatile_img)

I'm concerned that this code is executed as root, while the user
controls the path to self.volatile_img. It seems like this would
allow someone to remove any file on the system by tweaking the
destination file.

With this change and relocating /var/log, var/cache, /var/lib/xen, and
/etc/libvrt/libxl to the writable partition, I'm now able to boot Qubes
with a read-only / filesystem and mostly read-only template VMs.
libvrt re-writes the config files to /etc on every vm startup, which
seems a little odd, but I haven't tracked it down yet.

--
Trammell

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Aug 12, 2016 at 09:24:44AM -0600, Trammell Hudson wrote:
> On Fri, Aug 12, 2016 at 09:18:52AM +0200, Marek Marczykowski-Górecki wrote:
> > > > [...]
> > > > Also search the list archive for relocating volatile.img files - AFAIR
> > > > there was some script for that.
> >
> > Besides the above, for volatile.img it should be enough to modify this
> > script.
>
> It seems that script is insufficient for a read-only / filesystem
> since the prepare-volatile-img.sh script calls truncate on the file
> and qubes/storage/__init__.py calls os.remove() on the file.
>
> I modified the python code to check for a symlink and remove the
> destination of the link instead:
>
> # Re-create only for template based VMs
> if source_template is not None and self.volatile_img:
> if (os.path.islink(self.volatile_img)):
> if os.path.exists(self.volatile_img):
> os.remove(os.readlink(self.volatile_img))
> elif os.path.exists(self.volatile_img):
> os.remove(self.volatile_img)
>
> I'm concerned that this code is executed as root, while the user
> controls the path to self.volatile_img. It seems like this would
> allow someone to remove any file on the system by tweaking the
> destination file.

Have you read /etc/sudoers.d/qubes? ;)

> With this change and relocating /var/log, var/cache, /var/lib/xen, and
> /etc/libvrt/libxl to the writable partition, I'm now able to boot Qubes
> with a read-only / filesystem and mostly read-only template VMs.
> libvrt re-writes the config files to /etc on every vm startup, which
> seems a little odd, but I haven't tracked it down yet.

Yes, I think it needs a cache there, for persistence across daemon
restarts. It's perfectly fine to restart libvirtd while some domains are
still running.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXrjnVAAoJENuP0xzK19csocAH/33j/2+HIA/Y/G40xP+Tzh4g
8RkXF8f+bPVOWnPEvb3nBrwt/Tth8fILq4V8tH5EO/GDDOf10G8ZeLqLFEaPNOPc
dmFNJHa7pD9uKBGAcpRmFZTcrsdsDNWeKv0+GVL6c7ndofCWZcArry3p1R6b8wIl
g9fOhi+9SV+kqsdkSOXG5PycSPPnlMLE/QqYsJN32ppFrr7E929gjRKHx18owTSt
iw6YG0CWL4RVlZNzIZ8JPrvCiFrdkbDk3kr0J6s8dPGwI8yUfW0CKkQJGSPgXwF1
Lhgm9JWGWcTkLgnOJcZms0yBBzgxd+WtNP1aCf4zdaOLvUxmqGECrrGw/O3opAU=
=/EZ3
-----END PGP SIGNATURE-----