Booting Cubes, Migration

[johny...@sigaint.org]

I've finally got Qubes set up in a way I'm comfortable working every day.

Now I wanted to move that same installation to another drive for its
permanent home.

The current drive has a standard bios /boot partition (sda1), and an
encrypted extended partition (#5) containing lvm with swap and /.

The destination drive has a few existing partitions on it, #2 - Extended
partition containing #5/#6/#7 which are a few encrypted partitions
containing some data I'd like to keep.

The destination drive has a 500M empty space at the start ready for
partition #1 (/boot) and 550G at the end, ready for partition #8, a future
encrypted partition, which will contain a physical volume/volume group
containing the logical volumes for swap and /.

1) First attempt to migrate, I created the new /boot partition by hand,
rsync'd from the old. Similarly, I created the new encrypted partition,
added the pv,vg,lvm's for swap/root, and rsync'd all the data from the old
root to the new root.

I tweaked up the UUID's in /boot/grub2/grub.cfg, /etc/crypttab,
/etc/fstab, and (I think) in the initrd.

However, that failed to boot. It got as far as dropping into the dracut
shell, and when I manually set up the qubes_dom0/root to be /dev/root and
exited, systemd went a little ways, but then went into some loop where it
kept saying target paths were ready, but did nothing further. Argh.

2) Second attempt was a raw DD of both /boot and the whole encrypted lvm
partition. (The destination partitions, luckily, were bigger.)

I then unplugged the original drive (since there'd be conflicting UUID's
now), rebooted, and same deal... The boot wouldn't mount the crypted
drive, and went into some loop.

Third attempt, I figured I'd do it the more official way, and do a fresh
install of Qubes, and then restore a backup from my old setup.

But with 3.2rc3, I couldn't get the installer to do what I wanted.
Occasionally I'd get a weird pop-up error about "e not being defined yet"
or something like that. :S

I tried and tried to coax it into creating a biosboot partition in
/dev/sda1, and making a /dev/sda8 as an encrypted, lvm-containing
partition, to no avail.

The closest I came, when it described the steps it was about to take (nice
feature!), it said it was going to create a partition in /dev/sda8, put an
LVM in that, and then create two Luks partitions, one for swap and one for
root.

That's not what I had before, and not what I wanted. I wanted the
encryption outside the LVM, which I thought was the standard way of doing
things.

If I choose "automatically partition the drive", will it preserve my
existing #5/#6/#7 extended partitions and just add new partitions as
needed? I don't want to nuke everything by mistake.

Any suggestions?

Also, on my first attempt to set it up, for the heck of it, I made the
/boot partition encrypted.
http://www.pavelkogan.com/2014/05/23/luks-full-disk-encryption/

That part seemed to work fine; asked for passphrase right away, decrypted
/boot, loaded xen/vmlinux/initrd and started things rolling. But the same
systemd failure happened as described above. So I went back to
non-encrypted /boot. I do think encrypted /boot might be a good default
or optional feature for Qubes. Why leave more attack surface than
necessary, and the /boot partition is a pretty big glaring one.

(Why would an attacker care if root is encrypted, or bother with it, if
they have free reign to tamper with an un-encrypted /boot with all it's
kernel-ey and initrd goodness. It's like locking your front door, but
leaving your basement door wide open.)

Thanks.

JJ

[johny...@sigaint.org]

> I've finally got Qubes set up in a way I'm comfortable working every day.
>
> Now I wanted to move that same installation to another drive for its
> permanent home.

Oh, I also meant to ask this:

Does all of the Template/VM state live in /var/lib/qubes? Obviously the
machines' disks do, and it appears that the metadata associated with them
lives in /var/lib/qubes/qubes.xml (not human readable, tho').

Is this correct? If I do a fresh install of Qubes, and (with no VM's
running) bring over my whole /var/lib/qubes directory from another
installation, reboot, would I be good to go? Or are there other subtle
things in dom0's root drive that would be out of kilter?

(It'd be nice to see the VM's and their state not being tied to anything
beyond /var/lib/qubes. Seems a lot cleaner that way.)

Thanks.

JJ

[Chris Laprise]

On 09/19/2016 04:25 PM, johny...@sigaint.org wrote:
> Any suggestions?

Anaconda is notorious for messing up specific requests for volume
layout. You would stand a much better chance of getting help in a fedora
or redhat forum... they have many more people experienced with this.

As for copying all of /var/lib/qubes, I'm not sure its a 100% clean way
to do it.... transferring old qubes.xml directly would be a concern (but
I may be wrong). Qubes does make it possible to copy a vm's folder into
/vm-templates or /appvms, then run qvm-add-template or qvm-add-appvm to
add vms one-by-one into the current system.

Chris

[johny...@sigaint.org]

> Anaconda is notorious for messing up specific requests for volume
> layout. You would stand a much better chance of getting help in a fedora
> or redhat forum... they have many more people experienced with this.

Cool, thanks. I guess it is a more general grub/luks/lvm issue, and not
necessarily Qubes-specific.

> As for copying all of /var/lib/qubes, I'm not sure its a 100% clean way
> to do it.... transferring old qubes.xml directly would be a concern (but
> I may be wrong). Qubes does make it possible to copy a vm's folder into
> /vm-templates or /appvms, then run qvm-add-template or qvm-add-appvm to
> add vms one-by-one into the current system.

Interesting. In the past, I have had success in moving a single VM by
creating a new VM with the same settings as the old one, then replacing
the files in the vm's directory. (Sort of the reverse order of what you
mentioned.)

Now, if I can only get the booting working . . . :)

JJ

[Andrew David Wong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 2016-09-19 13:36, johny...@sigaint.org wrote:
>> I've finally got Qubes set up in a way I'm comfortable working every day.
>>
>> Now I wanted to move that same installation to another drive for its
>> permanent home.
>
> Oh, I also meant to ask this:
>
> Does all of the Template/VM state live in /var/lib/qubes? Obviously the
> machines' disks do, and it appears that the metadata associated with them
> lives in /var/lib/qubes/qubes.xml (not human readable, tho').
>
> Is this correct? If I do a fresh install of Qubes, and (with no VM's
> running) bring over my whole /var/lib/qubes directory from another
> installation, reboot, would I be good to go? Or are there other subtle
> things in dom0's root drive that would be out of kilter?
>

I'm not sure whether this would work (never tried it). If possible, I'd
recommend using the built-in backup/restore system instead. This is the
recommended way to migrate to a fresh installation.

https://www.qubes-os.org/doc/backup-restore/

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJX4Sv/AAoJENtN07w5UDAwf1MQALeO+ai7E2z0NiJJTPKa+NwB
68J3DWKf/wss00VeVaiPNqBtyaAgwNnXcDx2Ay87gt6UIVntQC3GYoFYjWtnYM5b
6tTCPIotZ8U1ma4FpYUMz9v944Qqs7nviV45QRGgkzCrnlk21Gjz2yA2hzbT/Slw
Fv8RYxCaHdi+3FIpknxQB/bAtnM353fVRxBHADCY1XSbhUKqoEyhy/Z1G7ZlVkvE
qKoTimTBLFp565z0OIDMLA/kFvJBav51g2Nfsz30szlzx2IxEL970E4PcyXeTFXu
6CzuSIWHXPD7d9BDHKKFal024IvZMDBHRZ7ZWt/xOSNAeJM4nKwqBa5pg7OwKEYT
gwJJmsCNHccxbRAHtSw2ifNrX2iTvub4qNZEb8Ll4jWOLU6nVhJxG8cnFdcGe8SQ
UKhWlGdqfdI4zELuXHxvjtkCdMvHC1ACwUdHggDt/uw01t3XX/OlCEsSVUQ48iQa
rtZ3B8UAmxiX65e81UAGQtxTFmSUMwuO1Fma9bpOLjjXgFp8Be3SfdjTHsb4qvd0
oHFbmpt2qxVzO4pIWQynX4s8zVoxBdtS3Qg4sf7/Pv/xR05EYQANBXLTe+OavG3B
zVuQdIX9PgtmAHljIJRPjyrHU8CCvBkHaSuvLb3o8yHwhS2IQY2vkd1DsNACTD60
ctKniI1LTcqlWOYdfaTI
=KR2L
-----END PGP SIGNATURE-----

[johny...@sigaint.org]

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On 2016-09-19 13:36, johny...@sigaint.org wrote:
>>> I've finally got Qubes set up in a way I'm comfortable working every
>>> day.
>>>
>>> Now I wanted to move that same installation to another drive for its
>>> permanent home.
>>
>> Oh, I also meant to ask this:
>>
>> Does all of the Template/VM state live in /var/lib/qubes? Obviously the
>> machines' disks do, and it appears that the metadata associated with
>> them
>> lives in /var/lib/qubes/qubes.xml (not human readable, tho').
>>
>> Is this correct? If I do a fresh install of Qubes, and (with no VM's
>> running) bring over my whole /var/lib/qubes directory from another
>> installation, reboot, would I be good to go? Or are there other subtle
>> things in dom0's root drive that would be out of kilter?
>>
>
> I'm not sure whether this would work (never tried it). If possible, I'd
> recommend using the built-in backup/restore system instead. This is the
> recommended way to migrate to a fresh installation.
>
> https://www.qubes-os.org/doc/backup-restore/

Understood. I just find the backup/restore painfully slow as compared to
a quick (and incremental, as I work) rsync.

And I did try to do the backup/restore thing, but my destination drive had
some partitions on it I wanted to preserve (as mentioned), and I just
couldn't get the layout I wanted (partition->luks->lvm->swap/root, plus a
sda1 bios boot).

I guess I'll move those three existing encrypted partitions (1.5T, ugh) to
another drive (external USB, blah), do a fresh stock install of Qubes,
then copy those partitions back. See ya in a week or two. :P

I'll probably toy around with (and study up on)
grub/dracut/luks/crypttab/initrd a bit more first, though, since I'm so
close. A little more debugging might do the trick. I have a perfect
image of the old encrypted swap/root and bios boot partitions on the new
drive. So it's just a matter of the right grub/dracut magic. So close,
to avoid a couple of days of copying.

If I do find the magic tweak to make systemd roll along further, I will
follow up here.

JJ