Corrupted LVM Pool Metadata - no free space (recoverable?)
joev...@gmail.com
Well, I broke my Qubes.
I was messing with a large vmdk converted to raw image, expanded to 100G on a Volume Group that could not handle it.
Some errors did pop up, but with the space filled up, I could not do anything.
After reboot, no vms would start.
After attempting to repair using lvm2 tools, will no longer boot up.
I don't know enough about thin provisioning. I may have made it worse by attempting to add another Physical Volume. The volume I added was a re-partitioned disk that had a previous Qubes 4 install on it... which also had the same volume group name of qubes_dom0. This was causing all sorts of trouble like causing the PV to be MISSING (which stops most commands from executing). While the VG included this new PV, I attempted to expand the Pool00 and its metadata. Expanding to a new PV had resolved the last time I ran out of space. But this time it seemed to make things much worse.
I think I am going to have to reinstall Qubes onto some new drives that I will buy and possibly RAID.
My question is, what is the easiest way to recover the qubes?
I do have some backups, but...
If I wanted to move LVs over to the new Qubes, is that as easy as changing the VG name, and moving the whole pool?
Should I mount each LV and attempt to import the root.img and private.img files? How would I mount those to get to the files, and how would import to Qubes?
Most LVs say, "Thin's thin-pool needs inspection."
I have tried repairing:
root@kali:~# lvconvert --repair qubes_dom0/pool00
Using default stripesize 64.00 KiB.
terminate called after throwing an instance of 'std::runtime_error'
what(): transaction_manager::new_block() couldn't allocate new block
Child 15634 exited abnormally
Repair of thin metadata volume of thin pool qubes_dom0/pool00 failed (status:-1). Manual repair required!
I could not fix this by adding another disk to the VG.
Here are the outputs you may be wanting:
root@kali:~# lvmdiskscan
/dev/nvme0n1 [ 232.89 GiB]
/dev/loop0 [ 2.45 GiB]
/dev/mapper/q1 [ 231.88 GiB] LVM physical volume
/dev/nvme0n1p1 [ 1.00 GiB]
/dev/sda1 [ 232.88 GiB] LVM physical volume
/dev/nvme0n1p2 [ 231.88 GiB]
/dev/sdb3 [ 3.64 TiB]
/dev/sdd1 [ 58.84 GiB]
/dev/sdd4 [ 667.00 MiB]
0 disks
7 partitions
0 LVM physical volume whole disks
2 LVM physical volumes
I got /dev/mapper/q1 after cryptsetup luksOpen /dev/nvme0n1p2 q1
root@kali:~# pvs
PV VG Fmt Attr PSize PFree
/dev/mapper/q1 qubes_dom0 lvm2 a-- 231.88g 15.80g
/dev/sda1 qubes_dom0 lvm2 a-- 232.88g 0
root@kali:~# vgs
VG #PV #LV #SN Attr VSize VFree
qubes_dom0 2 59 0 wz--n- 464.76g 15.80g
root@kali:~# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
pool00 qubes_dom0 twi-cotzM- 425.47g 90.48 99.95
root qubes_dom0 Vwi-a-tz-- 192.59g pool00 99.95
swap qubes_dom0 -wi-a----- 23.29g
...
Please let me know how much, if anything is recoverable.
Thanks.
awokd
> Should I mount each LV
> and attempt to import the root.img and private.img files? How would I
> mount those to get to the files, and how would import to Qubes?
them mounted you might be able to recover contents. Boot from some
recovery distribution and get the Qubes PV attached, then "mount <subdir>
/dev/mapper/qubes_dom0-vm--vmname--private" for example.
Chris Laprise
'vgchange -ay' then 'lvchange -ay' on the drive and then look for the
vm*private volumes in /dev/mapper.
Once you got that, you can do stuff like this to backup:
# dd if=/dev/mapper/qubes_dom0-vm--work--private | gzip >work_img.gz
and this to restore:
# gunzip -c work_img.gz | dd
of=/dev/mapper/qubes_dom0-vm--work--private conv=sparse
This example is unencrypted, so take care with the remaining backup details.
Also keep in mind that thin-provisioned lvm is still pretty young -- it
is NOT 'like' regular non-thin lvm, but more like a new filesystem on
top of regular lvm. If reliability is high on your list then Btrfs may
be better... it seems to be older with a lot more people using it, and
has more internal error-correction mechanisms (but it still isn't
recommended for RAID5/6). Btrfs is worth considering as an alternative.
--
Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886
joev...@gmail.com
> On 08/13/2018 06:48 AM, 'awokd' via qubes-users wrote:
> > On Mon, August 13, 2018 7:59 am,
> >
> >> and attempt to import the root.img and private.img files? How would I
> >> mount those to get to the files, and how would import to Qubes?
> >
> > Sounds pretty FUBARd. LVs don't have .img files; if you can manage to get
> > them mounted you might be able to recover contents. Boot from some
> > recovery distribution and get the Qubes PV attached, then "mount <subdir>
> > /dev/mapper/qubes_dom0-vm--vmname--private" for example.
>
> Thin LVs can be imaged like normal volumes, however. I would try
> 'vgchange -ay' then 'lvchange -ay' on the drive and then look for the
> vm*private volumes in /dev/mapper.
>
> Once you got that, you can do stuff like this to backup:
>
> # dd if=/dev/mapper/qubes_dom0-vm--work--private | gzip >work_img.gz
>
> and this to restore:
>
> # gunzip -c work_img.gz | dd
> of=/dev/mapper/qubes_dom0-vm--work--private conv=sparse
>
> This example is unencrypted, so take care with the remaining backup details.
>
> Also keep in mind that thin-provisioned lvm is still pretty young -- it
> is NOT 'like' regular non-thin lvm, but more like a new filesystem on
> top of regular lvm. If reliability is high on your list then Btrfs may
> be better... it seems to be older with a lot more people using it, and
> has more internal error-correction mechanisms (but it still isn't
> recommended for RAID5/6). Btrfs is worth considering as an alternative.
>
> --
>
> Chris Laprise
Thanks.
Is Btrfs an option when installing Qubes 4.0?
BTW,
Qubes fully recovered.
Like last time I messed things up by filling up the LVM,
Here are the steps:
Insert a bigger drive
create PV from new drive
added PV to VG
LVmove pool00_tdata from full drive to new PV
LVextended Pool00 to give room
LVextended pool00_tmeta (this was the corrupted part)
LVconvert --repair qubes_dom0/pool00
I did try all this yesterday, but the drive I put in, had a previous Qubes 4 install. Even with deleting the partitions, it seemed to still recognize the Volume Group information on it. And since it is also named qubes_dom0... it really messed things up and had me going down rabbit holes with a VG with a MISSING PV. No commands really work when in that stuck state.
The issue still remains if anything will be done with Thin Provisioning to prevent this from happening. Last time this happened, there was no disk storage applet. This time, there was, but I just didn't see what a vmdk file was going to be on disk.
Should VMs get automatically paused when approaching the limit?
Also, should dom0 be inside the thin pool? Or would it make sense to keep it as its own LV? Yeah, it means dom0 cannot grow with the pool, but should it?
joev...@gmail.com
If I installed Qubes and used LUKS encryption (I have to run cryptsetup openLuks just to see the LVM inside)... then I add physical drives to my Volume Group, and start adding more AppVMs to the pool, that starts writing to the PV...
Is the data on the new drive, encrypted?
Can anyone forensically pull data from those new AppVMs since it wasn't originally a part of the LUKS encrypted drive?
Chris Laprise
> Thanks.
>
> Is Btrfs an option when installing Qubes 4.0?
>
> BTW,
> Qubes fully recovered.
>
> Like last time I messed things up by filling up the LVM,
> Here are the steps:
>
> Insert a bigger drive
> create PV from new drive
> added PV to VG
> LVmove pool00_tdata from full drive to new PV
> LVextended Pool00 to give room
> LVextended pool00_tmeta (this was the corrupted part)
> LVconvert --repair qubes_dom0/pool00
> I did try all this yesterday, but the drive I put in, had a previous Qubes 4 install. Even with deleting the partitions, it seemed to still recognize the Volume Group information on it. And since it is also named qubes_dom0... it really messed things up and had me going down rabbit holes with a VG with a MISSING PV. No commands really work when in that stuck state.
>
> The issue still remains if anything will be done with Thin Provisioning to prevent this from happening. Last time this happened, there was no disk storage applet. This time, there was, but I just didn't see what a vmdk file was going to be on disk.
>
> Should VMs get automatically paused when approaching the limit?
computer to take immediate action when the space crisis occurs. But I
think the resolution will also involve moving dom0 root out of the thin
pool to a static-sized lv.
> Also, should dom0 be inside the thin pool? Or would it make sense to keep it as its own LV? Yeah, it means dom0 cannot grow with the pool, but should it?
>
Chris Laprise
encrypted. You have to add separate LUKS/dmcrypt block layers to those
new devices and then treat those dmcrypt block devices as the new pvs.
If you're doing this to qubes_dom0, then it could be a little tricky
getting all of the encrypted "pvs" to unlock at the same time during the
boot process. You'd need to investigate how crypttab and grub
accommodate that multi-volume setup.
joev...@gmail.com
> On 08/13/2018 04:47 PM,
> >
> > If I installed Qubes and used LUKS encryption (I have to run cryptsetup openLuks just to see the LVM inside)... then I add physical drives to my Volume Group, and start adding more AppVMs to the pool, that starts writing to the PV...
> > Is the data on the new drive, encrypted?
> > Can anyone forensically pull data from those new AppVMs since it wasn't originally a part of the LUKS encrypted drive?
>
> Based on the sparse description I'd say No, the new space is not
> encrypted. You have to add separate LUKS/dmcrypt block layers to those
> new devices and then treat those dmcrypt block devices as the new pvs.
>
> If you're doing this to qubes_dom0, then it could be a little tricky
> getting all of the encrypted "pvs" to unlock at the same time during the
> boot process. You'd need to investigate how crypttab and grub
> accommodate that multi-volume setup.
>
> --
>
> Chris Laprise
But it seems I have an LVM over LUKS configuration... when what I want is a LUKS over LVM.
Here's what I have:
[root@dom0]# lsblk | grep -v "\-\-"
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 3.7T 0 disk
└─sdb1 8:17 0 3.7T 0 part
├─qubes_dom0-pool00_tmeta 253:1 0 2.1G 0 lvm
│ └─qubes_dom0-pool00-tpool 253:3 0 1T 0 lvm
│ ├─qubes_dom0-pool00 253:6 0 1T 0 lvm
│ ├─qubes_dom0-root 253:4 0 192.6G 0 lvm /
├─qubes_dom0-pool00_meta0 253:63 0 2.1G 0 lvm
└─qubes_dom0-pool00_tdata 253:2 0 1T 0 lvm
└─qubes_dom0-pool00-tpool 253:3 0 1T 0 lvm
├─qubes_dom0-pool00 253:6 0 1T 0 lvm
├─qubes_dom0-root 253:4 0 192.6G 0 lvm /
sr0 11:0 1 1024M 0 rom
loop0 7:0 0 500M 0 loop
sda 8:0 0 232.9G 0 disk
└─sda1 8:1 0 232.9G 0 part
nvme0n1 259:0 0 232.9G 0 disk
├─nvme0n1p1 259:1 0 1G 0 part /boot
└─nvme0n1p2 259:2 0 231.9G 0 part
└─luks-bfcca13a-213d-46ec-b156-53df348dba30 253:0 0 231.9G 0 crypt
├─qubes_dom0-pool00_tdata 253:2 0 1T 0 lvm
│ └─qubes_dom0-pool00-tpool 253:3 0 1T 0 lvm
│ ├─qubes_dom0-pool00 253:6 0 1T 0 lvm
│ ├─qubes_dom0-root 253:4 0 192.6G 0 lvm /
└─qubes_dom0-swap 253:5 0 23.3G 0 lvm [SWAP]
Even better, I should look into a RAID setup too.
If I choose btrfs for my next install, I can avoid the LVM problems, but can I expand onto new physical volumes by adding more drives?
Chris Laprise
> On Monday, 13 August 2018 17:13:06 UTC-4, Chris Laprise wrote:
>> On 08/13/2018 04:47 PM,
>>> Related question.
>>>
>>> If I installed Qubes and used LUKS encryption (I have to run cryptsetup openLuks just to see the LVM inside)... then I add physical drives to my Volume Group, and start adding more AppVMs to the pool, that starts writing to the PV...
>>> Is the data on the new drive, encrypted?
>>> Can anyone forensically pull data from those new AppVMs since it wasn't originally a part of the LUKS encrypted drive?
>>
>> Based on the sparse description I'd say No, the new space is not
>> encrypted. You have to add separate LUKS/dmcrypt block layers to those
>> new devices and then treat those dmcrypt block devices as the new pvs.
>>
>> If you're doing this to qubes_dom0, then it could be a little tricky
>> getting all of the encrypted "pvs" to unlock at the same time during the
>> boot process. You'd need to investigate how crypttab and grub
>> accommodate that multi-volume setup.
>>
>> --
>>
>> Chris Laprise
>> PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886
>
>
> I would imagine it would require a longer grub entry with rd.luks attributes for other UUIDs.
keyfile or password then dracut may be able to setup the grub entries
automatically.
> But it seems I have an LVM over LUKS configuration... when what I want is a LUKS over LVM.
>
The problem is that Qubes 4.0 manipulates lvm directly so unless you opt
for Qubes managing your vms with Ext4 image files (not a nice way to
go), what you could end up with is:
lvm --> luks --> lvm --> device
And I don't think that could truly work under ideal conditions because
thats COW on top of COW. This is also the reason that Btrfs on lvm is
frowned upon... stacking this way is not reliable.
> If I choose btrfs for my next install, I can avoid the LVM problems, but can I expand onto new physical volumes by adding more drives?
be added and removed with ease, even when it reduces overall space. But
since encryption is still a separate luks layer you're still faced with
managing multiple luks volumes under Btrfs.
The only other possibility I can think of is to experiment with ecryptfs
on top of Btrfs. If you got it to work with Qubes it could span physical
volumes easily and the main (only?) downside would be increased
vulnerability to _physical_ (not remote) attacks.
joev...@gmail.com
Also, I may want to stick with LUKS instead of ecryptfs because yubikey support.
Can you elaborate on the steps for using Btrfs during install of Qubes, the documentation is still pretty old and doesn't really go over the non standard options.
Does Qubes 4.0 need LVM for snapshots and backups while vm is running? I haven't seen anything about why LVM thin provisioning is done on Qubes, or what I would lose if I went with something else like Btrfs or even Ext4.
Here's what I think my setup will have to be:
nvme0n1 (2 drives in hw RAID 0)
├─nvme0n1p1 part /boot
└─nvme0n1p2 part
└─luks (same key) crypt
├─qubes_dom0-pool00_tmeta lvm
├─qubes_dom0-pool00_tdata lvm
│ └─qubes_dom0-pool00-tpool lvm
│ ├─qubes_dom0-pool00 lvm
│ ├─qubes_dom0-root lvm /
│ └─ ... vm lvm
└─qubes_dom0-swap lvm [SWAP]
sda (2 drives in hw RAID 0)
└─sda1 part
└─luks (same key) crypt
└─qubes_dom0-pool00_tdata lvm
└─qubes_dom0-pool00-tpool lvm
├─qubes_dom0-pool00 lvm
├─qubes_dom0-root lvm /
└─ ... vm lvm
And if I need another drive added (sdb),... I just insert the drive, create another luks partition, unlock, add to Volume Group (qubes-dom0), lvextend pool00?
-----------------
What would Btrfs look like?
This?:
nvme0n1 (2 drives in hw RAID 0)
├─nvme0n1p1 part /boot
└─nvme0n1p2 part
└─luks (same key) crypt
├─qubes (btrfs) part /
└─qubes_dom0-swap part [SWAP]
sda (2 drives in hw RAID 0)
└─sda1 part
└─luks (same key) crypt
└─qubes (btrfs) part /
Under Btrfs, does qubes simply have each vm as a image file under the /var/lib/qubes/appvms/<appvm>/ directory? Do I lose the ability to backup live, or any other features? How does performance compare to LVM
And to add another drive (sdb), how would I extend with Btrfs?
Thanks Chris.