Custom installation: "You must create a new filesystem"

60 views
Skip to first unread message

Claudia

unread,
Aug 31, 2019, 12:25:04 PM8/31/19
to qubes-users
The "Custom Installation" doc gives instructions about how to create a
non-default dm-crypt partition, or other custom setup, and install Qubes
to it. But when I follow these instructions on R4.0.1, and try to assign
my dm-crypt device to "/", I get a message something like

"You must create a new filesystem for the root filesystem."

W.T.F... That's a major limitation for anyone trying to do anything even
slightly custom, such as using stronger dm-crypt parameters than the
defaults. I can sort of understand the rationale -- you don't want to
install on a dirty filesystem or whatever -- but still, I know if the
filesystem is empty or not, don't treat me like an idiot.

This message appears whether my dm-crypt partition is empty or has a
filesystem on it. It seems there's no way to tell the installer "okay,
create a new filesystem if you must, but create it on this specific
block device, not on a physical partition or LVM partition or whatever."
The installer insists on *replacing* the dm-crypt device with a filesystem.

I remember running into this before, and I thought I eventually got
around it somehow after playing with it for an hour or two. But I don't
remember. I might have just ended up using the default dm-crypt parameters.

Idea 1: LUKS allows you to change some, but not all, parameters after
installation. So you can change the iter-time, for example, but not the
key size, cipher, or hash size(?). Not great, but might work for some
situations.

Idea 2: In my case I want to use btrfs, so I'm thinking I can create a
small standard partition at the end of the disk, install to that, then
once installed, `btrfs device add` my custom dm-crypt root partition and
`btrfs device remove` the original, and optionally delete the temporary
partition and grow the real one. I don't yet know what changes will have
to be made to the bootloader/dracut config, but I'm assuming the UUID at
the very least.

Aside from dm-crypt and btrfs, there are also plenty of situations where
someone might want to install to an existing device or filesystem.

Some of this has been talked about in #2294, but it's not quite the same
thing.

So I guess this is mostly just a rant. But I was also wondering
1) Am I doing something wrong? Should I not be seeing this message? Is
it a bug?
2) Why isn't this addressed in the custom installation tutorial? Why do
we have a tutorial that cannot be followed?
3) Is there a way around it that doesn't involve the hacky
post-installation migration?
4) Does qubes provide any way to sidestep the graphical installer, i.e.
something akin to debootstrap or arch-bootstrap?

-------------------------------------------------
This free account was provided by VFEmail.net - report spam to ab...@vfemail.net

ONLY AT VFEmail! - Use our Metadata Mitigator to keep your email out of the NSA's hands!
$24.95 ONETIME Lifetime accounts with Privacy Features!
15GB disk! No bandwidth quotas!
Commercial and Bulk Mail Options!

Andrew David Wong

unread,
Aug 31, 2019, 9:00:55 PM8/31/19
to Claudia, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 31/08/2019 11.23 AM, Claudia wrote:
> The "Custom Installation" doc gives instructions about how to
> create a non-default dm-crypt partition, or other custom setup, and
> install Qubes to it. But when I follow these instructions on
> R4.0.1, and try to assign my dm-crypt device to "/", I get a
> message something like
>
> "You must create a new filesystem for the root filesystem."
>

That's odd. I don't remember getting a message like that when I
installed 4.0 this way.

> W.T.F... That's a major limitation for anyone trying to do anything
> even slightly custom, such as using stronger dm-crypt parameters
> than the defaults. I can sort of understand the rationale -- you
> don't want to install on a dirty filesystem or whatever -- but
> still, I know if the filesystem is empty or not, don't treat me
> like an idiot.
>
> This message appears whether my dm-crypt partition is empty or has
> a filesystem on it. It seems there's no way to tell the installer
> "okay, create a new filesystem if you must, but create it on this
> specific block device, not on a physical partition or LVM partition
> or whatever." The installer insists on *replacing* the dm-crypt
> device with a filesystem.
>

Well, the Qubes installer is mostly just the upstream Fedora
installer, so you might want to file bug reports with them about these
issues.

> I remember running into this before, and I thought I eventually
> got around it somehow after playing with it for an hour or two. But
> I don't remember. I might have just ended up using the default
> dm-crypt parameters.
>
> Idea 1: LUKS allows you to change some, but not all, parameters
> after installation. So you can change the iter-time, for example,
> but not the key size, cipher, or hash size(?). Not great, but might
> work for some situations.
>
> Idea 2: In my case I want to use btrfs, so I'm thinking I can
> create a small standard partition at the end of the disk, install
> to that, then once installed, `btrfs device add` my custom dm-crypt
> root partition and `btrfs device remove` the original, and
> optionally delete the temporary partition and grow the real one. I
> don't yet know what changes will have to be made to the
> bootloader/dracut config, but I'm assuming the UUID at the very
> least.
>
> Aside from dm-crypt and btrfs, there are also plenty of situations
> where someone might want to install to an existing device or
> filesystem.
>
> Some of this has been talked about in #2294, but it's not quite the
> same thing.
>
> So I guess this is mostly just a rant. But I was also wondering 1)
> Am I doing something wrong? Should I not be seeing this message?
> Is it a bug?

Could be. As mentioned above, I don't remember seeing this when I went
through it myself.

> 2) Why isn't this addressed in the custom installation tutorial?
> Why do we have a tutorial that cannot be followed?

I wrote this version of the tutorial because I couldn't find any
information about how to do this sort of thing on R4 anywhere. I went
through the trial-and-error and documented my findings for the benefit
of others (and my future self). But I'm certainly not an expert in all
the underlying technologies. It worked for me when I wrote it, and no
one else has contributed to it since then. I'm honestly sorry to hear
that it didn't work for you, but I don't know why it didn't. I can
simply remove it from the documentation if it's no longer working.

> 3) Is there a way around it that doesn't involve the hacky
> post-installation migration?

Not that I know of.

> 4) Does qubes provide any way to sidestep the graphical installer,
> i.e. something akin to debootstrap or arch-bootstrap?
>

Not that I know of. (Again, it's mostly the stock Fedora installer.)

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEZQ7rCYX0j3henGH1203TvDlQMDAFAl1rGCoACgkQ203TvDlQ
MDAHxA/9GF+5FeinrQIkejuunv56tbgFmbCPjjsqmWnvHFnJcj95EzfswrJ/xeqM
EAmJ8P/U9nHxbY00NApCJxYcCq2rpXdxpFHB6Ur4kgtOvfA6npAq8E8/HWC63vOj
+rJ6WqeOu1ivT5rLIdW+HSVAtEvujna36WNqTPj7t57ti2seTLE75oR8EIcwCfsN
EYp6c9BD1O9ZLqDuu8M52gPnEm6oHqSTzRHsytFGPVDND9b4qnHPS1UIIW4vOhM3
xy6Z8N12Dra7c6aP9FpgX/bHqiraeDRjohaXpA/fNx+Gj1T0vihGv5Qdpa/bq4H7
+USjM6lraCnPg2gaEBXBlRnml4HDXov6vA1a1SdzF1zVeqyTMOgIRFqnLHC9A5Sp
IjPq9Nyy1dYKHbIwQaI8TqAsF5KDQL5jfKv3ObjrpzCwjiXmi4XmEBhhJHdcjpys
lhnnkrwoxo0Z3yoSt3pZocTJv22E1GB7GKnXUxNQr7idHSAT7jh5QGCkjl66Q+35
1dKx+/QOreV5aR+cdV78p5jvLAud+UdNtttktyzlnB8qg+ZVQFhfdgMruLK1kClz
blyPdCLvHhfDRcbqVEvdIQup+pQyehFJsp3hFJ2D7J/t23GI9ksN3rdxo0V+21LF
J7lr14RC1zmvptsr0Q1ewh1kdqdX1VluTeNXa+3VI46m6lGloFM=
=FIqm
-----END PGP SIGNATURE-----

Claudia

unread,
Sep 1, 2019, 7:04:48 PM9/1/19
to Andrew David Wong, qubes-users
Andrew David Wong:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On 31/08/2019 11.23 AM, Claudia wrote:
>> The "Custom Installation" doc gives instructions about how to
>> create a non-default dm-crypt partition, or other custom setup, and
>> install Qubes to it. But when I follow these instructions on
>> R4.0.1, and try to assign my dm-crypt device to "/", I get a
>> message something like
>>
>> "You must create a new filesystem for the root filesystem."
>>
>
> That's odd. I don't remember getting a message like that when I
> installed 4.0 this way
First, thanks for your reply!

BTW, the actual message is "You must create a new file system on the
root device." (I was going from memory.)

Okay, so I think I might have figured it out: The tutorial should work
for any filesystem other than btrfs, provided you check the "Reformat"
option. Upon closer examination, your tutorial covers creation of
dm-crypt and LVM containers, but not any filesystems. The installer does
create the actual filesystem, so that's why the tutorial doesn't cause
the message about creating a new filesystem. It's just that btrfs isn't
one of the options.

When there is an empty dm-crypt partition on the disk, under "Unknown"
category it shows up as "luks-<uuid>" and asks for a password. Once
unlocked, all options are greyed out, including Mountpoint, except Label
and Reformat. When check Reformat, the File System drop down is enabled,
but btrfs is not an option. So at this point I could use another
filesystem, just not btrfs. The "Encrypt" checkbox is also enabled and
checked by default.

When I manually format that partition with btrfs, it shows up under
"Unknown" as "Encrypted (LUKS)" and asks for a password. Once unlocked,
it shows under "Unknown" as "btrfs" and all options are greyed out
except Mountpoint and Label. But when I enter "/" as mountpoint I get
that message. I would be fine with replacing the filesystem in the
container, but the "Reformat" box is unchecked and greyed out.

Like I said, I thought I got around it somehow, but I don't remember for
sure. I might have given up and used the default cryptsetup options.


>
> Well, the Qubes installer is mostly just the upstream Fedora
> installer, so you might want to file bug reports with them about these
> issues.

I was afraid of that. I may try to look into it some more and perhaps
see if it's a reportable bug. But the more I'm looking at it, I think
they would call it a "feature" of this deranged installer. (See below.)
I really just want to get past it.
Did you happen to do any testing with btrfs when you wrote the tutorial?
At this point I don't think the tutorial is faulty, I think it just
cannot be used with btrfs.

Like I said, though, bug #2294 talks about this very problem. So I'm at
least not imagining things. Although it doesn't mention the exact error
message (I could have sworn it did).

In #2294, under "General Notes" > "Not Workarounds:"
"If you also manually create a new btrfs filesystem inside the LUKS
container, the installer will unselect and gray out the Reformat check
box and then complain that reformatting the root mountpoint is required
to continue..."

That pretty much exactly describes what I'm running into.

The bug itself apparently was fixed in R4.0-rc4, but I'm assuming the
btrfs problem has not.


>> 3) Is there a way around it that doesn't involve the hacky
>> post-installation migration?
>
> Not that I know of.
>
>> 4) Does qubes provide any way to sidestep the graphical installer,
>> i.e. something akin to debootstrap or arch-bootstrap?
>>
>
> Not that I know of. (Again, it's mostly the stock Fedora installer.)
>
> - --


So I think the real problem here is that the installer doesn't treat
btrfs the same way as the rest of the filesystems. Btrfs is a "Device
Type" not a "File System". i.e. You can't put btrfs on an existing
device, you have to choose "Btrfs" for "Device Type". Thus, it can't be
installed to a preexisting dm-crypt container.

Note that when you select Btrfs for Device Type, Encrypt will be greyed
out and unchecked. You have to click Modify, select the device, and
check the Encrypt checkbox, and click save. Then the Encrypt checkbox
will be checked next to Device Type on the previous screen. However this
will be created using cryptsetup defaults!

There are several workaround ideas which after much testing and
reconfiguration eventually may or may not work. But right now I don't
have that kind of time, and I just want something to work. So for now I
think I'll just accept defeat: settle for cryptsetup defaults and just
let the installer do its thing. You can't always get what you want.

Thanks again for your input. If you get the chance to try the tutorial
using btrfs on a custom dm-crypt container, I'd be interested in your
results. You might have better luck, or perhaps a whole new tutorial.

Andrew David Wong

unread,
Sep 1, 2019, 11:50:42 PM9/1/19
to Claudia, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 01/09/2019 8.03 AM, Claudia wrote:
> Andrew David Wong:
I'm afraid not. I don't have any experience with btrfs.
An alternative is to follow the tutorial without using btrfs (which is
what I did). I suppose it's up to you to decide which of the two is
more important to you.

> Thanks again for your input. If you get the chance to try the
> tutorial using btrfs on a custom dm-crypt container, I'd be
> interested in your results. You might have better luck, or perhaps
> a whole new tutorial.
>

I was going to suggest the same to you, i.e., please consider
submitting a PR to update that page with your findings! :)

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEZQ7rCYX0j3henGH1203TvDlQMDAFAl1skXsACgkQ203TvDlQ
MDC0WQ//dTFwoQaQde3+spQVvh0X7tyPhAIdIO/ybr8rnJLCSy7SrOItS7ufqwyi
1GpMdlijFDNeqiZZsbs6BjTkFbcDnqr7GsBdKYfvlJIyS/PbJqO0BazzMDM5zGJ7
a4kmuLV+fh0ivygslOgoe9N0wGfSWQMURTi1Jn/G6X083KGKwGJmyK1c3De6FgRa
q2TWWIwiO55H36Anvlmzbr7za8MlJn8bzoRggaMSGnIEvQAGQUFkj7R1h7TTU1EB
dUufpskmnktseg+eU1tyXJd5gWP873jur6oei/kcjvwss1n0oJT/4iKg2a3dz7MH
f8fQMtMfYiYqX5gk6doMwn9+leN8VCCYN09CERIqk3nP+LktLvf79NGZ9t4WW7cB
Yj1sXHihaHPEJpWdR/9a1FVmlQWmm1HbwtVMuiFDIdluDa+/a7BLUrqugD94U3d1
/xhCXncqkrxI8DIG4Xl2FPV/4R4F9t4DanPWidOndmTsJZa9qawhM2ByLAVfVaFZ
Fub60jAUGvVy3Iz6UQpe/Mq7SZwcCZuAFiy7l7U8RF+feUrVTp4ZlY6tVJgx0OOj
lgRjYbB9pyochmhL9LE2BFEHadZUg6m8xNuw/hXuyo02V8F/O9hal1Ne9iVKpzt4
QfUPrNIery09vTZZGV293QWpg19aQ7lSM0rYRS6bzQg+O7bO3lY=
=P/Kk
-----END PGP SIGNATURE-----

Claudia

unread,
Sep 2, 2019, 8:35:27 AM9/2/19
to Andrew David Wong, qubes-users
Andrew David Wong:
Precisely.

I generally prefer btrfs (due to compression, self-healing, subvolumes,
and so on), but I don't know if it has any advantages with regards to
Qubes specifically.

The conventional wisdom within the Xen community is that LVM volumes are
more efficient than any filesystem, although I recall seeing an analysis
that actually found the difference to be insignificant on modern
systems. I try to avoid LVM when I can, just because I've had problems
with it in the past and it adds another layer of complexity.

Likewise, with cryptsetup, a lot of people say the default crypto is too
weak, though the developers are confident that the defaults are as
strong as the maximums, in practice. I really don't know.

So no strong arguments either way, as far as I know.

>> Thanks again for your input. If you get the chance to try the
>> tutorial using btrfs on a custom dm-crypt container, I'd be
>> interested in your results. You might have better luck, or perhaps
>> a whole new tutorial.
>>
>
> I was going to suggest the same to you, i.e., please consider
> submitting a PR to update that page with your findings! :)
Yes, I'm hoping to, in the near future, after I get my main system
installed and running. I just can't dedicate my main machine to it right
now. I was actually wondering if you have any advice on how to go about
testing this kind of thing in a VM?

I don't suppose it's possible to install Qubes in a Qubes VM? Because
Xen won't run under Xen :-( Can it run in a sort of "pseudo" mode,
where Xen is actually disabled, just for the purposes of doing the
installation and testing the boot config?

It's a bit inconvenient to test it on bare metal, until I can come up
with an unused machine to dedicate to that purpose for some time.

Perhaps it would be possible to play around with the vanilla Fedora
installer under a Qubes VM, and then do a final test with Qubes on bare
metal. I guess this could be accomplished by creating a Qubes HVM with a
cdrom, as in [1]?

[1]
https://www.qubes-os.org/doc/standalone-and-hvm/#installing-an-os-in-an-hvm

>
> - --
> Andrew David Wong (Axon)
> Community Manager, Qubes OS
> https://www.qubes-os.org
>
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCgAdFiEEZQ7rCYX0j3henGH1203TvDlQMDAFAl1skXsACgkQ203TvDlQ
> MDC0WQ//dTFwoQaQde3+spQVvh0X7tyPhAIdIO/ybr8rnJLCSy7SrOItS7ufqwyi
> 1GpMdlijFDNeqiZZsbs6BjTkFbcDnqr7GsBdKYfvlJIyS/PbJqO0BazzMDM5zGJ7
> a4kmuLV+fh0ivygslOgoe9N0wGfSWQMURTi1Jn/G6X083KGKwGJmyK1c3De6FgRa
> q2TWWIwiO55H36Anvlmzbr7za8MlJn8bzoRggaMSGnIEvQAGQUFkj7R1h7TTU1EB
> dUufpskmnktseg+eU1tyXJd5gWP873jur6oei/kcjvwss1n0oJT/4iKg2a3dz7MH
> f8fQMtMfYiYqX5gk6doMwn9+leN8VCCYN09CERIqk3nP+LktLvf79NGZ9t4WW7cB
> Yj1sXHihaHPEJpWdR/9a1FVmlQWmm1HbwtVMuiFDIdluDa+/a7BLUrqugD94U3d1
> /xhCXncqkrxI8DIG4Xl2FPV/4R4F9t4DanPWidOndmTsJZa9qawhM2ByLAVfVaFZ
> Fub60jAUGvVy3Iz6UQpe/Mq7SZwcCZuAFiy7l7U8RF+feUrVTp4ZlY6tVJgx0OOj
> lgRjYbB9pyochmhL9LE2BFEHadZUg6m8xNuw/hXuyo02V8F/O9hal1Ne9iVKpzt4
> QfUPrNIery09vTZZGV293QWpg19aQ7lSM0rYRS6bzQg+O7bO3lY=
> =P/Kk
> -----END PGP SIGNATURE-----
>


Andrew David Wong

unread,
Sep 2, 2019, 12:19:56 PM9/2/19
to Claudia, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 01/09/2019 12.51 PM, Claudia wrote:
> I don't suppose it's possible to install Qubes in a Qubes VM?

Yes, Qubes OS can be installed inside of itself, with certain
limitations. (IIRC: must be an HVM, must contain only HVMs, and only
nested one level deep. But it's been a long time since I've done this.)

- --
Andrew David Wong (Axon)
Community Manager, Qubes OS
https://www.qubes-os.org

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEZQ7rCYX0j3henGH1203TvDlQMDAFAl1tQQYACgkQ203TvDlQ
MDD7dRAAhG/7LS4k3YFaMaBBF2nTXslPakIiJfE/QeKotiuOZl76gLe+hcQ1xl7l
M5ck4fKp/5CK+EsDoJqe6IK9qkpBwNNyzZtFnY4nJ8q5iv44B2/SPFsQsZzSRZoP
CxO+OrNaVVxUcReczmwbg/Q+2QTKTSWhVQSggChnzFvGDf+FhNMa6ZSE4Vwuj78/
TwvDQnqy8RnURZQ29MTh+vKgICw/H6RQx3dWufLTyuquxnoWizGmYKABSv2dn4Lo
ShMrA/iXWq0GNBVcIYGyaQySGAik270ZgDC97ydCkTN1JHHUpBMMTXP1AQeed7jf
ndDd1culelLs2ISeG3a6Ub1EdLLP2zRMxJ0JVLZAlyUBiiPXFtTQL94lVQ62DKYB
jT/M8yODj7g4EVkhbk8ya1rrFRIw1L6mFsx7a2t/6fQDvWjclCChN2vDNWco3OvB
nXfoSmcj56LBX3+eKHVioc6JnG/93WKGXobqmU2Y6WHIQ8+muRW/lZzaDH9Lm39y
9v8CVkrBUjjH+8mgtkk642pE4q5IWXFD8KJL5Bm6741WhkQ9Hhmq5aIBrzUvUOZV
sZNaiVD6GDTNoUfV+Y7Pmx52Fy2Niq+4YQpRJD9B5zoiOdr9DvFTUit0DQE+ztcP
w30T1sFIpNfaRGMmy41WTiaFEIJE3frfElYMF4RGbG4gFGT+174=
=1f+A
-----END PGP SIGNATURE-----

Reply all
Reply to author
Forward
0 new messages