[qubes-devel] [GSoc] Template Manager PoC

[WillyPillow]

On Monday, June 15, 2020 10:51 AM, Marek Marczykowski-Górecki <marm...@invisiblethingslab.com> wrote:

> On Tue, Jun 09, 2020 at 12:35:48PM +0000, WillyPillow wrote:
>

> > On Sunday, June 7, 2020 11:39 PM, Marek Marczykowski-Górecki marm...@invisiblethingslab.com wrote:
> >

> > > [1] https://dev.qubes-os.org/projects/core-admin/en/latest/qubes-features.html#qvm-features-request-qubes-postinstall-service
> > > [2] https://dev.qubes-os.org/projects/core-admin-client/en/latest/manpages/qvm-features.html
> > >

> >

> > Having looked through them, the following features seem useful.
> >

> > - `no-monitor-layout`
> >

> > - `net.fake-ip`
> >

> > - `net.fake-gateway`
> >

> > - `net.fake-netmask`
> > - Hide info from the template
> > - `video-model`
> >

> > - `pci-e820-host`
> >

> > - `linux-stubdom`
> > - Compatibility-related?
>

> Yes, especially video-model. I don't see any specific situation where
> other two would be useful, but they exists specifically for cases I
> can't think of ;)
>

> > Other features:
> >

> > - `gui`
> > - `gui-emulated`
> > - `qrexec`
> > - These seem to be set in `CoreFeatures`, and the defaults also seem to make
> > sense for VMs not supporting the discovery mechanism.
> >

>

> Almost. MirageOS-based unikernel VMs do support qrexec (and in some
> cases even gui!), but currently do not support discovery mechanism.

TIL! So perhaps they should be placed in the metadata as well?

Also, currently it seems that the flags are all related to VM configuration.
As such, processing it in `qvm-template-postprocess`, after importing the image and before starting the VM, seems to make sense.

> > - `appmenus-legacy`
> > - Not sure if this is useful, especially when qubes.StartApp seems to be
> > around for a pretty long time.
> >

>

> I agree. This exists mostly for compatibility with VMs imported from
> Qubes 3.2 (using backup&restore). Definitely not needed for new templates.
>

> > > > Also, I noticed that in `qvm_template_postprocess.py`
> > > > (https://github.com/QubesOS/qubes-core-admin-client/blob/e700af9eb2c1b39b065c476b2502a1f6a94d054e/qubesadmin/tools/qvm_template_postprocess.py#L145),
> > > > it is mentioned that the call to `qvm-appmenus` (which, from what I've gathered,
> > > > only works in dom0) will be switched to qrexec calls in the future. Is there
> > > > currently a qrexec call for this?
> >

> > > It does work in non-dom0 too (if installed). This is what's happening in
> > > GUI domain. But still it operates on local files, so running it outside
> > > of GUI domain (whether it's dom0 or something else), doesn't really
> > > makes sense.
> > > Yes, we'll need to come up with some qrexec calls for that.
> >

> > > BTW, I'd store default appmenus list into qvm-features or similar place
> > >

> > > - to have access to those files outside of dom0 too.
> >

> > Perhaps this also makes the qrexec call more simple, since the information does
> > not need to be explicitly passed?
>

> Are you referring to not-yet-existing qrexec calls for handling menu
> entires? Yes, that's why I'm proposing qvm-features for this. But you
> need to extract it from the rpm package anyway.

Yes, indeed.

So the call only needs to accept one argument indicating the VM to be worked on, correct?

>

> > Anyhow, if the package format has been established to a certain extent, maybe I
> > can start building a PoC that, say, imports a image from an existing RPM?
>

> Yes :)

A PoC using the original file format can be found at
<https://gist.github.com/WillyPillow/61ee5f48b7c5b7cc90c9fd2ec5c1b20d>.

The script is tested with `qubes-template-fedora-31-minimal` from the official repos.

(I have yet to modify `qvm-template-postprocess` to handle, e.g., tarballs that are not split.)

Schedule-wise, should I modify `qvm-template-postprocess` to handle the new file format and `qvm-feature` flags next (and perhaps implement a qrexec call for `qvm-appmenus`), to try to finish the RPM-installation part?

As a side note, I *might* be a bit busy for the next week or so, due to having to help migrate the server closet of the school department to another building (due to construction work).
Apologies for not putting this out earlier. The event was planned for late-May to early-June, but was ultimately postponed until now.
Anyways, I will try not to let it interfere with my progress.

Thanks,
WillyPillow

> https://blog.nerde.pw/
>
> PGP fingerprint = 6CCF 3FC7 32AC 9D83 D154 217F 1C16 C70E E7C3 1C84
>
> Protonmail PGP = D02D CEFF ACE5 5A7B FF5D 871E 4004 1CB1 F52B 127E

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Yes, I think so.

> Also, currently it seems that the flags are all related to VM configuration.
> As such, processing it in `qvm-template-postprocess`, after importing the image and before starting the VM, seems to make sense.

I agree.

> > > - `appmenus-legacy`
> > > - Not sure if this is useful, especially when qubes.StartApp seems to be
> > > around for a pretty long time.
> > >
>
> >
>
> > I agree. This exists mostly for compatibility with VMs imported from
> > Qubes 3.2 (using backup&restore). Definitely not needed for new templates.
> >
>
> > > > > Also, I noticed that in `qvm_template_postprocess.py`
> > > > > (https://github.com/QubesOS/qubes-core-admin-client/blob/e700af9eb2c1b39b065c476b2502a1f6a94d054e/qubesadmin/tools/qvm_template_postprocess.py#L145),
> > > > > it is mentioned that the call to `qvm-appmenus` (which, from what I've gathered,
> > > > > only works in dom0) will be switched to qrexec calls in the future. Is there
> > > > > currently a qrexec call for this?
> > >
>
> > > > It does work in non-dom0 too (if installed). This is what's happening in
> > > > GUI domain. But still it operates on local files, so running it outside
> > > > of GUI domain (whether it's dom0 or something else), doesn't really
> > > > makes sense.
> > > > Yes, we'll need to come up with some qrexec calls for that.
> > >
>
> > > > BTW, I'd store default appmenus list into qvm-features or similar place
> > > >
>
> > > > - to have access to those files outside of dom0 too.
> > >
>
> > > Perhaps this also makes the qrexec call more simple, since the information does
> > > not need to be explicitly passed?
> >
>
> > Are you referring to not-yet-existing qrexec calls for handling menu
> > entires? Yes, that's why I'm proposing qvm-features for this. But you
> > need to extract it from the rpm package anyway.
>
> Yes, indeed.
>
> So the call only needs to accept one argument indicating the VM to be worked on, correct?

Yes, I think that should be enough.

I'm not yet sure about interactions with qubes.SyncAppmenus call (used
to extract actually available applications). This is normally done when
the template is initially started during installation (and also later
after installing updates). Before starting the template (specifically
before making qubes.PostInstall call to it), you don't have desktop
files templates, so menu entries cannot be created yet. This probably
means various menu related calls will need to be after qubes.PostInstall
call.

> > > Anyhow, if the package format has been established to a certain extent, maybe I
> > > can start building a PoC that, say, imports a image from an existing RPM?
> >
>
> > Yes :)
>
> A PoC using the original file format can be found at
> <https://gist.github.com/WillyPillow/61ee5f48b7c5b7cc90c9fd2ec5c1b20d>.
>
> The script is tested with `qubes-template-fedora-31-minimal` from the official repos.

extract_rpm is not called from anywhere?
Be careful about TemporaryDirectory - by default it puts it in /tmp,
which is RAM-based tmpfs - may be too small for the whole template.
/var/tmp should be better (AFAIR dir= argument).

Also, look specifically for a requested template name, not just anything
you find in the package. Currently "for name in os.listdir(target +
PATH_PREFIX)" will happily accept whatever template is there (possibly
also multiple templates in a single rpm).

> (I have yet to modify `qvm-template-postprocess` to handle, e.g., tarballs that are not split.)
>
> Schedule-wise, should I modify `qvm-template-postprocess` to handle the new file format and `qvm-feature` flags next (and perhaps implement a qrexec call for `qvm-appmenus`), to try to finish the RPM-installation part?

Yes, that sounds like a good plan. Should allow you to have testable
state most of the time.

> As a side note, I *might* be a bit busy for the next week or so, due to having to help migrate the server closet of the school department to another building (due to construction work).
> Apologies for not putting this out earlier. The event was planned for late-May to early-June, but was ultimately postponed until now.
> Anyways, I will try not to let it interfere with my progress.
>
> Thanks,
> WillyPillow
>
> > https://blog.nerde.pw/
> >
> > PGP fingerprint = 6CCF 3FC7 32AC 9D83 D154 217F 1C16 C70E E7C3 1C84
> >
> > Protonmail PGP = D02D CEFF ACE5 5A7B FF5D 871E 4004 1CB1 F52B 127E

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl7qPNwACgkQ24/THMrX
1yxNdAf+Mcw0Hu+/d4IVccLXoTlGG6yP+/fSkpMeAmYBeHmvdze0UvvVhJdH0+ji
C0d7CmyTwItRzkKEGlpWXHoSUtQayf1j2IE14q6ec6ofh8ApG+oX6zk4MiVZSiaO
V9aOoPsx/b9s7HGuomYym7FRuNunjEHuaqKR4xj0hbntzkEzgv16K04EQM+T771f
tAT39JxaDBBoXRfyr5b1GwupKf1C2kgQZHWk58mCVZnIv6TGgp8KtCrN8BOqKQ48
zpYr3DvdamudcXmmQ2pLxHiWrS4Qu9orvhtNFmyi0TwUf0UodsSunq0oZVsn959k
+pHBzwh0jKPnr7elSo/RUzrEL0AbYA==
=8j0K
-----END PGP SIGNATURE-----

[WillyPillow]

As far as I can tell from reading the source code, the main things that
`qvm-appmenus` do are i) update the whitelists (which is the part that needs to
be replaced in `qvm-template-postprocess`) and ii) update the appmenus
according to the whitelist and desktop file templates. The latter is
(indirectly) invoked by qubes.SyncAppmenus after updating the desktop file
templates. As such, the new qrexec call, to my understanding, only has to
implement the former functionality.

However, as qubes.SyncAppmenus updates the appmenus, my understanding is that
the orders in which they are called do not matter?

By the way, upon inspecting the code more closely, `qvm-appmenus` merely writes
the whitelist to a file and reads from it later. As we are storing the list in
qvm-features anyway, is it still necessary to have the qrexec call?

> > > > Anyhow, if the package format has been established to a certain extent, maybe I
> > > > can start building a PoC that, say, imports a image from an existing RPM?
> >

> > > Yes :)
> >

> > A PoC using the original file format can be found at
> > https://gist.github.com/WillyPillow/61ee5f48b7c5b7cc90c9fd2ec5c1b20d.
> > The script is tested with `qubes-template-fedora-31-minimal` from the official repos.
>

> extract_rpm is not called from anywhere?

That's... very embarrassing :(

I may have accidentally removed it when cleaning up the comments.

> Be careful about TemporaryDirectory - by default it puts it in /tmp,
> which is RAM-based tmpfs - may be too small for the whole template.
> /var/tmp should be better (AFAIR dir= argument).

I see. Did not realize that /tmp is a tmpfs.

An updated version with the issues above fixed (and some other slight changes)
has been posted.

> Also, look specifically for a requested template name, not just anything
> you find in the package. Currently "for name in os.listdir(target +
> PATH_PREFIX)" will happily accept whatever template is there (possibly
> also multiple templates in a single rpm).

The reason I did it this way was because i) there doesn't seem to be a clean
way of extracting the template name from the package, and ii) it might
conceivably be a feature useful in cases like installing whonix-gw and
whonix-ws at the same time.

That being said, the idea of looking for a requested template name makes a lot
of sense as a way of dealing with the first problem. I am unsure what the user
interface should look like when installing downloaded RPMs though.

(Thinking about it, being able to install multiple templates this way would
probably create issues when implementing force-reinstall operations later on,
so sticking to one template in one RPM is probably a better idea after all.)

> > (I have yet to modify `qvm-template-postprocess` to handle, e.g., tarballs that are not split.)
> > Schedule-wise, should I modify `qvm-template-postprocess` to handle the new file format and `qvm-feature` flags next (and perhaps implement a qrexec call for `qvm-appmenus`), to try to finish the RPM-installation part?
>

> Yes, that sounds like a good plan. Should allow you to have testable
> state most of the time.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sat, Jun 20, 2020 at 04:47:49PM +0000, WillyPillow wrote:
> On Wednesday, June 17, 2020 11:55 PM, Marek Marczykowski-Górecki <marm...@invisiblethingslab.com> wrote:
> > I'm not yet sure about interactions with qubes.SyncAppmenus call (used
> > to extract actually available applications). This is normally done when
> > the template is initially started during installation (and also later
> > after installing updates). Before starting the template (specifically
> > before making qubes.PostInstall call to it), you don't have desktop
> > files templates, so menu entries cannot be created yet. This probably
> > means various menu related calls will need to be after qubes.PostInstall
> > call.
>
> As far as I can tell from reading the source code, the main things that
> `qvm-appmenus` do are i) update the whitelists (which is the part that needs to
> be replaced in `qvm-template-postprocess`) and ii) update the appmenus
> according to the whitelist and desktop file templates. The latter is
> (indirectly) invoked by qubes.SyncAppmenus after updating the desktop file
> templates. As such, the new qrexec call, to my understanding, only has to
> implement the former functionality.
>
> However, as qubes.SyncAppmenus updates the appmenus, my understanding is that
> the orders in which they are called do not matter?
>
> By the way, upon inspecting the code more closely, `qvm-appmenus` merely writes
> the whitelist to a file and reads from it later. As we are storing the list in
> qvm-features anyway, is it still necessary to have the qrexec call?

If we rely on the side effect of qubes.SyncAppmenus to not only extract
available desktop files, but also (re)generate actual menu, then indeed
for template installation no extra qrexec services are necessary.

But in general, qrexec services for manipulating menu will be needed for
all kind of other cases, like creating/removing VMs, updating menu after
setting what applications are included, after changing template, after
changing guivm etc. Sounds like a task unrelated to template manager.

> > > > > Anyhow, if the package format has been established to a certain extent, maybe I
> > > > > can start building a PoC that, say, imports a image from an existing RPM?
> > >
>
> > > > Yes :)
> > >
>
> > > A PoC using the original file format can be found at
> > > https://gist.github.com/WillyPillow/61ee5f48b7c5b7cc90c9fd2ec5c1b20d.
> > > The script is tested with `qubes-template-fedora-31-minimal` from the official repos.
> >
>
> > extract_rpm is not called from anywhere?
>
> That's... very embarrassing :(
>
> I may have accidentally removed it when cleaning up the comments.
>
> > Be careful about TemporaryDirectory - by default it puts it in /tmp,
> > which is RAM-based tmpfs - may be too small for the whole template.
> > /var/tmp should be better (AFAIR dir= argument).
>
> I see. Did not realize that /tmp is a tmpfs.
>
> An updated version with the issues above fixed (and some other slight changes)
> has been posted.

If the intention of nogpgcheck is similar to dnf --nogpgcheck option,
then it would be simpler to skip the check completely instead of
handling various error conditions.

Actually, why both --nogpgcheck and --allow-unsigned? If you don't care
if the signature is there, one of those options is enough. And skipping
the check will be simpler.

As for the comment about setting installed_by_rpm - yes, better drop
setting it in qvm-template-postprocess.

> > Also, look specifically for a requested template name, not just anything
> > you find in the package. Currently "for name in os.listdir(target +
> > PATH_PREFIX)" will happily accept whatever template is there (possibly
> > also multiple templates in a single rpm).
>
> The reason I did it this way was because i) there doesn't seem to be a clean
> way of extracting the template name from the package, and ii) it might
> conceivably be a feature useful in cases like installing whonix-gw and
> whonix-ws at the same time.
>
> That being said, the idea of looking for a requested template name makes a lot
> of sense as a way of dealing with the first problem. I am unsure what the user
> interface should look like when installing downloaded RPMs though.

I would go with the package name (in rpm header). And if it doesn't
conform to "qubes-template-(template name)" pattern, then require
explicit option from the user, or even reject as an invalid package.

And when the user request to download the template, you already have
expected name.

> (Thinking about it, being able to install multiple templates this way would
> probably create issues when implementing force-reinstall operations later on,
> so sticking to one template in one RPM is probably a better idea after all.)

Yes, one template - one rpm. Otherwise there will be a lot of complexity
in various corner cases.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl7wL24ACgkQ24/THMrX
1yyyTQgAhIR22DsnTtE+xToZ4KEH1Y/zMUQm1WVcbBUqk3/QtVVUbr+54Ruhq8Ni
yMgp3Iqwy2PgrcsCh2cVDgB/MvctKx1J91/Gn4tuFBEQe7w8uWCg5VLRgLWhIwrt
pv/UC11swf4oCto26KugInqMhVDHKgogKGELCH+X8DADTFaDZISIPTMSwmKxajBV
p1JjnGwVqg70QUpvtf3lkLTcO9KvdiNHa/vx8MEkW1s2lgjeeVWPqJZIWdgy2x7C
AF7XDFsP3inDPuz2+nGx4VF3kK7Ne7IFx2g4W7x/K2+1eli+F8iXWN4bzgeR4lm0
mIB7RBrbxC6ggMQSnuYSQb8vKtdEPA==
=OYDe
-----END PGP SIGNATURE-----

[WillyPillow]

Ah, I see. Perhaps I can, for the time being, have `qvm-appmenus` read the
whitelists from `qvm-features` as i) that should be sufficient to make the
template manager work, and ii) it seems to be something that needs doing
regardless of the qrexec call design.

> > > > > > Anyhow, if the package format has been established to a certain extent, maybe I
> > > > > > can start building a PoC that, say, imports a image from an existing RPM?
> >

> > > > > Yes :)
> >

> > > > A PoC using the original file format can be found at
> > > > https://gist.github.com/WillyPillow/61ee5f48b7c5b7cc90c9fd2ec5c1b20d.
> > > > The script is tested with `qubes-template-fedora-31-minimal` from the official repos.
> >

> > > extract_rpm is not called from anywhere?
> >

> > That's... very embarrassing :(
> > I may have accidentally removed it when cleaning up the comments.
> >

> > > Be careful about TemporaryDirectory - by default it puts it in /tmp,
> > > which is RAM-based tmpfs - may be too small for the whole template.
> > > /var/tmp should be better (AFAIR dir= argument).
> >

> > I see. Did not realize that /tmp is a tmpfs.
> > An updated version with the issues above fixed (and some other slight changes)
> > has been posted.
>

> If the intention of nogpgcheck is similar to dnf --nogpgcheck option,
> then it would be simpler to skip the check completely instead of
> handling various error conditions.
>

> Actually, why both --nogpgcheck and --allow-unsigned? If you don't care
> if the signature is there, one of those options is enough. And skipping
> the check will be simpler.

That indeed makes sense. I was blinded by the fact that `rpm -K` handles the
two cases differently.

> As for the comment about setting installed_by_rpm - yes, better drop
> setting it in qvm-template-postprocess.
>

> > > Also, look specifically for a requested template name, not just anything
> > > you find in the package. Currently "for name in os.listdir(target +
> > > PATH_PREFIX)" will happily accept whatever template is there (possibly
> > > also multiple templates in a single rpm).
> >

> > The reason I did it this way was because i) there doesn't seem to be a clean
> > way of extracting the template name from the package, and ii) it might
> > conceivably be a feature useful in cases like installing whonix-gw and
> > whonix-ws at the same time.
> > That being said, the idea of looking for a requested template name makes a lot
> > of sense as a way of dealing with the first problem. I am unsure what the user
> > interface should look like when installing downloaded RPMs though.
>

> I would go with the package name (in rpm header). And if it doesn't
> conform to "qubes-template-(template name)" pattern, then require
> explicit option from the user, or even reject as an invalid package.

I am leaning towards the latter, as otherwise the interface might be a bit
clunky if the user were to install multiple templates at once (`qvm-template
install templateA templateB`).

(Because of this, the current implementation behaves this way.)

> And when the user request to download the template, you already have
> expected name.
>

> > (Thinking about it, being able to install multiple templates this way would
> > probably create issues when implementing force-reinstall operations later on,
> > so sticking to one template in one RPM is probably a better idea after all.)
>

> Yes, one template - one rpm. Otherwise there will be a lot of complexity
> in various corner cases.

Most of the required modifications to `qvm-template` and
`qvm-template-postprocess` should be finished. I should be able to test the
changes more thoroughly and put it out in a day or so.

I also changed `linux-template-builder` to generate packages *without* the
split tarballs, and realized that `rpm2cpio` refuses to work with large files.
As a result, I probably have to move to `rpm2archive` instead.

As for including the config file in the RPM, am I correct in saying that it
suffices to deal with it similar to appmenu whitelists -- have a generic
version but allow builder plugins to override it?

As a side note, is `netvm-whitelisted-appmenus.list` actually used anywhere? I
can't find references to it in `qubes-desktop-linux-common` aside from
`appmenus_init`, which merely copies it without doing anything else.

Also, the design gist [^gist] has been slightly updated to reflect the
discussions.

[^gist]: <https://gist.github.com/WillyPillow/b8a643ddbd9235a97bc187e6e44b16e4/>

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Thu, Jun 25, 2020 at 05:22:12PM +0000, WillyPillow wrote:
> On Monday, June 22, 2020 12:11 PM, Marek Marczykowski-Górecki <marm...@invisiblethingslab.com> wrote:
> > If we rely on the side effect of qubes.SyncAppmenus to not only extract
> > available desktop files, but also (re)generate actual menu, then indeed
> > for template installation no extra qrexec services are necessary.
> >
>
> > But in general, qrexec services for manipulating menu will be needed for
> > all kind of other cases, like creating/removing VMs, updating menu after
> > setting what applications are included, after changing template, after
> > changing guivm etc. Sounds like a task unrelated to template manager.
>
> Ah, I see. Perhaps I can, for the time being, have `qvm-appmenus` read the
> whitelists from `qvm-features` as i) that should be sufficient to make the
> template manager work, and ii) it seems to be something that needs doing
> regardless of the qrexec call design.

Yes, that should works perfectly here.

> > Actually, why both --nogpgcheck and --allow-unsigned? If you don't care
> > if the signature is there, one of those options is enough. And skipping
> > the check will be simpler.
>
> That indeed makes sense. I was blinded by the fact that `rpm -K` handles the
> two cases differently.

In fact, `rpm -K` is dumb here - reporting "everything is fine" when
calling "rpm --checksig" on an unsigned file is plainly misleading.

> > > That being said, the idea of looking for a requested template name makes a lot
> > > of sense as a way of dealing with the first problem. I am unsure what the user
> > > interface should look like when installing downloaded RPMs though.
> >
>
> > I would go with the package name (in rpm header). And if it doesn't
> > conform to "qubes-template-(template name)" pattern, then require
> > explicit option from the user, or even reject as an invalid package.
>
> I am leaning towards the latter, as otherwise the interface might be a bit
> clunky if the user were to install multiple templates at once (`qvm-template
> install templateA templateB`).
>
> (Because of this, the current implementation behaves this way.)

+1

> > And when the user request to download the template, you already have
> > expected name.
> >
>
> > > (Thinking about it, being able to install multiple templates this way would
> > > probably create issues when implementing force-reinstall operations later on,
> > > so sticking to one template in one RPM is probably a better idea after all.)
> >
>
> > Yes, one template - one rpm. Otherwise there will be a lot of complexity
> > in various corner cases.
>
> Most of the required modifications to `qvm-template` and
> `qvm-template-postprocess` should be finished. I should be able to test the
> changes more thoroughly and put it out in a day or so.
>
> I also changed `linux-template-builder` to generate packages *without* the
> split tarballs, and realized that `rpm2cpio` refuses to work with large files.
> As a result, I probably have to move to `rpm2archive` instead.

That is interesting development given the RPM package is cpio
internally. Needing to convert that to tar sounds weird...
Be careful to put appropriate defensive tar options (expected paths,
perhaps force specific tar format etc).

> As for including the config file in the RPM, am I correct in saying that it
> suffices to deal with it similar to appmenu whitelists -- have a generic
> version but allow builder plugins to override it?

Yes, that sounds fine.

> As a side note, is `netvm-whitelisted-appmenus.list` actually used anywhere? I
> can't find references to it in `qubes-desktop-linux-common` aside from
> `appmenus_init`, which merely copies it without doing anything else.

That's a good question. It is supposed to be for network-providing VMs,
like sys-net or sys-firewall. But indeed it isn't the case right now.
And with the current architecture it may be hard to achieve, as the VM
(including appmenus) is created before setting `provides_network`
property. Anyway, I'd keep this information (save as a separate feature)
even if it isn't used right now.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl71OOsACgkQ24/THMrX
1ywd8wf8DPlfYLfQOcF4HXP1DWDYmAS/o7jehLrAj0DuBzMLSYl06BVET1ZgHfyt
9PD2QwZ5uZsnc/rYxtLrtcanUlF9qJDlmzNx1/7hUjtW/apyEYlva3ZZGXl33hDW
6VTrkKw8u3WmeNC/swAzNkPMTBGeS4GNRWTdlY2wC7Uods8JgwK3PpMm66MH+TDL
zSrI+JBtDRYHrchJCYvCE6h07IfUKq3H4A+i4NI1KzOEK+12PflcNl0JC+5GoD+/
0XsPfpeKQ8FLPxLXcPuzxsGcmj9PKRLwB/1UkO4MV9tecsUe1RSBtBMX+zd9C2th
6ixMD4GHEFAPIEktUcM58hX5Vn222w==
=8knw
-----END PGP SIGNATURE-----

[WillyPillow]

On Friday, June 26, 2020 7:53 AM, Marek Marczykowski-Górecki <marm...@invisiblethingslab.com> wrote:

> On Thu, Jun 25, 2020 at 05:22:12PM +0000, WillyPillow wrote:
>

> > On Monday, June 22, 2020 12:11 PM, Marek Marczykowski-Górecki marm...@invisiblethingslab.com wrote:
> >

> > > If we rely on the side effect of qubes.SyncAppmenus to not only extract
> > > available desktop files, but also (re)generate actual menu, then indeed
> > > for template installation no extra qrexec services are necessary.
> >

> > > But in general, qrexec services for manipulating menu will be needed for
> > > all kind of other cases, like creating/removing VMs, updating menu after
> > > setting what applications are included, after changing template, after
> > > changing guivm etc. Sounds like a task unrelated to template manager.
> >

> > Ah, I see. Perhaps I can, for the time being, have `qvm-appmenus` read the
> > whitelists from `qvm-features` as i) that should be sufficient to make the
> > template manager work, and ii) it seems to be something that needs doing
> > regardless of the qrexec call design.
>

> Yes, that should works perfectly here.

As `qvm-appmenus` copies whitelists around, if the source is file-based, should I convert the target to `qvm-features` anyway?

> > I also changed `linux-template-builder` to generate packages without the

> > split tarballs, and realized that `rpm2cpio` refuses to work with large files.
> > As a result, I probably have to move to `rpm2archive` instead.
>

> That is interesting development given the RPM package is cpio
> internally. Needing to convert that to tar sounds weird...
> Be careful to put appropriate defensive tar options (expected paths,
> perhaps force specific tar format etc).

According to <https://bugzilla.redhat.com/show_bug.cgi?id=1662481>, vanilla cpio does not support files over 4G unfortunately.

One downside of using rpm2archive is that it converts to .tar.gz, creating an extra compression/decompression step and slowing down the process. Setting the environment variable GZIP="-1" does not change anything unfortunately.

> > As for including the config file in the RPM, am I correct in saying that it
> > suffices to deal with it similar to appmenu whitelists -- have a generic
> > version but allow builder plugins to override it?
>

> Yes, that sounds fine.
>

> > As a side note, is `netvm-whitelisted-appmenus.list` actually used anywhere? I
> > can't find references to it in `qubes-desktop-linux-common` aside from
> > `appmenus_init`, which merely copies it without doing anything else.
>

> That's a good question. It is supposed to be for network-providing VMs,
> like sys-net or sys-firewall. But indeed it isn't the case right now.
> And with the current architecture it may be hard to achieve, as the VM
> (including appmenus) is created before setting `provides_network`
> property. Anyway, I'd keep this information (save as a separate feature)
> even if it isn't used right now.

Anyways, sorry for the delay. The updated version of `qvm-template` has been uploaded [^gist], and the modified version of `qvm-template-postprocess` can be found at <https://github.com/WillyPillow/qubes-core-admin-client>. I should be able to put out the modified versions of `qvm-appmenus` and `linux-template-builder` soon.

[^gist]: <https://gist.github.com/WillyPillow/61ee5f48b7c5b7cc90c9fd2ec5c1b20d>

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sat, Jun 27, 2020 at 08:05:49PM +0000, WillyPillow wrote:
> On Friday, June 26, 2020 7:53 AM, Marek Marczykowski-Górecki <marm...@invisiblethingslab.com> wrote:
>
> > On Thu, Jun 25, 2020 at 05:22:12PM +0000, WillyPillow wrote:
> >
>
> > > On Monday, June 22, 2020 12:11 PM, Marek Marczykowski-Górecki marm...@invisiblethingslab.com wrote:
> > >
>
> > > > If we rely on the side effect of qubes.SyncAppmenus to not only extract
> > > > available desktop files, but also (re)generate actual menu, then indeed
> > > > for template installation no extra qrexec services are necessary.
> > >
>
> > > > But in general, qrexec services for manipulating menu will be needed for
> > > > all kind of other cases, like creating/removing VMs, updating menu after
> > > > setting what applications are included, after changing template, after
> > > > changing guivm etc. Sounds like a task unrelated to template manager.
> > >
>
> > > Ah, I see. Perhaps I can, for the time being, have `qvm-appmenus` read the
> > > whitelists from `qvm-features` as i) that should be sufficient to make the
> > > template manager work, and ii) it seems to be something that needs doing
> > > regardless of the qrexec call design.
> >
>
> > Yes, that should works perfectly here.
>
> As `qvm-appmenus` copies whitelists around, if the source is file-based, should I convert the target to `qvm-features` anyway?

Not necessarily. `qvm-features` are already copied when VM is cloned, so
it should work out of the box.

> > > Most of the required modifications to `qvm-template` and
> > > `qvm-template-postprocess` should be finished. I should be able to test the
> > > changes more thoroughly and put it out in a day or so.
> > > I also changed `linux-template-builder` to generate packages without the
> > > split tarballs, and realized that `rpm2cpio` refuses to work with large files.
> > > As a result, I probably have to move to `rpm2archive` instead.
> >
>
> > That is interesting development given the RPM package is cpio
> > internally. Needing to convert that to tar sounds weird...
> > Be careful to put appropriate defensive tar options (expected paths,
> > perhaps force specific tar format etc).
>
> According to <https://bugzilla.redhat.com/show_bug.cgi?id=1662481>, vanilla cpio does not support files over 4G unfortunately.

Ah, so that's why we have the root.tar split into parts.

> One downside of using rpm2archive is that it converts to .tar.gz, creating an extra compression/decompression step and slowing down the process. Setting the environment variable GZIP="-1" does not change anything unfortunately.

So, maybe better revert to the old approach - split root.tar?

> Anyways, sorry for the delay. The updated version of `qvm-template` has been uploaded [^gist],

When extracting the package, I'd recommend extracting just specific
directory - if the rpm would contains also other files/dirs outside
/var/lib/qubes/vm-templates/<NAME>, then don't extract them at all.
Simply add extra argument to tar/cpio.

> and the modified version of `qvm-template-postprocess` can be found at <https://github.com/WillyPillow/qubes-core-admin-client>.

Added some comments there.

> I should be able to put out the modified versions of `qvm-appmenus` and `linux-template-builder` soon.
>
> [^gist]: <https://gist.github.com/WillyPillow/61ee5f48b7c5b7cc90c9fd2ec5c1b20d>
>

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl731U8ACgkQ24/THMrX
1ywhMQf8DtsqF5hcbtk6ZnJpKE5U7LwZ8FcjR/nN/KLqaQwojlW+TKNWKPXuGdGG
ETzldIXb1qnf9lFGHynVxGF/eDWmJzC/o7NXK1rptAJiG31k9MRNAp3NsgdusL8T
VeOYyo8B2I9ZoMvep1A1qmh3e7vhP08noBkiOw7QwnXS2Y1jL5RhoTYd+j9+cyun
UHoZyXjshyVKnoRtgI1jsrU9MSNhwhSm6G8Vbif3bqoih0JuIQo52pOB3X0xH7Tg
dkeMTUYmuVGFRLZcpI9FLPresiNmaSNgFDwguC724FyGM5ALzn1llVfNaAaV8tuM
meNf2fVHUdgT07/xntvAqYuGo++EuQ==
=Co5c
-----END PGP SIGNATURE-----

[WillyPillow]

On Sunday, June 28, 2020 7:25 AM, Marek Marczykowski-Górecki <marm...@invisiblethingslab.com> wrote:

> On Sat, Jun 27, 2020 at 08:05:49PM +0000, WillyPillow wrote:
>

> > On Friday, June 26, 2020 7:53 AM, Marek Marczykowski-Górecki marm...@invisiblethingslab.com wrote:
> >

> > > On Thu, Jun 25, 2020 at 05:22:12PM +0000, WillyPillow wrote:
> >

> > > > On Monday, June 22, 2020 12:11 PM, Marek Marczykowski-Górecki marm...@invisiblethingslab.com wrote:
> >

> > > > > If we rely on the side effect of qubes.SyncAppmenus to not only extract
> > > > > available desktop files, but also (re)generate actual menu, then indeed
> > > > > for template installation no extra qrexec services are necessary.
> >

> > > > > But in general, qrexec services for manipulating menu will be needed for
> > > > > all kind of other cases, like creating/removing VMs, updating menu after
> > > > > setting what applications are included, after changing template, after
> > > > > changing guivm etc. Sounds like a task unrelated to template manager.
> >

> > > > Ah, I see. Perhaps I can, for the time being, have `qvm-appmenus` read the
> > > > whitelists from `qvm-features` as i) that should be sufficient to make the
> > > > template manager work, and ii) it seems to be something that needs doing
> > > > regardless of the qrexec call design.
> >

> > > Yes, that should works perfectly here.
> >

> > As `qvm-appmenus` copies whitelists around, if the source is file-based, should I convert the target to `qvm-features` anyway?
>

> Not necessarily.`qvm-features` are already copied when VM is cloned, so
> it should work out of the box.

I probably could have worded that better :). I meant to ask that, for example, when there is a template VM using file-based whitelists and an AppVM based on it is created, should the AppVM whitelist be file-based or qvm-feature-based? Currently I'm implementing the former as there are less changes to be made, though I'm not sure if the latter is better for future migration.

> > > > Most of the required modifications to `qvm-template` and
> > > > `qvm-template-postprocess` should be finished. I should be able to test the
> > > > changes more thoroughly and put it out in a day or so.
> > > > I also changed `linux-template-builder` to generate packages without the
> > > > split tarballs, and realized that `rpm2cpio` refuses to work with large files.
> > > > As a result, I probably have to move to `rpm2archive` instead.
> >

> > > That is interesting development given the RPM package is cpio
> > > internally. Needing to convert that to tar sounds weird...
> > > Be careful to put appropriate defensive tar options (expected paths,
> > > perhaps force specific tar format etc).
> >

> > According to https://bugzilla.redhat.com/show_bug.cgi?id=1662481, vanilla cpio does not support files over 4G unfortunately.

>

> Ah, so that's why we have the root.tar split into parts.
>

> > One downside of using rpm2archive is that it converts to .tar.gz, creating an extra compression/decompression step and slowing down the process. Setting the environment variable GZIP="-1" does not change anything unfortunately.
>

> So, maybe better revert to the old approach - split root.tar?

Perhaps that is indeed the better way. I'll keep the relevant parts in `qvm-template-postprocess` for a while in case a better solution is found, though.

Side note, I experimented with cramming a bunch of files in a RPM, and indeed as long as the individual files are not too large, rpm2cpio works fine.

> > Anyways, sorry for the delay. The updated version of `qvm-template` has been uploaded ^gist,

>

> When extracting the package, I'd recommend extracting just specific
> directory - if the rpm would contains also other files/dirs outside
> /var/lib/qubes/vm-templates/<NAME>, then don't extract them at all.
> Simply add extra argument to tar/cpio.
>

> > and the modified version of `qvm-template-postprocess` can be found at https://github.com/WillyPillow/qubes-core-admin-client.
>

> Added some comments there.
>

> > I should be able to put out the modified versions of `qvm-appmenus` and `linux-template-builder` soon.

The modified versions are at <https://github.com/WillyPillow/qubes-desktop-linux-common/tree/qvm-template> and <https://github.com/WillyPillow/qubes-linux-template-builder/tree/qvm-template>. The other repos have also been updated.

For the next step, should I start implementing the interactions with the repos?

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Jun 30, 2020 at 05:41:39AM +0000, WillyPillow wrote:
> I probably could have worded that better :). I meant to ask that, for example, when there is a template VM using file-based whitelists
> and an AppVM based on it is created, should the AppVM whitelist be file-based or qvm-feature-based? Currently I'm implementing the former as there are less changes to be made, though I'm not sure if the latter is better for future migration.

I'd go with qvm-features for all. At some (not very distant) point,
we'll also add forced migration of existing whitelist files to
qvm-features and files will be no more.

> > > One downside of using rpm2archive is that it converts to .tar.gz, creating an extra compression/decompression step and slowing down the process. Setting the environment variable GZIP="-1" does not change anything unfortunately.
> >
>
> > So, maybe better revert to the old approach - split root.tar?
>
> Perhaps that is indeed the better way. I'll keep the relevant parts in `qvm-template-postprocess` for a while in case a better solution is found, though.
>
> Side note, I experimented with cramming a bunch of files in a RPM, and indeed as long as the individual files are not too large, rpm2cpio works fine.

Good.

> The modified versions are at <https://github.com/WillyPillow/qubes-desktop-linux-common/tree/qvm-template> and <https://github.com/WillyPillow/qubes-linux-template-builder/tree/qvm-template>. The other repos have also been updated.

Can you open (WIP) pull requests with those? It's easier to add comments
in github interface then.

> For the next step, should I start implementing the interactions with the repos?

Yes :)

While you are writing/testing those things, please also keep notes about
testing scenarios. Can be sketch quality (list of commands to test or
such). This will be useful later when integrating all of this together.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl77uPUACgkQ24/THMrX
1yymvAf+IeQsFO0RVGmhLzsUGeglFj3QqlI05D3OE7FW2l7wwBGIhtuiOobJsfJM
PTtCKZ9rVzh8ueLy496v06lmmkq7o+xlrXg4WNcsF9/NOf10TxyAcwnVnTecFn44
6slEldlxEjryWgCnX3dCSmqsvfeGK6txuZKTK7353foc7pMzDkAanP27XWyC3xar
1VsbRraJSUcO4RsbUxCud3z2vwoBkSI6j91tiAn5PponjPf3bkvdhXgVgmkEv7BT
aCESaGvkvpWMl5r5fZGFDOyol/k6iH+RS0TQEuiR2LM6IBLJ6cDHbK84gKmFAFL/
792Qog+260u9AYMwr/fJgRTQYNAkpQ==
=uvDL
-----END PGP SIGNATURE-----

[WillyPillow]

On Wednesday, July 1, 2020 6:13 AM, Marek Marczykowski-Górecki <marm...@invisiblethingslab.com> wrote:

> On Tue, Jun 30, 2020 at 05:41:39AM +0000, WillyPillow wrote:
>

> > I probably could have worded that better :). I meant to ask that, for example, when there is a template VM using file-based whitelists
> > and an AppVM based on it is created, should the AppVM whitelist be file-based or qvm-feature-based? Currently I'm implementing the former as there are less changes to be made, though I'm not sure if the latter is better for future migration.
>

> I'd go with qvm-features for all. At some (not very distant) point,
> we'll also add forced migration of existing whitelist files to
> qvm-features and files will be no more.

I see. I've pushed a version that implements the latter.

> > > > One downside of using rpm2archive is that it converts to .tar.gz, creating an extra compression/decompression step and slowing down the process. Setting the environment variable GZIP="-1" does not change anything unfortunately.
> >

> > > So, maybe better revert to the old approach - split root.tar?
> >

> > Perhaps that is indeed the better way. I'll keep the relevant parts in `qvm-template-postprocess` for a while in case a better solution is found, though.
> > Side note, I experimented with cramming a bunch of files in a RPM, and indeed as long as the individual files are not too large, rpm2cpio works fine.
>

> Good.
>

> > The modified versions are at https://github.com/WillyPillow/qubes-desktop-linux-common/tree/qvm-template and https://github.com/WillyPillow/qubes-linux-template-builder/tree/qvm-template. The other repos have also been updated.

>

> Can you open (WIP) pull requests with those? It's easier to add comments
> in github interface then.

Done :)

> > For the next step, should I start implementing the interactions with the repos?
>

> Yes :)
>

> While you are writing/testing those things, please also keep notes about
> testing scenarios. Can be sketch quality (list of commands to test or
> such). This will be useful later when integrating all of this together.

Got it.