What's your flow for new templateVM?

[Ryan Tate]

Saw the new f31 templateVM (thanks for that) and just curious how
folks generally migrate to a new templateVM.

I manually maintain this big text list of packages and just use
that to manually update the fresh templateVM to what I
need. There's typically also some non package installs, which I
include basic pointers for (think downloaded rpms and so forth),
as well as some outside repos to add (e.g. keybase). There's also
typically some packages I forgot to put on the list, which I can
usually suss out by going through the bash history for the old
template, although often there's one or two that slip through the
cracks, which I find out about eventually and it's not a huge
deal.

I'm particularly curious if anyone does anything more
sophisticated than that, using salt or some other automated deploy
system to prep new template images.

Thanks for any tips!

[Stumpy]

Ditto, would really be interested as well, I have a similar system but i
am sure there are better ways to do it.

[unman]

Salt it - if you get used to using salt, it's simple to use.
If you want to install a package, don't open the template and install it
there, edit the install.sls file to include the package, and run
`qubesctl --skip-dom0 --targets=<template> state.apply install`

That *should* install the package, and you have a record of what you've
done. So our "big text file" becomes functional.
You can also leverage salt to apply the same packages to Debian and
Fedora templates - where names differ, you can apply packages by
checking OS.
And, of course, you can add/edit sources.files, insert gpg keys, copy in
rpms/source, and your salt files will be a record of what you want.

On a new system, or a new template, all you have to do is run the
`install` state targeting the template(s) you want.
Really, a great system, and I suspect sadly under used.

I have full systems set up in salt to customise a new install as I want,
with new templates and different setups. Sometimes it can be a bit
shaky, and you *have* to check the logs, but it's great to run the full
state, have a coffee, and come back to a fully configured system.
For travel, I have a minimum state I can download and apply, to get a
workable system with gpg, vpn, ssh set up out of the box. So cool.

unman

[Steve Coleman]

On Mon, May 11, 2020, 10:26 AM 'Ryan Tate' via qubes-users <qubes...@googlegroups.com> wrote:

I manually maintain this big text list of packages and just use
that to manually update the fresh templateVM to what I
need. There's typically also some non package installs, which I
include basic pointers for (think downloaded rpms and so forth),
as well as some outside repos to add (e.g. keybase). There's also
typically some packages I forgot to put on the list, which I can
usually suss out by going through the bash history for the old
template, although often there's one or two that slip through the
cracks, which I find out about eventually and it's not a huge
deal.

I'm particularly curious if anyone does anything more
sophisticated than that, using salt or some other automated deploy
system to prep new template images.

I was just playing with creating a domain bash script that runs "qvm-run -p"  in one template to extract the list of packages (dnf list), then subtracts the list from the second template, pulls that difference list up in an editor, and then pushes the manually edited list to the next template ( dnf install $list). Then I found there were so many packages I did not particularly want to carry forward without proper investigation that I essentially put that script on hold. 

I obviously need to take my time to decide what I want to bring forward. Things like python2 packages need to be weeded out, as well as other packages that I was merely investigating for use at work but don't have any need for now that I am retired. 

I think the general script/process has merit but I have far too many packages to evaluate in a single session. Simply pushing everything in one go would merely add a lot of stuff I did not need. When I have time I may just delete most of the packages from the editor and do this in chunks as I find time to work on it. 

[Elliot Killick]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I also keep a list of packages I want to have on each of my templates.
For the switching part of each AppVM to the new TemplateVM (e.g.
fedora-30 to fedora-31) I use a simple Python script that utilizes the
Qubes Admin API:

#!/usr/bin/python3

import qubesadmin
from qubesadmin.exc import *

from_templatevm = 'fedora-30'
to_templatevm = 'fedora-31'

qubes = qubesadmin.Qubes().app.domains

for qube in qubes:
if qube.name == from_templatevm:
appvms = qube.appvms
for appvm in appvms:
print('Changing TemplateVM of:', appvm.name)
try:
appvm.__setattr__('template', to_templatevm)
except QubesVMNotHaltedError:
print("Cannot change TemplateVM while qube is turned on!
")

I usually just run this after my next reboot when almost (don't forget
to shutdown the NetVMs) all my qubes will be turned off. This is much
better than switching them manually one by one in Qubes Manager.

After that all I do is clone, for example, fedora-30-dvm to
fedora-31-dvm with its template now changed.


-----BEGIN PGP SIGNATURE-----

iHUEARYIAB0WIQQBj7nebfoT+xj7VVL5uQ1E+D3V8gUCXrm5oQAKCRD5uQ1E+D3V
8qS/AQCtW8COW7f2ndgpDTAJD/VYRzfqgo333UeR7EmcC1JKqAD+O0Jh2Z3tseRn
cmEcBRFQyOYmPjvGCdHfq/Ypnj66VQo=
=9oMW
-----END PGP SIGNATURE-----

[ryan...@ryantate.com]

On Monday, May 11, 2020 at 11:11:45 AM UTC-4, unman wrote:

If you want to install a package, don't open the template and install it
there, edit the install.sls file to include the package, and run
`qubesctl --skip-dom0 --targets=<template>  state.apply install`

Dumb question, where is install.sls (or where should I create it)? I poked around in /srv in dom0 but still not sure

 

I have full systems set up in salt to customise a new install as I want,
with new templates and different setups. Sometimes it can be a bit
shaky, and you *have* to check the logs, but it's great to run the full
state, have a coffee, and come back to a fully configured system.
For travel, I have a minimum state I can download and apply, to get a
workable system with gpg, vpn, ssh set up out of the box. So cool.

Very intriguing. I am looking at

https://docs.saltstack.com/en/latest/ref/states/all/salt.states.pkg.html

https://www.qubes-os.org/doc/salt/

....lmk if there is anything else I should read!

[ryan...@ryantate.com]

On Monday, May 11, 2020 at 6:02:06 PM UTC-4, ryan...@ryantate.com wrote:

Very intriguing. I am looking at

https://docs.saltstack.com/en/latest/ref/states/all/salt.states.pkg.html

https://www.qubes-os.org/doc/salt/

....lmk if there is anything else I should read!

(To be clear, I'm just starting to read these....)

[unman]

On Mon, May 11, 2020 at 03:02:06PM -0700, ryantate via qubes-users wrote:
>
>
> On Monday, May 11, 2020 at 11:11:45 AM UTC-4, unman wrote:
>
> > If you want to install a package, don't open the template and install it
> > there, edit the install.sls file to include the package, and run
> > `qubesctl --skip-dom0 --targets=<template> state.apply install`
> >
>
>
> Dumb question, where is install.sls (or where should I create it)? I poked
> around in /srv in dom0 but still not sure
>

You create it - you can bundle relevant files under subdirectory in
/srv/salt and then refer to them like this:
state.apply <subdirectory>.state

>
> >
> > I have full systems set up in salt to customise a new install as I want,
> > with new templates and different setups. Sometimes it can be a bit
> > shaky, and you *have* to check the logs, but it's great to run the full
> > state, have a coffee, and come back to a fully configured system.
> > For travel, I have a minimum state I can download and apply, to get a
> > workable system with gpg, vpn, ssh set up out of the box. So cool.
> >
> >
> Very intriguing. I am looking at
>
> https://docs.saltstack.com/en/latest/ref/states/all/salt.states.pkg.html
>
> https://www.qubes-os.org/doc/salt/
>
> ....lmk if there is anything else I should read!
>

From another post:
I ran some training a few years back, and the notes are here:
https://github.com/unman/notes/tree/master/salt

They start with the simplest use of `qubesctl`, and work up to quite
complex configurations, but should be easy to understand.
There are examples in (naturally) "examples".

For some real world cases look in notes/config.

unman

[unman]

Just for the record there's a GUI tool to do this.
System->ManageTemplates in Qube Manager

[lik...@gmx.de]

On 2020-05-11 16:11, unman wrote:
> On Mon, May 11, 2020 at 10:52:32AM -0400, Stumpy wrote:
>> On 2020-05-11 10:26, 'Ryan Tate' via qubes-users wrote:
>>> Saw the new f31 templateVM (thanks for that) and just curious how folks
>>> generally migrate to a new templateVM.
>>>
>>> I manually maintain this big text list of packages and just use that to
>>> manually update the fresh templateVM to what I need. There's typically
>>> also some non package installs, which I include basic pointers for
>>> (think downloaded rpms and so forth), as well as some outside repos to
>>> add (e.g. keybase). There's also typically some packages I forgot to put
>>> on the list, which I can usually suss out by going through the bash
>>> history for the old template, although often there's one or two that
>>> slip through the cracks, which I find out about eventually and it's not
>>> a huge deal.
>>>
>>> I'm particularly curious if anyone does anything more sophisticated than
>>> that, using salt or some other automated deploy system to prep new
>>> template images.
>>>
>>> Thanks for any tips!
>>>
>>
>> Ditto, would really be interested as well, I have a similar system but i am
>> sure there are better ways to do it.
>>
>
> Salt it - if you get used to using salt, it's simple to use.
> If you want to install a package, don't open the template and install it
> there, edit the install.sls file to include the package, and run
> `qubesctl --skip-dom0 --targets=<template> state.apply install`
>

What's the advantage of using Salt in comparison to 'qvm-run ...' commands bundled to a script to manage template modifications? In this case you've also a executable documentation.
What's I'm missing?

[Elliot Killick]

Huh, I've seen that before but I guess I kind of forgot about it!

Thanks, unman.

[unman]

You're missing the range and flexibility of salt.

As soon as you move beyond the most basic provisioning of a system, the
scripting approach starts to become a messy bundle.
I used to use bash scripts, qvm-tools, sed, and STILL I'd have to go in
and finish off by hand.
With Salt, I build and configure templates, individual qubes.
It's simple to restore a subset of my system for when I'm travelling, or
rebuild complete configurations. I wouldn't go back.

Of course you *can* do everything using qvm-tools in scripts, but the
salt tools are cleaner and more flexible. .

unman

[gal...@gmail.com]

I had a try with salt today and made a VM from a template. If like to go further with this.

I found that the /svr/salt directory was owned by root so I had to use sudo to copy anything to it. Is that normal? I also had to use sudo on the qubesctl commands.

One of the things I need to go is install a printer driver for a network printer. Is this sort of thing easy to do with salt? I normally download the driver from the brother website and follow the instructions for a fedora install.

[lik...@gmx.de]

>>
>
> From another post:
> I ran some training a few years back, and the notes are here:
> https://github.com/unman/notes/tree/master/salt
>
> They start with the simplest use of `qubesctl`, and work up to quite
> complex configurations, but should be easy to understand.
> There are examples in (naturally) "examples".
>

Let's assume I'm managing all my configuration in /srv/salt/myConfig

From this post, it seems that these directories are not backuped by default by the qubes-backup:
https://groups.google.com/forum/#!searchin/qubes-users/doesn$27t$20fully$20restore%7Csort:date/qubes-users/FiGImnZ87sY/yK5h7gcsAgAJ

How do you backup them? By linking from /home? Copying all to /home before a backup? What's the best practice?

[unman]

My practice is to maintain an encrypted tar archive - I can pull that
down, copy it to offline disposableVM, decrypt there, and copy in to
dom0.
For general system configuration, I load the directories from an encrypted USB drive.