Thoughts on Salt in Qubes in practice?

[Jean-Philippe Ouellet]

Previously...
https://groups.google.com/d/topic/qubes-devel/VimLbPHwJRw/discussion
> Let me be the first to admit that salt appears difficult to understand at
> first. Their documentation drives me up the wall. It strikes me as the kind
> of project that gained followers early when it was still easy to grasp, then
> built up an obscurantist mindset.

https://github.com/QubesOS/qubes-issues/issues/1983
> I've been trying to learn saltstack for a few weeks now, but the
> documentation is outright novice-hostile. It spouts off about advanced
> options in the second or third paragraphs of each topic and rarely
> returns to Earth.

I'm curious how many people actually use Salt to manage their Qubes
machines. If you do, I want to hear your thoughts on it.

Over the past few years my main daily Qubes system has grown to ~150
VMs, and over 20 templates. In an effort to keep things clean, I don't
upgrade my templates across fedora / debian versions, but instead
regularly create fresh clean templates with only the intended packages
and local config explicitly desired.

Every time my templates go EOL and its time to bump them to new
versions I say to myself "okay, this time I'm going to learn salt and
do my template management the official Qubes way". But, each time I
try, I get annoyed at Salt's learning curve, annoyed at the upstream
docs, annoyed at how they've reinvented their own terminology for
every simple thing, and annoyed at subtle differences in behavior
documented upstream and behavior as integrated in Qubes. Then, I
invariably start digging into the implementation and Qubes' existing
use of salt as an example to learn from, get annoyed at how many
layers of abstraction there are, annoyed at how every part of the
documentation already assumes familiarity with the entire rest of
stack, and conclude it's more painful to use than it's worth.

Also, I've on occasion had to set up a number of Qubes machines for
use in various organization-managed settings. This should be the
perfect use case for Salt, but each time I spend a number of hours
trying before finally resorting back to simple shell scripts which end
up being 1) maintainable by others without too much trouble and 2)
just as idempotent as salt aims to be with its declarative nature but
without requiring so much abstraction. It's hard to see the benefit.

Digging into the implementations of things to figure out how they work
is second nature to me, but salt has been particularly unpleasant to
adopt. The only other users of Salt on Qubes I'm aware of are Whonix
and SecureDrop-Workstation (authors found in commit logs of both
CC'd), and Qubes itself. I'd love to hear how other peoples'
experience compares.

Regards,
Jean-Philippe

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Jul 01, 2018 at 03:06:06PM -0400, Jean-Philippe Ouellet wrote:
> Previously...
> https://groups.google.com/d/topic/qubes-devel/VimLbPHwJRw/discussion
> > Let me be the first to admit that salt appears difficult to understand at
> > first. Their documentation drives me up the wall. It strikes me as the kind
> > of project that gained followers early when it was still easy to grasp, then
> > built up an obscurantist mindset.
>
> https://github.com/QubesOS/qubes-issues/issues/1983
> > I've been trying to learn saltstack for a few weeks now, but the
> > documentation is outright novice-hostile. It spouts off about advanced
> > options in the second or third paragraphs of each topic and rarely
> > returns to Earth.
>
> I'm curious how many people actually use Salt to manage their Qubes
> machines. If you do, I want to hear your thoughts on it.

I do, to some extend (templates only). And I feel your pain. Salt is
definitely not easy thing to learn. When you get it, it's quite
useable, but I wouldn't call it comfortable.

> Over the past few years my main daily Qubes system has grown to ~150
> VMs, and over 20 templates. In an effort to keep things clean, I don't
> upgrade my templates across fedora / debian versions, but instead
> regularly create fresh clean templates with only the intended packages
> and local config explicitly desired.

Initially I have upgraded templates, but since about R3.0 I use fresh
templates as you do. Initially I did this with just a text file listing
packages to be installed + tar archive with few configs I've extracted
over /etc. But now I use salt for this.
I don't do that with AppVMs on my main system, at least not yet.

I do that also for test machines, which I reinstall quite frequently. I
have a set of salt configs for installing extra software in templates,
required to run all the tests, configure dom0 to my liking, copy helper
scripts where needed, add my local repositories etc.
And also to install updates in dom0 and all templates at once.

> Every time my templates go EOL and its time to bump them to new
> versions I say to myself "okay, this time I'm going to learn salt and
> do my template management the official Qubes way". But, each time I
> try, I get annoyed at Salt's learning curve, annoyed at the upstream
> docs, annoyed at how they've reinvented their own terminology for
> every simple thing, and annoyed at subtle differences in behavior
> documented upstream and behavior as integrated in Qubes. Then, I
> invariably start digging into the implementation and Qubes' existing
> use of salt as an example to learn from, get annoyed at how many
> layers of abstraction there are, annoyed at how every part of the
> documentation already assumes familiarity with the entire rest of
> stack, and conclude it's more painful to use than it's worth.

I think the major pain points of Salt are:
- completely different terminology you need to learn (state, formula,
pillar, grain, top, ...)
- top files - the fact you define in separate places _what_ to do and
_where_ to do - this makes it hard to grasp the whole picture

IMO the second thing is a bigger problem than the first one. But when
you get initial thing working, adding more states (one more package, one
more config file etc) is straightforward. You can find some basic
example on https://www.qubes-os.org/doc/salt/
When you look at documentation, focus on "Configuration management"
section on docs.saltstack.com, specifically "state modules". Salt in
theory can to much more, but other things are far less useful on Qubes.
Personally I mostly use "pkg" module (for package management) and "file"
module (setting config files, either as a whole, or individual
settings).

When I look at it now, Ansible is probably much easier to learn. And if
you like, you can use it, with some limitations, using:
https://github.com/Rudd-O/ansible-qubes

> Also, I've on occasion had to set up a number of Qubes machines for
> use in various organization-managed settings. This should be the
> perfect use case for Salt, but each time I spend a number of hours
> trying before finally resorting back to simple shell scripts which end
> up being 1) maintainable by others without too much trouble and 2)
> just as idempotent as salt aims to be with its declarative nature but
> without requiring so much abstraction. It's hard to see the benefit.

Of course you can do all of this with own scripts. You know, Salt is
also a bunch of python code ;) For me, the 2) point above is an
important reason to use Salt. How many times you forgot -n option to
ln -sf and ended up with a symlink inside a linked directory, instead of
updating the symlink? With Salt I'm pretty sure I'll get the state I've
described regardless what was there initially (unless I opt for cmd.run
state, which I try to avoid whenever possible...).

> Digging into the implementations of things to figure out how they work
> is second nature to me, but salt has been particularly unpleasant to
> adopt. The only other users of Salt on Qubes I'm aware of are Whonix
> and SecureDrop-Workstation (authors found in commit logs of both
> CC'd), and Qubes itself. I'd love to hear how other peoples'
> experience compares.

There is one more:
https://github.com/inversepath/qubes-qrexec-to-tcp

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAls5asUACgkQ24/THMrX
1ywsfQf/e8gwUegan+mFGclXf6bZBY7ybd6IG6cyq+77uU4GNKIgqyJfBpZ6vgIx
mOrCa6OgHmecvFTkpE/RZkrv3o9AAgoLoSG4AgFBc9BH4+RYqyMX/AcFzKm2lIU2
Q9IWPRNf6l/fai9MkG6YQUs/nBM6gPgrkwzO95zWNzUcx2Fx85wwy/fCDS+bbWrI
avWydfXBf0sK6wo2thDkZL1W+hwrhOYx866CWgqnzHc4qt9Hyz3GninXeKL48Z7d
vYu95INd0UMjlVtbcDaSET6+FOAl58sRtqO0wiAaHD6zkioHX8yWwxqS1m3jO6YD
72dScce6a7NDhEOLHYTIikHVLyrf9A==
=eBD6
-----END PGP SIGNATURE-----

[airele...@tutanota.com]

annoyed at how they've reinvented their own terminology for
every simple thing,

This. I never got past the strange terminology and still use shell scripts for everything.

 

The only other users of Salt on Qubes I'm aware of are Whonix

I think whonix had to actually get outside help for Salt, for whonix-14.

[Johannes Graumann]

I was contemplating to ask about salt for a while now ... I'm in a
long, slow process to move to Qubes as my general purpose OS from an
archlinux environment with home-grown lxc-based 'isolation' of
processes/installs I don't like close to the core OS. In the process of
setting up that infrastructure I had come to use ansible and found it
approachable and easy to learn.

Then I became familiar with Qubes and the transition started. Naturally
I was drawn to management with ansible and under Qubes 3 had a working
setup going using https://github.com/Rudd-O/ansible-qubes, which Marek
Marczykowski-Górecki also references.

With Qubes 4 being released and the integration of salt into its core,
I have made several earnest, lengthy and utterly unsuccessful attempts
to move from a 3rd party add-on (Rudd-O's ansible) to an infrastructure
maintained along with the core of Qubes (salt). It's very possible that
my dabbling with ansible has poisoned my ways, but I have a very hard
time to even understand the concepts of salt. I'm giving up. It's
either manual management or a return to ansible.

When reading on the two I never really grasped why salt was chosen. If
I understand it correctly the system is centered on infrastructures
where tearing down images at a whim (think docker etc.) is the norm. I
can see how that model of thinking applies to the early stages of
setting up a Qubes infrastructure (and thus the developers), but I
cannot press my interaction with Qubes into what little I grasp from
salt: the frequent upgrade of various base templates (fedora, minimal,
whonix gw and ws, archlinux hopefully soon again), the maintenance of
VMs using flatpak user installs to isolate applications I want to have
seperate machines for AND do not want to have to install into templates
(think skype, signal, etc.) ... I'm happy to be proven wrong and if
someone can point me at a tutorial (maybe even Qubes-based) to get
started with a reasonable amount of time investment would retry.

Would there be possibilities to bring a in my experience much more
approachable ansible option closer to the core and integrate it into
the code base overseen by Invisible Things? Maybe by contracting Rudd-
O?

Thank you for your consideration.

Sincerely, Joh

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Mon, Jul 02, 2018 at 05:17:31PM +0200, Johannes Graumann wrote:
> Would there be possibilities to bring a in my experience much more
> approachable ansible option closer to the core and integrate it into
> the code base overseen by Invisible Things? Maybe by contracting Rudd-
> O?

I think yes. But someone would need to implement it. Having Ansible as
first-class citizen in Qubes requires:

1. Direct integration with Admin API / qvm-* commands / qubesadmin python
module, instead of converting ansible -> salt -> qvm-* commands.
Generally make managing VMs with Ansible independent of Salt. Admin API
allows to do all that from selected VM, instead of dom0 (as it was
before Qubes 4.0).

2. Make VM management more isolated - namely do not parse complex data
returned from managed VM. Displaying success/fail info and a text
message should be ok, but an interactive protocol is not.
Salt (namely: salt-ssh) provides a method to package all the
required configuration into a single tarball, which then can be send
and executed - this was AFAIR one of main reasons why we've chosen Salt.
But later it turned out making that tarball needs some input from "remote"
system ("grains" - things like what OS is there, various tools versions etc), so
we've added an intermediate DispVM which gets all salt configuration,
ask target VMs for "grains", then create a tarball and sends it there.
Each target VM have own DispVM for that created on demand.
This way if anything compromise the code parsing "grains" (or any
related structure), it will not gets an access to neither dom0, nor
other VMs. See relevant ticket[1] for design discussion about this.
We need something with similar properties for Ansible. If there is a
mode with uni-directional communication with target VM, it should be
enough, otherwise a similar scheme as for Salt needs to be done.

Manuel, would you be interested in working on this?

Since this isn't the first time people ask about Ansible integration, I
think we can use some of money collected on OpenCollective to help make
it happen.

[1] https://github.com/QubesOS/qubes-issues/issues/1541

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAls6TyUACgkQ24/THMrX
1yxNGQgAmYW7yd7LnJzVfDVtzxBeta/pwfnE2Px3E9Lo8+kCP1avwGb3GaFaaGbt
/ljsamBCqLTD5ExZFunHXDVtoLnI/c/GoEgRpTPv1NpP3RhsDolEr87P9dzK+YF4
v+XR0f3mev/L35k+ssHktW3MUUYserV7FLlL0bSkba2q4iYQwk+QkK0bOHuzPW+R
2S8RPE32nFS41Oq9Y/vXdyU3I1lSLlgOZJOdBUVPPUES+Nhw4vJdklWEwaEptTyA
78AdhaJcCxCVYISgkMFyepydjKq559KpD9PvG94UjCtlWNETU9eLEH3L/l8hBbIN
1AaEu2bzqq+t6ftxoP46979oWdC90g==
=WlJD
-----END PGP SIGNATURE-----

[Chris Laprise]

On 07/01/2018 07:59 PM, Marek Marczykowski-Górecki wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On Sun, Jul 01, 2018 at 03:06:06PM -0400, Jean-Philippe Ouellet wrote:
>> Previously...
>> https://groups.google.com/d/topic/qubes-devel/VimLbPHwJRw/discussion
>>> Let me be the first to admit that salt appears difficult to understand at
>>> first. Their documentation drives me up the wall. It strikes me as the kind
>>> of project that gained followers early when it was still easy to grasp, then
>>> built up an obscurantist mindset.
>>
>> https://github.com/QubesOS/qubes-issues/issues/1983
>>> I've been trying to learn saltstack for a few weeks now, but the
>>> documentation is outright novice-hostile. It spouts off about advanced
>>> options in the second or third paragraphs of each topic and rarely
>>> returns to Earth.
>>
>> I'm curious how many people actually use Salt to manage their Qubes
>> machines. If you do, I want to hear your thoughts on it.
>
> I do, to some extend (templates only). And I feel your pain. Salt is
> definitely not easy thing to learn. When you get it, it's quite
> useable, but I wouldn't call it comfortable.

Unfortunately I haven't made another effort to get into salt since Fall;
no change. But I did feel like I had made some progress at the time.

The subject probably needs someone who can explain in some detail what
the new concepts and unique challenges are at the level of a
garden-variety sysadmin or app developer.

The main thing that bothers me about salt (besides being scattershot
with files -- I hate that) is the pretentiousness. I recall some of
their examples were just finding ways to do things procedurally without
much assurance of state.

The protocol salt uses to propagate states is deemed secure and I credit
Marek's intelligence in recognizing that and being able to absorb and
apply it (on top of everything else) to Qubes.

With that said, I do still follow a textfile recipe (stored in vault vm)
approach with a smattering of maintenance/configuration scripts; I
update the recipes whenever appropriate to capture essential
configuration goals and steps. It also hasn't hurt that some template
upgrades like debian 8-9 behaved very well, saving me a lot of hassle.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[Daniel Wilcox]

Hi there!

I use Salt almost everyday for work so I can run over some high level basics, sans Qubes.  I did read through the qubes salt modules and they're a touch difficult even for me to understand (having written a couple myself), so while not an expert I do have a good amount of hands on experience with Salt.

I recently did modify the sys-usb salt recipe to replace my sys-usb with a debian based one. I can dig up my notes on that and send them your way if you are interested.

The easiest way to learn Salt is *outside* of playing with Qubes directly -- I tried to start with Salt + Qubes before my current gig and also gave up, so I totally understand your pain.  After that at least you'll have the basics and can muddle through some things.

I would still be reluctant to give up on Salt in Qubes however because of how powerful it is, having used both Ansible and Salt in production there isn't really a comparison.  If we, collectively, can improve our knowledge of Salt and documentation on how to use it in Qubes I think it would be a huge boon to automating (and sharing!) more complicated Qubes configurations.  (this is doubly the case since salt 'formulas' -- I promise to define below -- are easy to move to different environments)

Terminology

States - A recipe or set of execution steps and checks to make sure a "minion" (or host) matches a given configuration.

Pillars - Actual configuration data that is separated from the state execution itself, can also be used to modify the flow of control in a state.

Formulas - A state packaged for distribution to others, usually has some dependencies regarding pillars that must be defined, and map.jinja lookup files that really just store common config data (location of nginx pid file, name of pkg Y, etc) for distributions (debian, fedora, etc).

Minion - Host running the 'salt-minion' binary, gets its minion-id from the vault written in /etc/salt/minion_id, usually the hostname but that may have to be setup.

Top.sls - One of these exists in the top-level directory for each Salt states + pillars, it maps minion-id patterns (aka hostnames hopefully) to lists of states, or pillars.

Highstate - the states mapped to a given minion in top.sls *and* it's compiled pillar data

  -- there is a corresponding low state that has more grimey details includes hooks to the real functions being run and things that get evaluated at runtime and so may go missing from highstate if that conditional evaluates to false on a given minion

  -- double note, you can also have dependencies between states provided one includes the other *or* if they are only applied via highstate, where-in both states will have all their steps loaded at once, and so can make strict orderings (read about 'requisites' if interested)

Network of relationships:

Salt Master <-----> Salt Minion -- standard configuration, minions ask the master over a 0mq channel for states + pillars and then apply it to themselv

Host ---ssh---> Host -- salt-ssh can be used with a simple directory structure to compile and apply states over ssh (really shuffling base64 encoded python bootstrap environment into temp files and executing state data on the 'minion' side).  tldr; ansible mode

Salt Minion -- so-called master-less minions, minions have their own copy of states + pillars, run salt-minion and apply state either manually or on a cron job (I have no experience with this configuration but have read about it)

Example directory structure:

/srv/salt -- default salt states directory

/srv/pillar -- default salt pillar directory

Files: .sls files are jinja2 files that are evaluated and then loaded as yaml, .jinja files are typically used as templates ala ansible

Live Demo on Your Dom0:

Bumbling around right now in dom0 I can do this, fairly quickly, please follow along:

vi /srv/salt/hello.sls # contents follow, note spaces very carefully it is yaml

hello:

  cmd.run:

    - name: echo hello from $(hostname)

EOF

To run this state in dom0:

qubesctl state.apply hello

OK -- so I'll be the first to admit I don't have a great grasp of *why* there is so much additionally complexity in the Qubes salt config versus normal -- but to extend this to use a pillar, let's just play along with it and do this:

vi /srv/pillar/hello.sls

hello: world

EOF

vi /srv/pillar/topd/config.top (this is an edit, it is harmless to add pillar data that isn't use provided it doesn't overwrite something critical)

# vim blah blah

base:

  '*':

    - topd.config

    - hello

EOF

Unlike states which can be applied to a minion *without* being mapped in highstate -- pillars *must* be mapped before they are able to be used.  Which is what we just did, map all hosts (*) to have topd.config (whatever this does, some dynamic module config goodness) and our new 'hello' pillar.

qubesctl saltutil.refresh_pillar  # required, this is normally done periodically by the salt-master, but without one running it'll be stale -- it does just what it says, refreshes in memory pillars from those on disk

Now let's check and see it if is in there:

qubesctl pillar.get hello

# qubesctl pillar.items |grep hello or whatever works too

Finally let's update the salt state to use our new pillar data:

vi /srv/salt/hello.sls

hello:

  cmd.run:

    - name: echo hello from $(hostname) and {{ pillar.get('hello') }}

EOF

And run it:

qubesctl state.apply hello

# and you should see 'hello from $HOSTNAME and world'

This model *should* work on appvms as well, but it wasn't for me just now (using --targets 'dev,www,whatever', or --all) so more digging will be needed.  Glad to hear folks are interested in even looking at this, I've been long overdue in digging in and contributing.

Cheers!

=D

--
You received this message because you are subscribed to the Google Groups "qubes-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscribe@googlegroups.com.
To post to this group, send email to qubes...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/32c079aa-dd89-d174-7afe-c7a674e0f771%40posteo.net.

For more options, visit https://groups.google.com/d/optout.

[Kushal Das]

Over the weekend I actually thought over the problem, and wanted to have
something as close as possible to the upstream Ansible for the same.

The result is is availble at [1]. This has three major things.

1. One *qubes* connection plugin for Ansible
This allows dom0 and any domU (with proper policy) to do things
inside of a VM. Means installing packages, copy/fetch files etc.

I have also opened a PR to the upstream Ansible to add this in the
core.

2. To make 1 happen, I added a small qrexec service *qubes.Ansible*.
To do things from dom0, we only need that service in the target AppVMs
or templates. There is also a command line tool (basically service
name changed from
qvm-run-vm command) *qvm-ansible* which will be used by domU VMs to connect
and do things inside of other VMs.

3. A pure Python Ansible module (named: qubesos) to
create/destroy/manage state of the
VMs.

Now, for now I have tested point 3 only from dom0. Point was tested
from both dom0 and domU VMs.

The Python module will require a lot of other things to make it 100%
compatible with
standard qvm*/qubes-* tools.

I have added examples in the repo. I managed to ran random playbooks
(which I use
in other places) using this. I would love to have feedback on this.

Note: This does not use Salt anywhere.

[1] https://github.com/kushaldas/qubes_ansible


Kushal
--
Staff, Freedom of the Press Foundation
CPython Core Developer
Director, Python Software Foundation
https://kushaldas.in

[Johannes Graumann]

Great to see work on this! While I haven't had time (yet) to play
around with some of the salt introductions given in this thread,
actually sticking with ansible would be my preferred solution ...

Can you summarize where your approach differs from
https://github.com/Rudd-O/ansible-qubes ?

Will it be possible to have an admin domU running ansible, that also
updates/configures dom0?

Sincerely, Joh

[Kushal Das]

On Wed, Oct 17, 2018 at 3:28 AM Johannes Graumann
<nons...@graumannschaft.org> wrote:
>
>
> Great to see work on this! While I haven't had time (yet) to play
> around with some of the salt introductions given in this thread,
> actually sticking with ansible would be my preferred solution ...
>
> Can you summarize where your approach differs from
> https://github.com/Rudd-O/ansible-qubes ?
>

I am not using Salt anywhere and using the standard qubesadmin
API to create/delete and other instructions on the vm.

> Will it be possible to have an admin domU running ansible, that also
> updates/configures dom0

I personally don't know how to do that from an admin domU, can you please
tell me the steps?

Kushal

Kushal