What are the benefits and drawbacks of having user_salt as a separate env?

[Zaz Brown]

I'm working to make SALT more accessible (#8218) and ensure user SALT states are backed up by default (#8853).

It seems the user environment was introduced 10 years ago somewhere around qubes-mgmt-salt commit 1890a4e7ec77009b81faecaa25606dfee2e7633e by nrgaway. Having another env seems to add subtle complexity. What is the advantage to having a user env instead of just a user directory in the base env?

I wouldn't suggest we add multiple file_roots to the base env, because you then have the potential of colliding subdirectory names causing confusion. But why not add a /srv/salt/user/ directory? The only downside I see there would be that users would have to prefix their states with user.

Also, I have found debugging SALT to be a nightmare, with critical errors hidden under DEBUG.  Is this something Qubes-specific? E.g. because there is output in a management disp VM that I am not seeing? If so, what is the easiest way to debug SALT scripts in Qubes?

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Apr 08, 2025 at 06:18:05PM +0000, 'Zaz Brown' via qubes-devel wrote:
> I'm working to make SALT more accessible ([#8218](https://github.com/QubesOS/qubes-issues/issues/8218)) and ensure user SALT states are backed up by default ([#8853](https://github.com/QubesOS/qubes-issues/issues/8853)).
>
> It seems the user environment was introduced 10 years ago somewhere around [qubes-mgmt-salt commit 1890a4e7ec77009b81faecaa25606dfee2e7633e by nrgaway](https://github.com/QubesOS/qubes-mgmt-salt/commit/1890a4e7ec77009b81faecaa25606dfee2e7633e). Having another env seems to add subtle complexity. What is the advantage to having a user env instead of just a user directory in the base env?

Mostly not having to worry about conflicts with standard formulas
(virtual-machines and few more in more specialized cases), and all the
automation to manage merged top file (qubesctl top.{enable,disable}).
I don't think it's a huge concern, maybe avoiding the few names used
there is still easier than dealing with two envs?

> I wouldn't suggest we add multiple file_roots to the base env, because you then have the potential of [colliding subdirectory names causing confusion](https://docs.saltproject.io/en/latest/ref/configuration/master.html#file-roots). But why not add a /srv/salt/user/ directory? The only downside I see there would be that users would have to prefix their states with user.

Actually, there are already multiple file_roots for the base env, see
/etc/salt/minion.d/*.conf. Adding another one should be fine IMO.
Anyway, I'd rather avoid /srv/salt specifically, as this one is covered
by the magic top management (see /srv/salt/top.sls), not exactly a thing
for manual user changes.

> Also, I have found debugging SALT to be a nightmare, with critical errors hidden under DEBUG. Is this something Qubes-specific? E.g. because there is output in a management disp VM that I am not seeing? If so, what is the easiest way to debug SALT scripts in Qubes?

That's kinda my experience too, also outside of qubes. If something
breaks, I usually add `-l all` and then search through pages and pages
of output...
As for dispvm, yes, there are also cases where some output may not be
retrieved, I think internally salt makes some call with `-l quiet` or
such. I don't think I needed such logs often, but when I do, I
usually comment out `dispvm.kill()` line in qubessalt/__init__.py and
then call relevant salt-ssh command manually (as seen in
/etc/qubes-rpc/qubes.SaltLinuxVM). Far from ideal...

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmf3HIEACgkQ24/THMrX
1yzBrggAh1ye0txzUxp1cT+dLKwSQ+tKD/hu5BaRnnobise58Vv0/GJnXGkitbpn
3gxIzc4WVFLyM4QeXFvvpMimw6qNEqxtYAXlcNCpLpsIQeaXv34ie1o8+g7kXX4m
Ig9NPUjJ2RGenx3MJvhDlDq+Tu9QH9vTnwpWSMzmAtVGVCe9ad2OhFk1UPZWfLN9
7tJ4FZaWJwzn4qiXbL9O3fT9gYcRB2N3QbYqyFdM8q7DUEupwhc7LRtuMgY6IPhh
2aH8ieSlVE985s+fgGZedjpd0JPAkOvFomgM9eWFUYIQhJv5BijWExJj0D4vReLt
9kE75rV8biGqtDB6H+ICXqj6RLbnuw==
=o15z
-----END PGP SIGNATURE-----

[Zaz Brown]

> Mostly not having to worry about conflicts with standard formulas
> (virtual-machines and few more in more specialized cases), and all the
> automation to manage merged top file (qubesctl top.{enable,disable}).
> I don't think it's a huge concern, maybe avoiding the few names used
> there is still easier than dealing with two envs?

If we were to use two file_roots, I see the concern about conflicts. But
I was proposing simply a user directory: /srv/salt/user/ That should
result in no conflicts, while still allowing the user states to utilize
all the stuff available in the base environment.

As for qubesctl top.enable, are you saying that a new env allows us to:

qubesctl top.enable myState saltenv=user

and that's convenient? Wouldn't it be straightforward to change the
above to work as so (if we have /srv/salt/user/):

qubesctl top.enable user.myState

> Actually, there are already multiple file_roots for the base env, see
> /etc/salt/minion.d/*.conf. Adding another one should be fine IMO.
> Anyway, I'd rather avoid /srv/salt specifically, as this one is covered
> by the magic top management (see /srv/salt/top.sls), not exactly a thing
> for manual user changes.

I only see one file_root on my machine and on GitHub: /srv/salt.
Although there are two base pillar_roots: /srv/pillar and
/srv/pillar/base, but that seems very unusual. If I understand
correctly, that means anything in /srv/pillar/base can be accessed as
myfile OR base.myfile. My instinct is that this is not necessary and has
potential to cause confusion down the line.

> That's kinda my experience too, also outside of qubes. If something
> breaks, I usually add `-l all` and then search through pages and pages
> of output...
> As for dispvm, yes, there are also cases where some output may not be
> retrieved, I think internally salt makes some call with `-l quiet` or
> such. I don't think I needed such logs often, but when I do, I
> usually comment out `dispvm.kill()` line in qubessalt/__init__.py and
> then call relevant salt-ssh command manually (as seen in
> /etc/qubes-rpc/qubes.SaltLinuxVM). Far from ideal...

Is there any possibility of transitioning Qubes to an alternative, such
as Ansible? It seems to be much larger with more momentum and easier to
get started with, which is crucial to appeal to an audience who are
technically savvy, but not professional sysadmins.

I see we have https://github.com/Rudd-O/ansible-qubes by Rudd-O and
https://github.com/fepitre/qubes-ansible by Kushas Das and fepitre.

--

[Marek Marczykowski-Górecki]

On Thu, Apr 17, 2025 at 10:32:05AM +0000, Zaz Brown wrote:
>
> > Mostly not having to worry about conflicts with standard formulas
> > (virtual-machines and few more in more specialized cases), and all the
> > automation to manage merged top file (qubesctl top.{enable,disable}).
> > I don't think it's a huge concern, maybe avoiding the few names used
> > there is still easier than dealing with two envs?
>
> If we were to use two file_roots, I see the concern about conflicts. But
> I was proposing simply a user directory: /srv/salt/user/ That should
> result in no conflicts, while still allowing the user states to utilize
> all the stuff available in the base environment.
>
> As for qubesctl top.enable, are you saying that a new env allows us to:
>
> qubesctl top.enable myState saltenv=user
>
> and that's convenient? Wouldn't it be straightforward to change the
> above to work as so (if we have /srv/salt/user/):
>
> qubesctl top.enable user.myState

Honestly, for states that user create manually I find it easier to
simply edit the actual top file (top.sls) directly, instead of doing it
in two steps: creating new .top file and then enabling it. But if you
have more modular setup, the top.enable may be useful.

> > Actually, there are already multiple file_roots for the base env, see
> > /etc/salt/minion.d/*.conf. Adding another one should be fine IMO.
> > Anyway, I'd rather avoid /srv/salt specifically, as this one is covered
> > by the magic top management (see /srv/salt/top.sls), not exactly a thing
> > for manual user changes.
>
> I only see one file_root on my machine and on GitHub: /srv/salt.

Look closer, see files in /etc/salt/minion.d/*.conf. Or simply ask salt
about the final value:

qubesctl pillar.get master:file_roots

> Although there are two base pillar_roots: /srv/pillar and
> /srv/pillar/base, but that seems very unusual. If I understand
> correctly, that means anything in /srv/pillar/base can be accessed as
> myfile OR base.myfile. My instinct is that this is not necessary and has
> potential to cause confusion down the line.

Indeed, looks like a mistake. But removing it now will likely break some
systems, as users may rely on one or the other already...

> > That's kinda my experience too, also outside of qubes. If something
> > breaks, I usually add `-l all` and then search through pages and pages
> > of output...
> > As for dispvm, yes, there are also cases where some output may not be
> > retrieved, I think internally salt makes some call with `-l quiet` or
> > such. I don't think I needed such logs often, but when I do, I
> > usually comment out `dispvm.kill()` line in qubessalt/__init__.py and
> > then call relevant salt-ssh command manually (as seen in
> > /etc/qubes-rpc/qubes.SaltLinuxVM). Far from ideal...
>
> Is there any possibility of transitioning Qubes to an alternative, such
> as Ansible? It seems to be much larger with more momentum and easier to
> get started with, which is crucial to appeal to an audience who are
> technically savvy, but not professional sysadmins.
>
> I see we have https://github.com/Rudd-O/ansible-qubes by Rudd-O and
> https://github.com/fepitre/qubes-ansible by Kushas Das and fepitre.

Yes, Ansible is definitely easier to use. And yes, there is a plan to
add official Ansible support. But I don't have any specific timeline for
that yet (I hope we'll squeeze it into R4.3, but we'll see).
But Salt will also remain there, at least for now.

[Zaz Brown]

> Yes, Ansible is definitely easier to use. And yes, there is a plan to
> add official Ansible support. But I don't have any specific timeline for
> that yet (I hope we'll squeeze it into R4.3, but we'll see).
> But Salt will also remain there, at least for now.

Ah. So the plan is to gradually migrate to Ansible?

In that case, would it make sense to drop my plans to improve SALT and
instead focus on testing and setting up states for the new Ansible?

If so, it might be worth publicizing the move more widely, because there
seems to be a vague understanding among community developers that SALT
is the "proper" way to do things. E.g. Arkenoi's Liteqube fork states
translating everything to SALT as a pre-1.0 goal.
https://github.com/arkenoi/liteqube

I'm sure I'm not the only dev out there who's been getting frustrated
playing around with SALT and would be more than willing to test Ansible.

[unman]

On Fri, Apr 18, 2025 at 10:36:13AM +0000, 'Zaz Brown' via qubes-devel wrote:

> If so, it might be worth publicizing the move more widely, because there
> seems to be a vague understanding among community developers that SALT
> is the "proper" way to do things. E.g. Arkenoi's Liteqube fork states
> translating everything to SALT as a pre-1.0 goal.
> https://github.com/arkenoi/liteqube

In the current state of things salt IS the "proper" way, and will
remain so for some time.

> I'm sure I'm not the only dev out there who's been getting frustrated
> playing around with SALT and would be more than willing to test Ansible.

It would help if you explained what the source of your frustration is,
and so what could be improved. In my experience, most users are able to
get a grip on using salt fairly quickly, and I dont see why "devs"
should find this more difficult.

--
I never presume to speak for the Qubes team.
When I comment in the mailing lists I speak for myself.

[Ken Rosen]

Should those of us looking to update or plan for an ansible migration look to the new tool here: https://forum.qubes-os.org/t/ansible-in-qubes-os/33309

[Zaz Brown]

> In my experience, most users are able to get a grip on using salt fairly quickly

I suspect you have a biased sample or that said users are not
experimenting as heavily as I am. But I am not experimenting that
heavily: I'm doing things like trying to copy files with files.recurse,
and change file_roots to a directory under /home/user.

> It would help if you explained what the source of your frustration is,
> and so what could be improved.

Firstly, there are some Qubes issues, such as files.recurse permission
copy and changing file_roots to somewhere outside of /srv not being
supported. But the main issue is the error messages. As Marek has also
experienced, often when SALT fails, no error message is displayed. When
I did get a stack trace, it was often followed by "During handling of
the above exception, another exception occurred".

As for what could be improved with Qubes, I think we have a lot of
things going on that don't need to be going on and it causes confusion.
Some super simple features are, IMO, obfuscated. For example, to enable a
top file, you just drop it (or a link to it) in /srv/salt/_tops/base or
/srv/salt/_tops/user. To add user states, you simply drop them in
/srv/user_salt. That's all great! But then instead of just using those,
we have top.enable and state.sls qubes.user-dirs. Those two commands
together, along with issue #9885 caused some people, myself included
initially, to think that user_dirs should be enabled.

The addition of SALT environments, along with how they are set up to
work with Qubes and the lack of documentation about such, is also
confusing. I would expect if we have two different environments, that
they would be totally separate and that adding saltenv=user would run
only the user states, saltenv=base would run only the base states, etc.
In reality, it seems that running all envs and running with saltenv=base
will both pull in user SALT states enabled by top.enable but not user
SALT states defined in /srv/user_salt/top.sls, whereas running with
saltenv=user will only look at user's top.sls file, but not the user
states enabled by top.enable.

It seems that because /srv/salt/top.sls pulls in files from
/srv/user_salt/, the environments get mixed and we end up with issues
like #9916 where modifying the user env causes the base env to fail with
error 20 and no error message. I suspect its because of the same error as
in #9985, but its hard to tell because there's no error message.

Another random issue is that top.disable <state that does not exist>
throws a stack trace. It even sometimes throws a stack trace when the
state does exist. Issue #9915.

The real question though is should developers target Ansible? Why keep
targeting SALT if we are hoping to slowly phase it out? Or is the Qubes'
team of two minds as to whether we're moving to Ansible or not and want
to try it out a bit before committing?


#9885: https://github.com/QubesOS/qubes-issues/issues/9885
#9915: https://github.com/QubesOS/qubes-issues/issues/9915
#9916: https://github.com/QubesOS/qubes-issues/issues/9916

--