saltstack: user specific pillars in qubes

[lik...@gmx.de]

Hi!

What's the correct way to use user specific pillars in qubes salt stack?

My pillars are located in
/srv/salt/user_pillar/

To enable during a run them I'm running:
sudo qubesctl --show-output state.highstate --pillar-root=/srv/salt/user_pillar/

This works quite well, but when I'm running it towards a target:
sudo qubesctl --show-output --target myTarget state.highstate --pillar-root=/srv/salt/user_pillar/

I receive this error:
ERROR (exception list index out of range)


Any ideas to investigate? Should pillar files be enabled in a different way?


Thanks in advance! P.

[hut7no]

I personally have pillars in /srv/pillar/tops_d and added the top file /srv/pillar/_tops/base/tops_d.top.
The top file includes relative paths from /srv/pillar/ with a dot instead of a slash:
base:
'*':
- tops_d.statefile1
- tops_d.statefile2
- tops_d.statefile3
- tops_d.statefile4

I do not use any commandline arguments specifying pillars, just {% salt['pillar.get']('pillar_variable') %} from the statefiles in /srv/salt.

[unman]

Have you enabled the pillar in /srv/salt/user_pillar/top.sls?

Example:

In /srv/salt/user_pillar/top.sls I have a stanza:
user:
'target':
- get

In /srv/salt/user_pillar/get.sls, I set the pillar:
host: www.qubes-os.org

Then I have the state and top files, get.sls and get.top:
get file:
cmd.run:
- name: wget {{pillar['host'] }}

base:
target:
- get


qubesctl --skip-dom0 --targets=target state.apply get

[lik...@gmx.de]

>>
>
> Have you enabled the pillar in /srv/salt/user_pillar/top.sls?
>
> Example:
>
> In /srv/salt/user_pillar/top.sls I have a stanza:
> user:
> 'target':
> - get
>
> In /srv/salt/user_pillar/get.sls, I set the pillar:
> host: www.qubes-os.org
>
> Then I have the state and top files, get.sls and get.top:
> get file:
> cmd.run:
> - name: wget {{pillar['host'] }}
>
> base:
> target:
> - get
>
>
> qubesctl --skip-dom0 --targets=target state.apply get
>

First of all I'd like to thank you for your answer unman.

I've tried your example (and after modifying it to be able to run it) I get exact the same error as before.

Here's the setup:

> Have you enabled the pillar in /srv/salt/user_pillar/top.sls?

As far as I know, I have only to enable state.top files. But I have a file like you suggest located in /srv/salt/user_pillar/:

> In /srv/salt/user_pillar/top.sls I have a stanza:
> user:
> 'target':
> - get

Ok!

> In /srv/salt/user_pillar/get.sls, I set the pillar:
> host: www.qubes-os.org

Ok!

>/srv/salt/user_salt/get.sls

> get file:
> cmd.run:
> - name: wget {{pillar['host'] }}

Ok!

>/srv/salt/user_salt/get.top
> base:
> target:
> - user_salt.get

This one I had to modify as the get.sls is located in "/srv/salt/user_salt/". Above is already the modified version (user_salt is prefixed). I also enabled this top file by executing:
sudo qubesctl top.enable user_salt.get

Now, let's run this:

> qubesctl --skip-dom0 --targets=target state.apply get

When I'm executing this, I get the error logged in /var/log/qubes/mgmt-target.log:
calling 'state.apply'...
2020-09-13 10:08:14,729 output: target:
2020-09-13 10:08:14,730 output: - Rendering SLS 'base:user_salt.get' failed: Jinja variable 'salt.utils.context.NamespacedDictWrapper object' has no attribute 'host'

When I'm executing this (additional parameter --pillar-root=/srv/salt/user_pillar/):
sudo qubesctl --skip-dom0 --targets=target state.apply --pillar-root=/srv/salt/user_pillar/

The result is the same as I described before:
target: ERROR (exception list index out of range)

I also tried to modify the pillar top file to meet the different location:

> In /srv/salt/user_pillar/top.sls I have a stanza:
> user:
> 'target':

> - user_pillar.get

But this haven't changed the results.

Any further suggestions to investigate are highly appreciated.

Best. P.

[lik...@gmx.de]

Thanks hut7no.

I'm trying to find a solution using user defined directories as it's suggested in /srv/salt/qubes/user-dirs.sls:
##
# qubes.user-dirs
# ===============
#
# Install and maintain user salt and pillar directories for personal state
# configurations:
#
# Includes a simple locale state file
#
# User defined scripts will not be removed on removal of qubes-mgmt-salt
# by design nor will they be modified on any updates, other than permissions
# being enforced.
#
# Execute:
# --------
# qubesctl state.sls qubes.user-dirs
#
# Note:
# Using using custom ID's to prevent possible conflicts
##

But if I'll not manage to finding this out, I'll try your suggestion.

[unman]

OK, it might have been better if you had NOT modified my example, as you
should then have been able to verify that user pillars can be referenced
in this way.
I always find it better to take one small step at a time when
troubleshooting or trying to learn something new.

In the new case, you are using the user environment. I would try the
following:
/srv/salt/user_salt/get.top
user:
target:
- get

/srv/salt/user_salt/get.sls
get file:
cmd.run:
- name: wget {{pillar['host'] }}

qubesctl --skip-dom0 --targets=target state.apply saltenv=user get


You might like to look at:
https://docs.saltstack.com/en/latest/ref/states/top.html#environments

New environments can be defined in
`/etc/salt/minions.d/f_defaults.conf`, but I rarely do this in Qubes.

[lik...@gmx.de]

>
> OK, it might have been better if you had NOT modified my example, as you
> should then have been able to verify that user pillars can be referenced
> in this way.
> I always find it better to take one small step at a time when
> troubleshooting or trying to learn something new.
>
> In the new case, you are using the user environment. I would try the
> following:
> /srv/salt/user_salt/get.top
> user:
> target:
> - get
>
> /srv/salt/user_salt/get.sls
> get file:
> cmd.run:
> - name: wget {{pillar['host'] }}
>
> qubesctl --skip-dom0 --targets=target state.apply saltenv=user get
>
>
> You might like to look at:
> https://docs.saltstack.com/en/latest/ref/states/top.html#environments
>
> New environments can be defined in
> `/etc/salt/minions.d/f_defaults.conf`, but I rarely do this in Qubes.
>

unman, as always you pointed me into the right direction. The whole reason it was not working as I missed up with the directories.

Instead of using:
/srv/user_salt

I was using:
/srv/salt/user_salt

That's why your first example didn't work in my environment and I had to adjust it.

Now everything is running perfectly with user defined pillars!

Thank you very much!

[David Hobach]

I also got this running, thanks!

However when I run

qubesctl --all --show-output state.highstate saltenv=user

I get for my states:
State 'qvm.present' was not found in SLS 'mysls'
Reason: 'qvm.present' is not available.

I guess the custom qvm.* states are not available in the /srv/user_salt folder, but only in /srv/salt?

Is there any way to make them available in /srv/user_salt as well?

Side Note: I ran `qubesctl --show-output state.sls qubes.user-dirs` to obtain the folders.

[David Hobach]

P.S.: I also noticed that right after I did
sudo qubesctl --all --show-output state.highstate saltenv=user
I get
```
sudo qubesctl top.enabled
[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable
'top.enabled' is not available.
DOM0 configuration failed, not continuing
```
(It did work right before.) It doesn't change after a reboot, only reinstalling the salt packages helps.
I guess that's not normal?

[David Hobach]

On 10/20/20 9:06 PM, David Hobach wrote:
>>> Thank you very much!
>>
>> I also got this running, thanks!
>>
>> However when I run
>>
>> qubesctl --all --show-output state.highstate saltenv=user
>>
>> I get for my states:
>> State 'qvm.present' was not found in SLS 'mysls'
>> Reason: 'qvm.present' is not available.
>>
>> I guess the custom qvm.* states are not available in the /srv/user_salt folder, but only in /srv/salt?
>>
>> Is there any way to make them available in /srv/user_salt as well?
>>
>> Side Note: I ran `qubesctl --show-output state.sls qubes.user-dirs` to obtain the folders.

For reference I got it running with symlinks this way:

#!/bin/bash
#
# Run the salt configuration of _this_ folder in dom0.
#
# Assumes that you have `user_[formulas|pillar|salt]` directories in _this_ folder.
#
# NOTE: If even `sudo qubesctl top.enabled` failed for you, you can try re-installing `qubes-mgmt-salt-* salt salt-minion`
# (first via `sudo qubes-dom0-update`, then via `sudo dnf reinstall`.

set -e -o pipefail

[[ "$(whoami)" != "root" ]] && >&2 echo "This script must be run as root." && exit 1

#path of this directory (hopefully...)
SCRIPT_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"

#saltModSymlink [target]
function saltModSymlink {
local target="$1"
local tpath="/srv/user_salt/$target"
rm -f "$tpath"
ln -s "/srv/salt/$target" "$tpath"
}

#create user_ symlinks @/srv/ for the saltenv=user (explicitly configured by Qubes OS)
echo "Creating user_ symlinks in /srv/..."
for file in "$SCRIPT_DIR"/* ; do
if [ -d "$file" ] && [[ "$file" == *"user_"* ]] ; then
target="/srv/${file##*/}"

#remove previous instances & update new
rm -f "$target"
ln -s "$file" "$target"
fi
done

#create module symlinks
echo "Creating Qubes module symlinks..."
saltModSymlink "_grains"
saltModSymlink "_modules"
saltModSymlink "_pillar"
saltModSymlink "_states"
saltModSymlink "_utils"

#sync modules (we just added some via the symlinks above)
#echo "Syncing modules..."
#qubesctl saltutil.sync_all saltenv=user

#call
if [ $# -gt 0 ] ; then
echo "Calling qubesctl saltenv=user with your arguments..."
qubesctl --show-output "$@" saltenv=user
else
echo "Using qubesctl to apply the top.sls state..."
qubesctl --show-output state.apply saltenv=user
fi

> P.S.: I also noticed that right after I did
> sudo qubesctl --all --show-output state.highstate saltenv=user
> I get
> ```
> sudo qubesctl top.enabled
> [CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable
> 'top.enabled' is not available.
> DOM0 configuration failed, not continuing
> ```
> (It did work right before.) It doesn't change after a reboot, only reinstalling the salt packages helps.
> I guess that's not normal?

With the above & a fresh reinstall, this didn't happen anymore. It's odd anyway that it happened.