Some elements of saltext.vault configuration appears to be ignored?
20 views
Skip to first unread message
Steve Scotter
Jun 21, 2024, 7:02:32 AM (8 days ago) Jun 21
to Salt-users
Advance warning, I'm a salt and vault newbie, this may be a ID10T error. I full expect this to be a syntax issue in my vault configuration. Any assistance will be gratefully received!
Apologies in advance if this isn't the correct place to post requests for assistance with new vault extension. If that's the case I'd be grateful if you could point me in the correct direction. Thank you.
I have successfully gotten salt/vault integration using Token based authentication, but I'm really struggling to get it working using the recommended AppRole method.
It appears that my salt-master is ignoring the "mount" and "policies" configuration options.
Here's my redacted config
vault:
auth:
method: approle
mount: salt-master-approle
role_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
secret_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
issue:
type: approle
approle:
mount: salt-minions-approle
server:
url: https://some.internal.host.com:8200
verify: /etc/ssl/certs/ca-certificates.crt
policies:
- salt-minions
------------------------------------------------------------------
My vault has two approles enabled at /auth/salt-master-approle and /auth/salt-minions-approle
I created a role and secret ID within /auth/salt-master-approle for use by the salt-master.
When I try and fetch a secret from a salt-minion I can see in the vault audit logs that the salt-master is making a request to auth/approle/login rather than auth/salt-master-approle/login as configured.
Entry from vault's audit log...
{"auth":{"token_type":"default"},"error":"permission denied","request":{"data":{"role_id":"hmac-sha256:REDACTED","secret_id":"hmac-sha256:REDACTED"},"id":"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX","namespace":{"id":"root"},"operation":"update","path":"auth/approle/login","remote_address":"127.0.0.1","remote_port":38802},"time":"2024-06-21T08:47:36.935378548Z","type":"request"}
The following error is returned to the salt-minion.
[ERROR ] Failed to get Vault connection from master! An error was returned: VaultPermissionDeniedError: permission denied
[ERROR ] Failed to read secret! CommandExecutionError: {'error': 'VaultPermissionDeniedError: permission denied'}
Error running 'sdb.get': {'error': 'VaultPermissionDeniedError: permission denied'}
The documentation available at https://salt-extensions.github.io/saltext-vault/topics/basic_configuration.html#complete-examples suggests the mount key should be at the same indentation as the method key (which it is in my config). I wondered if the hyptens in the value was causing the problem (clutching at straws) but changing the auth->mount value to testtest, restarting the salt-master service and attempting to fetch a secret from a salt minion still resulted in a request to /auth/approle/.
As a troubleshooting step I enabled a approle at /auth/approle/, generated a role and secret within that approle, updated my config and restarted the salt-master service. Doing so got me a bit further. The next time I requested a secret from the salt minion it returned the following error
[ERROR ] Failed to get secret metadata VaultPermissionDeniedError: preflight capability check returned 403, please ensure client's policies grant access to path "secrets/"
[ERROR ] Failed to get secret metadata VaultPermissionDeniedError: preflight capability check returned 403, please ensure client's policies grant access to path "secrets/"
The entry in the audit log looked as follows.
{"auth":{"accessor":"hmac-sha256:REDACTED","client_token":"hmac-sha256:REDACTED","display_name":"salt-minions-approle","entity_id":"XXXXXXXX-XXXX-XXXX-XXX-XXXXXXXXXXXX","metadata":{"role_name":"minion-name","saltstack-jid":"20240621093014556744","saltstack-minion":"minion-name","saltstack-user":"salt"},"policies":["default","salt_minion"],"policy_results":{"allowed":false},"remaining_uses":10,"token_policies":["default","salt_minion"],"token_issue_time":"2024-06-21T10:30:15+01:00","token_ttl":60,"token_type":"service"},"error":"1 error occurred:\n\t* permission denied\n\n","request":{"client_id":"XXXXXXXX-XXXX-XXXX-XXX-XXXXXXXXXXXX","client_token":"hmac-sha256:REDACTED","client_token_accessor":"hmac-sha256:REDACTED","id":"XXXXXXXX-XXXX-XXXX-XXX-XXXXXXXXXXXX","namespace":{"id":"root"},"operation":"read","path":"secrets","remote_address":"X.X.X.X","remote_port":56042},"response":{"data":{"error":"hmac-sha256:REDACTED"}},"time":"2024-06-21T09:30:15.39546514Z","type":"response"}
You can see it was using the default and salt_minion policies, even though I've configured it to use a salt-minions policy.
Reviewing the documentation again I found the config syntax appears to have changed from the old internal vault plugin to the new vault extension (https://salt-extensions.github.io/saltext-vault/topics/basic_configuration.html#policies)
so I replaced this ...
policies:
- salt-minions
with this
policies:
assign:
- salt-minions
and restarted the salt-master. However, after retrying to get a secret from the salt minion I could see it's still attempting to use a policy called salt_minion.
As a further troubleshooting step in Vault I duplicated my salt-minions policy to a policy called salt_minion. Doing so allowed the salt minion to successfully fetch the secret.
I have successfully gotten salt/vault integration using Token based authentication, but I'm really struggling to get it working using the recommended AppRole method.
It appears that my salt-master is ignoring the "mount" and "policies" configuration options.
Here's my redacted config
vault:
auth:
method: approle
mount: salt-master-approle
role_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
secret_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
issue:
type: approle
approle:
mount: salt-minions-approle
server:
url: https://some.internal.host.com:8200
verify: /etc/ssl/certs/ca-certificates.crt
policies:
- salt-minions
------------------------------------------------------------------
My vault has two approles enabled at /auth/salt-master-approle and /auth/salt-minions-approle
I created a role and secret ID within /auth/salt-master-approle for use by the salt-master.
When I try and fetch a secret from a salt-minion I can see in the vault audit logs that the salt-master is making a request to auth/approle/login rather than auth/salt-master-approle/login as configured.
Entry from vault's audit log...
{"auth":{"token_type":"default"},"error":"permission denied","request":{"data":{"role_id":"hmac-sha256:REDACTED","secret_id":"hmac-sha256:REDACTED"},"id":"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX","namespace":{"id":"root"},"operation":"update","path":"auth/approle/login","remote_address":"127.0.0.1","remote_port":38802},"time":"2024-06-21T08:47:36.935378548Z","type":"request"}
The following error is returned to the salt-minion.
[ERROR ] Failed to get Vault connection from master! An error was returned: VaultPermissionDeniedError: permission denied
[ERROR ] Failed to read secret! CommandExecutionError: {'error': 'VaultPermissionDeniedError: permission denied'}
Error running 'sdb.get': {'error': 'VaultPermissionDeniedError: permission denied'}
The documentation available at https://salt-extensions.github.io/saltext-vault/topics/basic_configuration.html#complete-examples suggests the mount key should be at the same indentation as the method key (which it is in my config). I wondered if the hyptens in the value was causing the problem (clutching at straws) but changing the auth->mount value to testtest, restarting the salt-master service and attempting to fetch a secret from a salt minion still resulted in a request to /auth/approle/.
As a troubleshooting step I enabled a approle at /auth/approle/, generated a role and secret within that approle, updated my config and restarted the salt-master service. Doing so got me a bit further. The next time I requested a secret from the salt minion it returned the following error
[ERROR ] Failed to get secret metadata VaultPermissionDeniedError: preflight capability check returned 403, please ensure client's policies grant access to path "secrets/"
[ERROR ] Failed to get secret metadata VaultPermissionDeniedError: preflight capability check returned 403, please ensure client's policies grant access to path "secrets/"
The entry in the audit log looked as follows.
{"auth":{"accessor":"hmac-sha256:REDACTED","client_token":"hmac-sha256:REDACTED","display_name":"salt-minions-approle","entity_id":"XXXXXXXX-XXXX-XXXX-XXX-XXXXXXXXXXXX","metadata":{"role_name":"minion-name","saltstack-jid":"20240621093014556744","saltstack-minion":"minion-name","saltstack-user":"salt"},"policies":["default","salt_minion"],"policy_results":{"allowed":false},"remaining_uses":10,"token_policies":["default","salt_minion"],"token_issue_time":"2024-06-21T10:30:15+01:00","token_ttl":60,"token_type":"service"},"error":"1 error occurred:\n\t* permission denied\n\n","request":{"client_id":"XXXXXXXX-XXXX-XXXX-XXX-XXXXXXXXXXXX","client_token":"hmac-sha256:REDACTED","client_token_accessor":"hmac-sha256:REDACTED","id":"XXXXXXXX-XXXX-XXXX-XXX-XXXXXXXXXXXX","namespace":{"id":"root"},"operation":"read","path":"secrets","remote_address":"X.X.X.X","remote_port":56042},"response":{"data":{"error":"hmac-sha256:REDACTED"}},"time":"2024-06-21T09:30:15.39546514Z","type":"response"}
You can see it was using the default and salt_minion policies, even though I've configured it to use a salt-minions policy.
Reviewing the documentation again I found the config syntax appears to have changed from the old internal vault plugin to the new vault extension (https://salt-extensions.github.io/saltext-vault/topics/basic_configuration.html#policies)
so I replaced this ...
policies:
- salt-minions
with this
policies:
assign:
- salt-minions
and restarted the salt-master. However, after retrying to get a secret from the salt minion I could see it's still attempting to use a policy called salt_minion.
As a further troubleshooting step in Vault I duplicated my salt-minions policy to a policy called salt_minion. Doing so allowed the salt minion to successfully fetch the secret.
This seems to confirm to me that the defined configuration isn't being respected.
As I said at the top, I fully expect this to be config syntax issue on my part.
# salt --versions-report
Salt Version:
Salt: 3007.1
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.4
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.16.0
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Extensions:
saltext.vault: 1.0.0
Salt Package Information:
Package Type: onedir
System Versions:
dist: ubuntu 24.04 noble
locale: utf-8
machine: x86_64
release: 6.8.0-31-generic
system: Linux
version: Ubuntu 24.04 noble
Many thanks
Steve
As I said at the top, I fully expect this to be config syntax issue on my part.
# salt --versions-report
Salt Version:
Salt: 3007.1
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.4
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.16.0
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Extensions:
saltext.vault: 1.0.0
Salt Package Information:
Package Type: onedir
System Versions:
dist: ubuntu 24.04 noble
locale: utf-8
machine: x86_64
release: 6.8.0-31-generic
system: Linux
version: Ubuntu 24.04 noble
Many thanks
Steve
Steve Scotter
Jun 21, 2024, 8:23:59 AM (8 days ago) Jun 21
to Salt-users
Hi all,
Salt Version:
Salt: 3007.1
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
I've had two additional thoughts since my original post and have chased down the rabbit holes...
- Does the salt-minion need the saltext-vault package installing?
- Does the salt-minion need any additional configuration?
In regards to saltext-vault, it occurred to me I'd installed the extension on the master, but not the minion I'm testing on (I spotted a DeprecationWarning: The 'vault' functionality in Salt has been deprecated and its functionality will be removed in version 3009.0 (Potassium) in favor of the saltext.vault Salt Extension. (https://github.com/salt-extensions/saltext-vault) message while developing some templating). Installing the saltext-vault didn't resolve my issue in regards to wrong paths being requested from vault.
It has however raised the following question.... does anyone know for certain one way or the other if it's necessary to install the saltext-vault extension on the minion? Doing so in my environment on my test minion required me to allow the minion to access https://pypi.org/ and https://files.pythonhosted.org, which is suboptimal from a security perspective.
In regards to a salt-minion configuration, I wondered if I needed to set some vault based configuration in the minion (over and above the driver, which I failed to mention in my original post I'd done in order to get it working at all).
I already had the following in /etc/salt/minion.d/myvault.conf.
myvault:
driver: vault
driver: vault
Changing it to the following and restarting the salt minion service made no difference, I could see in the vault audit logs it was still using the default and salt_minion policies
myvault:
driver: vault
policies:
- salt-minions
vault:
auth:
method: approle
policies:
- salt-minions
driver: vault
policies:
- salt-minions
vault:
auth:
method: approle
policies:
- salt-minions
I also tried the new saltext configuration syntax (as below), restarting the salt minion service and that too made no difference, I could see in the vault audit logs it was still using the default and salt_minion policies
myvault:
driver: vault
policies:
assign:
- salt-minions
vault:
auth:
method: approle
policies:
assign:
- salt-minions
driver: vault
policies:
assign:
- salt-minions
vault:
auth:
method: approle
policies:
assign:
- salt-minions
Both of those above changes where a scatter gun attempt to find a working configuration, neither was successful :(
Minion version report :-
# salt-minion --versions-report
Salt Version:
Salt: 3007.1
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: 18.8.0
dist: ubuntu 22.04.4 jammy
locale: utf-8
machine: x86_64
release: 5.15.0-112-generic
system: Linux
version: Ubuntu 22.04.4 jammy
locale: utf-8
machine: x86_64
release: 5.15.0-112-generic
system: Linux
version: Ubuntu 22.04.4 jammy
Many thanks
Steve
Steve Scotter
Jun 21, 2024, 11:27:03 AM (8 days ago) Jun 21
to Salt-users
Hi all,
I've had some partial success. I've managed to resolve the fact the salt-master was ignoring the mount configuration.
The documentation at https://salt-extensions.github.io/saltext-vault/topics/basic_configuration.html says it should be
vault:
auth:
method: approle
mount: salt-master-approle
role_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
secret_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
However https://github.com/salt-extensions/saltext-vault/blob/main/docs/ref/configuration.md#approle_mount indicates it should be
vault:
auth:
method: approle
approle_mount: salt-master-approle
role_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
secret_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Using approle_mount works as expected.
I've raised a issue on the repo https://github.com/salt-extensions/saltext-vault/issues/68
Note, the salt_minion policy issue is still eluding me. Any help will be gratefully received!!
PS. It is my intent to document from beginning to end getting salt and vault working together because I've been unable to find any decent documentation on it, relying on snip bits all over the place.
Regards
Steve
I've had some partial success. I've managed to resolve the fact the salt-master was ignoring the mount configuration.
The documentation at https://salt-extensions.github.io/saltext-vault/topics/basic_configuration.html says it should be
vault:
auth:
method: approle
mount: salt-master-approle
role_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
secret_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
vault:
auth:
method: approle
approle_mount: salt-master-approle
role_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
secret_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Using approle_mount works as expected.
I've raised a issue on the repo https://github.com/salt-extensions/saltext-vault/issues/68
Note, the salt_minion policy issue is still eluding me. Any help will be gratefully received!!
PS. It is my intent to document from beginning to end getting salt and vault working together because I've been unable to find any decent documentation on it, relying on snip bits all over the place.
Regards
Steve
Steve Scotter
Jun 21, 2024, 11:43:45 AM (8 days ago) Jun 21
to Salt-users
Hi all,
issue:
type: approle
approle:
mount: salt-minions-approle
server:
url: https://some.internal.host.com:8200
verify: /etc/ssl/certs/ca-certificates.crt
metadata:
entity:
minion-id: '{minion}'
secret:
saltstack-jid: '{jid}'
saltstack-minion: '{minion}'
saltstack-user: '{user}'
policies:
assign:
Sorry for talking to myself this afternoon.. hopefully the last message on this matter for today.
In the past few minutes my minions started using the correct policy. I'm not 100% sure why, but I believe it may be due to running or more of the following commands on the master...
salt-run vault.auth_info
salt-run vault.sync_approles
salt-run vault.list_approles
salt-run vault.sync_entities
salt-run vault.list_entities
salt-run vault.cleanup_auth
salt-run vault.clear_cache
salt-run vault.sync_approles
salt-run vault.list_approles
salt-run vault.sync_entities
salt-run vault.list_entities
salt-run vault.cleanup_auth
salt-run vault.clear_cache
Here's my working configuration (/etc/salt/master.d/vault.conf)
vault:
auth:
method: approle
approle_mount: salt-master-approle
role_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
secret_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
auth:
method: approle
approle_mount: salt-master-approle
role_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
secret_id: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
issue:
type: approle
approle:
mount: salt-minions-approle
server:
url: https://some.internal.host.com:8200
verify: /etc/ssl/certs/ca-certificates.crt
metadata:
entity:
minion-id: '{minion}'
secret:
saltstack-jid: '{jid}'
saltstack-minion: '{minion}'
saltstack-user: '{user}'
policies:
assign:
- salt-minions
- saltstack/minions
- saltstack/{minion}
cache_time: 60
refresh_pillar: null
- saltstack/minions
- saltstack/{minion}
cache_time: 60
refresh_pillar: null
I hope this helps those that follow in my footsteps.
As I said earlier, it is my intent to document from beginning to end getting salt and vault working together because I've struggled for days to get this working and there seems little in the way of decent documentation available in the places I've been looking.
Have a good weekend all.
Steve
Reply all
Reply to author
Forward
0 new messages