Puppet 6 client lookup secret from Hashicorp Vault

[comport3]

Mentioned in the Puppet 6 release notes are the ability for a client to lookup secret data from Vault.

Is there any more info on how to implement this?

I have done extensive work on POC environments that use Vault as a top level in Hierarchy and mark the secrets as 'sensitive' so they do not appear in logs and reports, but do not want to continue deploying this methodology if it's not the way the technology is headed.

https://github.com/comport3/puppet5-hiera-vault-poc

[Thomas Müller]

Am Dienstag, 9. Oktober 2018 14:12:39 UTC+2 schrieb comport3:

Mentioned in the Puppet 6 release notes are the ability for a client to lookup secret data from Vault.

Is there any more info on how to implement this?

I think the press release only mentioned the ability to query something from while applying the catalog instead of on the puppetserver while compiling the catalog.

The deferred data type takes a puppet function (only ruby functions supported) that is only executed on the client while applying the catalog. So you have the ability to query vault, consul or whatever from the client without the puppetserver ever knowing the secret at all.

But the function you need to implement yourself.

- Thomas

[Thomas Müller]

Am Dienstag, 9. Oktober 2018 14:12:39 UTC+2 schrieb comport3:

from https://puppet.com/docs/puppet/6.0/using_a_deferred_function.html  :

The Forge already hosts some community modules that provide integrations with secret store, like the following:

• Azure Key Vault: works on both the master and the server

• Cyberark Conjur: works on the master

• Cyberark AIM: works on the agent

• Hashicorp Vault: works on the agent

• AWS Secrets Manager: works on the agent

but it does not directly link the modules.

[Lindsey Smith]

Apologies for not updating the docs in the last couple of days. The agent-side Vault integration lives here: https://github.com/voxpupuli/puppet-vault_lookup (coming soon to the Forge)

See also the related blog post: https://puppet.com/blog/secret-agents-man-secrets-store-integrations-puppet-6

 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/c44e5f05-fefe-40d6-90d0-4471fb33a9a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[comport3]

Hi Lindsay and Thomas,

Thanks for your documentation - I'm having some problems getting the client lookup to work.

I have the Puppetserver CA setup in Vault, and the Vault servers Puppet certificate and private key configured.

I have added the Puppetserver CA to the trusted roots, per: https://github.com/hashicorp/vault/issues/438

I have configured Vault ('auth enable cert', 'vault write auth/cert/certs/puppetserver...') successfully (or so it seems) and Vault is unlocked.

Now I can get a test lookup to work using this CURL command -

curl -X GET -H "X-Vault-Token:$VAULT_TOKEN" https://vault1.domain.com:8200/v1/secret/test

But configuring via Puppet code, I get -

Error: Failed to apply catalog: Received 403 response code from vault at vault1.domain.com for secret lookup (api errors: ["1 error occurred:\n\t* permission denied\n\n"])

Any ideas what I'm missing?

[comport3]

We now have Vault lookups working using the Deferred functionality that debuted in Puppet6x.

Here are my high level note on "how", hoping they help someone else in the future (lots of assumptions, but if you have questions please reach out...)

#export VAULT_SKIP_VERIFY=true

#mkdir /var/lib/vault

#chown puppet:puppet /var/lib/vault

#vault operator init -key-shares=1 -key-threshold=1

# (securely record keys and tokens)

#export VAULT_TOKEN=TheTokenFromAbove

#vault operator unseal

# (unseal key from above)

#vault auth enable cert

#vault write auth/cert/certs/puppetserver display_name=puppet policies=prod,test certificate=@/etc/puppetlabs/puppet/ssl/ca/ca_crt.pem ttl=3600

#vault kv put secret/test hello=world

#echo "path \"secret/test\" {capabilities = [\"read\",\"list\"]}" > test.hcl

#vault policy write test test.hcl

#  (enable vault profile::vaulttest in Hiera & deploy, test Puppet and see the KV output)

Our next challenge is how to isolate clients to see ONLY their own secrets within Vault based on the incoming name and a policy to suit (or a similar workflow that results in client isolation).

If any Vault users has any advice it would be welcome.

[Peter M Souter]

Hi All!

 

I've been working on a Vagrant repo that sets everything up: https://github.com/petems/puppet-vault-function-vagrant

> #export VAULT_SKIP_VERIFY=true

This one you can avoid by pointing to the puppet CA cert with VAULT_CACERT, means you're avoiding the cert skipping.

> Our next challenge is how to isolate clients to see ONLY their own secrets within Vault based on the incoming name and a policy to suit (or a similar workflow that results in client isolation).

Yeah, this part has been difficult so far. The only way to match lookups based on the cert doing the lookup would be with Sentinel using request.data.cert matching it to the kv path. https://www.vaultproject.io/docs/enterprise/index.html

However, you could re-write the vault_lookup function to only allow lookups with the trusted cert certname in it, handling it on the Puppet side instead, similar to the hiera logic. I've not been able to find an example of doing this, but I'm looking into it.

Also, you can restrict the policy on the Vault side based on Certificate attributes, so you could have a more general policy based on OIDs etc:

 vault write auth/cert/certs/puppetserver \

    display_name=puppet \
    policies=all_secrets \
    certificate=@/etc/vault/keys/ca_cert.pem \
    required_extensions="1.3.6.1.4.1.34380.1.1.22:vaultok" \
    ttl=3600

  cat << EOF > /etc/puppetlabs/puppet/csr_attributes.yaml
---
extension_requests:
  pp_securitypolicy: "vaultok"
EOF

[comport3]

We've made some more progress integrating Puppet 6+ Deferred lookups with Vault for secrets storage.

The basic principle we've used for the isolation is to upload and sync a Puppet TLS certificate per host, and lookup the relevant keys under there for the secret storage.

```

vault write secret/test1.exampledomain.com policies=test1.exampledomain.com certificate=@/etc/puppetlabs/puppet/ssl/ca/signed/test1.exampledomain.com.pem

vault kv put secret/test1.exampledomain.com mysql_root=TheVerySecureMySQLRootPassword123!

echo "path \"secret/test1.exampledomain.com\" {capabilities = [\"read\"]}" > test1.exampledomain.com.hcl

vault policy write test1.exampledomain.com test1.exampledomain.com.hcl

```

We can then see the above working on the client with this code -

```

$mysql_root = Deferred('vault_lookup::lookup', ["secret/test1.exampledomain.com", 'https://puppet.exampledomain.com:8200'])

notify {mysql_root: message => $mysql_root}

```

What we can't figure out is how to reference the KV pair inside a Puppet manifest as a parameter. Eg, in YAML:

```

---

mysql::server::root_password: "%{something wonderful happens here}"

```

Any ideas?

[Chadwick Banning]

If using a different Vault auth method is an option, you could use the AppRole method and define a role and policies in Vault. The Puppet agent then authenticates under a specific role (and instance of that role) that is governed by the policy.