Hiera 5 and Deferred / Sensitive

[Aaron Russo]

We're using the vault_lookup[1] module to retrieve secrets from Vault via mTLS. It works fairly well when grabbing secrets within a manifest.

However it feels like an anti-pattern by forcing lookups into our manifests when we want to keep that in Hiera. I found a previous related thread[2] where Henrik suggested writing a custom backend for Hiera and return a Deferred.

However after doing what I thought was the correct thing, and returning a Deferred in our custom backend, the value in the file ends up being the literal string 'Deferred ...' and not being evaluated. I even wrote a quick manifest to check if a Deferred is being returned by Hiera/APL and it does not seem to be the case -- Hiera is returning a String representation of it.

So my question is -- is it possible to actually return a Deferred via a Hiera lookup_key backend and if so, what might I be doing wrong? Sanitized code / outputs / etc provided[3] for mocking.

Versions:

 puppet: 7.20.0

 puppetserver: 7.8.0

 puppetlabs/stdlib: 8.30

Thanks!

Aaron

[1] https://forge.puppet.com/modules/puppet/vault_lookup

[2] https://groups.google.com/g/puppet-users/c/E-Q-ok-B0gQ/m/h-tYJFPdBwAJ

[3] https://gist.github.com/arusso/9eed3cac93e02aa270b6811b560b2093

[Aaron Russo]

So I managed to get this to work by changing hiera data slightly, replacing the `lookup` function with `alias` so that the type wasn't automatically converted to a string.

```

# this works! note the weird quoting is to avoid hiera treating fqdn dots as sub-keys and is intentional.

profile::gitlab_runner::lookup_test: "%{alias(\"'vault_lookup::kv/data/host/gitlab-runner-31.example.com/gitlab-ci>registration_token'\")}"

```

Unfortunately, this breaks down when I want to embed that key in a config hash within hiera, like this:

```

gitlab_ci_runner::runners:
  'instance':
    config:
      name: "Instance Runner on %{::hostname}"
      registration-token: "%{alias(\"'vault_lookup::kv/data/host/gitlab-runner-31.example.com/gitlab-ci>registration_token'\")}"

  ...

```

I'm guessing without some changes in Hiera/Puppet, this part may not be possible? Anyone have experience with this?

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/e5e12ede-e33f-440a-b13f-ccd221110f9dn%40googlegroups.com.

--

Aaron Russo (He/Him/His)

PIXAR | Network & Server Admins (NSA) | Senior Systems Engineer

aru...@pixar.com

[Aaron Russo]

Closing the loop on this for anyone else trying this in the future.

The custom backend was correct the entire time and was returning the Deferred function. However since I was defining a key prefix that wasn't actually used in Hiera for the custom backend to key off of, I was making use of the `lookup` function to grab perform the lookup of that key. After re-reading the docs, I realized that `lookup` will always convert the value to a string, which was exactly my problem. Switching to use the `alias` function brought me success, as it does not attempt a conversion to string if it is the only value being interpolated.

I was then able to use that value as desired, provided whatever was consuming the value accepted (and unwrapped) the Sensitive[String] that was returned from the Deferred function.

Cheers,

Aaron

[Dirk Heinrichs]

Am Freitag, dem 21.10.2022 um 11:49 -0700 schrieb Aaron Russo:

However it feels like an anti-pattern by forcing lookups into our manifests when we want to keep that in Hiera. I found a previous related thread[2] where Henrik suggested writing a custom backend for Hiera and return a Deferred.

hiera-eyaml has a plugin for retrieving secrets from Vault. Did you try that?

HTH...

Dirk

-- 

Dirk Heinrichs

Senior Systems Engineer, Delivery Pipeline

OpenText ™ Discovery | Recommind

Phone: +49 2226 15966 18

Email: dhei...@opentext.com

Website: www.recommind.de

Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach

Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, Registernummer HRB 10646

This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.

[Aaron Russo]

If you're referring to hiera-eyaml-vault, that's not pulling secrets out of Vault -- it's using the transit encryption provider in place of gpg keys and storing those in yaml. It's a neat approach but not what I'm looking for.

There is petems-hiera_vault which is close -- it retrieves secrets straight from Vault, but the puppet server is doing the retrieving and means that the server needs privileged access to all the secrets in Vault that agents' would need.

vault_lookup uses Deferred functions to have the agent authenticate and retrieve secrets from Vault, which lets me assign a policy based on the host, so it can only see the secrets it needs. It works great! I simply want that functionality in hiera.

What I've done is similar to petems-hiera_vault except I return a Deferred function to perform the vault_lookup::lookup on the agent side rather than perform the vault lookup on the server side.

Thanks,

Aaron

-- 

Aaron Russo (He/Him/His)

PIXAR | Senior Systems Engineer

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7897bf9d6301f9bad84d762de8a0e7d35dfd2572.camel%40opentext.com.

[Dirk Heinrichs]

Am Mittwoch, dem 02.11.2022 um 00:21 -0700 schrieb Aaron Russo:

There is petems-hiera_vault which is close --

Didn't know this one.

 it retrieves secrets straight from Vault, but the puppet server is doing the retrieving and means that the server needs privileged access to all the secrets in Vault that agents' would need.

Yes, of course. The server does the catalog compilation, so that's the place where the values are needed.

Bye...