Referencing Deferred functions in Puppet Code or Hiera YAML

[comport3]

Does anyone have any real world examples of referencing Deferred lookups, either in Puppet code or Hiera that you could share?

[Lindsey Smith]

On Sun, Feb 17, 2019 at 4:17 PM comport3 <comp...@gmail.com> wrote:

Does anyone have any real world examples of referencing Deferred lookups, either in Puppet code or Hiera that you could share?

Ben Ford just gave a talk on this at Config Management Camp and his slides are here: https://twitter.com/binford2k/status/1092806318501576706

I also gave a talk late last year and if you skip to the middle of the video you can see the agent-side code I used: https://www.brighttalk.com/webcast/15621/335134

Lindsey

 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/899f155f-799c-4c32-bad0-9d6f75ce6a23%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[comport3]

Thank you very much Lindsay, I will lock in some time to digest and understand this material.

Are their any Hiera YAML example? The specific use case I'm trying to solve is for 'mysql::server::root_password: <magic happens here>'.

[Lindsey Smith]

Here is an example of fetching secrets from Vault via Hiera: https://github.com/hashicorp/webinar-vault-hiera-puppet

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/746f0cf7-8e37-4d2c-9e81-f36483027ec6%40googlegroups.com.

[chengkai liang]

I don't recall that puppet had this feature but I may be wrong. What are you looking to do?

On Sun, Feb 17, 2019 at 16:17 comport3 <comp...@gmail.com> wrote:

Does anyone have any real world examples of referencing Deferred lookups, either in Puppet code or Hiera that you could share?

--

[Henrik Lindberg]

On 2019-02-18 01:17, comport3 wrote:
> Does anyone have any real world examples of referencing Deferred
> lookups, either in Puppet code or Hiera that you could share?

Vault Lookup shows examples of how to use it to lookup secrets on the
agent side.

https://github.com/voxpupuli/puppet-vault_lookup

You mention hiera as well - what were you thinking about there wrt.
Deferred? Being able to return deferred values from hiera, to be
resolved on the agent? Or something else...

Best,
- henrik

>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com

> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/899f155f-799c-4c32-bad0-9d6f75ce6a23%40googlegroups.com

> <https://groups.google.com/d/msgid/puppet-users/899f155f-799c-4c32-bad0-9d6f75ce6a23%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Grant Davies]

Yes exactly, using the deferred Function to lookup secrets on the agent side defined in Hiera. So I want to write some Puppet code in a Profile class that specifies where (Vault for us, but could be anything) to fetch the secret agent side, how, which key etc. Then reference that in Hiera for use in pre existing modules as parameters.

You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/E-Q-ok-B0gQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/q4e2bv%247pd%241%40blaine.gmane.org.

[Henrik Lindberg]

On 2019-02-19 20:15, Grant Davies wrote:
> Yes exactly, using the deferred Function to lookup secrets on the agent
> side defined in Hiera. So I want to write some Puppet code in a Profile
> class that specifies where (Vault for us, but could be anything) to
> fetch the secret agent side, how, which key etc. Then reference that in
> Hiera for use in pre existing modules as parameters.

If you want hiera to return instances of Deferred, you can do that by
writing your own simple hiera kind of backend function (much simpler
than it sounds).

All it has to do is to return a hash along the lines of:

{ some_key => Deferred('vault_lookup', [params, ...]),
some_other_key => Deferred(...)
}

The hiera 5 documentation should show how to write those functions.
Also checkout video of my Hiera 5 talk from Puppetconf and ConfigMgmntCamp.

best,
- henrik

>
> On Wed, Feb 20, 2019, 4:42 AM Henrik Lindberg
> <henrik....@puppet.com <mailto:henrik....@puppet.com> wrote:
>
> On 2019-02-18 01:17, comport3 wrote:
> > Does anyone have any real world examples of referencing Deferred
> > lookups, either in Puppet code or Hiera that you could share?
>
> Vault Lookup shows examples of how to use it to lookup secrets on the
> agent side.
>
> https://github.com/voxpupuli/puppet-vault_lookup
>
> You mention hiera as well - what were you thinking about there wrt.
> Deferred? Being able to return deferred values from hiera, to be
> resolved on the agent? Or something else...
>
> Best,
> - henrik
>
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Puppet Users" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to puppet-users...@googlegroups.com

> <mailto:puppet-users%2Bunsu...@googlegroups.com>
> > <mailto:puppet-users...@googlegroups.com
> <mailto:puppet-users%2Bunsu...@googlegroups.com>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/puppet-users/899f155f-799c-4c32-bad0-9d6f75ce6a23%40googlegroups.com
>
> >
> <https://groups.google.com/d/msgid/puppet-users/899f155f-799c-4c32-bad0-9d6f75ce6a23%40googlegroups.com?utm_medium=email&utm_source=footer>.
> > For more options, visit https://groups.google.com/d/optout.
>
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/E-Q-ok-B0gQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users...@googlegroups.com

> <mailto:puppet-users%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/q4e2bv%247pd%241%40blaine.gmane.org.
> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/CAE1da7JSO_hCiosb_5sTtAsE4qOx9K6XKzTB6C1Udj1WVKUzCA%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-users/CAE1da7JSO_hCiosb_5sTtAsE4qOx9K6XKzTB6C1Udj1WVKUzCA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[comport3]

Hi Henrik and Group,

Thank you very much, this sounds like exactly what we are after. After reviewing https://puppet.com/docs/puppet/6.3/hiera_custom_backends.html we are a little lost as to how to get started. Does anyone have any example code of using a Customer Backend or Puppet Function in conjunction with a Deferred data type lookup for us to review? Anything including the Vault lookup logic or some 'proof of concept' code would be a much welcomed starting point.

[Henrik Lindberg]

It is actually dead simple - here is an example that returns a hard
coded deferred. The example is written in the Puppet Language, and is
autoloaded just like other functions from "mymodule".

This simple example is a "data hash" kind of backend - it is called once
and is expected to return a hash with key => value bindings.

function mymodule::deferred_example(
Hash $options,
Puppet::LookupContext $context,
) {
# Return a hash with key(s) bound to Deferred value(s)
{ 'the_key' => Deferred('vault_lookup', ['the key']) }
}

Then there are lots of different things you could do.

The $options hash contains information from hiera.yaml:
- if a path/URI was given or not (and that path existed)
- any custom options given in hiera.yaml for this entry

This means you could configure what the backend would do based on either
options given directly (a list of keys for which this backend should
return a Deferred), or you can use the path to read such data from a
file, using say a function to read that file as json from the path in
options.

To use this backend simply enter its name in hiera.yaml like you do for
other backend functions.

Another alternative is to write a backend of "lookup key" kind. The
contract there is to return a value per key or that the
$context.not_found() is called (if it does not have a value for the key).

If you go this route, then the function could for example lookup a key
in hiera that holds the names of keys to lookup in a deferred way.
Or, if you design it so that all deferred parameters can be identified
via their name then you could simply return a Deferred for all keys that
match a pattern.

Read all the details starting from here:
https://puppet.com/docs/puppet/latest/hiera_custom_backends.html

Hope this helps.
- henrik

[comport3]

Hi Henrik,

Thanks for your reply.

I still don't really understand how to use your example - is it an additional custom function written in Ruby that would be synchronised from the master?

How would this be structured or implemented step by step please?

This is what's in my current class, with specifically what doesn't work commented out at the bottom (ignore my custom Vault port, please) -

class profile::vaulttest {

 $secret_lookup = Deferred('vault_lookup::lookup', ["secret/client.example.com", 'https://puppet.example.com:8228'])

  ## Works, returns a hashed array of key/value pairs, I want to look up a SPECIFIC key and it's value, eg 'mysql_root_password'

  notify {mysql_root: message => $secret_lookup}

  ## Trying to lookup a key within the array, fails with error

  ## Evaluation Error: Operator '[]' is not applicable to an Object.

  #notify {mysql_root: message => $secret_lookup[mysql_root_password]}

}

Help?! :-)

[Henrik Lindberg]

On 2019-03-08 03:13, comport3 wrote:
> Hi Henrik,
>
> Thanks for your reply.
>
> I still don't really understand how to use your example - is it an
> additional custom function written in Ruby that would be synchronised
> from the master?
>

I showed a hiera backend that returns Deferred - but it seems that is
not really what you were asking about (your example is different).

> How would this be structured or implemented step by step please?
>
> This is what's in my current class, with specifically what doesn't work
> commented out at the bottom (ignore my custom Vault port, please) -
>
> class profile::vaulttest {
>
>  $secret_lookup = Deferred('vault_lookup::lookup',
> ["secret/client.example.com", 'https://puppet.example.com:8228'])
>
>   ## Works, returns a hashed array of key/value pairs, I want to look
> up a SPECIFIC key and it's value, eg 'mysql_root_password'
>   notify {mysql_root: message => $secret_lookup}
>
>   ## Trying to lookup a key within the array, fails with error
>   ## Evaluation Error: Operator '[]' is not applicable to an Object.
>   #notify {mysql_root: message => $secret_lookup[mysql_root_password]}
>
> }
>
> Help?! :-)
>

The reason your commented out code does not work is because you are
trying to get the key "mysql_root_password" from a Deferred that has not
yet been resolved. (It will be resolved later when the catalog is
processed on the agent, but while compiling the Deferred is just an
instruction to do something later).

You need the following:

message => Deferred('get', [$secret_lookup, 'mysql_root_password'])

To get what I think you want.

- henrik

> <http://puppet-on-the-edge.blogspot.se/>
>
> --

> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/2ecccdee-e130-4f44-b6a1-b1a0c8532bbd%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/2ecccdee-e130-4f44-b6a1-b1a0c8532bbd%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.

[comport3]

Hi Henrik,

You're correct - this sample code precedes getting the Hiera function/lookup working, I want to be able to do it in Puppet code first.

I've ended up with the following -

```

 $secret_lookup = Deferred('vault_lookup::lookup', ["secret/client.example.com", 'https://puppet.example.com:8228'])

  notify {mysql_root: message => Deferred('get', [$secret_lookup, 'mysql_root_password'])}

```

It returns this error -

Error: Failed to apply catalog: 'dig' parameter 'data' expects a value of type Undef or Collection, got Sensitive[Hash]

Any ideas?

> <mailto:puppet-users+unsub...@googlegroups.com>.

[Henrik Lindberg]

On 2019-03-11 04:01, comport3 wrote:
> Hi Henrik,
>
> You're correct - this sample code precedes getting the Hiera
> function/lookup working, I want to be able to do it in Puppet code first.
>
> I've ended up with the following -
> ```
>  $secret_lookup = Deferred('vault_lookup::lookup',
> ["secret/client.example.com", 'https://puppet.example.com:8228'])
>   notify {mysql_root: message => Deferred('get', [$secret_lookup,
> 'mysql_root_password'])}
> ```
>
> It returns this error -
>

> *Error: Failed to apply catalog: 'dig' parameter 'data' expects a value
> of type Undef or Collection, got Sensitive[Hash]*
>
> Any ideas?
>

Yeah - sigh... The vault lookup returns an instance of Sensitive which
get / dig cannot dig into. We could possibly allow digging into a
Sensitive value as the result would also be made sensitive, but that
function does not do that. (Please file a ticket).

There is unfortunately no way of solving the issue with just Deferred
since unwrapping a Sensitive is done with a lambda and those cannot be
deferred. There is a proposal (and an implementation) in ticket
https://tickets.puppetlabs.com/browse/PUP-9254 where an eval() function
is added.

Another approach of getting what you want is to write your own custom
ruby function that does the unwrap and dig.

And lastly - you could avoid having to dig out a detailed value and
instead look it up separately.

- henrik

>
> On Friday, March 8, 2019 at 8:43:55 PM UTC+11, Henrik Lindberg wrote:
>
> On 2019-03-08 03:13, comport3 wrote:
> > Hi Henrik,
> >
> > Thanks for your reply.
> >
> > I still don't really understand how to use your example - is it an
> > additional custom function written in Ruby that would be
> synchronised
> > from the master?
> >
> I showed a hiera backend that returns Deferred - but it seems that is
> not really what you were asking about (your example is different).
>
> > How would this be structured or implemented step by step please?
> >
> > This is what's in my current class, with specifically what
> doesn't work
> > commented out at the bottom (ignore my custom Vault port, please) -
> >
> > class profile::vaulttest {
> >
> >   $secret_lookup = Deferred('vault_lookup::lookup',

> > ["secret/client.example.com <http://client.example.com>",

> > an email to puppet-users...@googlegroups.com <javascript:>
> > <mailto:puppet-users...@googlegroups.com <javascript:>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/puppet-users/2ecccdee-e130-4f44-b6a1-b1a0c8532bbd%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/2ecccdee-e130-4f44-b6a1-b1a0c8532bbd%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/puppet-users/2ecccdee-e130-4f44-b6a1-b1a0c8532bbd%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/puppet-users/2ecccdee-e130-4f44-b6a1-b1a0c8532bbd%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/
> <http://puppet-on-the-edge.blogspot.se/>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com

> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/d7900a19-e1c4-4ffa-8581-ef48f66192b8%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/d7900a19-e1c4-4ffa-8581-ef48f66192b8%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Karsten Heymann]

Hi,
Am Di., 12. März 2019 um 11:54 Uhr schrieb Henrik Lindberg
<henrik....@puppet.com>:

> Yeah - sigh... The vault lookup returns an instance of Sensitive which
> get / dig cannot dig into. We could possibly allow digging into a
> Sensitive value as the result would also be made sensitive, but that
> function does not do that. (Please file a ticket).

Just for completeness and in case anyone stumbling over this thread
might not know it: You always have also the option to deploy a
consul-template (https://github.com/hashicorp/consul-template) config
with puppet and then let consult-template request the secrets from the
vault instance locally on the server. What's nice about that is that
it decouples for example vault secret renewals from the actual puppet
runs.

Best regards,
Karsten

[Vincent Lamers]

isn't better not to return a Sensitive data in vault_lookup.rb (just plain text), but wrap the Deferred function with Sensitive?

Like this:

$secret_lookup = Sensitive(Deferred('vault_lookup::lookup', ["secret/client.example.com", 'https://puppet.example.com:8228']))

notify {mysql_root: 

  message => Deferred('get', [$secret_lookup.unwrap, 'mysql_root_password'])

}