New Deferred type and agent data lookups in Puppet 6

[Lindsey Smith]

Hi all,

We wanted to let you know about an upcoming capability, the Deferred type, that is now present in Puppet 6 nightlies and will be part of the Puppet 6.0 release.

A longstanding request has been to allow agents to fetch data for themselves at catalog application time. One key use case for this is getting secrets directly from a store like Conjur, Vault or Consul. Without this capability the master has to be in the middle and secret values are passed in catalogs around more than is necessary.

The solution in Puppet 6 is the Deferred type. A Deferred value describes a function call to be made in the future and when placing it in a catalog the agent will replace it with the result of calling the wrapped function before it continues with application as normal.

Of course, for the agent to actually fetch data from a keystore the function has to exist on the agent side and be loaded during a run. In Puppet 6.0, these functions will be downloaded from the master via pluginsync from the lib/puppet/functions directory in modules, then loaded during an agent run. Though Deferred is intended primarily for agents running with a master, it does work in the same way with an agent only.

https://gist.github.com/turbodog/06d3fecef403bfefd9c8174ede4d9174 has more explanation and walks you through a simple Deferred function example. Work on this is tracked in PUP-8711 and updating the Puppet specification for Deferred is a work in progress happening here: https://github.com/puppetlabs/puppet-specifications/pull/122

If you have other use cases for Deferred we’d love to hear what those are.

Lindsey

[bert hajee]

Lindsy,

Is it just ment for this use case? I can think of other situations where in might be vary valuable to fetch a value at run-time on the agent. Now whenever we have to get the current state, we need to make a fact. If we can make deferred functions for that that would make things much more simple. I'm not sure if it the still is "The Puppet way". Like to hear any thoughts on that.

Bert

[Chadwick Banning]

Would it be safe to consider this in a general context i.e. as enabling agent-side function execution?

[R.I.Pienaar]

On Fri, 31 Aug 2018, at 15:03, Chadwick Banning wrote:
> Would it be safe to consider this in a general context i.e. as enabling
> agent-side function execution?

I dont think so - for general function calls to be usable you want to get the value and then do some conditional logic on it. or put it in a variable and use it in another resource etc.

That is not what this is for, this is a based placeholder to later be replaced by the value - you cannot do any conditionals etc with it.

Imagine something like:

mysql::user{"bob":
password => Deferred(vault_lookup, "bob_pass")
}

(I am just making this syntax up, this is presumably not how it will look)

Here its fine because its a simple interpolation into a value, you cant do more complex things with this design.

Anyway thats my understanding, Henrik might chime in too

[Lindsey Smith]

On Friday, August 31, 2018 at 4:33:02 AM UTC-7, bert hajee wrote:

Lindsy,

Is it just ment for this use case? I can think of other situations where in might be vary valuable to fetch a value at run-time on the agent. Now whenever we have to get the current state, we need to make a fact. If we can make deferred functions for that that would make things much more simple. I'm not sure if it the still is "The Puppet way". Like to hear any thoughts on that.

Yes I think that's totally valid and would like to hear what cases you are thinking of for Deferred. As RI said there are some limitations but with specific uses we can see if we need to do more in this area.

[Chadwick Banning]

So for this example, there are some sort of limitations as to what the 'vault_lookup' function is able to do internally? I had just assumed that as long as the function returned a simple value, what the function does internally was open.

As an example, could Deferred be used to read and extract a value from a file agent-side?

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/DurqiLnVWMk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1535721137.3301091.1492516568.3EB7087A%40webmail.messagingengine.com.
For more options, visit https://groups.google.com/d/optout.

--

Chadwick Banning

[KevinR]

I would love to be able to query live system information during the run, instead of having to rely on facts for this in all use cases.

For example, I recently had a case where I wanted to make the code more resilient by using a Windows environment variable instead of a hardcoded path for something, but this variable wasn't available in any facts. So today, that means creating another fact for it, so you can use it in your code. It feels like unnecessary bloat to the PuppetDB to me. With the deferred function (and a function to query Windows environment variables in this case), I can do that live and not burden PE with it at all.

Another use case is resource utilization values. Creating facts for CPU or RAM usage is fairly useless as the fact in PuppetDB is out-of-date a minute later. Also since the value keeps changing every run, it unnecessarily taxes the PuppetDB, while there is little value in having the last metric in the database.

Instead, if you need to evaluate such a value in a puppet run, it would be much better to do so through a deferred function and query it during the run.

The function can also be useful for situations where only the agent is able to reach a specific resource on the network, while the Puppet master isn't. The deferred function allows that processing to happen on the agent side, bypassing the need for PE to be able to reach that network resource. This could be useful in multi-tenant network scenarios, where PE is on an isolated network, with agents on customers networks that use NAT to reach the PE master.

-KevinR

[Luke Bigum]

On Friday, 31 August 2018 16:41:34 UTC+1, Chadwick Banning wrote:

So for this example, there are some sort of limitations as to what the 'vault_lookup' function is able to do internally? I had just assumed that as long as the function returned a simple value, what the function does internally was open.

As an example, could Deferred be used to read and extract a value from a file agent-side?

In theory, you probably could.  The ruby code probably just executes on the Agent.  Personally though if I were to find myself in this situation, I'd really think about what I was trying to achieve and why I am in this situation in the first place...  Perhaps it's Puppet design that needs to be refactored, or if I really wanted to "do" something on an Agent, a Custom Type / Provider might be a better vehicle.

I can partially see the argument in proceeding posts for making decisions on run time environment data...  I'd argue that if you're writing a Deferred Type in Ruby, it's not that much further to write a Fact.  Also, unit and acceptance testing code that relies on a run time Fact seems very difficult.

[R.I.Pienaar]

On Fri, 31 Aug 2018, at 17:41, Chadwick Banning wrote:
> So for this example, there are some sort of limitations as to what the
> 'vault_lookup' function is able to do internally? I had just assumed that
> as long as the function returned a simple value, what the function does
> internally was open.
>
> As an example, could Deferred be used to read and extract a value from a
> file agent-side?
>

The function can do whatever it wants but needs to return a value and that value has to be used by a resource and not mixed in with some other strings etc. In my example deferred as the entire value to a resource property is good, deferred and then any derived data from it wont work.

You cant for example, as far as I understand it, have the function lookup a piece of data and then use that data in a template via template() and epp(). Those are processed entirely on the master.

> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/CAPwwW9GmasAV7Yz98Nq4YawYoehCuxK_JpH5iTj7sAKMRXeHXw%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.


--

R.I.Pienaar / www.devco.net / @ripienaar

[Chadwick Banning]

In general, I would agree. But enterprises often have limiting factors that make rewriting/redesign very difficult. Also, IMHO newer Puppet users can understand function creation a lot easier than building out custom resources and types.

Facts work well for a lot of use cases, but as mentioned earlier they can quickly become a dumping ground and get out of control. Also, in an enterprise environment you could find yourself needing to access sensitive data on disk thats managed by another individual or group who hasn't got around to putting things in Puppet or a secrets management solution and you'd probably want to avoid exposing the data as a Fact. Using Deferred to extract and use the data without having to know what it is would be helpful in that case.

[Henrik Lindberg]

On 2018-08-31 18:28, R.I.Pienaar wrote:
>
>
> On Fri, 31 Aug 2018, at 17:41, Chadwick Banning wrote:
>> So for this example, there are some sort of limitations as to what the
>> 'vault_lookup' function is able to do internally? I had just assumed that
>> as long as the function returned a simple value, what the function does
>> internally was open.
>>
>> As an example, could Deferred be used to read and extract a value from a
>> file agent-side?
>>
>
> The function can do whatever it wants but needs to return a value and that value has to be used by a resource and not mixed in with some other strings etc. In my example deferred as the entire value to a resource property is good, deferred and then any derived data from it wont work.
>
> You cant for example, as far as I understand it, have the function lookup a piece of data and then use that data in a template via template() and epp(). Those are processed entirely on the master.

You can have nested Deferred values, they are resolved as you expect.

Have not tried, but I think inline templates would work so you get
template rendering on the agent side.

- henrik

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Henrik Lindberg]

Your example is good, except arguments are in an array - so

password = Deferred(vault_lookup, ["bob_pass"])

Note that a Deferred can take Deferred arguments. That means that the
deferred arguments will be resolved before the Deferred they are
arguments to is resolved. It does however not resolve Deferred values
created on the agent side during the resolution process (so you cannot
build loops with this).

If wanting to do really advanced things - it is possible to write
functions that take puppet language source code as argument and
evaluates that in the agent context.

The puppet context that is available is a "scripting context" with the
same restrictions as for Bolt (you cannot evaluate catalog related
expressions).

The order among the top-level deferred (i.e. various resource attributes
containing a deferred value) is undefined. And all resolution occurs
before the catalog is applied. The puppet context used during resolution
is gone by the time the catalog application starts.

Hope that provides a glimpse into some advanced stuff that could be
implemented with this, as well as some of the constraints regarding
what you cannot do.

- henrik

[Henrik Lindberg]

On 2018-08-31 13:33, bert hajee wrote:
> Lindsy,
>
> Is it just ment for this use case? I can think of other situations where
> in might be vary valuable to fetch a value at run-time on the agent. Now
> whenever we have to get the current state, we need to make a fact. If we
> can make deferred functions for that that would make things much more
> simple. I'm not sure if it the still is "The Puppet way". Like to hear
> any thoughts on that.
>
> Bert
>

Consider a cached catalog - with Deferred values that cached catalog can
stay the same for as long as it is only the deferred values that need to
change (and that they can be obtained locally or from service).

Without use of cached catalog, Deferred values are good for things
that must or are much better suited to be computed fresh on the agent.

- henrik

> On Friday, 31 August 2018 02:11:52 UTC+2, Lindsey Smith wrote:
>
> Hi all,
>
> We wanted to let you know about an upcoming capability, the Deferred
> type, that is now present in Puppet 6 nightlies and will be part of
> the Puppet 6.0 release.
>
> A longstanding request has been to allow agents to fetch data for
> themselves at catalog application time. One key use case for this is
> getting secrets directly from a store like Conjur, Vault or Consul.
> Without this capability the master has to be in the middle and
> secret values are passed in catalogs around more than is necessary.
>
> The solution in Puppet 6 is the Deferred type. A Deferred value
> describes a function call to be made in the future and when placing
> it in a catalog the agent will replace it with the result of calling
> the wrapped function before it continues with application as normal.
>
> Of course, for the agent to actually fetch data from a keystore the
> function has to exist on the agent side and be loaded during a run.
> In Puppet 6.0, these functions will be downloaded from the master

> via pluginsync from the lib/puppet/functionsdirectory in modules,

> then loaded during an agent run. Though Deferred is intended
> primarily for agents running with a master, it does work in the same
> way with an agent only.
>
> https://gist.github.com/turbodog/06d3fecef403bfefd9c8174ede4d9174
> <https://gist.github.com/turbodog/06d3fecef403bfefd9c8174ede4d9174>has
> more explanation and walks you through a simple Deferred function
> example. Work on this is tracked in PUP-8711

> <https://tickets.puppetlabs.com/browse/PUP-8711>and updating the

> Puppet specification for Deferred is a work in progress happening
> here: https://github.com/puppetlabs/puppet-specifications/pull/122
> <https://github.com/puppetlabs/puppet-specifications/pull/122>
>
> If you have other use cases for Deferred we’d love to hear what
> those are.
>
> Lindsey
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com

> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/7c1b7417-abcd-4c17-a237-0681bc9a14bb%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/7c1b7417-abcd-4c17-a237-0681bc9a14bb%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.

[Ben Ford]

In general, you're going to want to be cautious about using agent-side functions. It might be simpler to gather information during run time in some cases, but it makes your catalog more black-box and unpredictable. Here are just a couple concerns:

You lose some of the ability to look at the facts and look at the code and know what the outcome will be because the enforcement also depends on the output of some lazy bound agent function. For example, if you have code that uses the $hostname fact, you can validate that the $hostname fact actually resolved to something useful and raise an error if not. You can even fail the compile and enforce the last cached catalog until you can fix it. If that were an agent side function, you won't have any warning before things break.

You also make your catalogs less repeatable. For example, if you push out a codebase update and it doesn't work how you want it to, then you can roll back the code, or the hiera data, or the classification as needed. You've got a pretty good expectation that your configuration will return to a working state (with obvious exceptions). You could even revert to that config version a year later. But if you depend on agent side functions, you don't have as strong an expectation because the agent state that might affect the output of the function isn't versioned with the codebase. It's effectively an uncontrolled variable.

I'm not trying to discourage you from using agent side functions. In fact, I'm about to do some work on node_encrypt[1] to make it usable as a Deferred function and I'm really excited because that means that on Puppet 6, it will no longer be limited to only File types. But I do want you to be aware of what you're trading it for so you can make an informed decision.

The tl;dr is that these concerns essentially boil down to the fact that your configuration state will no longer be fully represented in the catalog. So make sure you account for that when evaluating your options.

[1] https://forge.puppet.com/binford2k/node_encrypt

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/pmbugj%2437k%241%40blaine.gmane.org.

[Henrik Lindberg]

There is an Encrypted data type coming soon along with a decrypt
function (there is a PR for it, but we are running out of time to
complete it for puppet 6). Want to help out on that? Ping me.

- henrik

> <mailto:puppet-users%2Bunsu...@googlegroups.com>
> > <mailto:puppet-users...@googlegroups.com
> <mailto:puppet-users%2Bunsu...@googlegroups.com>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/puppet-users/7c1b7417-abcd-4c17-a237-0681bc9a14bb%40googlegroups.com
>
> >
> <https://groups.google.com/d/msgid/puppet-users/7c1b7417-abcd-4c17-a237-0681bc9a14bb%40googlegroups.com?utm_medium=email&utm_source=footer>.
> > For more options, visit https://groups.google.com/d/optout.
>
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to puppet-users...@googlegroups.com

> <mailto:puppet-users%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/pmbugj%2437k%241%40blaine.gmane.org.
> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/CACkW_L7zW1s4e9YJpHjPd0S%3DHevgwdEHbSxi3RQRMQgy-0d3ig%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-users/CACkW_L7zW1s4e9YJpHjPd0S%3DHevgwdEHbSxi3RQRMQgy-0d3ig%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Rob Nelson]

This sounds great. My only concern is that I don’t see tests mentioned in the example. Will rspec-puppet be updated at the same time so that we can test deferred functions out of the gate? If so, I’d love to see what that looks like, in case we have any comments on that. Mocking some things like facts and functions is a little clunky at times. Hopefully deferred function mocking will be a little easier. Thanks.

[Henrik Lindberg]

On 2018-09-01 01:40, Rob Nelson wrote:
> This sounds great. My only concern is that I don’t see tests mentioned in the example. Will rspec-puppet be updated at the same time so that we can test deferred functions out of the gate? If so, I’d love to see what that looks like, in case we have any comments on that. Mocking some things like facts and functions is a little clunky at times. Hopefully deferred function mocking will be a little easier. Thanks.
>

There is no difference between a regular function and one that is called
in a deferred way. You can unit test functions that will run on the
agent the same way as regular functions.

- henrik