Custom Hiera Interpolation Functions

[adam.g...@magicmemories.com]

So, I'm aiming to have a method of interpolating Facter facts in Hiera data files that works more or less exactly like the `scope` lookup function, except that if the Facter fact is not defined, this interpolation should result in Hiera moving on to the next hierarchy level as if the key were not found, rather than simply interpolating an empty string.

I've managed to make a rough hack of https://github.com/puppetlabs/puppet/blob/master/lib/puppet/pops/lookup/interpolation.rb that seems to accomplish what I'm looking for, but I'm not really sure how to work with Puppet's test infrastructure, nor whether this is really an idea the Puppet team would actually be interested in.

My actual goal is to allow Jenkins to run the Puppet Agent on a CI node while setting an environment variable such as `FACTER_myapp_version`, and have a node-specific YAML file that reads `profile::myapp::version: "%{maybe('facts.myapp_version')}"`, and thus allow Jenkins to control the version of the app that gets deployed; at the same time, during a run which doesn't set that particular environment variable I'd want Hiera to continue looking in other hierarchy levels, not just say that profile::myapp::version is an empty string. And, I do want to control this in Hiera if at all possible, not using `pick($facts['myapp_version'],lookup('profile::myapp::version'))` in puppet code, or anything similar.

So, I have three questions:

1. Is there an easier way to accomplish this that I'm not seeing, with a custom Hiera backend or something similar?
2. If not, is there a system for implementing custom interpolation functions as anything other than a patch to the `interpolation_method` method in lib/puppet/pops/lookup/interpolation.rb? (I couldn't find one, but that doesn't mean one doesn't exist)
   
3. If not, would anyone actually be interested in a pull request for this?

Thanks,

- Adam

[Eric Sorenson]

On Apr 17, 2018, at 6:56 PM, adam.g...@magicmemories.com wrote:

So, I'm aiming to have a method of interpolating Facter facts in Hiera data files that works more or less exactly like the `scope` lookup function, except that if the Facter fact is not defined, this interpolation should result in Hiera moving on to the next hierarchy level as if the key were not found, rather than simply interpolating an empty string.

I've managed to make a rough hack of https://github.com/puppetlabs/puppet/blob/master/lib/puppet/pops/lookup/interpolation.rb that seems to accomplish what I'm looking for, but I'm not really sure how to work with Puppet's test infrastructure, nor whether this is really an idea the Puppet team would actually be interested in.

My actual goal is to allow Jenkins to run the Puppet Agent on a CI node while setting an environment variable such as `FACTER_myapp_version`, and have a node-specific YAML file that reads `profile::myapp::version: "%{maybe('facts.myapp_version')}"`, and thus allow Jenkins to control the version of the app that gets deployed; at the same time, during a run which doesn't set that particular environment variable I'd want Hiera to continue looking in other hierarchy levels, not just say that profile::myapp::version is an empty string. And, I do want to control this in Hiera if at all possible, not using `pick($facts['myapp_version'],lookup('profile::myapp::version'))` in puppet code, or anything similar.

The `pick` approach would seem to be very straightforward, can you explain why that's not suitable for you? 

So, I have three questions:

1. Is there an easier way to accomplish this that I'm not seeing, with a custom Hiera backend or something similar?
2. If not, is there a system for implementing custom interpolation functions as anything other than a patch to the `interpolation_method` method in lib/puppet/pops/lookup/interpolation.rb? (I couldn't find one, but that doesn't mean one doesn't exist)
   
3. If not, would anyone actually be interested in a pull request for this?

Just IMHO - conceptually, this approach feels very strange to me because it's effectively injecting client-side source of truth (facts) into server-side truth (hiera data). The natural place to figure out which should "win" is in Puppet code, where you can codify the business logic in an expressive way.  

Not to say it's not possible to get this to work. (Though I confess I don't know how) 

--eric0

[Henrik Lindberg]

On 18/04/18 03:56, adam.g...@magicmemories.com wrote:
> So, I'm aiming to have a method of interpolating Facter facts in Hiera
> data files that works more or less exactly like the `scope` lookup
> function, except that if the Facter fact is not defined, this
> interpolation should result in Hiera moving on to the next hierarchy
> level as if the key were not found, rather than simply interpolating an
> empty string.
>
> I've managed to make a rough hack
> of https://github.com/puppetlabs/puppet/blob/master/lib/puppet/pops/lookup/interpolation.rb
> that seems to accomplish what I'm looking for, but I'm not really sure
> how to work with Puppet's test infrastructure, nor whether this is
> really an idea the Puppet team would actually be interested in.
>
> My actual goal is to allow Jenkins to run the Puppet Agent on a CI node
> while setting an environment variable such as `FACTER_myapp_version`,
> and have a node-specific YAML file that reads `profile::myapp::version:
> "%{maybe('facts.myapp_version')}"`, and thus allow Jenkins to control
> the version of the app that gets deployed; at the same time, during a
> run which doesn't set that particular environment variable I'd want
> Hiera to continue looking in other hierarchy levels, not just say that
> profile::myapp::version is an empty string. And, I do want to control
> this in Hiera if at all possible, not using
> `pick($facts['myapp_version'],lookup('profile::myapp::version'))` in
> puppet code, or anything similar.
>
> So, I have three questions:
>

> 1. Is there an easier way to accomplish this that I'm not seeing, with

> a custom Hiera backend or something similar?

With a custom hiera 5 backend function you can do interpolation any way
you like. The easiest is probably do do one that does exactly what you
want rather than a configurable one. For your case you can probably
write it in the puppet language in just a couple of lines of code.

> 2. If not, is there a system for implementing custom interpolation

> functions as anything other than a patch to the
> `interpolation_method` method in
> lib/puppet/pops/lookup/interpolation.rb? (I couldn't find one, but
> that doesn't mean one doesn't exist)

There is no such extension mechanism for hiera interpolation functions.

> 3. If not, would anyone actually be interested in a pull request for this?
>

Rather than adding/inventing new hiera interpolation functions (there
will never be an end to the requests people will have). I would rather
want to support a format where interpolation is done using puppet
notation. You can then call whatever functions you want.

That is, the function would be a lookup_key kind of function.
It would read a yaml file and cache it, then when returning a value it
would use the Puppet::Pops::EvaluatingParser class and call
evaluate_string. The evaluated string is returned.

Another alternative is to write a backend that supports a .pp file as
data with the expected result of a hash. Then the function is more or
less a wrapper for EvaluatingParser's method 'evaluate_file'.

If you want to do one of those, and you want to work on a generic
solution, (a .pp backend, either yaml + PP interpolation, or just .pp),
then I am happy to help out with the details, review etc.

- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/