lookup from external script

[Peter Meier]

Hi all,

I have a bunch of ruby script that are munging data around, part of that
data is based out of hiera.

Until now, with hiera 3, I just required hiera in my ruby script, called
lookup with a certain scope and I got my data out of hiera.

Now looking forward to puppet not anymore shipping hiera as a gem (as
everything moved to hiera 5 and hence lookup), how would I be able to
use the same functionality with lookup?

What I did (simplified):

require 'hiera'
def hiera
@hiera ||= Hiera.new(:config => '/etc/hiera.yaml')
end

def hiera_def_scope
@hiera_def_scope ||= { 'environment' => 'production' }
end

def hiera_lookup(key,default={})
hiera.lookup(key,default,hiera_def_scope)
end

mydata = hiera_lookup('mydata','a default')

While I see that with lookup this might get quickly very complex with
different environments, module data and so on. BUT I could imagine, that
given the data in hiera is central to your infra, lots of people might
also like to use that data for something else. So how to get that data out?

What would you recommend to do, how would you approach the problem with
getting data out of hiera in something else than puppet? Just stick
around with Hiera 3 and an old non-hiera-5 config file? Which would
likely be feasible if hiera stays somewhat supported, which it doesn't
from my understanding?! Write your own self-supported hiera?

best

~pete

[Craig Dunn]

What would you recommend to do, how would you approach the problem with
getting data out of hiera in something else than puppet? Just stick
around with Hiera 3 and an old non-hiera-5 config file? Which would
likely be feasible if hiera stays somewhat supported, which it doesn't
from my understanding?! Write your own self-supported hiera?

Maybe this is a bit overkill for your requirements, but this was actually one use case for Jerakia (http://jerakia.io).  Hiera 5 can use it as a backend from your Puppet implementation, and because it runs over an HTTP API other tools can easily hook into the same data lookups... for example there is now an Ansible lookup plugin that can pull the same data as Puppet.  It also has a client library written in Ruby which would hook into your script.

 

Craig

[Henrik Lindberg]

Over time the amount of features have grown for hiera and the move into
puppet was because of the increasing difficulties of making an external
hiera do all the required things as the external hiera would have needed
to know about environments and module paths and finding information in
modules, ability to reference to puppet variables etc. etc.

Thus, to use heira 5 as a library, it is basically the same as a
compilation + calls to the lookup function. To do that you would need to
do what the lookup CLI is doing in terms of setup. That would then give
you the full feature set.

There are however several use cases where people want to be able to
instantiate "a hiera" with a hierarchy of their own to do lookups. There
is the beginnings of such an API in puppet, but it is not well
documented and needs a bit more work.

In your case, do you want exactly the same as what a puppet compilation
would see (and support env, module data etc.) or do you need the
functionality of having your own hierarchy somewhere that is not mixed
into puppet?

Hiera 3, the freestanding gem will at some point no longer be supported.
It may be too early to drop it for Puppet 6, but it also depends on when
it will be released. We are currently just fixing critical problems with
the hiera 3 separate gem.

Best,
- henrik
--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Peter Meier]

> There are however several use cases where people want to be able to
> instantiate "a hiera" with a hierarchy of their own to do lookups. There
> is the beginnings of such an API in puppet, but it is not well
> documented and needs a bit more work.

And example, like the one I sent, would totally be enough documentation
for me. (and a stable API ;) )

> In your case, do you want exactly the same as what a puppet compilation
> would see (and support env, module data etc.) or do you need the
> functionality of having your own hierarchy somewhere that is not mixed
> into puppet?

The latter. In the context of external scripts (for me) it's not the
env- or module-specific data that matters. It's the global data that
matters. I would totally be fine, to replicate (part of) the hierarchy
in a separate config file, without all the benefits I get from the 5.x API.

This is more or less what I would do now when going to continue to use
the Hiera 3 freestanding gem for my scripts.

> Hiera 3, the freestanding gem will at some point no longer be supported.
> It may be too early to drop it for Puppet 6, but it also depends on when
> it will be released. We are currently just fixing critical problems with
> the hiera 3 separate gem.

This is what I got from the various announcements and so I though, when
now going down the move from Hiera 3 -> Hiera 5 for my puppet setup/code
I'd like to have a direction for the external scripts as well.

I could also see me replicating the trivial hiera stuff I do in my own
code, but I'd rather like to avoid that. ;)

best ~pete

[Peter Meier]

So are you reaching out to Hiera 5 from Jerakia to do that and how are
you doing it?

Are you as Hendrik suggested doing a compilation? What I saw as a big
difference is the whole context/scope that Hiera 5 is aware of, for
doing there lookups, which works for anything being called from puppet
code, but is kinda hard for an external query that must mimick that context.

best

~pete

[Craig Dunn]

> Maybe this is a bit overkill for your requirements, but this was
> actually one use case for Jerakia (http://jerakia.io).  Hiera 5 can use
> it as a backend from your Puppet implementation, and because it runs
> over an HTTP API other tools can easily hook into the same data
> lookups... for example there is now an Ansible lookup plugin that can
> pull the same data as Puppet.  It also has a client library written in
> Ruby which would hook into your script.

So are you reaching out to Hiera 5 from Jerakia to do that and how are
you doing it?

The other way around, Hiera  reaches out to Jerakia.   Jerakia is standalone but one way it can be used is as a backend to Hiera - it can integrate with Puppet using a Hiera 5 backend that ships with the crayfishx/jerakia Puppet module.  http://jerakia.io/integration/puppet

 

Are you as Hendrik suggested doing a compilation? What I saw as a big
difference is the whole context/scope that Hiera 5 is aware of, for
doing there lookups, which works for anything being called from puppet
code, but is kinda hard for an external query that must mimick that context.

No - if you are talking about looking up only data that is in your hieradata path in a specific environment, it would be easy to get Jerakia to read from the same file hierarchy, if you're using features inside of Hiera / Puppet such as module level Hiera lookups and Puppet variable interpolation, then this would, as Henrik said, require you to be in an actual Puppet environment so it would be difficult to expose this in the same way.  Your original post said you are on Hiera 3 though so it's highly the data you have to date would be very easy to expose using Jerakia.    Scope (eg: facts/variables..etc) shouldn't be an issue, you can hook up Jerakia to pull the scope data from PuppetDB when you query data from another tool.

Craig

[Reid Vandewiele]

For an abandoned experiment awhile back I went to the trouble of mostly getting an external ruby script working that used Hiera 5 as a library. I don't know that you still want to do this, sounds like there may be other options per the conversation in the thread, but I'll post the following link as reference code in case it's useful. (Ferreting out which specific parts of the code relate to using Hiera 5 as a library is left as an exercise to the reader.)

https://github.com/reidmv/r10k_puppetfile_ref_lookup/blob/master/r10k_puppetfile_ref_lookup.rb

~Reid