Hiera Questions: An array of :datadir: ?

[Dan White]

In a posting a few days ago was this hiera.yaml source listing:
---------------------------------
:backends: - json
:hierarchy: - %{fqdn}
- %{lmax_role}_role
- %{lmax_env}_server
- %{pop}.tradefair
- common
:json:
:datadir: - /etc/puppet/private/
- /etc/puppet/environments/%{environment}/hiera_data_store/
- /etc/puppet/environments/%{environment}/rebirth_data_store/
- /etc/puppet/environments/%{environment}/satellite_system_groups/
---------------------------------

I am curious how one might utilize this configuration.

Can the same key value appear in more than one datadir ?
Or does one use the multiple datadir's to group keys in some fashion ?

[Luke Bigum]

On 18/05/12 04:04, Dan White wrote:
> In a posting a few days ago was this hiera.yaml source listing:
> ---------------------------------
> :backends: - json
> :hierarchy: - %{fqdn}
> - %{lmax_role}_role
> - %{lmax_env}_server
> - %{pop}.tradefair
> - common
> :json:
> :datadir: - /etc/puppet/private/
> - /etc/puppet/environments/%{environment}/hiera_data_store/
> - /etc/puppet/environments/%{environment}/rebirth_data_store/
> - /etc/puppet/environments/%{environment}/satellite_system_groups/
> ---------------------------------
>
> I am curious how one might utilize this configuration.

Hi Dan,

That's my own terrible modifications to Hiera to support multiple data
stores of the same type. Code is here, but I warn you, I hacked in the
quickest way to get the functionality I wanted. I only modify
hiera-json, would be simple to do any others though:

https://github.com/lukebigum/hiera
https://github.com/lukebigum/hiera-json

As for use cases, a more elegant description is in this feature request:
http://projects.puppetlabs.com/issues/13954

> Can the same key value appear in more than one datadir ?

Yes, but precedence is taken just like Hiera's natural hierarchy, so
data stores at the start of the array have higher priority. In my case
for example, "hiera_data_store" overrides "satellite_system_groups".

> Or does one use the multiple datadir's to group keys in some fashion ?
>

That was the the original idea of the hack - I had data coming from
several places - some JSON written by hand, some generated by scripts
(the 'satellite_system_groups' store is a dump of our old Red Hat
Satellite System Groups and 'rebirth_data_store' is procedurally
generated). I could have modified the generating scripts to read any
existing JSON and write out a merged file, but I was rushing and I
didn't think of that at the time ;-)

It has managed to keep a nice logical separation of types of Hiera keys
though and you can put your data stores in different locations, like
having a 'private' dir for password hashes that's not under revision
control. This leads to giving our Development team a Git repo that's the
lowest priority data store and allow them to specify their own keys to
configure their Dev servers. Any Admin still has the ability to
overwrite any key in an data store above it, so it stops them doing
anything silly like changing passwords, LDAP servers, etc, as these keys
will already be set by us (Admins).

-Luke

--
Luke Bigum

Information Systems
Ph: +44 (0) 20 3192 2520
luke....@lmax.com | http://www.lmax.com
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN


FX and CFDs are leveraged products that can result in losses exceeding
your deposit. They are not suitable for everyone so please ensure you
fully understand the risks involved. The information in this email is not
directed at residents of the United States of America or any other
jurisdiction where trading in CFDs and/or FX is restricted or prohibited
by local laws or regulations.

The information in this email and any attachment is confidential and is
intended only for the named recipient(s). The email may not be disclosed
or used by any person other than the addressee, nor may it be copied in
any way. If you are not the intended recipient please notify the sender
immediately and delete any copies of this message. Any unauthorised
copying, disclosure or distribution of the material in this e-mail is
strictly forbidden.

LMAX operates a multilateral trading facility. Authorised and regulated
by the Financial Services Authority (firm registration number 509778) and
is registered in England and Wales (number 06505809).
Our registered address is Yellow Building, 1A Nicholas Road, London, W11
4AN.

[Dan White]

Thanks for responding, Luke.

This looks like a useful expansion of the hiera back-end, but as I am still fairly new to Puppet (and Python and Ruby for that matter), I think I will wait for this to be formally accepted and incorporated before trying it myself.

One parting question: Is the :hierarchy: repeated in each mamber of the :datadir: ? Logically, it should repeat, but I'd rather ask and be certain.

“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”
Bill Waterson (Calvin & Hobbes)

[Luke Bigum]

On 18/05/12 14:00, Dan White wrote:
> Thanks for responding, Luke.
>
> This looks like a useful expansion of the hiera back-end, but as I am still fairly new to Puppet (and Python and Ruby for that matter), I think I will wait for this to be formally accepted and incorporated before trying it myself.
>
> One parting question: Is the :hierarchy: repeated in each mamber of the :datadir: ? Logically, it should repeat, but I'd rather ask and be certain.
>

> �Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.�
> Bill Waterson (Calvin& Hobbes)
>

Hi Dan,

Last week I was going to reply with "Sure does, here's an example
debug..." and I released that my code didn't work they way I thought it
did, I had the inner and outer loops around the wrong way ;-)

So after much more butchering of RIP's code I can happily report that
the :hierarchy: is searched as the outer loop, the :datadir: Array as
the inner loop. This means that first the %{fqdn} files are searched in
each member of :datadir:, then if nothing is found it moves on to
'%{lmax_env}_role' and so on down the hierarchy.

A (working!) example:

[root@gs2puppet01 ~]# /opt/ruby-enterprise/bin/hiera -c
/etc/puppet/hiera.yaml -y
/var/lib/puppet/yaml/facts/test.example.com.yaml -a collective --debug

DEBUG: Mon May 21 07:59:39 +0000 2012: Hiera JSON backend starting
DEBUG: Mon May 21 07:59:39 +0000 2012: Looking up collective in JSON backend
DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source
test.example.com
DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an
Array, multiple data dirs to search
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/private/test.example.com.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile
/etc/puppet/environments/production/hiera_data_store/test.example.com.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/rebirth_data_store/test.example.com.json,
skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/satellite_system_groups/test.example.com.json,
skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from
/etc/puppet/environments/production/hiera_data_store/test.example.com.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source _role
DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an
Array, multiple data dirs to search
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/private/_role.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/hiera_data_store/_role.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/rebirth_data_store/_role.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/satellite_system_groups/_role.json,
skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source _server
DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an
Array, multiple data dirs to search
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/private/_server.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/hiera_data_store/_server.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/rebirth_data_store/_server.json,
skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/satellite_system_groups/_server.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source example.com
DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an
Array, multiple data dirs to search
DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile
/etc/puppet/private/example.com.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile
/etc/puppet/environments/production/hiera_data_store/example.com.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/rebirth_data_store/example.com.json,
skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/satellite_system_groups/example.com.json,
skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from
/etc/puppet/private/example.com.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from
/etc/puppet/environments/production/hiera_data_store/example.com.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source common
DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an
Array, multiple data dirs to search
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/private/common.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile
/etc/puppet/environments/production/hiera_data_store/common.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/rebirth_data_store/common.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile
/etc/puppet/environments/production/satellite_system_groups/common.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from
/etc/puppet/environments/production/hiera_data_store/common.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from
/etc/puppet/environments/production/satellite_system_groups/common.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Found key collective in file
/etc/puppet/environments/production/satellite_system_groups/common.json,
appending to Array
DEBUG: Mon May 21 07:59:39 +0000 2012: Lookup found answer ''
[""]