Duplicate hiera hashes in the same data source override instead of merge

[Matthew Burgess]

Hi folks,

In my common.yaml file I have the following hiera hashes:

system::packages:
    sshd:
        ensure: 'present'
    ...
...

more stuff here
...
system::packages:
    ntpd:
        ensure: 'present'

I naively assumed that Hiera would merge those 2 hashes together, just as it does if they were present in different layers of the hierarchy.  It appears, though, that the latter simply overrides the former;  I have an sshd service with a dependency on the sshd  package, but with the above hiera data, the catalog fails to compile as the sshd package isn't in the catalog.

Am I just missing some YAML syntax to stop the hash from being overwritten, or am I perhaps misunderstanding the behaviour I'm seeing here? I did test manually merging the hashes in the YAML file and that did indeed 'fix' the problem.

If that behaviour is simply a fact of life, that's fine too; I'll fix the data up, although there was an aesthetic/organisational reason for having it structured as it is currently.

Thanks,

Matt.

[Felix Frank]

Hi,

long story short: This is what the hiera_hash() function is for (as
opposed to hiera()).

HTH,
Felix

[Matthew Burgess]

Thanks Felix.

I'm using the system module (https://forge.puppetlabs.com/erwbgy/system) and the system::packages module *is* using hiera_hash() (https://github.com/erwbgy/puppet-system/blob/master/manifests/packages.pp) (as is system::services which is also causing me the same problem).

I assume hiera_hash()'s behaviour is default, i.e. I don't need to set :merge_behaviour in hiera.yaml?

In addition, running 'hiera' to debug this shows the same behaviour; I assume there isn't a native hiera equivalent to Puppet's hiera_hash() function?

Cheers,

Matt.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/533AE688.3070702%40alumni.tu-berlin.de.
For more options, visit https://groups.google.com/d/optout.

[Matthew Burgess]

Apologies for that last email; hiera_hash() is obviously doing *nearly* the right thing, as it *does* merge hashes across different levels of the hierarchy.  It *doesn't* seem to be merging hashes within the same level of the hierarchy.

Thanks,

Matt.

[Felix Frank]

Hmm, so you have the same key multiple times in one yaml file? Why?

[Robin Bowes]

I suspect duplicate keys are not valid YAML.

R.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAKUTv3JLCw_FWJTnf8ZCJ_RtWuLKvwG6afa5oECxB1R%3DxKdjHQ%40mail.gmail.com.

[Matthew Burgess]

Like I said, it was purely for cosmetic/aesthetic purposes; I wanted to write the YAML file in the same order as things were defined in our design document so that it was easier to review that the implementation matched the design.  Section 1 of that doc might cover something like NTP configuraiton, which requires a package, config file and service; Section 2 might cover sshd, again with another package, config file and service.  As it doesn't look like having multiple keys with the same name in the same file is supported, I'll get over my obsession with trying to lay the file out to match the order things appear in the design :-)

Perhaps I need to look at this another way; I could lay my tests out (I'm sure I have some somewhere!) in the same order as the design doc, then I get the implementation review for free by executing the tests!

Thanks for your help,

Matt.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/533AF084.9030509%40alumni.tu-berlin.de.

[Felix Frank]

Yeah, sounds about right.

Pardon my prejudice, but from your description I get the feeling that
you are not employing hiera to the height of its strengths. You want
your hierarchy to contain *data* for your nodes. While it's possible to
map all your resources to YAML and make heavy use of create_resources(),
it is certainly not beneficial in all cases.

I suggest at least a cursory glance at the roles/profiles pattern (or
what I lovingly refer to as the hiera ENC) to see if it fits you better.
An approach that has served me well is to make my manifests as
pseudo-intelligent as possible (allowing puppet to do the right thing
without us copy-pasting all the time) and sprinkle in hiera lookups only
where customization is needed or I cannot whip up a logic based on facts
alone.

Note that either way, you won't solve your issue of spacing out hiera
values per your given design structure. If you rely on manifests more,
you can get much closer to that. (If it's a sound goal at all is yet
another good question.)

Regards,
Felix

[Matthew Burgess]

On 2 April 2014 08:47, Felix Frank <felix...@alumni.tu-berlin.de> wrote:

Yeah, sounds about right.

Pardon my prejudice, but from your description I get the feeling that
you are not employing hiera to the height of its strengths. You want
your hierarchy to contain *data* for your nodes. While it's possible to
map all your resources to YAML and make heavy use of create_resources(),
it is certainly not beneficial in all cases.

I suggest at least a cursory glance at the roles/profiles pattern (or
what I lovingly refer to as the hiera ENC) to see if it fits you better.

​We're actually using the roles/profiles pattern too.

However, rather than create a separate module for our custom stuff​

 

​which is largely going to be install a package, a config file and a service, we're utilising the aforementioned 'system' module to prevent module-proliferation.

Our hiera structure mirrors the roles part of the roles/profiles pattern to enable us to customise the packages and services available for each unique role.  Our common.yaml provides the data for our base profile.
​

An approach that has served me well is to make my manifests as
pseudo-intelligent as possible (allowing puppet to do the right thing
without us copy-pasting all the time) and sprinkle in hiera lookups only
where customization is needed or I cannot whip up a logic based on facts
alone.

​It was that copy-pasting we were trying to get away from by utilising the system module.  I also really like the fact that we've now got pretty close to 'infrastructure as configuration data' as opposed to 'infrastructure as code' (obviously there'll always be some custom modules required, but our goal is to keep those to a minimum).
​

 

Note that either way, you won't solve your issue of spacing out hiera
values per your given design structure. If you rely on manifests more,
you can get much closer to that. (If it's a sound goal at all is yet
another good question.)

​Thanks for your comments; it's always good to hear other points of view.

Regards,

Matt.

[jcbollinger]

On Tuesday, April 1, 2014 12:00:45 PM UTC-5, Robin Bowes wrote:

I suspect duplicate keys are not valid YAML.

Indeed they are not.  The keys of a YAML mapping are required to be unique (http://www.yaml.org/spec/1.2/spec.html#id2764044).

How a particular YAML processor deals with duplicate mapping keys is implementation-defined, but the most likely behaviors are an error message and/or choosing one the two values presented for the duplicate key.  I don't see how merging the values is even viable.


John