converting Hiera v3 to v5 - :merge_behavior: deeper option?

[du]

Hi

I am in the process of migrating v3 hiera.yaml to v5 and I would like all hash merges to merge 'deep' by default, the same way :merge_behavior: deeper in v3 parameter provides.

I can't seem to find an easy way to do this after reading the docs - I understand lookup_options: need to be set, however it requires class parameter to be specified. I tried using regex as below and adding it to common.yaml however it does not produce the desired result (it works ok if I specify the hash name explicitly, however I am looking to expand this behaviour to all lookups, if possible).

Is there an easy way to just migrate :merge_behavior: deeper to v5?

Thanks,

D

lookup_options:
  '.*':
    merge:
      strategy: deep

[R.I.Pienaar]

On Fri, 27 Apr 2018, at 17:34, du wrote:
>
>
> Hi
>
> I am in the process of migrating v3 hiera.yaml to v5 and I would like all
> hash merges to merge 'deep' by default, the same way :merge_behavior:
> deeper in v3 parameter provides.
>
>
> I can't seem to find an easy way to do this after reading the docs - I
> understand lookup_options: need to be set, however it requires class
> parameter to be specified. I tried using regex as below and adding it to
> common.yaml however it does not produce the desired result (it works ok if
> I specify the hash name explicitly, however I am looking to expand this
> behaviour to all lookups, if possible).
>
>
> Is there an easy way to just migrate :merge_behavior: deeper to v5?

It's not there because its the way it works, there's no other option and nothing to enable.

[Henrik Lindberg]

On 27/04/18 17:34, du wrote:
> Hi
>
> I am in the process of migrating v3 hiera.yaml to v5 and I would like
> all hash merges to merge 'deep' by default, the same way
> :merge_behavior: deeper in v3 parameter provides.
>
>
> I can't seem to find an easy way to do this after reading the docs - I
> understand lookup_options: need to be set, however it requires class
> parameter to be specified. I tried using regex as below and adding it to
> common.yaml however it does not produce the desired result (it works ok
> if I specify the hash name explicitly, however I am looking to expand
> this behaviour to all lookups, if possible).
>
>
> Is there an easy way to just migrate :merge_behavior: deeper to v5?

Depends on what you mean. The 'deep' in hiera 5 is the same as 'deeper'
in hiera 3. It just works.

If you mean that all keys should lookup with strategy 'deep' then you
use the lookup_options to set that for all keys.

- henrik


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[du]

Thanks all for the responses

I realise that the global option is no longer there and I am trying to set v5 up so that I can get the similar behaviour.

On Friday, April 27, 2018 at 6:12:32 PM UTC+1, Henrik Lindberg wrote:

Depends on what you mean. The 'deep' in hiera 5 is the same as 'deeper'
in hiera 3. It just works.

Yes, it works absolutely fine if I specify hash name explicitly in lookup_options - my issue is that I can't seem to set it using regex for all hashes that I am retrieving from Hiera.

 

If you mean that all keys should lookup with strategy 'deep' then you
use the lookup_options to set that for all keys.

 Do you mind sharing an example what setting it for all keys potentially look like - I suspect my regex above may not be correct

Of course I always have alternatives like replacing all hiera_hash() functions in my manifests with lookup() with correct options however this is a fair bit of work and I wanted to make sure first I absolutely cannot achieve this by tweaking Hiera options.

Regards,

D

[Henrik Lindberg]

On 28/04/18 13:21, du wrote:
> Thanks all for the responses
>
> I realise that the global option is no longer there and I am trying to
> set v5 up so that I can get the similar behaviour.
>
> On Friday, April 27, 2018 at 6:12:32 PM UTC+1, Henrik Lindberg wrote:
>
> Depends on what you mean. The 'deep' in hiera 5 is the same as 'deeper'
> in hiera 3. It just works.
>
> Yes, it works absolutely fine if I specify hash name explicitly in
> lookup_options - my issue is that I can't seem to set it using regex for
> all hashes that I am retrieving from Hiera.
>
>
> If you mean that all keys should lookup with strategy 'deep' then you
> use the lookup_options to set that for all keys.
>
>  Do you mind sharing an example what setting it for all keys
> potentially look like - I suspect my regex above may not be correct

The regexp must be anchored at the beginning of the string to be
recognized as a key regexp.

So, use this:

'^.*'

[jcbollinger]

Doesn't that instruct Hiera to use the 'deep' strategy for all lookups where no other strategy is explicitly requested?  Will that not, therefore, alter the behavior of automated data binding and of most calls to the lookup() and hiera() functions?

I'm totally prepared to learn something new today, but to the best of my current knowledge, Hiera 5 has no mechanism for controlling merge behavior whose scope can be limited to just hiera_hash() calls.

John

[Henrik Lindberg]

> Doesn't that instruct Hiera to use the 'deep' strategy for /all/ lookups

> where no other strategy is explicitly requested?  Will that not,
> therefore, alter the behavior of automated data binding and of most
> calls to the lookup() and hiera() functions?
>
> I'm totally prepared to learn something new today, but to the best of my
> current knowledge, Hiera 5 has no mechanism for controlling merge
> behavior whose scope can be limited to just hiera_hash() calls.
>

True - the above general regexp rule will apply to all keys irrespective
of their data type. There is no mechanism to make a regexp rule apply to
only a particular data type.

[jcbollinger]

On Saturday, April 28, 2018 at 6:21:12 AM UTC-5, du wrote:

 

Of course I always have alternatives like replacing all hiera_hash() functions in my manifests with lookup() with correct options however this is a fair bit of work and I wanted to make sure first I absolutely cannot achieve this by tweaking Hiera options.

Coming back to that comment, I think you may have some wrong expectations here.  If you continue to use the hiera_hash() function then nothing should change for you.  It is still documented to honor the :merge_behavior key, and it is also documented to ignore the lookup_options, and to only lookup data from one hierarchy (not three, as Hiera 5 does).  If you observe different behavior then you should file a bug report.  But that means not moving to Hiera 5 API-wise.  Moreover, unless you also stick to a single, global hierarchy overall, Hiera 3 style, you will be setting yourself up for problems and confusion.  Moving to Hiera 5 means moving to lookup().

If it were me, I'd just bite the bullet and switch over.  It may be a bit of a pain to convert hiera(), hiera_hash(), and hiera_array() calls to lookup() calls, but inasmuch as each of these implies a different merge strategy, it should be possible to automate the changes.  Moreover, in so doing, you can set up pretty good emulation of the Hiera 3 hash-merge behavior for those lookup() calls that want to participate.  In particular, consider doing this at top scope in your site manifest:

$hash_merge_behavior = lookup(':merge_behavior', String, 'unique', 'hash')

... and then converting each

# original
$my_hash = hiera_hash('key', { ... })

to 

# updated
$my_hash = lookup('key', Hash, $::hash_merge_behavior, {...})

.  The hiera() and hiera_array() calls can be converted in a similar manner. The one thing to look out for is any three-parameter hiera*() calls, for, to the best of my knowledge, lookup() does not offer any hierarchy-modulation behavior corresponding to what the hiera* functions do with that third parameter.

John

[jcbollinger]

CORRECTION:

On Tuesday, May 1, 2018 at 9:52:31 AM UTC-5, jcbollinger wrote:

$hash_merge_behavior = lookup(':merge_behavior', String, 'unique', 'hash')

... but of course, I meant 

$hash_merge_behavior = lookup(':merge_behavior', String, 'first', 'hash')

John

[Henrik Lindberg]

On 01/05/18 17:00, jcbollinger wrote:
>
> CORRECTION:
>
> On Tuesday, May 1, 2018 at 9:52:31 AM UTC-5, jcbollinger wrote:
>
>
> |

> $hash_merge_behavior =lookup(':merge_behavior',String,'unique','hash')

> |
>
>
> ... but of course, I meant
>
> |$hash_merge_behavior = lookup(':merge_behavior',String,'first','hash')|
>
>

And drop the leading ':' in ":merge_behaviour" ;-)

- henrik

>
> John
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/874d8785-1411-4f6c-9aae-fa2b6de21d46%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/874d8785-1411-4f6c-9aae-fa2b6de21d46%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[du]

Thanks all for your responses 

On a closer inspection, it turns out that deep merger indeed works out of the box on v5 without any additional tweaks. (@Henrik - your regex suggestion also works and I can no longer see errors)

We are not looking to stick with v3 forever, just wanted to minimise the number of changes as we make a big leap from puppet 3 to puppet 4