migrating from calling_* in hiera 3 to hiera 5

[Rudy Gevaert]

Hallo,

Now that I have upgraded to the latest Puppet version I have the task to upgrade our hiera setup.

Unfortunately we are heavily using calling_class and calling_module in our setup.  And although the documentation says currently hiera5 is backward compatible.  This doesn't work anymore.

I would like to know how I can easily upgrade to hiera5 without needing to change all the lookups at once.

Is there any way to do that?  Or shall I need to do this all at once?

For now it seems impossible to have a situation that I can work in a branch to test things about.  Because to change to Hiera5, (and so being able to test the new system) I immediately remove my ability to leave calling_* in the global hiera config.

Thanks in advance,

Rudy

[Rudy Gevaert]

And to add the current hiera.yaml (3)

# Managed with Puppet

---

:backends:

  - eyaml

  - yaml

:yaml:

  :datadir: "/etc/puppetlabs/code/environments/%{::environment}/hieradata"

:eyaml:

  :extension: 'yaml'

  :datadir: "/etc/puppetlabs/code/environments/%{::environment}/hieradata"

  :pkcs7_private_key: /etc/puppetlabs/eyaml/private_key.pkcs7.pem

  :pkcs7_public_key:  /etc/puppetlabs/eyaml/public_key.pkcs7.pem

:hierarchy:

  - "nodes/%{::trusted.certname}"

  - "application_tier/%{::application_tier}/%{calling_class}"

  - "application_tier/%{::application_tier}/%{calling_module}"

  - "application_tier/%{::application_tier}"

  - "classes/%{calling_class}"

  - "modules/%{calling_module}"

  - "environment/%{::environment}"

  - "serverbeheer"

  - "common"

[Martin Alfke]

Hi Rudy,

I don’t know whether someone already replied.
https://docs.puppet.com/puppet/4.10/hiera_migrate_v3_yaml.html#remove-hierarchy-levels-with-callingmodule-and-friends

Quote from the link:

Remove hierarchy levels with calling_module and friends

Hiera 3 could use three special pseudo-variables (which weren’t available in Puppet code) in its hierarchy:

• calling_module
• calling_class
• calling_class_path
Hiera.yaml version 5 doesn’t support these, so you must drop any hierarchy levels that interpolate them.

These variables were added to support a hacky predecessor of module data; anything you were doing with them is better accomplished with the module layer.

Martin

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/a2fc367f-b591-49fa-b810-a0af926a0d7c%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Henrik Lindberg]

On 24/05/17 18:41, Martin Alfke wrote:
> Hi Rudy,
>
> I don’t know whether someone already replied.
> https://docs.puppet.com/puppet/4.10/hiera_migrate_v3_yaml.html#remove-hierarchy-levels-with-callingmodule-and-friends
>
> Quote from the link:
>
> Remove hierarchy levels with calling_module and friends
>
> Hiera 3 could use three special pseudo-variables (which weren’t available in Puppet code) in its hierarchy:
>
> • calling_module
> • calling_class
> • calling_class_path
> Hiera.yaml version 5 doesn’t support these, so you must drop any hierarchy levels that interpolate them.
>
> These variables were added to support a hacky predecessor of module data; anything you were doing with them is better accomplished with the module layer.
>
> Martin
>

It is worth pointing out that using calling_module/class etc. is
typically used to keep the data for a module/class (etc) in a separate
file/directory and then interpolate to get it. Most users do not use
these features for (the bad, anti-pattern) of binding the same key to
different values depending on from where it is looked up (that is the
main reason those keys are no longer supported, but it is also performance).

So - if not relying on the anti-pattern (which you should not), then all
the values in all the referenced files can actually be bound at the same
time and there is not need to differentiate on them using paths.

If there is a desire to keep data separate, they can be stored in
separate files and the `glob` variant can be used to read all of them.
Say you want to be able to override different classes at the environment
level - you could then create a directory called 'classes', and for each
class you would have a data file in that directory. Each data file by
convention would then only contain keys for that class. You then map all
of them with a glob in your hiera.yaml.

If the purpose was to provide defaults, simply place a hiera.yaml in
your module and keep the (default) data for the module there.

Hope that helps.
- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Rudy Gevaert]

On Thursday, May 25, 2017 at 6:51:32 PM UTC+2, Henrik Lindberg wrote:

It is worth pointing out that using calling_module/class etc. is
typically used to keep the data for a module/class (etc) in a separate
file/directory and then interpolate to get it. Most users do not use
these features for (the bad, anti-pattern) of binding the same key to
different values depending on from where it is looked up (that is the
main reason those keys are no longer supported, but it is also performance).

So - if not relying on the anti-pattern (which you should not), then all
the values in all the referenced files can actually be bound at the same
time and there is not need to differentiate on them using paths.

Thanks Hendrik.  I will look into the glob pattern as my aim was to override settings on the class level. (Not setting global defaults).  I'm also going to look at moving the hiera files to the specific module.  As all the modules that I'm overriding are custom modules here. 

[Rudy Gevaert]

On Thursday, May 25, 2017 at 6:51:32 PM UTC+2, Henrik Lindberg wrote:

If the purpose was to provide defaults, simply place a hiera.yaml in
your module and keep the (default) data for the module there.

Hope that helps.

Just to give some feedback.  I completed the migration.  I went for the route of moving the module specifc data into the hiera data dir in each module.

Some refactoring was needed.

An other issue that I ran into was that some module hiera data is encrypted with eyaml.  You can still use eyaml in module hieradata.  When running rspec tests you need to provide some fake private and public keys on the box doing the rspec tests.  (Assuming the rspec tests doesnt test for encryped values).  But the rspec still needs to be able to open those files and see them as valid private and public keys (but this can be different than on production).