Hiera data repetition

[Dmitry Nurislamov]

Hello. Sometimes it is desirable to define Hiera variables for usage in Hiera itself, i.e. variables that won't be looked up from Puppet manifests. We do this to avoid repetition. Here's an example of a variable used only in the file it is defined:

    _internal_api_host: 'int.api.example.com'

    profile::keepalived::vrrp_script:
      ping_internal_api:
        script: "curl https://%{lookup('_internal_api_host')}/ping"
    profile::nginx::servers:
      "%{alias('_internal_api_host')}":

        ...

Sometimes we also define this kind of variables in one file but do lookups somewhere else:

hieradata/project/foo.yaml:

    profile::base::network::interfaces:

      eth0:

        method: 'static'

        address: "%{lookup('_INTERNAL_IP')}/24"

hieradata/project/foo/foo1.yaml:

    _INTERNAL_IP: '10.1.2.3'

For us this became an issue. There are no conventions, and therefore no consistency in our quite large hiera. Sometimes we can't even tell whether some variable is looked up from manifests or is "local" for Hiera. For example, the "_INTERNAL_IP" above is named "profile::base::network::intern_ip" in some projects.

The question is... Is this a normal practice? How do other folks handle these cases? I couldn't find any information regarding this. Thanks.

[Martin Alfke]

Hi

This is part of the way how you can implement hiera.
We heavily use the same pattern to avoid duplicate data.
Unluckily there is (yet?) no best practices written down.

Usually we have a pattern that we strictly follow:
- Never set class parameters on module (technical component or library module) level. Only set data on profile classes.
- Only do automatic data binding, using profile class namespace on hiera data, never do explizit lookups in profile classes.
- e.g.:profile::base::usermgmt::users: {}
- nested lookups always use short key names, usually prefixed with company or department or team short name
- e.g. company_ldap_pass

This pattern allows us to strictly separate data which are needed multiple times in hiera.
Using automatic data binding and the name convention on nested lookups, we can easily identify if data is consumed by a class or needed hiera internally.

hth,
Martin


>
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/282b3812-287a-4eb0-a838-591f2c3f925an%40googlegroups.com.

[Karsten Heymann]

Hi Martin,

regarding your "no lookup" policy, how do you handle for example deep merge lookups. They cannot be used with automatic data binding, or am I wrong?

Regards

Karsten

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/77E8E860-ABA3-4B06-99C6-C294BB0EE750%40gmail.com.

[Henrik Lindberg]

On 2021-02-27 18:18, Karsten Heymann wrote:
> Hi Martin,
>
> regarding your "no lookup" policy, how do you handle for example deep
> merge lookups. They cannot be used with automatic data binding, or am I
> wrong?
>
> Regards
> Karsten

You can specify lookup_options per key in the hiera data itself. One of
the options is if the lookup should be a deep merge or not. This is
supported for all lookups (automatic or not) and was introduced
especially to make it possible to do deep merge (and other things) when
using APL).

Best,
- henrik

>
> Am Sa., 27. Feb. 2021 um 13:06 Uhr schrieb Martin Alfke

> <tux...@gmail.com <mailto:tux...@gmail.com>>:

>
> Hi
>
> > On 25. Feb 2021, at 11:59, Dmitry Nurislamov <dmn...@gmail.com

> <mailto:dmn...@gmail.com>> wrote:
> >
> > Hello. Sometimes it is desirable to define Hiera variables for
> usage in Hiera itself, i.e. variables that won't be looked up from
> Puppet manifests. We do this to avoid repetition. Here's an example
> of a variable used only in the file it is defined:
> >
> >     _internal_api_host: 'int.api.example.com

> <http://int.api.example.com>'

> <mailto:puppet-users%2Bunsu...@googlegroups.com>.

> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/282b3812-287a-4eb0-a838-591f2c3f925an%40googlegroups.com

> <https://groups.google.com/d/msgid/puppet-users/282b3812-287a-4eb0-a838-591f2c3f925an%40googlegroups.com>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to puppet-users...@googlegroups.com

> <mailto:puppet-users%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/77E8E860-ABA3-4B06-99C6-C294BB0EE750%40gmail.com
> <https://groups.google.com/d/msgid/puppet-users/77E8E860-ABA3-4B06-99C6-C294BB0EE750%40gmail.com>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com

> <mailto:puppet-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/CAL017hDOZ7zoJqT0pbAyXVq4665TPDbOJ5Y-ELwUgzbaSofpKg%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-users/CAL017hDOZ7zoJqT0pbAyXVq4665TPDbOJ5Y-ELwUgzbaSofpKg%40mail.gmail.com?utm_medium=email&utm_source=footer>.


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Martin Alfke]

Hi Karsten,

> On 27. Feb 2021, at 18:18, Karsten Heymann <karsten...@gmail.com> wrote:
>
> Hi Martin,
>
> regarding your "no lookup" policy, how do you handle for example deep merge lookups. They cannot be used with automatic data binding, or am I wrong?
>

we set the merge behavior using hiera data lookup_options.
https://blog.example42.com/2021/02/11/hiera-merge-behavior/ (at the end of the post you can see an example).
This allows you to not only set merge behaviour globally, but use the hierarchies to even set different lookup options.

e.g. you will deep merge the user hash from all hierarchies, except for one system:

# data/common.yaml
lookup_options:
profile::base::usermgmt::users:
merge: 'deep'

# data/nodes/<certname>.yaml
lookup_options:
profile::base::usermgmt::users:
merge: 'first'

Best,
Martin

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAL017hDOZ7zoJqT0pbAyXVq4665TPDbOJ5Y-ELwUgzbaSofpKg%40mail.gmail.com.