Hiera and roles/profiles

[Arnau]

Hi all,

I recently moved from puppet 3 to puppet 5 and hiera 1 to 3. I'm taking this opportunity to change a little bit the hierarchy setup I've been using for the last years.

In my previous conf I used to create a custom fact for each role so I could do something like:

- "%{environment}/hieradb/role/%{::role}"

and in each role file I specified the classes and any other role bases customizations for the role. So far so good. This approach is something that I'll continue using.

But now I'd like to do something similar with profiles: add a custom fact for each profile so I can later use something like:

- "%{environment}/hieradb/profile/%{::profile}"

Obviously this approach does not work when you have more than one profile:

First, I'd need to create the profile custom_fact as an array of several profiles and, then, I'd have to tell hiera to look in every profile from the same hierarchy level. In a node with profiles A/B/C hiera should have to look in the files:

"%{environment}/hieradb/profile/profileA.yaml"

"%{environment}/hieradb/profile/profileB.yaml" 

"%{environment}/hieradb/profile/profileC.yaml" 

Based on my tests (yup, I tried this) if hiera finds the value in, let's say, profileB.yaml it stops scanning this level and profileC.yaml is not evaluated.
In any case, even if hiera continues scanning all files in the same lelve, this approach will create a conflic in hiera itself whenever it finds the same variable defined in 2 profiles. ¿Which one should I choose?

So, the (obvious) solution is to move any profile customization to the role level, the same I've been doing till now.

But I was wondering if some part of my approach could be possible (without having to decalre all the possible profiles in my hiera  tree) or if someone was somehow using the "profile" in the hiera tree (and how)?.

TIA,

Arnau

[jcbollinger]

On Friday, May 18, 2018 at 3:44:24 AM UTC-5, Arnau wrote:

Hi all,

I recently moved from puppet 3 to puppet 5 and hiera 1 to 3. I'm taking this opportunity to change a little bit the hierarchy setup I've been using for the last years.

In my previous conf I used to create a custom fact for each role so I could do something like:

- "%{environment}/hieradb/role/%{::role}"

and in each role file I specified the classes and any other role bases customizations for the role. So far so good. This approach is something that I'll continue using.

But now I'd like to do something similar with profiles: add a custom fact for each profile so I can later use something like:

- "%{environment}/hieradb/profile/%{::profile}"

Obviously this approach does not work when you have more than one profile:

Indeed not.

 

First, I'd need to create the profile custom_fact as an array of several profiles and, then, I'd have to tell hiera to look in every profile from the same hierarchy level. In a node with profiles A/B/C hiera should have to look in the files:

Indeed, and that's a core problem.  With the standard YAML back-end, every hierarchy level maps to at most one data file.

 

"%{environment}/hieradb/profile/profileA.yaml"

"%{environment}/hieradb/profile/profileB.yaml" 

"%{environment}/hieradb/profile/profileC.yaml" 

Based on my tests (yup, I tried this) if hiera finds the value in, let's say, profileB.yaml it stops scanning this level and profileC.yaml is not evaluated.

I find that doubtful.  Given the hierarchy definition you specified earlier, I expect Hiera to read at most one of those files whether it finds the value there or not.

 

In any case, even if hiera continues scanning all files in the same lelve, this approach will create a conflic in hiera itself whenever it finds the same variable defined in 2 profiles. ¿Which one should I choose?

For data at different hierarchy levels, this is where Hiera's various merge behaviors come into play.  The default is that the first value found is used, but you can instead collect all values into an array, or, for hashes, merge the various hashes in one of two ways.

But note well that if you have different profiles trying to specify values for the same data then that's a serious conflict that you need to work out -- a data design problem, not inherently an Hiera problem.

 

So, the (obvious) solution is to move any profile customization to the role level, the same I've been doing till now.

But I was wondering if some part of my approach could be possible (without having to decalre all the possible profiles in my hiera  tree) or if someone was somehow using the "profile" in the hiera tree (and how)?.

You can parameterize your profile classes, and then rely on standard data binding to obtain the values from Hiera.  The data for different profiles are then distinguished by different keys, and they can appear at any Hierarchy level.

For data that you want to share among profiles without duplication, you can define well-known keys and explicitly call the lookup() function in each profile to retrieve the (same) associated data.  Or you can define a separate class to host the variables, set their values via automated data binding, and have all the profiles that want access obtain it via that one class.  Either way, again, the data can then appear at any hierarchy level.

If this is about class parameters for the classes your profiles use, then you could perhaps risk having the profiles declare those with resource-like class declarations (instead of include-like ones), whereby you can bind data to the classes' parameters via DSL code.

John

[Arnau]

Hi John,

first of all, thanks for your answer.

2018-05-18 14:15 GMT+02:00 jcbollinger <John.Bo...@stjude.org>:

 

"%{environment}/hieradb/profile/profileA.yaml"

"%{environment}/hieradb/profile/profileB.yaml" 

"%{environment}/hieradb/profile/profileC.yaml" 

Based on my tests (yup, I tried this) if hiera finds the value in, let's say, profileB.yaml it stops scanning this level and profileC.yaml is not evaluated.

I find that doubtful.  Given the hierarchy definition you specified earlier, I expect Hiera to read at most one of those files whether it finds the value there or not.

Ah, ok, so it only reads one file. Then the conclusions from my tests were wrong (and probbaly because of werid coincidence).

 

 

In any case, even if hiera continues scanning all files in the same lelve, this approach will create a conflic in hiera itself whenever it finds the same variable defined in 2 profiles. ¿Which one should I choose?

For data at different hierarchy levels, this is where Hiera's various merge behaviors come into play.  The default is that the first value found is used, but you can instead collect all values into an array, or, for hashes, merge the various hashes in one of two ways.

But note well that if you have different profiles trying to specify values for the same data then that's a serious conflict that you need to work out -- a data design problem, not inherently an Hiera problem.

Sure, I never said it was a hiera issue. Hiera itself, in the case of reading the same data from 2 differnet profiles, will never be able to decide which data to choose. Totally clear.

 

So, the (obvious) solution is to move any profile customization to the role level, the same I've been doing till now.

But I was wondering if some part of my approach could be possible (without having to decalre all the possible profiles in my hiera  tree) or if someone was somehow using the "profile" in the hiera tree (and how)?.

You can parameterize your profile classes, and then rely on standard data binding to obtain the values from Hiera.  The data for different profiles are then distinguished by different keys, and they can appear at any Hierarchy level.

That's what I was doing in my previous hiera tree. The issue now was that I wanted to use the "profile" custom_fact as a key to hiera.

 

For data that you want to share among profiles without duplication, you can define well-known keys and explicitly call the lookup() function in each profile to retrieve the (same) associated data.  Or you can define a separate class to host the variables, set their values via automated data binding, and have all the profiles that want access obtain it via that one class.  Either way, again, the data can then appear at any hierarchy level.

If this is about class parameters for the classes your profiles use, then you could perhaps risk having the profiles declare those with resource-like class declarations (instead of include-like ones), whereby you can bind data to the classes' parameters via DSL code.

Let me tell you what I wanted to do: I wanted to build the "classes" hash based on different profiles. So, insetad of writing classes for each profile and then "collect" them in the role in hiera:

roleA.yaml

classes:

  - profileA

  - profileB

and then declaring the classes:

class  profileA {

   include  apache

}

class profleB {

  include mysl

}

I wanted to do:

profileA.yaml

classes:

  - apache

profileB.yaml

classes:

  - mysql

roleA.yaml

custom_facts:

  - profileA

  - profileB

then "merge" the classes hash so it pick classes for all the profiles a node belongs to....

The main reason is to not having to write classes for each profile and do as much as I can in hiera.

I'm not sure if my idea makes sense to you, or if I explained it properly. But with teh potential of hiera3 + Puppetfile I wanted to avoid writing puppet code as much as possible.

John

Arnau

[jcbollinger]

I think you're trying to get Hiera to shoulder a bit too much of the load.  Nevertheless, you could make a similar data structure work for you with just a little help on the Puppet side.  For example, consider this class:

class site::role_classes(Array[String] $profiles) {
  $profiles.each |$profile| {
    $profile_classes = lookup("profile_${profile}_classes", Array[String[1]], 'unique', [])
    $profile_classes.each |$profile_class| {
      include $profile_class
    }
  }
}

Having that, instead of hiera_include('classes'), which seems to be what you were aiming toward, you can instead declare

include site::role_classes

Again, you cannot expect to put your per-profile class lists into different files at the same hierarchy level, at least with the standard YAML back-end.  You need to distinguish by keys instead.  But the above is all the manifest code you need around data that look like this:

roleA.yaml

site::role_classes::profiles:
 - profileA
 - profileB

profile_classes.yaml

profile_profileA_classes:
 - apache

profile_profileB_classes:
 - mysql

Note that only two hierarchy levels are represented there: one with a separate file per role, defining the profiles for that role, and one with a single file declaring all the classes for each profile.  The latter structure follows from the fact that, as you observed at the outset, most nodes have multiple profiles.

Alternatively, if having the profile data split out into separate files is important to you, but you insist on achieving that some other way than via DSL code, then perhaps you would be better off writing a bona fide ENC for yourself.  It probably would not have to be that much more complicated than the class above, and you then would not need any DSL code at all outside the module-level classes.

I would be remiss, however, to fail to note that actual profile classes can do more for you than your 100% data-driven approach can do.  Profile classes can set up relationships among the classes they declare, for example.  They can provide containment.  They can 'include' other profile classes to function as extensions.  They can perform arbitrary conditional logic to choose classes and / or class parameters.  Even though you might not usually want any of those things in your profiles, are you sure you don't want to reserve the ability to use those and similar capabilities?

John

[Arnau]

Hi John,

thanks for your answer and your examples.

2018-05-21 16:09 GMT+02:00 jcbollinger <John.Bo...@stjude.org>:

I think you're trying to get Hiera to shoulder a bit too much of the load. 

Yes, maybe, but I wanted to take all advantages from the new hiera and still use the old profiles/roles classes approach when needed. 

 

Nevertheless, you could make a similar data structure work for you with just a little help on the Puppet side.  For example, consider this class:

class site::role_classes(Array[String] $profiles) {
  $profiles.each |$profile| {
    $profile_classes = lookup("profile_${profile}_classes", Array[String[1]], 'unique', [])
    $profile_classes.each |$profile_class| {
      include $profile_class
    }
  }
}

Having that, instead of hiera_include('classes'), which seems to be what you were aiming toward, you can instead declare

include site::role_classes

Again, you cannot expect to put your per-profile class lists into different files at the same hierarchy level, at least with the standard YAML back-end.  You need to distinguish by keys instead.  But the above is all the manifest code you need around data that look like this:

roleA.yaml

site::role_classes::profiles:
 - profileA
 - profileB

profile_classes.yaml

profile_profileA_classes:
 - apache

profile_profileB_classes:
 - mysql

Note that only two hierarchy levels are represented there: one with a separate file per role, defining the profiles for that role, and one with a single file declaring all the classes for each profile.  The latter structure follows from the fact that, as you observed at the outset, most nodes have multiple profiles.

That's a nice approach, but, aprt from the classes, in my mind i also wanted to have some hiera data in each profile file... but it could be that, by mistake, i duplicate the data in two different profiles, then what is hiera supposed to do?

 

Alternatively, if having the profile data split out into separate files is important to you, but you insist on achieving that some other way than via DSL code, then perhaps you would be better off writing a bona fide ENC for yourself.  It probably would not have to be that much more complicated than the class above, and you then would not need any DSL code at all outside the module-level classes.

Too much for my approach. as i said, having hiera data in the same hierarchy level and expect hiera to take the right decission is not possible, 

I can use the typical profile/role with hiera approach and keep putting stuff in role and cert levels.

I would be remiss, however, to fail to note that actual profile classes can do more for you than your 100% data-driven approach can do.  Profile classes can set up relationships among the classes they declare, for example.  They can provide containment.  They can 'include' other profile classes to function as extensions.  They can perform arbitrary conditional logic to choose classes and / or class parameters.  Even though you might not usually want any of those things in your profiles, are you sure you don't want to reserve the ability to use those and similar capabilities?

Yes, I know, and i still use it, but I thought that my approach could simplify my life for some basic cases where it's clear what the class ha to do and has no depdency with other stuff. Clear example: autofs. If the node has the profile "autofs" I wanted it to include the class and read data from the profile/autofs.yaml file. 
But I can put that data in other hierarchy files and move the class include to the role level.

 

John

Thanks a lot for your answer,

Arnau