params.pp/inheritance/defaults/hiera/hiera functions?

[Christopher Wood]

(I'm not sure how to phrase my question precisely, so this may not all be totally clear.)

Has anybody else successfully moved away from using params.pp and default values, and if so, what did you do and how did it go? If not, what made you go back?

The only answer I can come up with is to stop using class parameters, change to hiera functions, and put all my data defaults in common.yaml.

The background:

I've had a couple of odd (lab) situations where much troubleshooting was occasioned by how I misspelled a key name in hiera and thus configuration data was pulled from a default value (either params.pp or the parameter default value). If there was no default value I would simply have gotten an error about the lookup failure and fixed that quickly.

In a related matter, sometimes I need a piece of configuration data from one module to appear in another module. Faked-up example, my SuperDaemon configuration data has to be used by monit to supervise all the worker daemons and by logrotate to rotate their individual log files. Some things are data bindings, some are hiera()/hiera_array()/hiera_hash() on a case by case basis, and it's messy.

In another related matter, explaining the combination of puppet inheritance from params.pp, default values, hiera, and hiera functions makes people's eyes glaze over. It would be easier to point people at hiera and say "source of data, learn these three functions".

Things are mostly fine, I may just be wanting to tweak things, but it strikes me there's a way to do all this with much less effort.

[Brian Mathis]

It sounds like you need to read through these:
    http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-1/
    http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/

I don't see why you'd want to remove parameters from classes.  They are pretty critical to having a flexible infrastructure.

Some of the smaller points in those posts which I think are important (in addition to the general pattern):

- Don't use default values in hiera.  If you don't supply (or typo) a variable name, then the puppet run should fail.

- Don't use automatic hiera lookups.  This removes the magic and makes it more clear to everyone that the data is coming from hiera.

For your config data showing up in other modules, I think you probably want role or profile modules, and pass in the data as a parameter.  Your roles/profiles should have all the custom stuff in it, not the component modules.

Also make liberal use of the validate_* functions from stdlib to further check your parameters have been supplied correctly.

Don't set default values to anything that is inappropriate.  If you don't want a default for your SuperDaemon config and it needs to be explicitly configured, then you want it to fail if no parameter is given.

❧ Brian Mathis

@orev

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20140529175818.GA4308%40iniquitous.heresiarch.ca.
For more options, visit https://groups.google.com/d/optout.

[Christopher Wood]

On Thu, May 29, 2014 at 02:26:21PM -0400, Brian Mathis wrote:

> It sounds like you need to read through these:

>     [1]http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-1/
>     [2]http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/

I've read those and recommended them, but I think there would be a scale problem if I took the advice literally. Quoth Ramin K. earlier, " I think there is a distinction between small systems and large ones which affect both our point of views."

https://groups.google.com/forum/#!msg/puppet-users/Lr6GrZziP6A/FpFqNbP9JeYJ

I'm not going for a large site, but still:

$ egrep -r '^(class|define)' modules | wc -l
709
$ egrep -r '^ \$' modules | wc -l
2094
$ egrep -r '^[a-z]' hieradata | wc -l
1690

If I put all the lookups used for one middling "technology-specific wrapper (class)" into that class I would start at tens of lookups in that class with some used in several places. That verges on difficult to understand, at least for me.

I'm also pondering the techniques used to figure out what information is used where. If hiera*() lookups are everywhere I can grep for the key name and find the classes using that key. If they're used in one place then I have to go hunting through the various class parameters and variable assignments. Still undecided.

Otherwise to the puppet workflow posts, misc points:

-params.pp, inheritance:

I disagree for the reason, stated differently below, that mixing data sources and lookup methods makes troubleshooting harder than it has to be.

-hiera lookups in components, visibility:

This is where I ponder whether I shouldn't just put all the variable data in hiera anyway, rather than struggle with where any data piece came from.

-compatibility

This sadly isn't a concern, for various corporate reasons likely none of the puppet stuff here is seeing the light of day.

-profiles are technology specific

I'd add the caveat that said technology also includes the various cron jobs, milters, modules, ancillary daemons, ACLs, monitors, and other shims that a mature system is encrusted with. Probably why I'm having trouble with magnitude.

> I don't see why you'd want to remove parameters from classes.  They are
> pretty critical to having a flexible infrastructure.

I think I'd phrase it better if I said that I don't necessarily need data bindings to have classes function in a parameterized fashion. But like the blog post, I will be happy to see the back of all my class parameters.

> Some of the smaller points in those posts which I think are important (in
> addition to the general pattern):
> - Don't use default values in hiera.  If you don't supply (or typo) a
> variable name, then the puppet run should fail.

There's a point, and per my grep I have two default values in use in manifests. I'll move those to hiera, since we have some legitimate cases for an empty default.

> - Don't use automatic hiera lookups.  This removes the magic and makes it
> more clear to everyone that the data is coming from hiera.

> For your config data showing up in other modules, I think you probably
> want role or profile modules, and pass in the data as a parameter.  Your
> roles/profiles should have all the custom stuff in it, not the component
> modules.
>
> Also make liberal use of the validate_* functions from stdlib to further
> check your parameters have been supplied correctly.

I hadn't thought of that! This will be a useful Friday activity in the dev repo.

> an email to [4]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [5]https://groups.google.com/d/msgid/puppet-users/20140529175818.GA4308%40iniquitous.heresiarch.ca.
> For more options, visit [6]https://groups.google.com/d/optout.

>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [7]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [8]https://groups.google.com/d/msgid/puppet-users/CALKwpEw0SPUjxYv1v2HAruacEP1XKUyvUn3dnhaE1No65ix9mQ%40mail.gmail.com.
> For more options, visit [9]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-1/
> 2. http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/
> 3. mailto:christop...@pobox.com
> 4. mailto:puppet-users%2Bunsu...@googlegroups.com
> 5. https://groups.google.com/d/msgid/puppet-users/20140529175818.GA4308%40iniquitous.heresiarch.ca
> 6. https://groups.google.com/d/optout
> 7. mailto:puppet-users...@googlegroups.com
> 8. https://groups.google.com/d/msgid/puppet-users/CALKwpEw0SPUjxYv1v2HAruacEP1XKUyvUn3dnhaE1No65ix9mQ%40mail.gmail.com?utm_medium=email&utm_source=footer
> 9. https://groups.google.com/d/optout

[Ramin K]

I'm curious about this as well. This was my pre parametrized classes
attempt to make it more understandable. I *think* this was a slight
improvement, but I'm not convinced that it adds much beyond pointing a
user where to look for the data.

class redis::params {
$packagename = $::osfamily ? {
/(?i:debian)/ => 'redis-server',
/(?i:redhat)/ => 'redis',
}
}

class redis::data {
$version = hiera('redis::version','present')
}

class redis::install {
package { $redis::params::packagename:
ensure => $redis::data::version;
}
}

I haven't switched over to parametrized classes mostly because the
above already works, the limited Hiera lookup available to data
bindings, and the great description you provided of the "where is this
data coming from" problem.

My current thinking is somewhat similar to yours. It might make sense
to pull in RI's module data patch and use explicit Hiera calls
everywhere. Data is always provided via Hiera, just from two locations.
The downsides are that PL is likely going to go a different direction
and it's a fair amount of code to change.
However moving the cognitive overhead to the writing of code instead of
the operating of it may be worth it till a better solution presents itself.

Ramin

[Daniele Sluijters]

> - Don't use automatic hiera lookups.  This removes the magic and makes it more clear to everyone that the data is coming from hiera.

Hold on a sec; if you mean data bindings with 'automatic hiera lookups', most certainly use them. Do not ever hardcode hiera() functions in parameter lookups in your module as you module can now _only_ work with Hiera and not everyone uses or wants to use hiera. Instead let data bindings take care of it so that everyone can benefit from your code, Hiera, ENC or by explicitly passing data in.

[Robin Bowes]

I think the most important thing is to not hardcode hiera() functions in
modules. That gives the flexibility for the users of the modules to
either rely on automatic parameter lookup (APL) or explicitly pass in
the parameters.

My own preference is to use explicit hiera lookups in "profiles" and
pass in the data to modules. APL is just a bit too magic, in my
experience, and avoiding it makes it easier to work out where data comes
from.

R.

[Brian Mathis]

Yes, this is what I mean.  Don't use or access hiera from with *component* modules.  You do need to use it from profile modules.  This assumes that one is using the profile/roles pattern, which seems to be the current best practice anyway.

❧ Brian Mathis

@orev

[jcbollinger]

On Friday, May 30, 2014 8:48:18 AM UTC-5, Brian Mathis wrote:

It sounds like you need to read through these:
    http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-1/
    http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/

I don't see why you'd want to remove parameters from classes.  They are pretty critical to having a flexible infrastructure.

No, as a matter of fact not.  What brings infrastructure flexibility in this area is relying on well-documented external data.  Automated data binding makes it easy to conflate class parameterization with data externalization, but they are not at all the same thing, as anyone who used(-es) parameterized classes in Puppet 2 can attest.  Obtaining data via explicit hiera*() calls for documented keys serves exactly the same data externalization aims as does automated class parameter binding.

Class parameterization does bring some limited advantages to modules, in that modules featuring parameterized classes lend themselves to an additional mode of use (class parameters declared in manifests via resource-like class declarations), but that yields module flexibility toward its users, not flexibility of the managed infrastructure.  I tend to discount that advantage, however, as it is usually poor practice to use resource-like class declarations even when your classes are parameterized (see, for example, PL's best-practices recommendation in the language reference).

John

[jcbollinger]

On Saturday, May 31, 2014 4:46:58 PM UTC-5, Robin Bowes wrote:

I think the most important thing is to not hardcode hiera() functions in
modules. That gives the flexibility for the users of the modules to
either rely on automatic parameter lookup (APL) or explicitly pass in
the parameters.

I am not particularly swayed by the advantages of enabling users to do something that they probably shouldn't do.  I'm not saying (these days) that you should avoid parameterizing your classes, but I hardly consider parameterization to be of great import.  Certainly not "the most important thing."


John