New to hiera

[Josh]

Hello,

I've never looked at heira before. I'm looking into upgrading from 2.7 to 3, and it seems that heira is the recommended way to do things now. From what I've read so far, it looks like one professed reason to use it is to go from parameterized classes to hiera overrides, because I guess parameterized classes are bad. So instead of:

class bar ($foo = 'blah') { ... }

One would have something like:

class bar { $foo = heira('bar::foo', 'blah') }

Is that correct?

I'm not yet sold on heira; so far it seems to just shift complexity outside of the classes and add a little more in the process, with hierarchies and stuff, and now I have data in multiple places... I still need case statements to handle different OSes and stuff. Eh, we'll see. Maybe I just haven't read something that explains the benefits well yet.

Josh

[Brian Lalor]

On May 4, 2013, at 9:50 PM, Josh <jo...@endries.org> wrote:

So instead of:

class bar ($foo = 'blah') { ... }

One would have something like:

class bar { $foo = heira('bar::foo', 'blah') }

Is that correct?

That's the hard way to do it.  The easy way is to do

class bar ($foo) { … }

and Hiera will look up the key "bar::foo" from the configured data sources.

--
Brian Lalor
bla...@bravo5.org

[Ashley Penney]

On Sat, May 4, 2013 at 9:50 PM, Josh <jo...@endries.org> wrote:

I'm not yet sold on heira; so far it seems to just shift complexity outside of the classes and add a little more in the process, with hierarchies and stuff, and now I have data in multiple places... I still need case statements to handle different OSes and stuff. Eh, we'll see. Maybe I just haven't read something that explains the benefits well yet.

You should check out https://puppetlabs.com/blog/hiera-for-pouncers-and-stalkers/ as it's a great guide to the benefits of hiera.  You really don't need case statements if you use heira, because you can make a hiera structure based on a fact like $::osfamily and then have:

osfamily/Redhat.yaml

osfamily/Debian.yaml

This gives you the benefit that if you drop in a new .yaml file supporting a new distribution you only have to drop it into heira, and not modify multiple manifests.  The article I linked probably does a better job of explaining it, but the goal is to remove all the data from your manifests leaving just variables behind, and then hiera fills in all the blanks.  It takes some getting used to but I think you'll find it leads to better, cleaner, manifests in the long run. 

[Larry Fast]

Hi Josh,

Here's what we found. I would love to hear about other approaches because we're still debating and building out our service.

1. Yes it's a great way to escape from parameterized classes.  If you ever needed to 'include' a class in two places, parameterized classes makes this difficult or impossible.

2. I don't recommend using automatic hiera lookup. There are some subtle gotchas in it especially when using hashes in hiera.  In my office our recommended practice is as almost as you wrote it, class bar ($foo = hiera('bar::foo')) { ... }.  Hiera automatic lookup is a good way to allow an easier transition from parameterized classes so it may still be a good choice for you.  As Ashley points out, read the latest docs!

3. If you explicitly use the hiera() call, you'll get nice "variable not found" messages when you make a mistake.  Clear error messages are rare in puppet so don't underestimate their value.

4. Don't over complicate your hiera tree.  It's not the place for everything.  Pick your largest collection of system-to-system variation and put that in hiera.

In our office we identified three variations that we *wanted* to put in hiera:

1. Variation between servers *within* an instance of our service

2. Variation between similar servers across different instances of our service (eg. Identity of the zookeeper in production, staging & QA)

3. Version differences between instances of our service.  IE. use hiera to manage the pipeline of releases.

What we *actually* put into hiera was only #2.  When #1 started creeping into hiera things got very messy. #1 is better handled by managing server Roles.  #3 should never go into Hiera - but I'm still arguing this point with my co-workers.

Caveate Emptor: We're still hemming and hawwing about all this.

Cheers, Larry Fast

[Drew Michel]

Hi Larry,

I'm curious to your opinion on point # 3, are you talking about OS packages or your organizations app version? If the latter, I was thinking of using hiera, maybe with a backend other than yaml such as redis, to store the version of the app, that way like you said it could be used in deploy pipeline. Why wouldn't you want to do this?

In addition to what everyone else has said, one of the bigger win's of using hiera, is the flexibility it offers you with overriding variables.

With the below hierarchy you can override based off an individual server, environment or datacenter. This could be powerful if you wanted to test some new JVM settings on just one server in production or an entire datacenter. You can use any facter fact you want to define your hierarchy. 

:hierarchy:

  - %{fqdn}

  - env/%{env}/common

  - datacenter/%{datacenter}

  - common

Drew

[Josh Endries]

Thanks for the replies!

While I don't like the idea of "reaching in" to a class to set its variables, given the design flaws in Puppet with parameterized classes, I agree that this is a strength of Heira (so far, the only one, IMHO). Maybe this irks me because I'm also a developer who started with C++.

I looked for docs and didn't find much beyond random blog posts, there are no "docs", really... I did read that article. I'm obviously a stalker. My (senior) co-worker that I'm training on Puppet is very much a pouncer. On Friday I had to say "no you actually need to read what it says" when trying to get him to use source instead of content (file type). He couldn't copy and paste, oh noes! I understand, though.

I read a couple of the articles linked from that one, too, and there are lots of problems with them, and with the example Ashley gave (which the articles mention also). The NTP argument is pretty weak, IMHO. I don't think it's a great NTP module, I guess (I don't use it). =P No offense. =)

I think the premise of supporting a new distro or OS (there is more than Linux out there, you know) with just variable flips almost silly. It practically requires modifying everything that will use that new OS. If all you do is flip a variable value, the module probably isn't very comprehensive. I almost always have functional differences for different OSes, like deploying an option file for a port, using .d files for some OSes and not for others, different templates, etc., not just a different package name.

I completely agree with your finding #4. I keep reading about using Heira to include classes and it makes me cringe. That's such a terrible idea, and totally unnecessary, IMHO. It's trivial to include classes based on facts with Puppet itself, where all the rest of your structure is declared... Introducing a second area of declaring that stuff is just...ugh.

I don't totally understand your variations...but I like the thought process and abstraction. It sounds like what I do all day long (on my "Puppet days"). I could see version numbers being useful in Heira: having a default and overriding it for dev or test. Instead of including instance::dev or instance::test classes, I guess, that do the same thing. We don't use version numbers at all in Puppet, though. =) This, however, is so far the only good use for Heira that I've found--overriding variables.

I organize things into classes within a single module. I make high-level "functional" classes that group things together based on mostly non-technical stuff, like workflow or purpose, kind of like roles, and include those in nodes. These grouping are usually pretty well-defined, with limited scope. E.g. I have a few different support-group sub-classes (e.g.: support-group::research) that create different user accounts and set or override some parameters, or high level role classes like research-group sub-classes. I almost never include a class in a node manifest directly (usually just when testing). I have no node variables or top scope variables. We have lots of HPC clusters, from 1 or 2 to entire racks of machines, so putting everything into classes as a policy and habit really helps us. This eliminates almost all problems with overriding variables so far. I guess if we need to split something up (like different "types" of NTP server), I would create classes for them and include those where necessary. I'm not sure if/how Heira can help with this yet.

The only real "problem" I have is user accounts integrating with various other things (like NFS/no NFS/root squash/pam_mount *and* AutoFS on the same box). We have a very big, very flexible user define with a layer on top of it for some corner cases. It's a lot of functional differences. I'm not yet sure if Heira could help with this, either, but it's something I'll be looking into...

I might try to track my journey and make blog posts to help codify what I learn. Maybe someone will find it interesting or useful if they get into the same situation...

GOodnight,
Josh

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/Z7GTiygw294/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[jcbollinger]

On Sunday, May 5, 2013 8:43:52 PM UTC-5, Josh wrote:

Thanks for the replies!

Although it's a bit tardy, I'd like to add a word or two here.

First, I think it's a mischaracterization to say "heira is the recommended way to do things now."  Certainly, hiera is the favored interface with external data, and I have said before that hiera finally makes parameterized classes usable.  That's not at all the same as hiera being recommended for everything, however.

You should not feel that hiera is being imposed on you.  Its integration with the Puppet core is a big advance -- the most significant in Puppet 3 -- but to the extent that you already have a manifest set that works in 2.7 without hiera, you have no particular need to convert to using hiera.
 

While I don't like the idea of "reaching in" to a class to set its variables,

Do you mean "to get its variables"?  Puppet variables can be set only in the scope to which they belong, and even then only once.  It is not possible to reach in to a class to set its variables.

On the other hand, a class can reach out to obtain values for its variables from an external source.  Prevailing Puppet wisdom holds that it is good practice to separate data from classes.

 

given the design flaws in Puppet with parameterized classes,

Although I have never been a fan of parameterized classes, with Puppet 3 I no longer attribute the related flaws to parameterized classes themselves, but rather to the parameterized-class declaration syntax, and somewhat to the intentional blurring of the distinction between classes and resources.  It's not a particular problem to single out certain variables that characterize the allowed variation in the state described by a given class -- i.e. parameters -- but it is a fundamental problem to bind data to parameters in a manner that depends on parse order.  Now that there are alternatives to doing the latter, those are separable issues.

I devote verbiage to this issue because the word "flaw" suggests something that can be fixed, but although class parameterization is basically ok (if overrated), the problems with binding data to classes via unrestricted DSL statements (parameterized-style class declarations) are not fixable.  That does not reflect a design flaw so much as the nature of the problem.

 

I agree that this is a strength of Heira (so far, the only one, IMHO). Maybe this irks me because I'm also a developer who started with C++.

This is actually a hard place from which to come to Puppet.  Puppet DSL uses some OO terms and has syntax and structure reminiscent of some OO languages, but it is essential to understand that it is NOT object-oriented in any but the most basic sense.  Your developer instincts will send you the wrong signals when you see the words "class" or "inherit", for example.  Coming from C/C++ may be especially hard, because "include" means something radically different in Puppet than in the C world, and "define" and defined types are very different beasts than C preprocessor macros.

Also, the overall mindset attending development in imperative languages is just plain wrong for writing Puppet manifests.  C++ (for example) focuses on actions, whereas Puppet DSL is almost entirely about state.  Imperative programmers are typically inclined to ask the wrong question: "How do I make Puppet do <foo>?".  The correct question is usually "How do I model <foo-result> in Puppet?"

If you've overcome these issues then I offer my congratulations to you.
 

I looked for docs and didn't find much beyond random blog posts, there are no "docs", really... I did read that article.

Yes, this a significant weakness for hiera.  I am sympathetic to your dissatisfaction there.

 

I'm obviously a stalker. My (senior) co-worker that I'm training on Puppet is very much a pouncer. On Friday I had to say "no you actually need to read what it says" when trying to get him to use source instead of content (file type). He couldn't copy and paste, oh noes! I understand, though.

I read a couple of the articles linked from that one, too, and there are lots of problems with them, and with the example Ashley gave (which the articles mention also). The NTP argument is pretty weak, IMHO. I don't think it's a great NTP module, I guess (I don't use it). =P No offense. =)

I think the premise of supporting a new distro or OS (there is more than Linux out there, you know) with just variable flips almost silly. It practically requires modifying everything that will use that new OS. If all you do is flip a variable value, the module probably isn't very comprehensive. I almost always have functional differences for different OSes, like deploying an option file for a port, using .d files for some OSes and not for others, different templates, etc., not just a different package name.

You're right, or course, that in many cases the differences from one OS to another requires more than just plugging in different package names, file names, etc..  Where you have deeper differences between configuration alternatives, it is entirely possible to use hiera to declare different classes for some nodes than for others, as appropriate to the node identity and fact pattern.

 

I completely agree with your finding #4. I keep reading about using Heira to include classes and it makes me cringe. That's such a terrible idea, and totally unnecessary, IMHO. It's trivial to include classes based on facts with Puppet itself, where all the rest of your structure is declared... Introducing a second area of declaring that stuff is just...ugh.

That's clearly a matter of taste.  Perhaps there's a bit of subconscious anti-hiera bias there, too?  Using hiera to identify classes to apply makes it serve essentially as an external node classifier.  Or at least semi-external -- although hiera itself is integrated, the characteristic data are external.  ENCs in general have a long history with Puppet and the concept is widely accepted.  That doesn't necessarily make it right for you, but prevailing opinion doesn't agree with "ugh" to the basic idea.

 

I don't totally understand your variations...but I like the thought process and abstraction. It sounds like what I do all day long (on my "Puppet days"). I could see version numbers being useful in Heira: having a default and overriding it for dev or test. Instead of including instance::dev or instance::test classes, I guess, that do the same thing. We don't use version numbers at all in Puppet, though. =) This, however, is so far the only good use for Heira that I've found--overriding variables.

I organize things into classes within a single module. I make high-level "functional" classes that group things together based on mostly non-technical stuff, like workflow or purpose, kind of like roles, and include those in nodes. These grouping are usually pretty well-defined, with limited scope. E.g. I have a few different support-group sub-classes (e.g.: support-group::research) that create different user accounts and set or override some parameters, or high level role classes like research-group sub-classes. I almost never include a class in a node manifest directly (usually just when testing). I have no node variables or top scope variables. We have lots of HPC clusters, from 1 or 2 to entire racks of machines, so putting everything into classes as a policy and habit really helps us. This eliminates almost all problems with overriding variables so far. I guess if we need to split something up (like different "types" of NTP server), I would create classes for them and include those where necessary. I'm not sure if/how Heira can help with this yet.

The only real "problem" I have is user accounts integrating with various other things (like NFS/no NFS/root squash/pam_mount *and* AutoFS on the same box). We have a very big, very flexible user define with a layer on top of it for some corner cases. It's a lot of functional differences. I'm not yet sure if Heira could help with this, either, but it's something I'll be looking into...

I might try to track my journey and make blog posts to help codify what I learn. Maybe someone will find it interesting or useful if they get into the same situation...

 

And that brings me back to the beginning: you should not feel that hiera is being imposed on you.  You especially should not get the impression that your approach to designing modules and classes is challenged by the integration of hiera into Puppet 3.  Your classes and modules do not need to be hiera-centric.

If you are not sold on externalizing your data, then hiera will be of limited interest to you.  And that's OK.  If you also do not use parameterized classes then (i) good for you, and (ii) hiera may be of no interest to you at all.  To the extent that you read external data or bind non-default data to class parameters, however, my technical advice to you is to do it via hiera.


John

[Larry Fast]

> I'm curious to your opinion on point # 3, are you talking about OS packages or your organizations app version? If the latter, I was thinking of using hiera, maybe with a backend other than yaml such as redis, to store the version of the app, that way like you said it could be used in deploy pipeline. Why wouldn't you want to do this?

Drew

Hi Drew,

I'm using a yaml backend so I can't speak about other options. So here's my disertation on why I (mostly) don't want version numbers in hiera.  In my world view, our puppet code is just another component of our system.  When I talk about our pipeline, I'm refering to deploying a version of our service into an environment.  At the highest level, the service version defines the version of all components - including puppet code.  BTW, it's not a big flat table, just the tip of the dependency tree.

I get my religion from Continuous Delivery (Jez Humble, David Farley) and I can only hope that I'm not misreading it.  Paraphrasing: You can break your service much faster with a configuration change than with a code change in one applications.  So configuration should be version controlled.

The direction I'm headed is to manage top level version control outside of our puppet code - with one caveate.  We also have {%fqdn} as the first entry in our hiera tree and the ability to override the version setting via hiera.  But it comes with the following warning:  

# FQDN should only be used for temporary overrides.

Cheers,

Larry