hiera - default parameter values and design questions

[Mike Reed]

Hello all,

I ran into a problem with hiera and in particular, assigning or not assigning default values to variables within a data source.  I'm using the mcollective module (currently using version 1.1.6) and I've built a very simple hiera backend to start adding site-specific data into a single place.  Within the mcollective init.pp manifest, I've identified four places for which I'd like to abstract data into hiera; they are:

$server        = hiera('::mcollective::server'),

$client         = hiera('::mcollective::client'),
$middleware = hiera('::mcollective::middleware'),
$middleware_hosts = hiera('::mcollective::middleware_hosts'),

I've created a fqdn specific hiera source (ourlocalhost.local.yaml) and added this:

classes:
  - mcollective

# mcollective parameters
mcollective::middleware: true
mcollective::server: true
mcollective::client: true
mcollective::middleware_hosts:
  - ourlocalhost.local

Upon running puppet on this specific host, the puppet run completes and I marvel at the wonder of puppet and heiras' beautiful co-existence.

Now I'd like to simply add mcollective to any node in my network so I go to common.yaml and add the following:

classes:
  - mcollective

# mcollective parameters
mcollective::server: true
mcollective::middleware_hosts:
  - outlocalhost.local

However, upon running puppet on a fresh machine, I get the following error:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item ::mcollective::client in any Hiera data file and no default supplied at /etc/puppet/environments/test/modules/mcollective/manifests/init.pp:5 on node mynode.local

If I add this line into common.yaml (mcollective::client: false) then I get the following failure:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item ::mcollective::middleware in any Hiera data file and no default supplied at /etc/puppet/environments/test/modules/mcollective/manifests/init.pp:6 on node mynode.local

If I then go back into common.yaml and add 'false' to the values I don't want, everything works:

classes: mcollective
mcollective::middleware: false
mcollective::server: true
mcollective::client: false
mcollective::middleware_hosts:
  - ourlocalhost.local


I've been searching for answers on the interwebs and haven't been able to find anything specific for this one so I figured I'd ask a few questions:

1.  Is this the expected functionality?  If so, am I to understand that I have to keep track of every hiera call (ex $server = hiera('::mcollective::server'), $client = hiera('::mcollective::client')), in my many modules and create defaults for each of these values in the various hiera source files that I create?  My understanding was that I could create these hiera calls in my classes and a al carte these values within my hiera source files.  Am I mistaken and if not, doesn't that create redundant data within my hiera source file tree?  Also, if future-me adds one additional hiera call to the init.pp, does that mean I have to track down every hiera data source and add a default value to each one? 

Which brings me to my next question:

2.  It seems that things could get messy very quickly if I am declaring parameters via hiera (let's take this mcollective example) and I have multiple hiera source files for which these values are declared.  As an example, let's say I have a custom fact which determines that a node is a 'server' based on hostname. I then create a hiera data source called 'server.yaml' and assign the mcollective parameters to that data source and all is well in the universe. 

But if I later I want to create a custom hostname data source for that server called 'nodename.yaml' and assign parameters from that source, I now have two places to check for declared values.  Additionally, if I go to the mcollective init.pp file, I simply see the source for that variable is being called from hiera but since I have two data sources, how will future-me know which one to check?  Further more, if I have a colleague that is just familiar enough with our infrastructure to check the init.pp and indeed he/she sees that the data is coming from hiera, he/she won't know where to look if we've defined the data in our 'server.yaml' our 'node.yaml' and our 'common.yaml'.  Obviously this is a simplistic example but am I thinking about this wrong?  Perhaps I've missed/overlooked a major piece of the overall hiera design?  Or, I'm simply a jackass and I'm "using it wrong?"

3.  If I want to add multiple 'classes' to say, common.yaml, is there a particular way in which I should is the below structure the preferred method and if so, does hiera parse these from top to bottom or am I likely to run into resource ordering problems in the future?:

classes:
  - one class
  - two class
  - three class

# class one parameters
stuff
blah

# class two parameters
stuff
blah

# class three parameters
stuff
blah


Thank you in advance to whomever replies with thoughts as I realize this is quite a long post.

Cheers,

Mike

[Xav Paice]

inline

There's two approaches to hiera - if you want to use the hiera() function, you can give it two args - hiera('::mcollective::client', false) would look at the content of ::mcollective::client and if there's nothing in hiera, would return a default of false.  That's probably the simplest approach for this example.

Also be aware that you don't have to use the hiera() function in parameterized classes (although it's nice and clear coding to do so).  You could, for example, declare a class like so:

class myclass (
  $param1 = 'defaultvalue'
) {
  codeblock
}

This would give $param1 a default value of 'defaultvalue'.  If you want to override that, you can call the class with param1 => 'thing', or you can add the value to hiera like this (assuming yaml, edit accordingly):
---
myclass::param1: 'overridevalue'

Puppet is smart enough to read hiera and use the value from hiera over the default in the class definition.

Which brings me to my next question:

2.  It seems that things could get messy very quickly if I am declaring parameters via hiera (let's take this mcollective example) and I have multiple hiera source files for which these values are declared.  As an example, let's say I have a custom fact which determines that a node is a 'server' based on hostname. I then create a hiera data source called 'server.yaml' and assign the mcollective parameters to that data source and all is well in the universe. 

But if I later I want to create a custom hostname data source for that server called 'nodename.yaml' and assign parameters from that source, I now have two places to check for declared values.  Additionally, if I go to the mcollective init.pp file, I simply see the source for that variable is being called from hiera but since I have two data sources, how will future-me know which one to check?  Further more, if I have a colleague that is just familiar enough with our infrastructure to check the init.pp and indeed he/she sees that the data is coming from hiera, he/she won't know where to look if we've defined the data in our 'server.yaml' our 'node.yaml' and our 'common.yaml'.  Obviously this is a simplistic example but am I thinking about this wrong?  Perhaps I've missed/overlooked a major piece of the overall hiera design?  Or, I'm simply a jackass and I'm "using it wrong?"

Valid concern in my experience but there's no guarantee I'm not using it wrong :)

You can check the result on the client, but that's not really practical all the time.  Agreeing a structure of hierarchy with your team, documenting that, and sticking to it, is quite important.

I also use grep to look for a value in the list of files - many many times I've found a value where I wasn't expecting to, and sometimes more than once in a file.  The results of the grep, combined with a dump of hiera.yaml, makes it reasonably easy to tell where a value is set unless it's set in a hundred node.yaml files.

3.  If I want to add multiple 'classes' to say, common.yaml, is there a particular way in which I should is the below structure the preferred method and if so, does hiera parse these from top to bottom or am I likely to run into resource ordering problems in the future?:

classes:
  - one class
  - two class
  - three class

# class one parameters
stuff
blah

# class two parameters
stuff
blah

# class three parameters
stuff
blah

Good question.  I have seen hiera data files with the same value set more than once, but don't recall if it took the top or bottom (or random) result.  I saw it as an error in the file and 'fixed' it.

To my mind, keeping the hiera data files logically split makes things easy for me to maintain.  E.g. classes in one set of files, then class params potentially in separate files (one per class, or one per kind of thing) - whatever makes most sense for your environment.  Directories of files makes things really easy at that point - e.g. all node data files in a 'nodes' directory or some such.

Thank you in advance to whomever replies with thoughts as I realize this is quite a long post.

Cheers,

Mike

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1c26c35b-f0c0-4caf-9ef7-9dfb9070fc2e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Reed]

@Xav, thanks for the information and pointers.  I've added the 'default' value into my arrays and things are running smoothly now.  I'm still a bit hazy on the resource ordering and how hiera by default, deals with applying resources within (for example) a hiera.yaml file but for now, I have enough into to get on with.

Thanks again for your info and your time.

Cheers,

m.

[jcbollinger]

On Monday, September 15, 2014 10:19:35 AM UTC-5, Mike Reed wrote:

@Xav, thanks for the information and pointers.  I've added the 'default' value into my arrays and things are running smoothly now.  I'm still a bit hazy on the resource ordering and how hiera by default, deals with applying resources within (for example) a hiera.yaml file but for now, I have enough into to get on with.

Hiera has no (direct) role in applying resources.  It is strictly a data service.

I think you may mean you're hazy about how Hiera handles interpolation of Puppet variables (%{variable_name}).  You are welcome to ask about that here, but you probably should also consult the documentation, and in particular the docs on interpolation tokens, to help fill in the picture.

Among the things it is important to understand is that each invocation of Hiera, whether explicitly via one of the DSL functions or implicitly for class parameter binding, is independent as far as Hiera itself is concerned.  That's part of why you can use Hiera to initialize variables that Hiera will interpolate on subsequent invocations.


John