Re: Puppet-Hiera: hiera_include and hiera_hash behaviour

[Ellison Marks]

quick definitions first:

hiera() gets a value from the backend. stops searching after it finds a value. the value can be any data type.
hiera_array() gets multiple values from the backend, combining them into one large array as possible.
hiera_hash() gets multiple values from the backend, combining them into one large hash as possible.
hiera_include() calls hiera_array() and then includes each classname returned.

Ok, so hiera_include will look in the backend for a variable named in the call, in your case 'classesA'. this should contain an array of class names to include. As your hierarchy apparently doesn't contain the variable 'classesA', this explains the second error you are getting. You also didn't include the variable 'classesH' in nodeA.yaml, so that might be the cause of the failure there. create_resources is also not used on classes, just types or defines. Generally, for hiera_include, it should look sort of like this.

class classA {
 notice("ClassA")
}

class classB ($param1 = '') { #This uses the auto lookup of parameters. It will perform a hiera search for 'classB::param1'.
 notice ("ClassB: ParamValue $param1")
}

NodeA.yaml
---
classes:
 - a

NodeB.yaml
---
classes:
 - b

'classB::param1': 'Puppet-Hiera'

node default {
   hiera_include('classes', []) #empty array, not empty string, which might also have been messing things up.

  }
}

On Thursday, February 21, 2013 10:55:59 AM UTC-8, Sai_Emc wrote:

I am looking for few clarifications on puppet-hiera integration.

Trying to move away completly from using site.pp. For that I started exploring Hiera. End of the day I want an external system prepare yaml files automatucally based on user requests.

Environment: Ubuntu12.04, PE 2.7, Hiera 1.1.2

EX:
class classA {
 notice("ClassA")
}

class classB ($param1 = '') {
 notice ("ClassB: ParamValue $param1")
}

Now I am want to include these two classes into two different nodes, so prepared two yaml files

hiera.yaml
---
:hierarchy:
    - %{::clientcert}
    - common
:backends:
    - yaml
:yaml:
    :datadir: '/etc/puppetlabs/puppet/hieradata'

NodeA.yaml
---
emcutil::a:
 - a

NodeB.yaml
---
classesH:
 emcutil::b:
     param1: 'Puppet-Hiera'

As I have classes as array and hash, so added below code in site.pp default section so that I can include hash and arrays.
  
node default {
   hiera_include('classesA','')
   $param_packagesH = hiera_hash('classesH')
   create_resources('class',$param_classesH)
  }
}

This approach does not work.

On NodeA following error:
 err: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `empty?' for nil:NilClass at /etc/puppetlabs/puppet/environments/development/manifests/site.pp:48 on node

On NodeB following error:
 err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item classesA in any Hiera data file and no default supplied at /etc/puppetlabs/puppet/environments/development/manifests/site.pp:46 on node
 
What I am doing here is valid?
If we are using hiera_include or hiera_hash all the node yaml files should have all the targets and the target returns not null values?

Do we have alternatives other than using hiera, hiera_include and hiera_hash?
PE 2.7 do we need to do any configuration for Hiera, apart from puppet-hiera package. Specifically related to these two attributes node_terminus, external_nodes?
With Hiera we can move away from site.pp completly. My understanding is correct on Hiera?

Any help on this greatly appreciated.

Sai.
EMC Corporation.
  
  

[Sai_Emc]

Mark,

 

Thanks for the quick reply.

 

"This uses the auto lookup of parameters. It will perform a hiera search for 'classB::param1'."  --> Auto lookup feature what you mentioned is in PE 2.7? I remember reading as 3.0 feature.

 

Regards

Sai.

EMC

[Ellison Marks]

Ah, sorry, my mistake, mised the version. That line would have to be

class classB ($param1 = hiera('classB::param1', '')) {

with the hiera call in the definition. Sorry about that :P

[Sai_Emc]

That's ok Mark. I kind of remember that from documentation.

 

When we use above approach puppet need to make multiple hiera() calls to get resources specific to a class.

 

I mean a class with say 4 arguments, puppet need to make a multiple calls to get all the parameters.

 

NodeB.yaml
---
classes:
 - b

'classB::param1': 'Puppet-Hiera'

'classB::param2': 'Puppet-Hiera'

'classB::param3': 'Puppet-Hiera'

'classB::param4': 'Puppet-Hiera'

 

If we have thousands of nodes we might see any performance issues with array approach? I might be totally wrong on this.

 

The same if I have declared as Hash then from puppet that is just one call, but it might look all defined hierarchies. So if we do use hiera_hash then without using create_resources() is there any other method I can include class definitions.

 

Thank you.

 

Regards
Sai.

EMC

[Ellison Marks]

It's not that bad performance wise. Hiera should only lode the applicable yaml files. If (god forbid) you had one yaml file per host and had thousands of hosts, hiera should still only load two files, the host's yaml file and the common yaml file. The only thing that would be bad is manually maintaining all the yaml files :)

As to the multiple parameters, it's also not that bad. You could format the parameters as an array and seperate them later, like this:

NodeB.yaml
---
classes:
 - b

'classB::params':
 - 'Puppet-Hiera'

 - 'Puppet-Hiera'

 - 'Puppet-Hiera'

 - 'Puppet-Hiera'

class classB ($params = hiera('classB::params', '')) {

  $param1=$params[0]

  $param2=$params[1]

  $param3=$params[2]

  $param4=$params[3]

but it's not that much better. The real trick is in using create resources correctly. You want to have a 3 depth array in hiera. top is the name you will look up with the hiera() call. Second level is the names of the resources you want to create. For example, if you were creating users, this level could have 'john' and 'jane'. Third level is the parameters for the individual resource. In this example, below john could be {'uid'=>501, 'group'=>john} and below jane could be {'uid'=>502, 'group'=>jane}

userhash:

  john:

    uid: 501

    group: john

  jane:

    uid: 502

    group: jane

then you would say

$userhash=hiera('userhash')

create_resources(user, $userhash)

So if you can structure your class in a way that supports that flow, it can be really good.

[Sai_Emc]

Wonderful...This is cool.

 

As you said, maintaining thousands yaml files manually might be difficlut. I have to automate this process to accept requests from an external system about which node needs which resources.

 

So I might use json over yaml, because at this point I dont know my external system can do yaml. Any idea json backend supported in 2.7?. I saw in the documentation but not sure supported in 2.7 or not. 

 

Once again thank you very much.

 

Regards

Sai.

EMC

[Sai_Emc]

I can see json_backend working in latest hiera. Tried from command line its working . puppetmaster was looking at old hiera-0.3.0 gem even whem latest hiera is installed. Posted a different topic for that.

 

Thanks.

Sai

EMC