Need help with simple hiera example

[Stack Kororā]

Greetings,

I don't know why, but I am having a rough time trying to get hiera to work. It seems to me that all the examples I see online are either absurdly complex or so stupidly simple that they are absolutely useless. Either way I have found the documentation for hiera completely lacking (it doesn't help that there is a lot of bad information pertaining old puppet releases that is no longer the correct way to do things...at least according to other sources which may or may not be the correct way either...bleck...). Of all the documentation/blogs/examples/ect I have been pouring over the last 2 hours, not one has given me a useful hint at getting this working.

*deep breath in an attempt to control my frustration so I can communicate on a somewhat intelligent level*

Whew...

OK. Lets start.

$ puppet -V

3.7.1

$ hiera -V

1.3.4

$ hostname

puppet.test.vm

$ sudo puppet cert list --all

+ "puppet.test.vm" <snip blah blah string>

This is as simple as I can make it.

$ cd /etc/puppet

$ find . -type f

./modules/testhiera/manifests/init.pp

./manifests/site.pp

./puppet.conf <- did not touch after test vm install

./auth.conf <- did not touch after test vm install

$ cat manifests/site.pp

node 'puppet.test.vm' {

class { 'testhiera': }

}

$ cat modules/testhiera/manifests/init.pp

class testhiera ( $test="blah") {

file { "/tmp/$test" : ensure => present}

}

$ puppet agent -t

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Caching catalog for puppet.test.vm

Info: Applying configuration version '1412771807'

Notice: /Stage[main]/Testhiera/File[/tmp/blah]/ensure: created

Notice: Finished catalog run in 0.04 seconds

$ rm /tmp/blah

Hooray! That works. Can't get much simpler then that, right? OK, lets add in hiera. Should be simple right? Ha!

$ sudo ln -s /etc/puppet/hiera.yaml /etc/hiera.yaml

$ find . -type f

./hiera.yaml

./hosts/puppet.test.vm.yaml

./modules/testhiera/manifests/init.pp

./manifests/site.pp

./puppet.conf

./auth.conf

# Only added these two files below; made no other changes

$ cat hiera.yaml

---

:hierarchy:

- hosts/%{clientcert}

:backends:

- yaml

:yaml:

:datadir: '/etc/puppet/'

$ cat hosts/puppet.test.vm.yaml

---

hieratest::test: yadda

$ hiera hieratest::test clientcert=puppet.test.vm

yadda

# Hiera on the command line works. I must be making progress!!

$ sudo puppet agent -t

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Caching catalog for puppet.test.vm

Info: Applying configuration version '1412771807'

Notice: /Stage[main]/Testhiera/File[/tmp/blah]/ensure: created

Notice: Finished catalog run in 0.03 seconds

What??? That is absolutely contrary to the documentation! It should have created /tmp/yadda! I am looking at the official docs right now on using a hiera variable with a default variable and I don't see how my example is any different in the slightest! Blah should have only been used as a default if the host wasn't found. Clearly either puppet can't find hiera and used default, or it simply ignored the hiera data.

Fine. We will take out the default blah and force it to use something from hiera.

$ rm /tmp/blah

$ cat modules/testhiera/manifests/init.pp

class testhiera ( ) {

file { "/tmp/$test" : ensure => present}

}

$ sudo puppet agent -t

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Caching catalog for puppet.test.vm

Info: Applying configuration version '1412773578'

Notice: Finished catalog run in 0.04 seconds

What? No blah, no yadda, nothing! I don't even get an error!!

OK. OK. Fine. Maybe it doesn't like something in hiera. I will give hiera a default.

$ find . -type f

./hiera.yaml

./modules/testhiera/manifests/init.pp

./puppet.conf

./hosts/common.yaml

./hosts/puppet.test.vm.yaml

./auth.conf

./manifests/site.pp

$ cat hiera.yaml

---

:hierarchy:

- hosts/%{clientcert}

- hosts/common

:backends:

- yaml

:yaml:

:datadir: '/etc/puppet/'

$ cat hosts/common.yaml

---

hieratest::test: blarg

$ hiera hieratest::test clientcert=puppet.test.vm

yadda

$ hiera hieratest::test clientcert=some.thing.else

blarg

Yeah...alright...hiera on the command line is giving me exactly what I want. I have feeling good about this one...

$ sudo puppet agent -t

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Caching catalog for puppet.test.vm

Info: Applying configuration version '1412773578'

Notice: Finished catalog run in 0.06 seconds

$ ls /tmp/blah /tmp/yadda /tmp/blarg

ls: cannot access /tmp/blah: No such file or directory

ls: cannot access /tmp/yadda: No such file or directory

ls: cannot access /tmp/blarg: No such file or directory

ACK!!!! What the hell?!?!! Nothing? Not even an error?? Even running –debug on that puppet run gives me jack-squat of information. There isn't anything in the puppet master logs either.

I have tried doing the hiera_include (even though several places say don't though they fail to mention why not). I have tried doing $t2 = hiera('test') and creating /tmp/$t2 and that still gave me nothing. I have tried referencing it by full scope hieradata::test (again, even though the docs say don't do this) and still nothing.

Clearly, the hiera data works because it gives me exactly what I want on the command line. Puppet just seems to flat out ignore it. I can't get any of the examples to work.

HOWEVER, I tested a few of the puppet forge programs and THEY can reference their hiera data just fine! Oh that really got me riled up...So it isn't puppet because it works with their code. It has to be something in my code. So I reset the VM back to my code and I fail to understand why my incredibly simple code isn't working and I am seriously frustrated why I can't get any of the official documentation examples to work either.

I need to go take a break from this...but I would be so very very grateful if someone could point out where I am going wrong. Hiera works, but Puppet+Hiera doesn't and I just don't understand why...Since the puppet forge code works, it *must* be something I have/haven't done but I can not seem to find it. The docs certainly aren't helping either. Any pointers/tips/examples/information would be greatly appreciated.

Thanks!

~Stack~

[Corey Osman]

Seems like  your clientcert is different during the puppet run than the command line since your explicitly passing in the clientcert.  I would change clientcert in your hiera config to fqdn.  You are not guaranteed to have a client cert, but all hosts will have a fqdn.  You can also use the hiera function in your puppet code so it makes that "automatic" lookup easier to understand since you control the lookup key.  Although, I don't recommend this since automated lookups are preferred as two lookups would actually occur if myfoo::bar doesn't exist.   If you run your puppet master in debug mode, it will actually detail when it can't find a lookup key, so have a look at the logs.

- hosts/%{fqdn}

ie. 

class foo(

    $bar = hiera('myfoo::bar', 'defaultvalue')

    # if the hiera function I specified as the default value for foo returns nil, the puppet automated lookup will occur and use 'foo::bar' lookup key

}

Hiera is pretty easy once you know the rules and I bet that your puppet agent clientcert must be different than what you have in your hiera datastore.

[Stack Kororā]

Greetings Corey,

Thank you for your advice.

If it was as simple as the clientcert being wrong, then it should have been hitting the default hiera option, but it wasn't pulling back anything at all. I honestly don't think it was even looking at hiera as I saw nothing about it in the logs.

We actually switched from fqdn to clientcert because we found clientcert more reliable. We have a significant number of laptops that travel between networks and would respond back with different fqdn depending on what wireless hub they were on (don't even get me started on the network team...we have frequent disagreements on things:-D ).

However, I did get it working and you reaffirmed something I had discovered. So thank you very much for responding. I do appreciate it.

______________________________________________________________


Greetings all that are following this saga...

For those following along with my code above, I mistyped hieratest in my first email but in the file it is correct (that's what I get for retyping instead of mucking with copy/paste from the VM :D ):

$ cat hosts/puppet.test.vm.yaml
---
hieratest::test: yadda

Since the documentation is near rubbish on hiera, I started digging around in the code itself for the hiera projects that _do_ work for me. I noticed that there were two methods that seemed to pop up frequently.

1) The method Corey mentioned:

$bar = hiera('myfoo::bar', 'defaultvalue')

However, the puppet docs basically say do this for 2.7 but not for 3+ [ https://docs.puppetlabs.com/hiera/1/puppet.html ]. Also note that there are two examples on that page which completely fail (silently...at least in my example code).
THIS DOESN'T WORK:
1.1) class myclass ( $parameter_one = hiera('myclass::parameter_one', 'default text')
1.2) class myclass ($parameter_one = "default text") { ...content => $parameter_one, ...}

The first I couldn't get working until I pulled it into the body of the class as Corey mentioned. Putting it in the class definition wouldn't work for me as it is listed in the documentation.
The second will /always/ go to 'default text' for me. It has yet to pull back the hiera data.

2) $ cat modules/testhiera/manifests/init.pp
class testhiera ( $test = $hieratest::test ) {

  file { "/tmp/$test" : ensure => present}
}

Since I started adopting the second method ( shortly after lunch ) it has been working well. I am migrating a few of the simpler puppet modules I have written to hiera and I totally get the value of hiera. I kinda wish I had started sooner. But to spend that much time and effort on such a simple example, it's no wonder so many people avoid starting with it. I wonder how many people just give up after the example code doesn't work for them...

But, its working now! Hooray!

~Stack~

[Stack Kororā]

In case anyone else struggles with Hiera and needs a simple example:
https://github.com/cstackpole/SimpleHiera

That is my code so you don't have to try to cut'n'paste out of a forum post. If I learn anything else useful, I will shove it into the repo too.

~Stack~

[Stack Kororā]

OK, I am beginning to question either puppet+hiera's reliability or my sanity...

I have code I am working on. It is existing code in which has a bunch of variables prime for hiera use. I pull one variable out of the code into hiera for testing and it works well. Tweak the code and now I am only using hiera for that variable. Hooray!

Then i pull a second variable out. Nothing in hiera works now. There is *nothing* in the --debug log about hiera except for when it reads the /etc/puppet/hiera.yaml file and a few SELinux context pieces. Well that is weird. I didn't do anything different that I can tell...so I will revert back to my git commit to the last working state (the first variable). Nothing. I can't get it to do anything at all. I am at a complete loss as to why it stopped working.

On a hunch I did a 'sudo service puppetmaster restart' and the first variable started working again! Hooray! Why puppetmaster needs a restart, I have NO idea but whatever. Lets add a test dummy variable into hiera and do a simple notify. Nope. Broken again. Maybe another puppetmaster restart? Still no.

So either hiera is only capable of doing a single variable, or it /really/ hates me. I am willing to bet on the latter.

My code is literally only a single line in the hiera file (one for the host and one for common) 'repos::testvar blah' and 'repos::testvar yadda' and in my init.pp file I just added the one line in the declaration '$testvar=$repos::testvar' and in the class body I added 'notify {"test $testvar":}'. That apparently is enough to break puppet's integration with hiera.... Even a 'sudo service puppetmaster restart' doesn't fix that..If I reset those three lines then reset the puppetmaster, it start working again...

Well, maybe it is the code? Other then the name change there is nothing different than my SimpleHiera test. So I threw the notify code into that, and it works! In fact, the code working in the SimpleHiera test case I built earlier /while the other class is still enabled/. The repos class fails while the test code works! o_O     Just to clarify that point. The SAME code works in one module but not in the other! (obviously I changed the top level scope of the variables to match the module it was being referenced in; but the code is otherwise identical) Thus, puppet+hiera has a selective memory on which time it feels like looking up hiera data...<sarcasm>That sounds like a product I want to trust in production! </sarcasm>

How do people debug hiera in puppet? Everything /always/ works on the command line so I really don't think it is hiera nor my configuration of hiera. And there is never anything of use in the puppet logs. At least nothing I see with a 'puppet agent -t | egrep "hiera|repos|testvar".

I have *got* to be missing something. I feel that the average puppet+hiera dev doesn't struggle constantly with puppet suddenly forgetting how to use hiera everytime they add a new variable. But I am at a loss as to what i am doing wrong...

Any help would be greatly appreciated.

Thanks!
~Stack~

[Stack Kororā]

Just so that the code is out there and my point made.

$ cat hieradata/hosts/puppet.test.vm.yaml
---
repos::reponetwork: one
repos::testvar: repotestvar
testhiera::test: blah
testhiera::testvar: testhieratestvar

$ cat modules/testhiera/manifests/init.pp
class testhiera ( $testvar = $testhiera::testvar ) {
#class testhiera ( $test = $testhiera::test ) {
  notify { "$testvar on $clientcert":}

}

$ puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin

Info: Loading facts

Info: Caching catalog for puppet.test.vm

Info: Applying configuration version '1412825217'
Notice: 2 testhieratestvar on puppet.test.vm
Notice: /Stage[main]/Testhiera/Notify[testhieratestvar on puppet.test.vm]/message: defined 'message' as '2 testhieratestvar on puppet.test.vm'
Notice:  on puppet.test.vm
Notice: /Stage[main]/Repos/Notify[ on puppet.test.vm ]/message: defined 'message' as ' on puppet.test.vm '

Notice: Finished catalog run in 0.04 seconds

Testhiera works, Repos doesn't.

Here is another interesting tidbit....If I pull that notify into a subclass of testhiera....it too stops working.... Ugh...I think it is time for me to go to sleep...this obviously isn't sinking in...

[Stack Kororā]

Ack! So I put the '$testvar = $repos::testvar' line in the class header for both the init.pp and the epel.pp (which is a subclass in repos). I removed it out of epel.pp and the init.pp started to print!!! That makes zero sense to me! So I removed it and the notify out of init.pp and put them in epel.pp and they both stop functioning!

Can subclasses not reference hiera data? That seems really weird....

OK...seriously...going to bed now... :-)

[jcbollinger]

On Wednesday, October 8, 2014 8:44:56 AM UTC-5, Stack Kororā wrote:

Greetings,

I don't know why, but I am having a rough time trying to get hiera to work. It seems to me that all the examples I see online are either absurdly complex or so stupidly simple that they are absolutely useless. Either way I have found the documentation for hiera completely lacking (it doesn't help that there is a lot of bad information pertaining old puppet releases that is no longer the correct way to do things...at least according to other sources which may or may not be the correct way either...bleck...). Of all the documentation/blogs/examples/ect I have been pouring over the last 2 hours, not one has given me a useful hint at getting this working.

*deep breath in an attempt to control my frustration so I can communicate on a somewhat intelligent level*

Whew...

Keep breathing.  Your frustration is coming through clearly -- and I don't blame you for it -- but Hiera really does work, and there's not much more to it than your research has already turned up.

 

OK. Lets start.

[...]
 

$ hiera hieratest::test clientcert=puppet.test.vm

yadda

# Hiera on the command line works. I must be making progress!!

$ sudo puppet agent -t

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Caching catalog for puppet.test.vm

Info: Applying configuration version '1412771807'

Notice: /Stage[main]/Testhiera/File[/tmp/blah]/ensure: created

Notice: Finished catalog run in 0.03 seconds

What??? That is absolutely contrary to the documentation! It should have created /tmp/yadda!

If the master had successfully looked up your datum then the result would have been as you expected.  The behavior you present is characteristic of (and well documented for) the case where the automatic lookup fails, leaving the master to fall back to the default value given in the class definition.  The problem is not in your Puppet manifests, and your CLI tests demonstrate that it is not in your data themselves, but those are not the only possibilities.

 

I am looking at the official docs right now on using a hiera variable with a default variable and I don't see how my example is any different in the slightest! Blah should have only been used as a default if the host wasn't found. Clearly either puppet can't find hiera and used default, or it simply ignored the hiera data.

Fine. We will take out the default blah and force it to use something from hiera.

$ rm /tmp/blah

$ cat modules/testhiera/manifests/init.pp

class testhiera ( ) {

file { "/tmp/$test" : ensure => present}

}

$ sudo puppet agent -t

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Caching catalog for puppet.test.vm

Info: Applying configuration version '1412773578'

Notice: Finished catalog run in 0.04 seconds

What? No blah, no yadda, nothing! I don't even get an error!!

How's hiera involved in that?  You now have $test as an uninitialized local variable (no longer a class parameter).  It expands to nothing when you interpolate it into your filename, so you're managing File["/tmp/"], which is equivalent to File["/tmp"].  That file (directory) already exists, and that's all you ask Puppet to ensure, so Puppet does nothing.

 

OK. OK. Fine. Maybe it doesn't like something in hiera. I will give hiera a default.

[...]

Unless you change the class back, you've already run off the rails here.  You have no hiera lookup involved any more.

John

[jcbollinger]

On Wednesday, October 8, 2014 8:09:28 PM UTC-5, Stack Kororā wrote:

 

However, I did get it working and you reaffirmed something I had discovered. So thank you very much for responding. I do appreciate it.

I'm glad you got it working.
 

Since the documentation is near rubbish on hiera, I started digging around in the code itself for the hiera projects that _do_ work for me. I noticed that there were two methods that seemed to pop up frequently.

1) The method Corey mentioned:
$bar = hiera('myfoo::bar', 'defaultvalue')

However, the puppet docs basically say do this for 2.7 but not for 3+ [ https://docs.puppetlabs.com/hiera/1/puppet.html ].

In Puppet 2, that was the only way you could do it.  After Puppet 3 was released, it was common for module authors to attempt to support both P2 and P3, which tied them to that form.  Even with official support for the P2 series ending, there is probably still interest in compatibility from some quarters.

As for using only automatic parameter binding in P3, that's basically a code style argument, its position right in the documentation notwithstanding.  The advice is not bad as far as it goes, but do note that it also presents an apparently acceptable (albeit downplayed) alternative: explicitly document the hiera keys your class uses.  It seems like you've probably been around enough to see a code-style war or two, so I'll leave it there.

 

Also note that there are two examples on that page which completely fail (silently...at least in my example code).
THIS DOESN'T WORK:
1.1) class myclass ( $parameter_one = hiera('myclass::parameter_one', 'default text')

That's hideous and wasteful (in current Puppet), but it ought to work.  Hiera calls (and other function calls) do work in class parameter lists, unless there's been a regression I don't know about.  In that particular case, Puppet 3.x should perform two lookups of the same key.  The only reason to do it is for compatibility with Puppet 2.x, which does not have automated data binding (and therefore will perform only one lookup).

 

1.2) class myclass ($parameter_one = "default text") { ...content => $parameter_one, ...}

[...] will /always/ go to 'default text' for me. It has yet to pull back the hiera data.

If that's true then something is dreadfully wrong in your environment.  Class parameter defaults do not interfere with automated data binding.  Puppet's automated test suite verifies this, so if you're seeing different behavior then there is some sort of local modification or environment issue that is causing breakage.  I hope you'll forgive me if I judge it more likely that your tests are misleading you.

 

2) $ cat modules/testhiera/manifests/init.pp
class testhiera ( $test = $hieratest::test ) {
  file { "/tmp/$test" : ensure => present}
}

How is that different from your 1.2, which you say doesn't work?

It seems like you may have a misconception: automated data binding applies specifically to class parameters, as an aspect of resolving their values.  It does not apply to class variables that are not parameters, and it is not automagically applied to parameters of classes that have not yet been evaluated.

Your code (2) is dangerous for one of two reasons:

1. If the code is written as you intended, then it relies for a class parameter default on a class variable of a different class, without doing anything to ensure that that other class (hieratest) has been evaluated.  You can find extensive discussion of this issue and others related to using other classes' variables for parameter defaults by googling "puppet params class pattern".
   
2. If you meant to write "class testhiera ( $test = $testhiera::test )", then you are specifying the default value for class parameter $teshiera::test to be itself.  In that case, that default value will be used only in situations where it has not been intialized, so I think it's functionally equivalent to "class testhiera ( $test = undef )".
   

Either way, your code will work fine as long as the hiera lookup for your class parameter succeeds (so that the default value is not consulted).  It's what will happen when the lookup accidentally or intentionally fails that may come back to bite you in the assets.

 

But, its working now! Hooray!

Hooray!


John

[Tony Thayer]

On Wednesday, October 8, 2014 6:44:56 AM UTC-7, Stack Kororā wrote:

Couple of things I noticed:

Your testhiera/manifests/init.pp file looks a little odd. I re-wrote it a bit and it ran without a hitch.

class testhiera {

  $test = hiera('testhiera::test')

  notice("Test is ${test}")

  file { "/tmp/${test}" :

    ensure => present,

  }

}

But keep in mind that I am leaning entirely on hiera in my environment. My site.pp file only contains hiera_include('classes') (and some other stuff to omit .svn/.git directories from being backed up).

My personal take is that you should go all or nothing with it. That way you aren't maintaining node definitions in site.pp AND in your node .yaml files.

- Tony

[Stack Kororā]

Wow! Thanks for the responses John!

> On Thursday, October 9, 2014 8:52:00 AM UTC-5, jcbollinger wrote:
> If the master had successfully looked up your datum then the result would
> have been as you expected.  The behavior you present is characteristic of
> (and well documented for) the case where the automatic lookup fails,
> leaving the master to fall back to the default value given in the class
> definition.  The problem is not in your Puppet manifests, and your CLI
> tests demonstrate that it is not in your data themselves, but those are not
> the only possibilities.

If it isn't hiera and it isn't puppet, what other possibilities are there? Any idea on how I can debug this better? I would love to see in the puppet logs something like "found a variable with no definition, looking up in hiera in this file...nope didn't find it...trying this other file...ah found it" but I haven't seen anything on how to better troubleshoot /how/ puppet is determining if a variable is in hiera or not.

    

> How's hiera involved in that?  You now have $test as an uninitialized local
> variable (no longer a class parameter).  It expands to nothing when you
> interpolate it into your filename, so you're managing File["/tmp/"], which
> is equivalent to File["/tmp"].  That file (directory) already exists, and
> that's all you ask Puppet to ensure, so Puppet does nothing.

Wow. That totally makes sense. I was under the impression (very likely a mistaken impression apparently) that puppet would look at $test, not find anything in it, and then do look up in hiera for the full scope of the variable (aka: class::variable -> testhiera::test). So this explains that some of the "oddity" i've seen is just a misunderstanding of the documentation. Thanks for pointing that out!

On Thursday, October 9, 2014 9:36:38 AM UTC-5, jcbollinger wrote:
>> $bar = hiera('myfoo::bar', 'defaultvalue')
>> However, the puppet docs basically say do this for 2.7 but not for 3+
>> [ https://docs.puppetlabs.com/hiera/1/puppet.html ].

> In Puppet 2, that was the only way you could do it.  After Puppet 3 was
> released, it was common for module authors to attempt to support both P2 and
> P3, which tied them to that form.  Even with official support for the P2
> series ending, there is probably still interest in compatibility from some
> quarters.
>
> As for using only automatic parameter binding in P3, that's basically a code
> style argument, its position right in the documentation notwithstanding.  The
> advice is not bad as far as it goes, but do note that it also presents an
> apparently acceptable (albeit downplayed) alternative: explicitly document
> the hiera keys your class uses.  It seems like you've probably been around
> enough to see a code-style war or two, so I'll leave it there.

OK. So it isn't going to cause me problems, it is just backwards compatibility and/or a code style choice. Then in that case, I may start using it just so that I have it explicitly called for my own use when I look at the code in 6 months. :-)

    
[snip]

>> 1.2) class myclass ($parameter_one = "default text") { ...content => $parameter_one, ...}
>> [...] will /always/ go to 'default text' for me. It has yet to pull back the
>> hiera data.

> If that's true then something is dreadfully wrong in your environment.

Hrm...that's not good. But this test VM is a very new build of Scientific Linux 6.5 with the latest puppet. I really haven't made many changes. Not sure what I could have goofed on it. Any suggestions for debugging what is wrong? I mean nothing is really being kicked out in the log files (but that doesn't mean there isn't a problem).

> Class parameter defaults do not interfere with automated data binding.
> Puppet's automated test suite verifies this, so if you're seeing different
> behavior then there is some sort of local modification or environment issue
> that is causing breakage.  I hope you'll forgive me if I judge it more likely
> that your tests are misleading you.

No worries. I get it. Chances are much more likely I goofed something up. I just don't know what. :-)

>> 2) $ cat modules/testhiera/manifests/init.pp
>>      class testhiera ( $test = $hieratest::test ) {
>>      file { "/tmp/$test" : ensure => present}
>>      }

> How is that different from your 1.2, which you say doesn't work?

Not sure i understand your question. The 1.2 that doesn't work I am setting a default value in hopes that the hiera value is taken. In this example, I am explicitly calling the hiera value...Now functionally it may be the same (or at least it is supposed to be), but that isn't how I see it behave.

> It seems like you may have a misconception: automated data binding applies
> specifically to class parameters, as an aspect of resolving their values.  It
> does not apply to class variables that are not parameters, and it is not
> automagically applied to parameters of classes that have not yet been evaluated.
>
> Your code (2) is dangerous for one of two reasons:
>
> If the code is written as you intended, then it relies for a class parameter
> default on a class variable of a different class, without doing anything to
> ensure that that other class (hieratest) has been evaluated.  You can find
> extensive discussion of this issue and others related to using other classes'
> variables for parameter defaults by googling "puppet params class pattern".
>
> If you meant to write "class testhiera ( $test = $testhiera::test )", then
> you are specifying the default value for class parameter $teshiera::test to
> be itself.  In that case, that default value will be used only in situations
> where it has not been intialized, so I think it's functionally equivalent to
> "class testhiera ( $test = undef )".
>
> Either way, your code will work fine as long as the hiera lookup for your
> class parameter succeeds (so that the default value is not consulted).  It's
> what will happen when the lookup accidentally or intentionally fails that may
> come back to bite you in the assets.

The second case was my intent, and that might explain a lot of the behaviour I have seen.

I have plans for tonight, but I am going to try hacking on hiera again this weekend. Fingers crossed I can make good progress in understanding hiera. :-)

Thank you for your help on this!
~Stack~

[Stack Kororā]

On Thursday, October 9, 2014 12:49:40 PM UTC-5, Tony Thayer wrote:
> Couple of things I noticed:
> Your testhiera/manifests/init.pp file looks a little odd. I re-wrote it a bit and it ran without a hitch.
> class testhiera {
>   $test = hiera('testhiera::test')
>   notice("Test is ${test}")
>   file { "/tmp/${test}" :
>   ensure => present,
>   }
> }
>
> But keep in mind that I am leaning entirely on hiera in my environment. My
> site.pp file only contains hiera_include('classes') (and some other stuff to
> omit .svn/.git directories from being backed up).
> My personal take is that you should go all or nothing with it. That way you
> aren't maintaining node definitions in site.pp AND in your node .yaml files.
>    - Tony

Thanks for the code fix! I would like to get to the point where all of the data is in hiera. But I have a pretty big environment and unfortunately it was one of those 'Boy, that escalated quickly' deals. It started off as 'Hey this puppet thing is cool!' and quickly tumbled into 5 sys-admins hacking/patching code together for their servers. A few months ago I really started to clean things up when I realized that an average puppet run was about 10 minutes...I fixed a few big things and re-wrote one module so now it is down to about 3 minutes but that is still far too long for what is going on. The more I looked at that code the more I really began dreading to look at the next module. Now that we have trimmed the number of OS's we support to just CentOS 5&6, Scientific Linux 5&6, RHEL 6, and SLES11, there is a /lot/ of dead puppet code for all the weird hacks we had to do. I want to rip all of that out too!

The point being, I can't just flip a switch and use hiera off the bat. This is going to be a tedious process as I switch out one module at a time with a re-write. So I need to make sure I do this right and balance things properly. This is also the biggest reason why I worry about a variable not being found and Puppet silently carrying on without telling anyone that there is a problem. The good thing is that I have a mostly-complete dev virtual environment on my laptop, plus a much-closer-to-production dev virtual environment at work, plus a good sampling of 'production boxes we can afford to test on' (and I test code in that order). So by the time we are ready to push to proper prod, things VERY rarely go completely sideways on us.

I really like the idea of one place to control the specific variables needed for a host and I /really/ like the idea of having our modules in such a manner that we can just hand that code over to someone without worrying about sensitive data in them (even if no one would ever use those modules...a lot of them are very specific to our environment but I still don't want that kind of information in them). So I think this is a good goal for me to move forward to. It just will take some time. :-)

~Stack~

[jcbollinger]

On Thursday, October 9, 2014 5:23:39 PM UTC-5, Stack Kororā wrote:

Wow! Thanks for the responses John!

> On Thursday, October 9, 2014 8:52:00 AM UTC-5, jcbollinger wrote:
> If the master had successfully looked up your datum then the result would
> have been as you expected.  The behavior you present is characteristic of
> (and well documented for) the case where the automatic lookup fails,
> leaving the master to fall back to the default value given in the class
> definition.  The problem is not in your Puppet manifests, and your CLI
> tests demonstrate that it is not in your data themselves, but those are not
> the only possibilities.

If it isn't hiera and it isn't puppet, what other possibilities are there? Any idea on how I can debug this better? I would love to see in the puppet logs something like "found a variable with no definition, looking up in hiera in this file...nope didn't find it...trying this other file...ah found it" but I haven't seen anything on how to better troubleshoot /how/ puppet is determining if a variable is in hiera or not.
    

I said the problem is not your data, but that doesn't cut Hiera out of the picture completely.

One possibility would be that the agent is using a different hiera configuration than you access from the command line, so even though your data are ok, the agent is not consulting them.  Since you now have hiera lookups apparently working (at least to some degree), we can probably reject that as an explanation for any remaining issues (but you may want to verify which issues do remain).

The main other category of possibilities I see is file access issues.  It is not an error for an hiera data file for any hierarchy level to be missing or inaccessible: if for any reason hiera cannot read the data file nominated for some hierarchy level, then it just moves on to the next hierarchy level.  Even if the agent runs as root, there can still be access controls (e.g. SELinux policy) that prevent it from reading certain files that can be read by commands you launch directly from a shell.

 

>> 1.2) class myclass ($parameter_one = "default text") { ...content => $parameter_one, ...}
>> [...] will /always/ go to 'default text' for me. It has yet to pull back the
>> hiera data.

> If that's true then something is dreadfully wrong in your environment.

Hrm...that's not good. But this test VM is a very new build of Scientific Linux 6.5 with the latest puppet. I really haven't made many changes. Not sure what I could have goofed on it. Any suggestions for debugging what is wrong? I mean nothing is really being kicked out in the log files (but that doesn't mean there isn't a problem).

Verify that the problem persists (you have presented conflicting data about that).  If so, turn on debug logging at the master, and see whether anything useful turns up in its log.  Also, make sure you don't anywhere have a declaration (as opposed to a definition) of the class that gives an explicit parameter value.  That is, make sure you don't anywhere have this:

    class { 'myclass': parameter_one => 'default_text' }

 

>> 2) $ cat modules/testhiera/manifests/init.pp
>>      class testhiera ( $test = $hieratest::test ) {
>>      file { "/tmp/$test" : ensure => present}
>>      }

> How is that different from your 1.2, which you say doesn't work?

Not sure i understand your question.

At (1.2) you say this doesn't work:

    class myclass ($parameter_one = "default text") { ... }

At (2) you say THIS (corrected per your response) does work:

    class testhiera ( $test = $testhiera::test ) { ... }

As far as I can tell, they are not different in any way that matters.  In particular, the reference to $testhiera::test in (2) (or anywhere else) does not itself cause Hiera to be consulted.  If you are now getting a value drawn from your Hiera data in (2) then you should continue to get the same value if you change that parameter's default to "default text".

 

The 1.2 that doesn't work I am setting a default value in hopes that the hiera value is taken. In this example, I am explicitly calling the hiera value...Now functionally it may be the same (or at least it is supposed to be), but that isn't how I see it behave.

I'm having trouble believing that changing the expression of a class parameter's default value between a variable reference and a string literal would affect whether that parameter's value is bound via Hiera.  If you can confirm such behavior then I'm sure PL would appreciate a bug report.

John

[Stack Kororā]

On Friday, October 10, 2014 9:18:03 AM UTC-5, jcbollinger wrote:

 Even if the agent runs as root, there can still be access controls (e.g. SELinux policy) that prevent it from reading certain files that can be read by commands you launch directly from a shell.

OMG. If that was the problem I am going to kick myself in shame. I did a restorecon on /etc/puppet and it changed the context of my hiera files and several files in my module directory. I, at this time, am not able to replicate the weird errors I had before.
*sigh*

OK. I am going to focus on writing a proper module with hiera and see if I run into any more issues. I will report back later if it works out.

Thanks! I do appreciate the help!
~Stack~

[Stack Kororā]

Greetings,

So yeah. I spent several hours hacking away at adding hiera data to a bunch of my modules. I haven't seen any weirdness going on. I wish there had been /any/ kind of indication in puppet about "trying to read this file but I don't have permission/access rights". That would have saved a lot of headache/frustration. Had I had any thought about it being permission based problems, I probably would have thought about SELinux a lot sooner but that didn't cross my mind since the command line version worked...oh well..

Thank you so much for that suggestion!
~Stack~

[Felix Frank]

Good point. Could you file that as a feature request?

https://tickets.puppetlabs.com

Cheers,
Felix