puppet agent gives error undefined method `empty?' for nil:NilClass

[Tom Hanstra]

I have a puppet master which has over 50 puppet agents talking to it just fine.  But I have one particular server on which the agent, after connecting and exchanging keys, gives the following error:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `empty?' for nil:NilClass at /etc/puppet/modules/server/iptables/manifests/init.pp:7 on node curatevanilla.library.nd.edu

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Having read other posts, and knowing that we do use hiera, this looks like a hiera problem.  If I comment out classes which don't use hiera, the agent will run them all.  But add any in which use hiera and that will be the class which fails (iptables above is just the first one it hits).

How do I best track down where the problem really is?  I have made sure that the /etc/hiera.yaml file is in place and it matches other servers which work.  I've tried clearing out /var/lib/puppet on the server and letting that get rebuilt.  If I move /var/lib/hiera out of the way, it does not get rebuilt, probably because the download does not get that far.  Any suggestions on what might allow me to find the conflict. 

Thanks,

Tom

[jcbollinger]

On Tuesday, February 25, 2014 5:17:04 PM UTC-6, Tom Hanstra wrote:

I have a puppet master which has over 50 puppet agents talking to it just fine.  But I have one particular server on which the agent, after connecting and exchanging keys, gives the following error:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `empty?' for nil:NilClass at /etc/puppet/modules/server/iptables/manifests/init.pp:7 on node curatevanilla.library.nd.edu

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Having read other posts, and knowing that we do use hiera, this looks like a hiera problem.

That's a reasonable possibility, but it's by no means the only one.  All the error really tells you is that at some point during compilation of the affected node's catalog, Puppet is running into an uninitialized Puppet variable.

 

 If I comment out classes which don't use hiera, the agent will run them all.  But add any in which use hiera and that will be the class which fails (iptables above is just the first one it hits).

So that does seem to tie the problem to hiera, but it's odd that it doesn't matter which hiera-using class you try to use.  That suggests a problem with your overall hiera configuration, rather than a problem with the data or any particular class.

 

How do I best track down where the problem really is?  I have made sure that the /etc/hiera.yaml file is in place and it matches other servers which work.  I've tried clearing out /var/lib/puppet on the server and letting that get rebuilt.  If I move /var/lib/hiera out of the way, it does not get rebuilt, probably because the download does not get that far.  Any suggestions on what might allow me to find the conflict. 

Have a look at your hiera.yaml, and carefully consider any variable interpolations in it.  When compiling a catalog for the affected node, are all those variables defined?  Such interpolations are usually based on node facts, but maybe your node is not providing one of the needed facts.

If that doesn't yield fruit, then create a separate node block just for the affected node, so that you can troubleshoot without affecting management of your other nodes.  Then use that node block to find a minimum set of classes that will trigger the error (perhaps just one).  Prefer simple classes for this purpose over more complex ones.  That should narrow the scope of the problem.  You can narrow it further by figuring out based on one or more failing classes the minimum it takes to build a class that will fail.

If you are still flummoxed after figuring out what minimum combination of declarations causes the failure then bring it back here.  We can probably help work out the specifics.


John