puppet master following 3.7 upgrade intermittently gives errors for a specific set of classes

[Lori Cho]

We have 2.7 agents, that are now hitting 3.7 masters.  

One of the masters is doing an odd thing.  We have a module called httpd, with a bunch of classes in that module, like httpd::common, httpd::init, etc.  

Sometimes, on a puppet run, we see:

Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class httpd::common for <node>

However, this does not happen on every run, so it's not an actual issue of the class not existing.  

This "Could not find class" error is only happening with these classes in the httpd module, even though we use subclasses all over the place.  

We pointed to a different 3.7 master, and it does not seem to happen there.  

Does anyone have any ideas? 

Thanks,

-Lori

[jcbollinger]

On Monday, March 9, 2015 at 6:37:19 PM UTC-5, Lori Cho wrote:

We have 2.7 agents, that are now hitting 3.7 masters.  

One of the masters is doing an odd thing.  We have a module called httpd, with a bunch of classes in that module, like httpd::common, httpd::init, etc.  

Sometimes, on a puppet run, we see:

Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class httpd::common for <node>

However, this does not happen on every run, so it's not an actual issue of the class not existing.  

This "Could not find class" error is only happening with these classes in the httpd module, even though we use subclasses all over the place.  

Terminology note: although you might indeed use subclasses, nothing you have shown actually indicates that, and it's not anyway much relevant.  In particular, the name "httpd::common" does not necessarily refer to a subclass of class "httpd", even if a class "httpd" exists.  Puppet namespacing is orthogonal to class inheritance (and class inheritance is not much used).

 

We pointed to a different 3.7 master, and it does not seem to happen there.  

Does anyone have any ideas? 

If you are observing normal Puppet behavior, as opposed to some sort of failure in the underlying system, then there is surely more of a pattern than you have so far recognized.  My first guess would be that the error occurs for nodes assigned to some specific environment.  Alternatively, if you have some kind automated process running alongside the master that might occasionally prevent the master from reading httpd/manifests/common.pp (for instance, maybe it looks that file for updating) then that could cause the behavior you describe.

Given that you see the problem only with a particular master from a load-balanced group, one alternative would be to rebuild the affected master.  That should be straightforward, supposing that your masters are identical but for host identifiers.  Although it may not by very satisfying to solve the problem without first understanding it, it is better than not solving it at all.


John

[Bostjan Skufca]

On Tuesday, 10 March 2015 00:37:19 UTC+1, Lori Cho wrote:

However, this does not happen on every run, so it's not an actual issue of the class not existing.

Try to restart puppet and rerun the faulty node.

Sometimes errors in your manifest are only fully logged after they are initially encountered, and later only "class not found" is repeated.

b.