what is actually "undefined method 'include?' for nil:NilClass on node" error?

[Sans]

 I have two identical nodes - serv106 and serv107 - one of which is working just fine but the other one failing with these error message:

err: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `include?' for nil:NilClass on node
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

running puppet master in the foreground, I see these on the screen:

err: undefined method `include?' for nil:NilClass on node
err: undefined method `include?' for nil:NilClass on node
debug: Received report to process from serv106.syst.local
debug: Processing report from serv106.syst.local with processor Puppet::Reports::Store
debug: Processing report from serv106.syst.local with processor Puppet::Reports::Http
err: Report processor failed: Connection refused - connect(2)
debug: Processing report from serv106.syst.local with processor Puppet::Reports::Log
err: //serv106.syst.local/Puppet: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `include?' for nil:NilClass on node
warning: //serv106.syst.local/Puppet: Not using cache on failed catalog
err: //serv106.syst.local/Puppet: Could not retrieve catalog; skipping run

a bit if google-search suggested that  removing certificates from both master and the agent (and recreating afterwards) is the solution to this issue. Which did but no joy so far. Has any one ever seen this error before or know what's I'm doing wrong here. Any help/pointer would be greatly appreciated.

Best!

[Felix Frank]

Hi,

this error can me pretty much anything - you're likely hitting some kind of bug (perhaps just a lack of error handling).

Run the puppet master with the --trace flag to get more details on what's happening specifically.

Felix

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/32a78b4c-2b58-454f-9e59-10c4a6081d4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sans]

Thanks Felix for the heads up.
Sorry for the long post. That's all I get using --trace option but I still can't figure out what's going wrong.

notice: Starting Puppet master version 2.7.23
info: access[^/catalog/([^/]+)$]: allowing 'method' find
info: access[^/catalog/([^/]+)$]: allowing $1 access
info: access[^/node/([^/]+)$]: allowing 'method' find
info: access[^/node/([^/]+)$]: allowing $1 access
info: access[/certificate_revocation_list/ca]: allowing 'method' find
info: access[/certificate_revocation_list/ca]: allowing * access
info: access[^/report/([^/]+)$]: allowing 'method' save
info: access[^/report/([^/]+)$]: allowing $1 access
info: access[/file]: allowing * access
info: access[/certificate/ca]: adding authentication any
info: access[/certificate/ca]: allowing 'method' find
info: access[/certificate/ca]: allowing * access
info: access[/certificate/]: adding authentication any
info: access[/certificate/]: allowing 'method' find
info: access[/certificate/]: allowing * access
info: access[/certificate_request]: adding authentication any
info: access[/certificate_request]: allowing 'method' find
info: access[/certificate_request]: allowing 'method' save
info: access[/certificate_request]: allowing * access
info: access[/]: adding authentication any
info: Inserting default '/status' (auth true) ACL because none were found in '/etc/puppet/auth.conf'
info: Expiring the node cache of
/usr/lib/ruby/site_ruby/1.8/puppet/node.rb:110:in `names'
/usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:212:in `evaluate_ast_node'
/usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:101:in `compile'
/usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:24:in `compile'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:84:in `compile'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
/usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:82:in `compile'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:42:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:195:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:111:in `do_find'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `call'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `synchronize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in `run_command'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:69:in `execute'
/usr/bin/puppet:4

err: undefined method `include?' for nil:NilClass on node

/usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:27:in `compile'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:84:in `compile'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
/usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:82:in `compile'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:42:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:195:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:111:in `do_find'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `call'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `synchronize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in `run_command'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:69:in `execute'
/usr/bin/puppet:4
err: /usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:27:in `compile'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:84:in `compile'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
/usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:82:in `compile'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:42:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:195:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:111:in `do_find'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `call'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `synchronize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in `run_command'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:69:in `execute'
/usr/bin/puppet:4

err: undefined method `include?' for nil:NilClass on node

/usr/lib/ruby/1.8/net/http.rb:560:in `initialize'
/usr/lib/ruby/1.8/net/http.rb:560:in `open'
/usr/lib/ruby/1.8/net/http.rb:560:in `connect'
/usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
/usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
/usr/lib/ruby/1.8/net/http.rb:560:in `connect'
/usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
/usr/lib/ruby/1.8/net/http.rb:542:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/reports/http.rb:18:in `process'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/report/processor.rb:37:in `process'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/report/processor.rb:55:in `processors'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/report/processor.rb:53:in `each'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/report/processor.rb:53:in `processors'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/report/processor.rb:30:in `process'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/report/processor.rb:14:in `save'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:271:in `save'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:169:in `do_save'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `call'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `synchronize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in `run_command'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:69:in `execute'
/usr/bin/puppet:4

err: Report processor failed: Connection refused - connect(2)

err: //serv106.syst.local/Puppet: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `include?' for nil:NilClass on node
warning: //serv106.syst.local/Puppet: Not using cache on failed catalog
err: //serv106.syst.local/Puppet: Could not retrieve catalog; skipping run

Anyone got a clue?

-San

[Henrik Lindberg]

On 2014-29-05 21:57, Sans wrote:
> I have two identical nodes - serv106 and serv107- one of which is

Is this using version 3.6.x of Puppet? We are currently working on
webrick issues for 3.6 that are caused by concurrency problems.
(I noticed in the trace that you are using webrick).

Note that Webrick is really just suitable for light out of the box,
simple development test work and is not recommended for production.

If you switch to using passenger, do you get the same error?

- henrik

[Sans]

Hi Henrik,

I don't really use WEBrick but apache/passenger for the normal operation. I think WEBrick is used to run puppet master in background in debug mode.  Is there a way to use apache/passenger for that?
Anyway, I'm running puppet v2.7.3 and getting the same error for one of those nodes even using passenger.

-San

[Henrik Lindberg]

On 2014-30-05 20:54, Sans wrote:
> Hi Henrik,
>
> I don't really use WEBrick but apache/passenger for the normal
> operation. I think WEBrick is used to run puppet master in background in
> debug mode. Is there a way to use apache/passenger for that?
> Anyway, I'm running puppet v2.7.3 and getting the same error for one of
> those nodes even using passenger.
>

Ok, then webrick is not the cause of your problem.

Run with --trace set to get a ruby stack trace. That will help to find
what is causing the problem.

- henrik

[Sans]

On Friday, May 30, 2014 11:13:51 PM UTC+1, Henrik Lindberg wrote:

Ok, then webrick is not the cause of your problem.

Run with --trace set to get a ruby stack trace. That will help to find
what is causing the problem.

- henrik

Yes, that was the output from --trace and that's why I where I'm a bit stuck as it's not saying that much.

-San

[Ellison Marks]

It's weird, but it looks like the hostname isn't being defined, or isn't being sent properly. See

"info: Expiring the node cache of"

With just a blank afterwards. Does running facter on the affected node show anything unusual?

[Sans]

Good catch Ellison; I haven't noticed that but nope, nothing "unusual" from facter on the affected node.
Now running puppet agent -tv with certificate cleared from both master and agent yields these:

notice: Starting Puppet master version 2.7.23

....
....

info: access[/certificate_request]: allowing * access
info: access[/]: adding authentication any
info: Inserting default '/status' (auth true) ACL because none were found in '/etc/puppet/auth.conf'

info: Could not find certificate for 'serv106.syst.local'
info: Could not find certificate_request for 'serv106.syst.local'
notice: serv106.syst.local has a waiting certificate request
notice: Signed certificate request for serv106.syst.local
notice: Removing file Puppet::SSL::CertificateRequest serv106.syst.local at '/var/lib/puppet/ssl/ca/requests/serv106.syst.local.pem'

info: Expiring the node cache of

/usr/lib/ruby/site_ruby/1.8/puppet/node.rb:110:in `names'
/usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:212:in `evaluate_ast_node'
/usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:101:in `compile'

....
....

It does sign the certificate just right but again the same wired "info: Expiring the node cache of" after that.

-San

[jcbollinger]

On Saturday, May 31, 2014 2:53:52 PM UTC-5, Sans wrote:

Good catch Ellison; I haven't noticed that but nope, nothing "unusual" from facter on the affected node.
Now running puppet agent -tv with certificate cleared from both master and agent yields these:

notice: Starting Puppet master version 2.7.23
....
....
info: access[/certificate_request]: allowing * access
info: access[/]: adding authentication any
info: Inserting default '/status' (auth true) ACL because none were found in '/etc/puppet/auth.conf'
info: Could not find certificate for 'serv106.syst.local'
info: Could not find certificate_request for 'serv106.syst.local'
notice: serv106.syst.local has a waiting certificate request
notice: Signed certificate request for serv106.syst.local
notice: Removing file Puppet::SSL::CertificateRequest serv106.syst.local at '/var/lib/puppet/ssl/ca/requests/serv106.syst.local.pem'
info: Expiring the node cache of
/usr/lib/ruby/site_ruby/1.8/puppet/node.rb:110:in `names'
/usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:212:in `evaluate_ast_node'
/usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:101:in `compile'
....
....

It does sign the certificate just right but again the same wired "info: Expiring the node cache of" after that.

Is there an ENC involved?
Is there a node declaration matching the affected node?


John

[Sans]

No such ENC (other than Hiera).
There is also no "static" node declaration in Puppet; that comes from Hiera. But again, rest of the 50-odd nodes are running just fine the very same way.

-San

[jcbollinger]

On Thursday, June 5, 2014 5:47:44 AM UTC-5, Sans wrote:

No such ENC (other than Hiera).
There is also no "static" node declaration in Puppet; that comes from Hiera. But again, rest of the 50-odd nodes are running just fine the very same way.

The exception is being thrown by this statement in puppet/node.rb (of Puppet 2.7.23):

    names += split_name(name) if name.include?(".")

Combined with the exception detail message, that shows the node object having a nil 'name' attribute, which is consistent with the odd log message Ellison pointed out.  Just how this particular node might have a nil name is unclear, particularly given that the Puppet::Node class's initializer has a specific guard against that.

I would scrutinize the affected node's certificate, especially in comparison with those of other nodes that do not experience this problem.  Perhaps there is something odd with the name it bears.

If that yields no fruit then my next guess would be that there is some custom component -- a custom type or function, or perhaps even a template -- taking inappropriate liberties with Puppet's internals.  That sort of thing could be data dependent, which might explain why only one node is affected (so far).


John

[Ed Deloye]

Did anyone find a solution to this. We suddenly started seeing this on 24 of our 251 nodes.

Thanks,
Ed

[Ed Deloye]

Discovered truncated yaml files on the puppet master in /var/lib/puppet/yaml/node for the 24 systems. Identified as each one was 4096 bytes. After deleting those files puppet runs successfully on the nodes.

On Thursday, May 29, 2014 at 3:57:49 PM UTC-4, Sans wrote:

[Tom Noonan]

Good catch. Out of curiosity, what lead you to look at file sizes
in /var/lib/puppet/yaml/node? Based on your initial problem
description that is not where I would have been looking to troubleshoot.

[Ed Deloye]

My co-worker said he got lucky on a google search a couple months ago and suggested that I look at that directory. when I saw a one to one correspondence between the short files and the systems not working it was a quick fix. We suspect that the disk ran out of room before a scheduled clean up ran and caused the truncated files.