Puppet first run issue: Error 400 on SERVER: Failed when searching for node

[A.K.]

Hi there,

Puppet client version 3.8.7
Every first puppet run I've got this error:

Nov 10 12:01:30 myinstance-172 puppet-agent[935]: (/File[/var/lib/puppet/lib/puppet/type/rabbitmq_vhost.rb]/ensure) defined content as '{md5}ff6fc35bb9c22b1c493b84adf6bb3167'
Nov 10 12:01:31 myinstance-172 puppet-agent[935]: (/File[/var/lib/puppet/lib/puppet/provider/rabbitmq_user/rabbitmqctl.rb]/ensure) defined content as '{md5}eeddc61ef6c57e7225c818e494fc6e47'
Nov 10 12:01:31 myinstance-172 puppet-agent[935]: (/File[/var/lib/puppet/lib/puppet/parser/functions/nslookup.rb]/ensure) defined content as '{md5}976cfe36eec535d97a17139c7408f0bd'
Nov 10 12:02:34 myinstance-172 puppet-agent[935]: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed when searching for node myinstance-9p23t0xkmtu6vxv: Failed to find myinstance-9p23t0xkmtu6vxv via exec: Execution of '/etc/puppet/node.rb myinstance-9p23t0xkmtu6vxv' returned 1:
Nov 10 12:02:34 myinstance-172 puppet-agent[935]: Using cached catalog
Nov 10 12:02:34 myinstance-172 puppet-agent[935]: Could not retrieve catalog; skipping run
Nov 10 12:02:48 myinstance-172 puppet: /usr/share/ruby/vendor_ruby/puppet/agent.rb:87:in `exit': no implicit conversion from nil to integer (TypeError)
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/agent.rb:87:in `block in run_in_fork'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/agent.rb:84:in `fork'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/agent.rb:84:in `run_in_fork'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/agent.rb:43:in `block in run'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/application.rb:179:in `call'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/application.rb:179:in `controlled_run'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/agent.rb:41:in `run'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/daemon.rb:175:in `block in run_event_loop'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/scheduler/job.rb:49:in `call'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/scheduler/job.rb:49:in `run'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/scheduler/scheduler.rb:39:in `block in run_ready'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/scheduler/scheduler.rb:34:in `each'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/scheduler/scheduler.rb:34:in `run_ready'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/scheduler/scheduler.rb:11:in `run_loop'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/daemon.rb:198:in `run_event_loop'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/daemon.rb:154:in `start'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/application/agent.rb:383:in `main'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/application/agent.rb:329:in `run_command'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `run'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
Nov 10 12:02:48 myinstance-172 puppet: from /usr/bin/puppet:8:in `<main>'Enter code here...

And on the master the same time:

Nov 10 11:59:40 ip-172-19-135-11 puppet-master[98099]: myinstance-9p23t0xkmtu6vxv has a waiting certificate request
Nov 10 11:59:40 ip-172-19-135-11 puppet-master[98099]: Signed certificate request for myinstance-9p23t0xkmtu6vxv
Nov 10 11:59:40 ip-172-19-135-11 puppet-master[98099]: Removing file Puppet::SSL::CertificateRequest myinstance-9p23t0xkmtu6vxv at '/var/lib/puppet/ssl/ca/requests/myinstance-9p23t0xkmtu6vxv.pem'
Nov 10 11:59:40 ip-172-19-135-11 puppet-master[98099]: Could not resolve 172.22.23.178: no name for 172.22.23.178
Nov 10 11:59:42 ip-172-19-135-11 puppet-master[98099]: Failed to find myinstance-9p23t0xkmtu6vxv via exec: Execution of '/etc/puppet/node.rb myinstance-9p23t0xkmtu6vxv' returned 1:

On the second and next runs everything is working perfectly.

Please advice where to look to fix this type of issue?

Thank you.

[Rob Nelson]

This looks like your external node classifier /etc/puppet/node.rb is having some issues on first run. 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/4f09c921-525a-4abb-a5ff-f3bab30f08e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Rob Nelson
rnel...@gmail.com

[Gavin Williams]

What Rob said... 

This error is fairly typical if you're using something like Foreman as an ENC, and not creating the node in Foreman before the first Puppet run. 

Basically, whatever your ENC is needs to "know" about the node before it first dials in... 

HTH

Gav

[A.K.]

Gavin,

Yes, Foreman it is (v.1.7.4).

> not creating the node in Foreman before the first Puppet run. 

Any idea how to work around this?
And why it is happening sometimes, not every first run?

Thank you!

[Gavin Williams]

Our current process is that we create the host entry in Foreman as part of our VM provision process.

Couple of options for interfacing with Foreman:

• cloud-init-foreman [1] - Not currently using this, but will be evaluating in near future
• python-foreman [2] - We use this in conjunction with a python bootstrap script that gets run when the VM boots, which installs Puppet agent, gathers host data and creates host in Foreman, then triggers a Puppet run.
• Provision the host using Foreman - Foreman can provision hosts on most common platforms, including VMWare, Openstack, AWS, GCE etc...

So largely depends on what your current provisioning process is and what the best way of integrating Foreman is in that process... 

With regards to "why it is happening sometimes", are you rebuilding VM's with the same hostname? As to Foreman, that would be the 'same' host, even though it's been rebuilt. 

HTH

Gav

 

[1] https://github.com/cernops/cloud-init-foreman

[2] https://github.com/david-caro/python-foreman

[A.K.]

Hi Gavin,

It's an AWS environment, so the same IP address of the machine and a hostname is reincarnated very often. Is there is some "auto cleaning" mechanism in Foreman 1.7.4? Or should I upgrade to some latest version?

[fatmcgav]

Afraid I can't help on that one, as haven't used AWS.

There might be some additional features in a more recent version of Foreman that will assist...

Cheers
Gav

You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/ei5OMmFZzVc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/83f403fb-89a2-4142-bdd0-bd0b4c86004e%40googlegroups.com.