PE 3.1 PuppetDB connection refused messages with ad hoc puppet agent runs. PE-initiated puppet agent runs are fine.

[Trevor (@VMTrooper)]

Hello Everyone,

PuppetDB is behaving strangely despite my fqdn, /etc/hosts files, firewalls, and ntp, being configured correctly.  FYI, I do not use DNS in my enviornment.  Also, I have followed previously posted suggestions regarding jetty.ini and confirming /etc/hosts.

After an initial install of Puppet Master (tried both CentOS 6.4 and Ubuntu 12.04 ) and the Puppet agent on one Ubuntu 12.04 node and one CentOS 6.4 node, my puppet agent --test runs will complete successfully 1 or 2 times.

Then, I start getting the following output:

root@puppetnode01:~# puppet agent --test

Info: Retrieving plugin

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/postgres_default_version.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/windows.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/ip6tables_version.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/iptables_version.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/custom_auth_conf.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/root_home.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/iptables_persistent_version.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/facter_dot_d.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/puppet_vardir.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/concat_basedir.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/puppetdb_server_status.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/pe_version.rb

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for puppetnode01.vmtrooper.local to PuppetDB at puppetmaster.vmtrooper.local:8081: Connection refused - connect(2)

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Eventually I get the following additional error messages:

root@puppetnode01:~# puppet agent --test

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: Error 400 on SERVER: Could not retrieve facts for puppetnode01.vmtrooper.local: Failed to find facts from PuppetDB at puppetmaster.vmtrooper.local:8081: Connection refused - connect(2)

Info: Retrieving plugin

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/postgres_default_version.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/windows.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/ip6tables_version.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/iptables_version.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/custom_auth_conf.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/root_home.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/iptables_persistent_version.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/facter_dot_d.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/puppet_vardir.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/concat_basedir.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/puppetdb_server_status.rb

Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/pe_version.rb

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for puppetnode01.vmtrooper.local to PuppetDB at puppetmaster.vmtrooper.local:8081: Connection refused - connect(2)

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Puppet Enterprise's own scheduled collections continue to run successfully.  It's just when I attempt to run my own puppet agent runs, it gives the errors.  Has anyone seen this?

Thanks,

Trevor

[Ken Barber]

> PuppetDB is behaving strangely despite my fqdn, /etc/hosts files, firewalls,
> and ntp, being configured correctly. FYI, I do not use DNS in my
> enviornment. Also, I have followed previously posted suggestions regarding
> jetty.ini and confirming /etc/hosts.

If you can send through your jetty.ini so we can take a look that
would be useful. Have you ran 'puppetdb-ssl-setup' yet? If no, try
running this and providing the output on this thread.

So the 'Connection refused' is the key here, its a client error - its
a very common error (not just for PuppetDB or Puppet) that usually
just means the service is not running on the host port you are
attempting to connect to. This can be because of a variety of reasons.

* The service is just not listening on port 8081, this could be
because the service has not started properly ...

Double check that the PuppetDB service is running, check your process
list for the 'java' process related to puppetdb. A check like: "ps
auxww | grep puppetdb" or "ps auxww | grep java" would show this. You
can also use tools such as netstat (ie. netstat -anp | grep 8081) to
check if its listening also, this should provide the PID of the
process as well.

If the service is indeed not running, you'll need to dig into this
further. Double check your /var/log/puppetdb/puppetdb.log and check
for any error messages (provide them in full here if they don't make
sense) - alternatively try to run it in the foreground
'puppetdb-foreground --debug' and see if that gives you a reasonable
error.

* The hostname 'puppetmaster.vmtrooper.local' does not resolve to the
IP address that you expect.

Double check your /etc/hosts and ensure this host resolves correctly.
You'll see where its trying to go by doing a basic telnet test such as
'telnet puppetmaster.vmtrooper.local 8081'.

* The service is not listening on the port you expect because of
misconfiguration.

If you haven't ran 'puppetdb-ssl-setup' before then the port won't be
configured. Provide your jetty.ini for us to look at. Double check
using a tool such as 'lsof -p <pid>' on the PID of the Java process to
see if the service is listening.

Also double check your jetty.ini ssl-host setting, and make sure that
is listening on the correct interface, change it to 0.0.0.0, restart
and see if this solves it.

* A firewall might be blocking the port.

This is fairly rare, since most firewalls drop the packet, as apposed
to rejecting it - and also it sounds like you're running the PuppetDB
instance on the same host as your master, so even weirder. But worth
mentioning.

ken.