PuppetDB connection issue - Failed to connect!!

[Sans]

Hi there,

Just started using PuppetDB (using the Puppetlabs' module) and getting issues with connection. First it was giving me server Not Found:

Error: Unable to connect to puppetdb server (puppet.internal:8081): [404] Not Found
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
....
....
Error: Unable to connect to puppetdb server (puppet.internal:8081): [404] Not Found
Notice: Failed to connect to puppetdb within timeout window of 15 seconds; giving up.
Error: Unable to connect to puppetdb server! (puppet.internal:8081)
Error: /Stage[main]/Puppetdb::Master::Config/Puppetdb_conn_validator[puppetdb_conn]/ensure: change from absent to present failed: Unable to connect to puppetdb server! (puppet.internal:8081)

but from the puppetdb ssl-setup, it looks okay:

root@puppet:/vagrant/PuppetConf/manifests# puppetdb ssl-setup -f
PEM files in /etc/puppetdb/ssl already exists, checking integrity.
Overwriting existing PEM files due to -f flag
Copying files: /var/lib/puppet/ssl/certs/ca.pem, /var/lib/puppet/ssl/private_keys/puppet.internal.pem and /var/lib/puppet/ssl/certs/puppet.internal.pem to /etc/puppetdb/ssl
Setting ssl-host in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-port in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-key in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-cert in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-ca-cert in /etc/puppetdb/conf.d/jetty.ini already correct.

 
since then, I'm still getting eventually the same error but with different Notice:

Warning: Puppet::Util::SUIDManager.run_and_capture is deprecated; please use Puppet::Util::Execution.execute instead.
   (at /vagrant/VagrantConf/modules/postgresql/lib/puppet/provider/postgresql_psql/ruby.rb:57:in `run_sql_command')
Notice: /Stage[main]/Puppetdb::Server::Jetty_ini/Ini_setting[puppetdb_sslhost]/value: value changed '0.0.0.0' to 'puppet.internal'
Info: Class[Puppetdb::Server::Jetty_ini]: Scheduling refresh of Service[puppetdb]
Notice: /Stage[main]/Puppetdb::Server/Service[puppetdb]: Triggered 'refresh' from 1 events
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
....
....
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb within timeout window of 15 seconds; giving up.
Error: Unable to connect to puppetdb server! (puppet.internal:8081)
Error: /Stage[main]/Puppetdb::Master::Config/Puppetdb_conn_validator[puppetdb_conn]/ensure: change from absent to present failed: Unable to connect to puppetdb server! (puppet.internal:8081)

Can anyone explain to me what actually going on/wrong please? Why it was "Not Found" before and now "Unable to connect"? Any help/pointer would be much appreciated. Best!

[Ken Barber]

> Just started using PuppetDB (using the Puppetlabs' module) and getting
> issues with connection. First it was giving me server Not Found:
>
>> Error: Unable to connect to puppetdb server (puppet.internal:8081): [404]
>> Not Found
>> Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
>> ....
>> ....
>> Error: Unable to connect to puppetdb server (puppet.internal:8081): [404]
>> Not Found
>> Notice: Failed to connect to puppetdb within timeout window of 15 seconds;
>> giving up.
>> Error: Unable to connect to puppetdb server! (puppet.internal:8081)
>> Error:
>> /Stage[main]/Puppetdb::Master::Config/Puppetdb_conn_validator[puppetdb_conn]/ensure:
>> change from absent to present failed: Unable to connect to puppetdb server!
>> (puppet.internal:8081)

This has retried to connect a few times then given up because of ...

... connection refused. This is the key error. Its a common TCP error,
and is something all administrators should know, because it doesn't
just affect PuppetDB.

What it means is that the hostname and port the client is trying to
connect to, in this case puppet.internal and 8081 respectively, is
either not the correct host or port, PuppetDB is not listening
correctly to this host or port because it is down or misconfigured or
some firewall is rejecting the connection (less likely, but worth
noting).

So what you need to check:

* From your puppet master try telnetting into the port and show your
results. "telnet puppet.internal 8081"
* Check that the PuppetDB instance is really listening on port 8081
and on a public interface on that host, you can usually check this
with netstat -anp | grep 8081 on the PDB host. Provide the results in
the thread if you can.
* Check the settings in your /etc/puppetdb/conf.d/jetty.ini, and
ensure that ssl-host is set to something like 0.0.0.0 to listen on all
interfaces. You can always refine this once you have it working, but
this is the recommend base setting. If you change it, restart
PuppetDB.
* Check that the PuppetDB java process is running also. "ps auxww |
grep java" is a good start, provide the results if you like in thread.
* Ensure that the hostname 'puppet.internal' resolves to what you
think it does. A large amount of these errors are due to assumptions
about what hostnames resolve to ... it must resolve to the IP that
PuppetDB is listening on. Check /etc/hosts on your puppet master ...

Let me know how you go.

ken.

[Sans]

Hi Ken,

Thanks for the heads up!
First of all, it's a VBox VM, provisioned by Vigrant. PuppetMaster and PuppetDB are on the same machine.
I did go through those basic checks before posting, which appeared fine:

root@puppet:~# telnet puppet.internal 8081
Trying 127.0.1.1...
Connected to puppet.internal.
Escape character is '^]'.

root@puppet:~# netstat -ntpl | grep 80
tcp        0      0 0.0.0.0:80              0.0.0.0:*        LISTEN      14345/apache2
tcp6       0      0 :::8080                 :::*             LISTEN      16301/java
tcp6       0      0 127.0.1.1:8081          :::*             LISTEN      16301/java

This is my jetty.ini:

root@puppet:~# awk '!/^($|#)/ {print}' /etc/puppetdb/conf.d/jetty.ini
[jetty]
host = 0.0.0.0
port = 8080
ssl-host = puppet.internal
ssl-port = 8081
ssl-key = /etc/puppetdb/ssl/private.pem
ssl-cert = /etc/puppetdb/ssl/public.pem
ssl-ca-cert = /etc/puppetdb/ssl/ca.pem

Java is also running:

root@puppet:~# ps auxww | grep java
puppetdb 16301  1.0 26.8 1558932 135336 ?      Sl   13:47   2:26 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -Djava.security.egd=file:/dev/urandom -cp /usr/share/puppetdb/puppetdb.jar clojure.main -m com.puppetlabs.puppetdb.core services -c /etc/puppetdb/conf.d

ping  can resolve:

root@puppet:~# ping -c2 puppet.internal
PING puppet.internal (127.0.1.1) 56(84) bytes of data.
64 bytes from puppet.internal (127.0.1.1): icmp_req=1 ttl=64 time=0.023 ms
64 bytes from puppet.internal (127.0.1.1): icmp_req=2 ttl=64 time=0.032 ms

--- puppet.internal ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.023/0.027/0.032/0.006 ms

but nslookup cannot:

root@puppet:~# nslookup puppet.internal
Server:        10.0.2.3
Address:    10.0.2.3#53

** server can't find puppet.internal: NXDOMAIN

(nslookup is fine though with localhost)

This is what my /etc/hosts looks like:

127.0.0.1    localhost
127.0.1.1    puppet.internal    puppet

It's Ubuntu 12.04 server and I heard that name resolving works differently in this version. I'm lost here. Best!!

[Ken Barber]

At first glance this all seems correct. Hrm.

Can you do the telnet test?

telnet puppet.internal 8081

Also, are you destroying and rebuilding these VM's each time and then
its failing? Or are you doing all of this _after_ the vm's are
launched. Its quite possible there is a race condition/ordering issue
in how the provisioning is occuring end-to-end.

ken.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/f1f592e5-c15f-407b-bf09-48ee28eb9ab7%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Ken Barber]

Oh ... and lets see the output of:

iptables -vnL

Perhaps there is a firewall here? Its worth double checking.

[Sans]

hi Ken,

Right now I'm creating only one VM, co-locating PuppetMaster and PuppetDB to make it simple - destroying and rebuilding. But it always fails - during the provisioning/building and also even after if I login to the machine and run puppet apply. Telnet works fine:

root@puppet:~# telnet puppet.internal 8081
Trying 127.0.1.1...
Connected to puppet.internal.
Escape character is '^]'.

Connection closed by foreign host.

and iptables is not configured at all but here is the output:

root@puppet:~# ufw status
Status: inactive

root@puppet:~# iptables -vnL
Chain INPUT (policy ACCEPT 349 packets, 47388 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 246 packets, 46964 bytes)
 pkts bytes target     prot opt in     out     source               destination

-San

[Ken Barber]

> Right now I'm creating only one VM, co-locating PuppetMaster and PuppetDB to
> make it simple - destroying and rebuilding. But it always fails - during the
> provisioning/building and also even after if I login to the machine and run
> puppet apply. Telnet works fine:
>
>> root@puppet:~# telnet puppet.internal 8081
>> Trying 127.0.1.1...
>> Connected to puppet.internal.
>> Escape character is '^]'.
>> Connection closed by foreign host.
>
>
> and iptables is not configured at all but here is the output:

Okay, let me zoom out a bit and rethink this. Going back to a point
you made originally:

> Can anyone explain to me what actually going on/wrong please? Why it was "Not Found" before and now "Unable to connect"? Any help/pointer would be much appreciated. Best!

I think the 404 is common while the application is loading within
Jetty. It might be that the timeout we have for waiting for PuppetDB
to be fully available needs to be adjusted? It could also be a legit
problem also.

So lets try a different tactic.

Lets adjust the setting puppetdb_startup_timeout for the
puppetdb::master::config class, and change it to something high, like
60 seconds or so.

If that still fails, I think we should grab the
/var/log/puppetdb/puppetdb.log output, can you put that in a gist or
something? The log immediately after the attempted provision is best -
so don't try to rerun puppet or anything, we want to see if purely
after the provisioning process fails.

ken.

[Sans]

It's very strange: Until I run puppetdb ssl-setup -f, I get

Error: Unable to connect to puppetdb server (puppet.internal:8081): [404] Not Found

but after that, I get

Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>

Since, I increased the timeout to 60, it changed into this:

Info: Class[Puppetdb::Server::Jetty_ini]: Scheduling refresh of Service[puppetdb]
Notice: /Stage[main]/Puppetdb::Server/Service[puppetdb]: Triggered 'refresh' from 1 events
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
Notice: Unable to connect to puppetdb server (puppet.internal:8081): #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry

Error: Unable to connect to puppetdb server (puppet.internal:8081): [404] Not Found
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
Error: Unable to connect to puppetdb server (puppet.internal:8081): [404] Not Found
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry
.....

.....

Error: Unable to connect to puppetdb server (puppet.internal:8081): [404] Not Found
Notice: Failed to connect to puppetdb; sleeping 2 seconds before retry

Error: Unable to connect to puppetdb server (puppet.internal:8081): [404] Not Found

Notice: Failed to connect to puppetdb within timeout window of 60 seconds; giving up.

Error: Unable to connect to puppetdb server! (puppet.internal:8081)
Error: /Stage[main]/Puppetdb::Master::Config/Puppetdb_conn_validator[puppetdb_conn]/ensure: change from absent to present failed: Unable to connect to puppetdb server! (puppet.internal:8081)

I'm  rebuilding the server now.

[Sans]

This is just after building a new server:

2014-06-17 23:20:05,246 INFO  [p.t.s.w.jetty9-core] Removing buggy security provider SunPKCS11-NSS version 1.7
2014-06-17 23:20:06,036 INFO  [p.t.s.w.jetty9-service] Initializing web server.
2014-06-17 23:20:06,132 INFO  [p.t.s.w.jetty9-service] Starting web server.
2014-06-17 23:20:06,134 INFO  [o.e.j.s.Server] jetty-9.1.z-SNAPSHOT
2014-06-17 23:20:06,177 INFO  [o.e.j.s.ServerConnector] Started ServerConnector@5c5a7bf{HTTP/1.1}{0.0.0.0:8080}
2014-06-17 23:20:06,437 INFO  [o.e.j.s.ServerConnector] Started ServerConnector@1bfc3437{SSL-HTTP/1.1}{puppet.internal:8081}
2014-06-17 23:20:06,531 INFO  [c.p.p.c.services] PuppetDB version 2.0.0
2014-06-17 23:20:06,900 WARN  [c.p.p.s.migrate] Adding additional indexes; this may take several minutes, depending on the size of your database. Trust us, it will all be worth it in the end.
2014-06-17 23:20:07,004 WARN  [c.p.p.s.migrate] Building resource parameters cache. This make take a few minutes, but faster resource queries are worth it.
2014-06-17 23:20:07,140 INFO  [c.p.p.c.services] Starting broker
2014-06-17 23:20:07,619 WARN  [o.a.a.b.BrokerService] Store limit is 100000 mb, whilst the data directory: /var/lib/puppetdb/mq/localhost/KahaDB only has 73883 mb of usable space
2014-06-17 23:20:07,620 INFO  [c.p.p.c.services] Starting 1 command processor threads
2014-06-17 23:20:07,639 INFO  [c.p.p.c.services] Starting query server
2014-06-17 23:20:07,642 WARN  [o.e.j.s.h.ContextHandler] Empty contextPath
2014-06-17 23:20:07,650 INFO  [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@3a3670a8{/,null,AVAILABLE}
2014-06-17 23:20:07,724 INFO  [c.p.p.c.services] Starting sweep of stale reports (threshold: 14 days)
2014-06-17 23:20:07,750 INFO  [c.p.p.c.services] Finished sweep of stale reports (threshold: 14 days)
2014-06-17 23:20:07,750 INFO  [c.p.p.c.services] Starting database garbage collection
2014-06-17 23:20:07,777 INFO  [c.p.p.c.services] Finished database garbage collection

-San

[Ken Barber]

The connection refused implies the server hasn't opened the port yet
for listening, the progression to 404 is probably the Jetty server
starting up but not yet being ready for serving the URL we test
against.

In particular, we test the URL similar to this curl request, so its
probably worth seeing if it returns anything on the command line:

curl 'http://puppet.local:8080/v2/metrics/mbean/java.lang:type=Memory'

Perhaps 60 seconds isn't enough? Its unlikely but worth trying to bump
it I guess.

Beyond that we'll need to see the puppetdb.log.

ken.

[Sans]

The puppetdb log is attached in the previous post.

And, here is the curl output:

root@puppet:~# curl 'http://puppet.internal:8080/v2/metrics/mbean/java.lang:type=Memory'
{
  "ObjectPendingFinalizationCount" : 0,
  "HeapMemoryUsage" : {
    "committed" : 86220800,
    "init" : 8052480,
    "max" : 194641920,
    "used" : 41882456
  },
  "NonHeapMemoryUsage" : {
    "committed" : 63504384,
    "init" : 24313856,
    "max" : 224395264,
    "used" : 63257288
  },
  "Verbose" : false,
  "ObjectName" : "java.lang:type=Memory"

[Rakesh Kathpal]

Santanu,. can you please try and update your jetti.ini as follows

[jetty]
host = 0.0.0.0
port = 8080

ssl-host = 0.0.0.0

ssl-port = 8081
ssl-key = /etc/puppetdb/ssl/private.pem
ssl-cert = /etc/puppetdb/ssl/public.pem
ssl-ca-cert = /etc/puppetdb/ssl/ca.pem

Restart puppetdb and verify...

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/36c1d8af-e7bb-4d95-960f-f00db7dc460e%40googlegroups.com.

[Sans]

Thanks Rakesh!
But, as you probably can guess, that didn't change a thing; still getting the same error. I'm not sure if PuppetDB look in the DNS record (as opposed to hosts file), in that case it never gonna work. Best!!

[Greg Cockburn]

On Thursday, June 19, 2014 12:17:41 AM UTC+10, Sans wrote:

Thanks Rakesh!
But, as you probably can guess, that didn't change a thing; still getting the same error. I'm not sure if PuppetDB look in the DNS record (as opposed to hosts file), in that case it never gonna work. Best!!

Santanu,

Did you ever fix this. I am having the same problem.

It is an issue with SSL.  Chrome gives this error: "Error code: ERR_SSL_PROTOCOL_ERROR".

Curl gives this error "curl: (35) Unknown SSL protocol error in connection"

If I figure more out, I will post the fix. 

[Ken Barber]

> Did you ever fix this. I am having the same problem.
>
> It is an issue with SSL. Chrome gives this error: "Error code:
> ERR_SSL_PROTOCOL_ERROR".
> Curl gives this error "curl: (35) Unknown SSL protocol error in connection"
>
> If I figure more out, I will post the fix.

These two errors are unrelated to the original problem described in
this thread, perhaps start a new thread?

Chrome will potentially throw an error here, because PuppetDB needs
client certificates to work, so nothing abnormal here - using chrome
is just not a good test unless you have the knowledge to load up the
CA and client certs yourself (not recommended really). I'd have to see
the full curl request to confirm, but it looks like you're tapping a
non-SSL port or somehow the openssl client is constraining the
protocols.

ken.

[Ken Barber]

Did you figure this out Sans?

My only next logical step would be to have you show me a full git
repository with a working copy of the code/vagrantfile/etc. that is
actually breaking.

ken.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/9c9cc6c3-1059-4b67-abd8-29f8d77e8527%40googlegroups.com.

[Matthias Raithel]

For the next one hitting this - in my case this was related to a old version of the puppetdb module (v2.0.0) which queried a URL that was no longer existent.
Updating the module cleared the error right up.

The module was previousely not update because of a dependency lockup through puppetlabs/postgresql at ripienaar/concat (which is lateron replaced by puppetlabs/concat)

Thanks to ken for helping me out through IRC