puppetdb setup issue

2,219 views
Skip to first unread message

Antidot SAS

unread,
Jun 12, 2012, 10:39:22 AM6/12/12
to puppet-users
Hi everyone,

I am trying to setup the new puppetdb on my environment (currently it worked great with mysql databases). All the setup was made by package for debian squeeze and puppet is used with passenger.


Here are the configuration files:
--
cat /etc/puppetdb/conf.d/jetty.ini
[jetty]
# Hostname to list for clear-text HTTP. Default is localhost
#host = localhost
# Port to listen on for clear-text HTTP.
host = puppetdb.fqdn
port = 8080
ssl-host = puppetdb.fqdn
ssl-port = 8081
keystore = /etc/puppetdb/ssl/keystore.jks
truststore = /etc/puppetdb/ssl/truststore.jks
key-password = uTyCY6damAQn9KInqCLuvAO53
trust-password = uTyCY6damAQn9KInqCLuvAO53
--
cat /etc/puppet/puppetdb.conf
[main]
server = pupperdb.fqdn
port = 8081
--
netstat -tulanp |egrep '808|543'
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 16224/postgres
tcp 0 0 127.0.0.1:5432 127.0.0.1:9232 ESTABLISHED 27554/postgres: pup
tcp 0 0 127.0.0.1:5432 127.0.0.1:9230 ESTABLISHED 27552/postgres: pup
tcp 0 0 127.0.0.1:5432 127.0.0.1:9229 ESTABLISHED 27551/postgres: pup
tcp 0 0 127.0.0.1:5432 127.0.0.1:9231 ESTABLISHED 27553/postgres: pup
tcp6 0 0 10.10.200.17:8080 :::* LISTEN 27496/java
tcp6 0 0 10.10.200.17:8081 :::* LISTEN 27496/java
tcp6 0 0 127.0.0.1:9232 127.0.0.1:5432 ESTABLISHED 27496/java
tcp6 0 0 127.0.0.1:9195 127.0.0.1:5432 TIME_WAIT -
tcp6 0 0 127.0.0.1:9230 127.0.0.1:5432 ESTABLISHED 27496/java
tcp6 0 0 127.0.0.1:9193 127.0.0.1:5432 TIME_WAIT -
tcp6 0 0 127.0.0.1:9194 127.0.0.1:5432 TIME_WAIT -
tcp6 0 0 127.0.0.1:9229 127.0.0.1:5432 ESTABLISHED 27496/java
tcp6 0 0 127.0.0.1:9231 127.0.0.1:5432 ESTABLISHED 27496/java
tcp6 0 0 127.0.0.1:9192 127.0.0.1:5432 TIME_WAIT -
--
Once everything is started:
2012-06-12 16:33:13,841 DEBUG [main] [bonecp.BoneCPDataSource] JDBC URL = jdbc:postgresql://localhost:5432/puppetdb, Username = puppetdb, partitions = 5, max (per partition) = 10, min (p
er partition) = 1, helper threads = 3, idle max age = 60 min, idle test period = 240 min
2012-06-12 16:33:13,979 INFO [main] [cli.services] Starting broker
2012-06-12 16:33:14,729 DEBUG [main] [page.PageFile] Page File: /usr/share/puppetdb/mq/localhost/KahaDB/db.data, Recovering page file...
2012-06-12 16:33:14,790 DEBUG [main] [index.BTreeIndex] loading
2012-06-12 16:33:14,795 DEBUG [main] [index.BTreeIndex] loading
2012-06-12 16:33:14,796 DEBUG [main] [index.BTreeIndex] loading
2012-06-12 16:33:14,796 DEBUG [main] [index.BTreeIndex] loading
2012-06-12 16:33:14,796 DEBUG [main] [index.BTreeIndex] loading
2012-06-12 16:33:14,796 DEBUG [main] [index.BTreeIndex] loading
2012-06-12 16:33:14,977 INFO [main] [journal.Journal] ignoring zero length, partially initialised journal data file: db-1.log number = 1 , length = 0
2012-06-12 16:33:14,987 DEBUG [main] [page.PageFile] Page File: /usr/share/puppetdb/mq/localhost/scheduler/scheduleDB.data, Recovering page file...
2012-06-12 16:33:15,031 DEBUG [main] [index.BTreeIndex] loading
2012-06-12 16:33:15,031 DEBUG [main] [index.BTreeIndex] loading
2012-06-12 16:33:15,031 DEBUG [main] [index.BTreeIndex] loading
2012-06-12 16:33:15,034 DEBUG [main] [index.BTreeIndex] loading
2012-06-12 16:33:15,109 INFO [main] [cli.services] Starting 2 command processor threads
2012-06-12 16:33:15,111 INFO [main] [cli.services] Starting query server
2012-06-12 16:33:15,111 INFO [main] [cli.services] Starting database compactor (60 minute interval)
2012-06-12 16:33:15,124 INFO [clojure-agent-send-off-pool-2] [mortbay.log] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-06-12 16:33:15,126 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Container Server@4f47afda + SocketC...@puppetdb.vitry.exploit.anticorp:8080 as connector
2012-06-12 16:33:15,131 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Container Server@4f47afda + SslSocket...@puppetdb.vitry.exploit.anticorp:8081 as connector
2012-06-12 16:33:15,131 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Container Server@4f47afda + AbstractHandler$0@4da4826b as handler
2012-06-12 16:33:15,132 INFO [clojure-agent-send-off-pool-2] [mortbay.log] jetty-6.1.x
2012-06-12 16:33:15,145 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Container Server@4f47afda + org.mortbay.thread.QueuedThreadPool@76bd92e4 as threadpool
2012-06-12 16:33:15,148 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started org.mortbay.thread.QueuedThreadPool@76bd92e4
2012-06-12 16:33:15,151 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] starting AbstractHandler$0@4da4826b
2012-06-12 16:33:15,151 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started AbstractHandler$0@4da4826b
2012-06-12 16:33:15,151 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] starting Server@4f47afda
2012-06-12 16:33:15,153 INFO [clojure-agent-send-off-pool-2] [mortbay.log] Started SocketC...@puppetdb.vitry.exploit.anticorp:8080
2012-06-12 16:33:15,153 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started SocketC...@puppetdb.vitry.exploit.anticorp:8080
2012-06-12 16:33:15,164 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Checking Resource aliases
2012-06-12 16:33:15,219 DEBUG [clojure-agent-send-off-pool-0] [listener.DefaultMessageListenerContainer] Established shared JMS Connection
2012-06-12 16:33:15,219 DEBUG [clojure-agent-send-off-pool-1] [listener.DefaultMessageListenerContainer] Established shared JMS Connection
2012-06-12 16:33:15,256 INFO [clojure-agent-send-off-pool-2] [mortbay.log] Started SslSocket...@puppetdb.vitry.exploit.anticorp:8081
2012-06-12 16:33:15,262 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started SslSocket...@puppetdb.vitry.exploit.anticorp:8081
2012-06-12 16:33:15,262 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started Server@4f47afda


and once I am trying to run any agent I am having the following error with the SSL port:
date && puppet agent -t --noop ; date
Tue Jun 12 16:31:16 CEST 2012
info: Retrieving plugin
info: Loading facts in meminbytes
info: Loading facts in facter_dot_d
info: Loading facts in root_home
info: Loading facts in puppet_vardir
info: Loading facts in meminbytes
info: Loading facts in facter_dot_d
info: Loading facts in root_home
info: Loading facts in puppet_vardir
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for lnk4c-cks01.vitry.exploit.anticorp to PuppetDB at puppetdb.vitry.exploit.anticorp:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run Tue Jun 12 16:31:23 CEST 2012
---
2012-06-12 16:31:23,054 WARN [1130816144@qtp-844964870-6] [mortbay.log] EXCEPTION
javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1763)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1006)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1190) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1217)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1201)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:675) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)


If I change the port:
cat puppetdb.conf
[main]
server = puppetdb.vitry.exploit.anticorp
port = 8080
--
date && puppet agent -t --noop ; date Tue Jun 12 16:36:58 CEST 2012
info: Retrieving plugin
info: Loading facts in meminbytes
info: Loading facts in facter_dot_d
info: Loading facts in root_home
info: Loading facts in puppet_vardir
info: Loading facts in meminbytes
info: Loading facts in facter_dot_d
info: Loading facts in root_home
info: Loading facts in puppet_vardir
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for lnk4c-cks01.vitry.exploit.anticorp to PuppetDB at puppetdb.vitry.exploit.anticorp:8080: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
Tue Jun 12 16:37:01 CEST 2012
--
2012-06-12 16:36:57,836 DEBUG [1255344208@qtp-1992135396-2] [mortbay.log] uri=
2012-06-12 16:36:57,836 DEBUG [1255344208@qtp-1992135396-2] [mortbay.log] fields=
2012-06-12 16:36:57,836 DEBUG [1255344208@qtp-1992135396-2] [mortbay.log] EXCEPTION
HttpException(400,null,null)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:361)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-06-12 16:36:57,844 DEBUG [1255344208@qtp-1992135396-2] [mortbay.log] BAD


Any idea, what could cause this error?


Regards,
JM

Nick Lewis

unread,
Jun 12, 2012, 4:46:28 PM6/12/12
to puppet...@googlegroups.com
2012-06-12 16:33:15,126 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Container Server@4f47afda + SocketConnector@puppetdb.vitry.exploit.anticorp:8080 as connector
2012-06-12 16:33:15,131 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Container Server@4f47afda + SslSocketConnector@puppetdb.vitry.exploit.anticorp:8081 as connector

2012-06-12 16:33:15,131 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Container Server@4f47afda + AbstractHandler$0@4da4826b as handler
2012-06-12 16:33:15,132 INFO [clojure-agent-send-off-pool-2] [mortbay.log] jetty-6.1.x
2012-06-12 16:33:15,145 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Container Server@4f47afda + org.mortbay.thread.QueuedThreadPool@76bd92e4 as threadpool
2012-06-12 16:33:15,148 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started org.mortbay.thread.QueuedThreadPool@76bd92e4
2012-06-12 16:33:15,151 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] starting AbstractHandler$0@4da4826b
2012-06-12 16:33:15,151 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started AbstractHandler$0@4da4826b
2012-06-12 16:33:15,151 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] starting Server@4f47afda
2012-06-12 16:33:15,153 INFO [clojure-agent-send-off-pool-2] [mortbay.log] Started SocketConnector@puppetdb.vitry.exploit.anticorp:8080
2012-06-12 16:33:15,153 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started SocketConnector@puppetdb.vitry.exploit.anticorp:8080

2012-06-12 16:33:15,164 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Checking Resource aliases
2012-06-12 16:33:15,219 DEBUG [clojure-agent-send-off-pool-0] [listener.DefaultMessageListenerContainer] Established shared JMS Connection
2012-06-12 16:33:15,219 DEBUG [clojure-agent-send-off-pool-1] [listener.DefaultMessageListenerContainer] Established shared JMS Connection
2012-06-12 16:33:15,256 INFO [clojure-agent-send-off-pool-2] [mortbay.log] Started SslSocketConnector@puppetdb.vitry.exploit.anticorp:8081
2012-06-12 16:33:15,262 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started SslSocketConnector@puppetdb.vitry.exploit.anticorp:8081

Did you run a puppet agent on the PuppetDB server before installing the PuppetDB package? In order to setup SSL correctly, this is currently necessary.

If you didn't, you can run a puppet agent to generate certificates and then run `/usr/sbin/puppetdb-ssl-setup` to redo the SSL setup. This will put your password in /etc/puppetdb/ssl/puppetdb_keystore_pw.txt, and you can update your jetty.ini with that.

Otherwise, please run these commands for some diagnostic output:

keytool -list -keystore /etc/puppetdb/ssl/keystore.jks
keytool -list -keystore /etc/puppetdb/ssl/truststore.jks

puppet cert --fingerprint ca <puppetdb hostname>

This will give some output to ensure that the certificates being used by PuppetDB are what we expect them to be.

As an aside, none of this output contains the timestamp of the puppet master (only the agent and PuppetDB). Can you also please ensure that's also correct?
 

Regards,
JM

Antidot SAS

unread,
Jun 13, 2012, 4:20:41 AM6/13/12
to puppet...@googlegroups.com
Hi thx for the reply here are the info:
--
nslookup puppetdb.fqdn
Server: 10.10.200.29
Address: 10.10.200.29#53

puppetdb.fqdn canonical name = puppetmaster.fqdn
Name: puppetmaster.fqdn
Address: 10.10.200.17
--
keytool -list -keystore /etc/puppetdb/ssl/keystore.jks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

puppetmaster.fqdn, Jun 12, 2012, PrivateKeyEntry,
Certificate fingerprint (MD5): 02:B5:21:B9:F7:72:4A:48:67:12:47:FF:0A:DE:B5:1D
--
keytool -list -keystore /etc/puppetdb/ssl/truststore.jks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

puppetdb ca, Jun 12, 2012, trustedCertEntry,
Certificate fingerprint (MD5): 1F:1B:E7:2A:89:B5:87:65:4F:91:1A:8B:75:8F:AD:60
--
puppet cert --fingerprint ca puppetmaster.fqdn
ca 1F:1B:E7:2A:89:B5:87:65:4F:91:1A:8B:75:8F:AD:60

So it seems that the certificates are not right?
--
On the master:
ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
+ntp1 198.82.1.204 3 u 986 1024 377 0.106 -1.399 0.323
*ntp2 129.70.132.32 3 u 54 1024 377 0.376 0.338 0.903
LOCAL(0) .LOCL. 12 l 14h 64 0 0.000 0.000 0.000


As you see the server is up to date.

Does that help?

Regards,
JM



2012-06-12 16:33:15,126 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Container Server@4f47afda + SocketC...@puppetdb.vitry.exploit.anticorp:8080 as connector
2012-06-12 16:33:15,131 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Container Server@4f47afda + SslSocket...@puppetdb.vitry.exploit.anticorp:8081 as connector

2012-06-12 16:33:15,131 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Container Server@4f47afda + AbstractHandler$0@4da4826b as handler
2012-06-12 16:33:15,132 INFO [clojure-agent-send-off-pool-2] [mortbay.log] jetty-6.1.x
2012-06-12 16:33:15,145 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Container Server@4f47afda + org.mortbay.thread.QueuedThreadPool@76bd92e4 as threadpool
2012-06-12 16:33:15,148 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started org.mortbay.thread.QueuedThreadPool@76bd92e4
2012-06-12 16:33:15,151 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] starting AbstractHandler$0@4da4826b
2012-06-12 16:33:15,151 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started AbstractHandler$0@4da4826b
2012-06-12 16:33:15,151 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] starting Server@4f47afda
2012-06-12 16:33:15,153 INFO [clojure-agent-send-off-pool-2] [mortbay.log] Started SocketC...@puppetdb.vitry.exploit.anticorp:8080
2012-06-12 16:33:15,153 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started SocketC...@puppetdb.vitry.exploit.anticorp:8080

2012-06-12 16:33:15,164 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] Checking Resource aliases
2012-06-12 16:33:15,219 DEBUG [clojure-agent-send-off-pool-0] [listener.DefaultMessageListenerContainer] Established shared JMS Connection
2012-06-12 16:33:15,219 DEBUG [clojure-agent-send-off-pool-1] [listener.DefaultMessageListenerContainer] Established shared JMS Connection
2012-06-12 16:33:15,256 INFO [clojure-agent-send-off-pool-2] [mortbay.log] Started SslSocket...@puppetdb.vitry.exploit.anticorp:8081
2012-06-12 16:33:15,262 DEBUG [clojure-agent-send-off-pool-2] [mortbay.log] started SslSocket...@puppetdb.vitry.exploit.anticorp:8081
--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/goDGIrarBNwJ.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Antidot SAS

unread,
Jun 13, 2012, 6:31:59 AM6/13/12
to puppet...@googlegroups.com
OK,

I have manged to have the same signature (Apparently using --config doesn't help for generating certificats :D)
So now is the deal:
# keytool -list -keystore /etc/puppetdb/ssl/keystore.jks

Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
puppetmaster.fqdn, Jun 13, 2012, PrivateKeyEntry,
Certificate fingerprint (MD5): FE:EA:B4:FE:C4:2C:07:9B:15:B7:F2:DB:3A:78:B3:47
--
# puppet cert fingerprint puppetmaster.fqdn --digest=md5 --config=/etc/puppet/conf/puppet.conf
puppetmaster.fqdn FE:EA:B4:FE:C4:2C:07:9B:15:B7:F2:DB:3A:78:B3:47
--

But still not the same for truststore.jks:
# keytool -list -keystore /etc/puppetdb/ssl/truststore.jks

Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
puppetdb ca, Jun 13, 2012, trustedCertEntry,
Certificate fingerprint (MD5): DA:38:CE:13:8A:20:8B:C1:4C:1C:2C:99:27:5F:53:05
--

And stil having the issue with the agent:
# date && puppet agent -t --noop ; date
Wed Jun 13 12:18:51 CEST 2012

info: Retrieving plugin
info: Loading facts in meminbytes
info: Loading facts in facter_dot_d
info: Loading facts in root_home
info: Loading facts in puppet_vardir
info: Loading facts in meminbytes
info: Loading facts in facter_dot_d
info: Loading facts in root_home
info: Loading facts in puppet_vardir
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for test-puppet.fqdn to PuppetDB at puppetmaster.fqdn:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client

warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
Wed Jun 13 12:18:54 CEST 2012

On the master:
2012-06-13 12:28:51,828 WARN [789688662@qtp-1034385146-6] [mortbay.log] EXCEPTION

javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1763)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1006)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1190) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1217)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1201)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:675)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)


As you can see in the log the date seems pretty the same.

Antidot SAS

unread,
Jun 13, 2012, 6:43:47 AM6/13/12
to puppet...@googlegroups.com
HI everyone,

Finally got it, once the certificats were recreated forgot to restart puppetdb... Sorry.


Thx for the tips :D
Reply all
Reply to author
Forward
0 new messages