Re: [Puppet Users] PuppetDB: SSL problems

[Stefan Schulte]

On Wed, 8 May 2013 07:01:56 -0700 (PDT)
kl.pup...@gmail.com wrote:

>
> Error: Could not retrieve catalog from remote server: Error 400 on
> SERVER: Failed to submit 'replace facts' command for gaia.local
> to PuppetDB at puppetdb.local:8081: SSL_connect SYSCALL returned=5
> errno=0 state=SSLv3 read finished A
> Warning: Not using cache on failed catalog
> Error: Could not retrieve catalog; skipping run
>

seems to be an issue with OpenJDK7. Reverting to Java6 solved the
problem for a lot of users.

issue is described here: http://projects.puppetlabs.com/issues/19884

-Stefan

[K L]

Hi Stefan,

On May 8, 10:36 pm, Stefan Schulte <stefan.schu...@taunusstein.net>
wrote:

> seems to be an issue with OpenJDK7. Reverting to Java6 solved the
> problem for a lot of users.
>
> issue is described here:http://projects.puppetlabs.com/issues/19884

Thanks for your reply. I tried it. Current output of `java -version`
on puppetdb is:
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.11)
(rhel-1.61.1.11.11.el6_4-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

It doesn't solve the issue.

I'm still thinking something might be wrong with my certificates,
though I can't be sure. (gaia = cname for puppet master).

puppetmaster(gaia)$ puppet cert fingerprint --all --digest=md5
gaia.kahuna.local (MD5) FB:8A:*:2D:A2
puppetdb.kahuna.local (MD5) 8A:70:*:0E:D4

Fingerprints from files on puppetdb:
puppetdb:ca_crt file E4:89:*:F2:FF
puppetdb:certs/puppetdb.local.pem: 8A:70:*:0E:D4

When I do `openssl x509 -in private_keys/puppetdb.local.pem -
fingerprint -noout -md5;`, I get the following. I don't know if this
is normal:

unable to load certificate
140457098893128:error:0906D06C:PEM routines:PEM_read_bio:no start
line:pem_lib.c:698:Expecting: TRUSTED CERTIFICATE


Can you please verify if I did everything correctly with setting up
the {key,trust}store.jks?
keystore.jks has 8A:70:*:0E:D4
truststore.jks has E4:89:*:F2:FF

It all seems good to me... But I might have done something wrong.
Thanks again for your reply.
kl

[kl.pup...@gmail.com]

I ran puppetdb-foreground --debug. Please find the output here:

http://pastebin.com/raw.php?i=Ra3BM3yf

Thanks again for your time!

[Ken Barber]

How did you setup your SSL certificates? You didn't mention a manual
certificate setup. Perhaps you can get away with just re-initializing
your certificates using 'puppetdb-ssl-setup'? Just backup your
/etc/puppetdb/ssl directory first, and then remove it and re-run the
tool and see if that helps:

# mv /etc/puppetdb/ssl /etc/puppetdb/ssl.bak
# puppetdb-ssl-setup

Try that first, and if it doesn't help let us know what any resulting
errors are ... even if its exactly the same error.

ken.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To post to this group, send email to puppet...@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[kl.pup...@gmail.com]

Thanks for your reply Ken,

On Fri, May 10, 2013 at 2:11 PM, Ken Barber <k...@puppetlabs.com> wrote:
> How did you setup your SSL certificates? You didn't mention a manual
> certificate setup.

I did it manually after the automatic way did not work. I followed
this guide ( http://goo.gl/m4PIH ) and reviewed your comments in this
thread: http://goo.gl/NzS5M .

>Perhaps you can get away with just re-initializing
> your certificates using 'puppetdb-ssl-setup'? Just backup your
> /etc/puppetdb/ssl directory first, and then remove it and re-run the
> tool and see if that helps:
>
> # mv /etc/puppetdb/ssl /etc/puppetdb/ssl.bak
> # puppetdb-ssl-setup

Just tried that. Also put the new pass in jetty.ini, as this was
changed. I also did:
# openssl verify -CAfile /etc/puppet/ssl/ca/ca_crt.pem `puppet master
--configprint hostcert`
/etc/puppet/ssl/certs/puppetdb.local.pem: OK

> Try that first, and if it doesn't help let us know what any resulting
> errors are ... even if its exactly the same error.

Exact output of puppet-onetime on a host after configuring puppetdb:

================================================
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb

Error: Could not retrieve catalog from remote server: Error 400 on

SERVER: Failed to submit 'replace facts' command for kayak.local to

PuppetDB at puppetdb.local:8081: SSL_connect SYSCALL returned=5
errno=0 state=SSLv3 read finished A
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

================================================


Tail of /var/log/puppetdb/puppetdb.log:
================================================
2013-05-10 15:12:55,421 INFO [main] [cli.services] Starting 1 command
processor threads
2013-05-10 15:12:55,432 INFO [main] [cli.services] Starting query server
2013-05-10 15:12:55,462 INFO [pool-2-thread-1] [cli.services] Starting
database garbage collection
2013-05-10 15:12:55,473 INFO [clojure-agent-send-off-pool-2]
[server.Server] jetty-7.x.y-SNAPSHOT
2013-05-10 15:12:55,494 INFO [pool-2-thread-1] [cli.services] Finished
database garbage collection
2013-05-10 15:12:55,505 INFO [pool-2-thread-1] [cli.services] Starting
sweep of stale reports (threshold: 14 days)
2013-05-10 15:12:55,525 INFO [pool-2-thread-1] [cli.services] Finished
sweep of stale reports (threshold: 14 days)
2013-05-10 15:12:55,545 INFO [clojure-agent-send-off-pool-2]
[server.AbstractConnector] Started
SelectChannelConnector@localhost:8080
2013-05-10 15:12:56,038 INFO [clojure-agent-send-off-pool-2]
[ssl.SslContextFactory] Enabled Protocols [SSLv2Hello, SSLv3, TLSv1]
of [SSLv2Hello, SSLv3, TLSv1]
2013-05-10 15:12:56,053 INFO [clojure-agent-send-off-pool-2]
[server.AbstractConnector] Started
SslSelectCha...@puppetdb.local:8081
2013-05-10 15:13:38,374 WARN [qtp283362979-38] [io.nio]
javax.net.ssl.SSLHandshakeException: null cert chain
================================================

Puppet master log line:
================================================
May 10 15:13:38 gaia puppet-master[5686]: Failed to submit 'replace
facts' command for kayak.kahuna.local to PuppetDB at
puppetdb.kahuna.local:8081: SSL_connect SYSCALL returned=5 errno=0
state=SSLv3 read finished A
================================================

Hope this helps. Thanks for your time (and the previous -comprehensive- responses on this mailing list),

kl

[kl.pup...@gmail.com]

Any idea on how I can do debugging?

Tried re-installing several times now. I'd like to be able to find out where the problem lies. 

Thanks,

kl

[Ken Barber]

Can we walk through your certificates again? Can you give the full
verbose output of the following?

* keytool -list -keystore /etc/puppetdb/ssl/keystore.jks # you'll need
the password from puppetdb_keystore_pw.txt
* keytool -list -keystore /etc/puppetdb/ssl/truststore.jks # same again
* puppet cert fingerprint --all --digest=md5
* facter fqdn
* puppet master --configprint hostcert
* cat /etc/puppet/puppetdb.conf
* echo "GET /" | openssl s_client -connect 127.0.1.1:8081 -cert
`puppet master --configprint hostcert` -key `puppet master
--configprint hostprivkey` -CAfile `puppet master --configprint
cacert` # obviously change 127.0.1.1 to whatever port puppetdb is
listening on

I get the feeling your problem is due to the client certificate being
used to connect is the issue, but I need to see all this data again to
be clear.

[kl.pup...@gmail.com]

Hi Ken, thanks for your reply,

On Tue, May 14, 2013 at 5:08 PM, Ken Barber <k...@puppetlabs.com> wrote:
> Can we walk through your certificates again? Can you give the full
> verbose output of the following?

I put the complete output here: http://pastebin.com/raw.php?i=iW44kACL . Hope this helps.

> I get the feeling your problem is due to the client certificate being

> used to connect is the issue, but I need to see all this data again to
> be clear.

There do indeed seem to be some problems with the certificate (especially with the [puppet cert fingerprint] command). This might be the main problem for puppetdb. The onetime command does work, however, but puppetdb might not like it. I don't know how to fix this. Other nodes seem to work fine. 

Thanks,

kl 

[Ken Barber]

I think the certificate fingerprint issue you received is a worry, but
might not indicate a problem per se. Lets use openssl instead to get
the fingerprint directly:

# openssl x509 -noout -in `puppet master --configprint hostcert`
-fingerprint -md5

So if I do the same exercise on my own host I get:
https://gist.github.com/kbarber/5592588

Notice how the fingerprints match? At first glance your failing
command seems to indicate the certificate in your JKS store is _not_
the same as the certificate being used by Puppet itself, but try the
openssl variant I showed you above instead and see how it goes.

If they do not match, it would make sense that you are receiving a
chain problem. The certificate in your keystore.jks file might not be
signed by the CA. Perhaps it is old and left over from another
certificate loading attempt?

What is weird is that you say you cleared /etc/puppetdb/ssl and re-ran
puppetdb-ssl-setup didn't you? This action should be enough to restore
the correct key in keystore.jks.

ken.

[kl.pup...@gmail.com]

Hi Ken,

On Thu, May 16, 2013 at 5:34 PM, Ken Barber <k..@puppetlabs.com> wrote:
> I think the certificate fingerprint issue you received is a worry, but
> might not indicate a problem per se. Lets use openssl instead to get
> the fingerprint directly:

Still get this problem.

> # openssl x509 -noout -in `puppet master --configprint hostcert`
> -fingerprint -md5
>
> So if I do the same exercise on my own host I get:
> https://gist.github.com/kbarber/5592588

I see, and I'va replicated this now. The hashes match.

> Notice how the fingerprints match? At first glance your failing
> command seems to indicate the certificate in your JKS store is _not_
> the same as the certificate being used by Puppet itself, but try the
> openssl variant I showed you above instead and see how it goes.

It indeed wasn't, now it is :).

> If they do not match, it would make sense that you are receiving a
> chain problem. The certificate in your keystore.jks file might not be
> signed by the CA. Perhaps it is old and left over from another
> certificate loading attempt?
>
> What is weird is that you say you cleared /etc/puppetdb/ssl and re-ran
> puppetdb-ssl-setup didn't you? This action should be enough to restore
> the correct key in keystore.jks.

I am not sure I did the ssl-setup command again. I started all over
again on the puppetdb. Deleted the package, all the logs and
configuration and reinstalled puppetdb. I included a complete output:
http://pastebin.com/raw.php?i=TDejFAvp

Does this make things more clear? I did a clean install of 1.3.0,
maybe there is a problem in that version?

Thanks,
Karlo

[Ken Barber]

> I am not sure I did the ssl-setup command again. I started all over
> again on the puppetdb. Deleted the package, all the logs and
> configuration and reinstalled puppetdb. I included a complete output:
> http://pastebin.com/raw.php?i=TDejFAvp
>
> Does this make things more clear? I did a clean install of 1.3.0,
> maybe there is a problem in that version?

Could very well be, however it seems so far you're the first unlucky
one to see this issue afaik :-). I've been trying to reproduce it on
my own setup with no luck yet, although I've got some ideas to try
today.

Also - remember this command?

echo "GET /" | openssl s_client -connect 127.0.1.1:8081 -cert
`puppet master --configprint hostcert` -key `puppet master
--configprint hostprivkey` -CAfile `puppet master --configprint
cacert`

Did you try running that from the puppet master node itself -
attempting to connect to puppetdb? I believe the last test you tried
was directly from the puppetdb node instead.

BTW - If you like, you can always get on Freenode IRC and chat to me
real time about this. Might speed things up. I'm usually on #puppet as
ken_barber.

ken.

[kl.pup...@gmail.com]

Ken, it's working now! "Solution" below.

On Fri, May 17, 2013 at 4:27 PM, Ken Barber <k...@puppetlabs.com> wrote:
> Could very well be, however it seems so far you're the first unlucky
> one to see this issue afaik :-). I've been trying to reproduce it on
> my own setup with no luck yet, although I've got some ideas to try
> today.

Thanks a lot for trying though. Your replies have been very helpful.

> Also - remember this command?
>
>     echo "GET /" | openssl s_client -connect 127.0.1.1:8081 -cert
> `puppet master --configprint hostcert` -key `puppet master
> --configprint hostprivkey` -CAfile `puppet master --configprint
> cacert`
>
> Did you try running that from the puppet master node itself -
> attempting to connect to puppetdb? I believe the last test you tried
> was directly from the puppetdb node instead.

Good catch. I was trying it from the puppetdb itself. That was working well.

I then tried from the puppet server itself. The problem was the following:
 - For everything puppet, I use puppet.local as the fqdn for the puppet master.
 - The actual hostname (and thus the cert) for the puppet master node
is gaia.local.
 - For some reason (config probably ;) ), puppet agents don't think
this is a problem.
 - When I tried your GET|openssl command, it was complaining about not
being able to find certs/puppet.local.something and
private_keys/puppet.local.something.
 - I symlinked puppet.local (to use gaia.local, the actual
certificate). This works. Probably not the nicest way, but it works!
Exported config now works.

I'm very happy it works now,
Thanks again!
/kl

[Ken Barber]

I'm glad you found a solution :-).

I think this is a bug though. Would you mind if you raised a ticket
for this in our redmine tracker with the details of your error and
solution? At least if we can record it for the purpose of errata, it
might help someone else - or we might come to a proper solution around
it eventually.

http://projects.puppetlabs.com/projects/puppetdb/issues/new

BTW, what does your puppet.conf look like?

[kl.pup...@gmail.com]

Opened bug 20838: http://projects.puppetlabs.com/issues/20838

Thanks,

kl