ssl alert: "Unknown CA"
Eric Sorenson
puppetd[4554]: could not retrieve catalog from remote server: ssl_connect returned=1 errno=0 state=sslv3 read server certificate b: certificate verify failed
puppetd[4554]: Not using cache on failed catalog
puppetd[4554]: Could not retrieve catalog; skipping run
puppetd[4961]: Retrieving plugin
puppetd[4961]: (/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
puppetd[4961]: (/File[/var/lib/puppet/lib]) Failed to retrieve current state of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for puppet://puppet/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
I've gone through the usual SSL troubleshooting: the clocks are in sync, the client cert matches the one issued to it by the server (and is decodable by the private_key).
When I use tshark to watch the ssl traffic, I see that the client is rejecting the server with the following ssl error. The connection never makes it to the back-end server, because the client hangs up. (10.1.1.1 is this client, 10.0.0.1 is the puppet vip)
[root@db9 /var/lib/puppet/ssl]# tshark -n -i bond0 -d tcp.port==8140,ssl -s2000 'port 8140 and len > 60'
0.000000 10.1.1.1 -> 10.0.0.1 TCP 29718 > 8140 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=1862094055 TSER=0 WS=7
0.001585 10.1.1.1 -> 10.0.0.1 SSLv2 Client Hello
0.001713 10.0.0.1 -> 10.1.1.1 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
0.002208 10.1.1.1 -> 10.0.0.1 TLSv1 Alert (Level: Fatal, Description: Unknown CA)
But openssl with the same cert and key that puppet is using passes verification and connects successfully:
openssl s_client -connect puppet:8140 -cert certs/db9.domain.com.pem -key private_keys/db9.domain.com.pem -showcerts -state -verify 2
103.115871 10.1.1.1 -> 10.0.0.1 TCP 40758 > 8140 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=1862197169 TSER=0 WS=7
103.116949 10.1.1.1 -> 10.0.0.1 SSLv2 Client Hello
103.117078 10.0.0.1 -> 10.1.1.1 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
103.121057 10.1.1.1 -> 10.0.0.1 TLSv1 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
103.122162 10.0.0.1 -> 10.1.1.1 TLSv1 Change Cipher Spec, Encrypted Handshake Message
Any thoughts on what could be causing this failure? I've seen quite a few odd ones (#3120, #4948 for example) but I've been gnawing at this one all day and haven't figured it out.
-=Eric
Brice Figureau
> I have a couple of hosts which are having trouble talking to the puppet VIP:
>
> puppetd[4554]: could not retrieve catalog from remote server: ssl_connect returned=1 errno=0 state=sslv3 read server certificate b: certificate verify failed
> puppetd[4554]: Not using cache on failed catalog
> puppetd[4554]: Could not retrieve catalog; skipping run
> puppetd[4961]: Retrieving plugin
> puppetd[4961]: (/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
> puppetd[4961]: (/File[/var/lib/puppet/lib]) Failed to retrieve current state of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for puppet://puppet/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
This certainly means your local node CA cert is not able to verify the
server proposed certificate (probably because the server advertised cert
hasn't been signed by this CA, or you use a CA chain but don't send the
full chain to the client).
> I've gone through the usual SSL troubleshooting: the clocks are in
> sync, the client cert matches the one issued to it by the server (and
> is decodable by the private_key).
>
> When I use tshark to watch the ssl traffic, I see that the client is
> rejecting the server with the following ssl error. The connection
> never makes it to the back-end server, because the client hangs up.
> (10.1.1.1 is this client, 10.0.0.1 is the puppet vip)
>
> [root@db9 /var/lib/puppet/ssl]# tshark -n -i bond0 -d tcp.port==8140,ssl -s2000 'port 8140 and len > 60'
> 0.000000 10.1.1.1 -> 10.0.0.1 TCP 29718 > 8140 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=1862094055 TSER=0 WS=7
> 0.001585 10.1.1.1 -> 10.0.0.1 SSLv2 Client Hello
> 0.001713 10.0.0.1 -> 10.1.1.1 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
> 0.002208 10.1.1.1 -> 10.0.0.1 TLSv1 Alert (Level: Fatal, Description: Unknown CA)
>
> But openssl with the same cert and key that puppet is using passes verification and connects successfully:
>
> openssl s_client -connect puppet:8140 -cert certs/db9.domain.com.pem -key private_keys/db9.domain.com.pem -showcerts -state -verify 2
You didn't ask openssl s_client to actually check the server certificate
against the CA cert of the client.
Can you try:
openssl s_client -connect puppet:8140 -CAfile certs/ca.pem -cert certs/db9.domain.com.pem -key private_keys/db9.domain.com.pem -showcerts -state -verify 2
> 103.115871 10.1.1.1 -> 10.0.0.1 TCP 40758 > 8140 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=1862197169 TSER=0 WS=7
> 103.116949 10.1.1.1 -> 10.0.0.1 SSLv2 Client Hello
> 103.117078 10.0.0.1 -> 10.1.1.1 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
> 103.121057 10.1.1.1 -> 10.0.0.1 TLSv1 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
> 103.122162 10.0.0.1 -> 10.1.1.1 TLSv1 Change Cipher Spec, Encrypted Handshake Message
>
> Any thoughts on what could be causing this failure? I've seen quite a
> few odd ones (#3120, #4948 for example) but I've been gnawing at this
> one all day and haven't figured it out.
For an unknown reason your local node CA cert is not correct.
You can solve this by overwriting it with the main CA cert, or check
that your server certificate is indeed correctly signed by the CA you
think it was signed with.
--
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!
Eric Sorenson
On Feb 18, 1:32 am, Brice Figureau <brice-pup...@daysofwonder.com>
wrote:
> server proposed certificate (probably because the server advertised cert
> hasn't been signed by this CA, or you use a CA chain but don't send the
> full chain to the client).
connecting to the same VIP are fine. There are just a couple of them
that exhibit this problem and scores that are fine.
> You didn't ask openssl s_client to actually check the server certificate
> against the CA cert of the client.
>
> Can you try:
> openssl s_client -connect puppet:8140 -CAfile certs/ca.pem -cert certs/db9.domain.com.pem -key private_keys/db9.domain.com.pem -showcerts -state -verify 2
>
it correctly. It also succeeds:
(echo "GET /" ; echo; ) | openssl s_client -connect puppet:8140 -cert
certs/db9.domain.com.pem -key private_keys/db9.domain.com.pem -
showcerts -state -verify 2 -CAfile certs/ca.pem 2>&1 | grep -i verify
verify depth is 2
verify return:1
verify return:1
SSL_connect:SSLv3 write certificate verify A
Verify return code: 0 (ok)
> For an unknown reason your local node CA cert is not correct.
> You can solve this by overwriting it with the main CA cert, or check
> that your server certificate is indeed correctly signed by the CA you
> think it was signed with.
shown with openssl -showcerts into puppet's certs dir and compared
them:
[/var/lib/puppet/ssl/certs]# openssl x509 -in ca.pem -modulus -noout
Modulus=B7491A13EFEDBBD259F90D914365DA8E2BF11F8AB6762723E3C91C1868AE999EAB3E0003F10C99363C7D9A5260B474EA193F52F05C42456FD0089E2284CDE9FD26F3F4F82D7A6B7E6A9B7A27EA2B2C94B46272926932A4FB6D6EDF7D6EEC720781B76261382927799E63FFA8DEF5EA6239D5590ADD34A0C06DB9818834DBB6BF
[/var/lib/puppet/ssl/certs]# openssl x509 -in ca-not.pem -modulus -
noout
Modulus=B7491A13EFEDBBD259F90D914365DA8E2BF11F8AB6762723E3C91C1868AE999EAB3E0003F10C99363C7D9A5260B474EA193F52F05C42456FD0089E2284CDE9FD26F3F4F82D7A6B7E6A9B7A27EA2B2C94B46272926932A4FB6D6EDF7D6EEC720781B76261382927799E63FFA8DEF5EA6239D5590ADD34A0C06DB9818834DBB6BF
I am really starting to think something is amiss with the ruby-
>openssl libraries on the affected hosts. They show the same behaviour
connecting to other puppet vips in the environment, including one that
has the SSL connection terminated at Apache instead of the
loadbalancer.
Thanks very much Brice. I will keep at it.
-=Eric
Eric Sorenson
puppetmaster is generating that kills the client.
When I mv $ssldir/ca/ca_crl.pem out of the way on the server (so it
can't be re-downloaded) and rm $ssldir/crl.pem on the client, they all
work. Put it back in place, or generate a new one with puppet cert --
clean <somehost>, and I get this error.
This explains why openssl was working - It doesn't use a CRL.
100% repeatable.
Still working out why the CRL is fatal. I had actually suspected this
because of my earlier trouble with it (#4948) but did not see the VIP
(nor the endpoints) certs in the CRL so I discounted it.
Jeff McCune
>
> Still working out why the CRL is fatal. I had actually suspected this
> because of my earlier trouble with it (#4948) but did not see the VIP
> (nor the endpoints) certs in the CRL so I discounted it.
>
Thanks for the follow up Eric, please let us know if you figure this
out. I suspect I'm going to run into this as well and may have
working with someone in training yesterday.
-Jeff
Eric Sorenson
On Feb 18, 2011, at 5:46 PM, Jeff McCune wrote:
>
> Thanks for the follow up Eric, please let us know if you figure this
> out. I suspect I'm going to run into this as well and may have
> working with someone in training yesterday.
I believe it's is related to changing the name of the issuer for the CA, from here:
http://www.mail-archive.com/puppe...@googlegroups.com/msg09176.html
When I have an empty CRL, the clients work fine. The crl looks like:
# openssl crl -noout -in ca_crl.pem -text
Certificate Revocation List (CRL):
Version 2 (0x1)
Signature Algorithm: sha1WithRSAEncryption
Issuer: /CN=ca
Last Update: Feb 19 01:46:28 2011 GMT
Next Update: Feb 18 01:46:28 2016 GMT
CRL extensions:
X509v3 CRL Number:
0
No Revoked Certificates.
and it passes validation:
# openssl crl -noout -CAfile ./ca_crt.pem -in ca_crl.pem -issuer
verify OK
issuer=/CN=ca
As soon as I --clean a client, the CRL gets rewritten and starts failing:
# openssl crl -noout -in ca_crl_fatal.pem -text
Certificate Revocation List (CRL):
Version 2 (0x1)
Signature Algorithm: sha1WithRSAEncryption
Issuer: /CN=Puppet CA: puppetmaster001
Last Update: Feb 19 01:21:00 2011 GMT
Next Update: Feb 18 01:21:00 2016 GMT
CRL extensions:
X509v3 CRL Number:
7
[some revoked certs here]
# openssl crl -noout -CAfile ./ca_crt.pem -in ca_crl_fatal.pem -text
Error getting CRL issuer certificate
Could it be that the issuer name change is causing the ssl client libraries to fail to match up the CRL with the issuing CA?
- Eric Sorenson - N37 17.255 W121 55.738 - http://twitter.com/ahpook -
Felix Frank
Definitely. This CRL is not related to your CA cert in any way. It's
probably a bug that the CRL is created this way.
Can you find a CA cert with the Subject CN as in the Issuer field above
anywhere on your system?
Regards,
Felix
Eric Sorenson
wrote:
however, a CRL, outside of the CA area, which puppet-cert read and
reused the Issuer field to create the new CRL. I think this is a bug
because the client and CA SSL areas ought to be distinct, but it's
probably quite rare for this situation to arise. I updated my
previous "bad ssl error messages" bug with this case and hope it will
help other people.
If I had to hazard a guess as to why it does this, it looks like the
ssl/certificate_revocation_list.rb uses the same indirector file
location (:hostcrl from indirector/certificate_revocation_list/
file.rb) whether its reading or writing a CRL. But I don't actually
understand the indirector so this could be completely off-base.