Re: [Puppet Users] sslv3 alert handshake failure

[Martin Alfke]

Hi,

- check time on client and server

- check ruby version on the 3 server which fail

- check SSLDir configuration in /etc/puppet/puppet.conf on the 3 systems.

Martin

On 06.07.2012, at 09:57, Martinus wrote:

I have a problem on 3 out of ~40 servers that gives the following error:

err: Could not request certificate: SSL_connect returned=1 errno=0 state=unknown state: sslv3 alert handshake failure

From previous posts, I made sure that SSLVerifyClient is set to optional.  I also cleared /var/lib/puppet/ssl/ client side, not that it should make any difference as this error is on the first run of Puppet.

When I try to run Puppet from either of these 3 servers, there is nothing noted in /var/log/apache2/* server side.  I have confirmed networking is ok with telnet and also checked that there is traffic with tcpdump.

Puppet server is at 2.7.11 and client is also at 2.7.11 both from Ubuntu repositories.

Any help would be appreciated to find why these 3 particular servers is giving me problems.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/mzcj4gN-AWQJ.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Martinus]

Martin,

Right.

Time is good (NTP) on all 3 clients and server.  And I double checked just now with ntpq -p (largest offset was -20).  There are different time zones, but then so has the working systems different time zones.

Ruby version on all 3 clients and server: ruby 1.8.7 (2011-06-30 patchlevel 352)

The SSLDir line looks like this: "ssldir = /var/lib/puppet/ssl" on all systems (config file is copied across systems).  I checked, and the standard set of directories are there and owned by Puppet.  However, crl.pem is not present like on the working systems.

Martinus.

To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.

[Martin Alfke]

On puppet master:

puppet cert --clean <fqdn>

on client:

rm -fr /var/lib/puppet/ssl/*

puppet agent --test

check on master for signing request:

puppet cert --list

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/ksgzsaL9g1MJ.

To post to this group, send email to puppet...@googlegroups.com.

To unsubscribe from this group, send email to puppet-users...@googlegroups.com.

[Martinus]

There is nothing to clean, as "puppet cert --list" or "puppet cert --list --all" does not have an entry for those 3 particular servers.

Deleting the client side ssl* makes no difference either.  The client will recreate the ssl (good) and the same error pops up, without anything showing up on the master (puppet cert --list).

And that is why I thought there is a communication problem.  But here is the tcpdump output to show that they are talking:

09:01:57.812646 IP my_client.46516 > my_server.8140: Flags [S], seq 1288389639, win 14600, options [mss 1460,sackOK,TS val 1052151283 ecr 0,nop,wscale 4], length 0

09:01:57.812700 IP my_server.8140 > my_client.46516: Flags [S.], seq 300735116, ack 1288389640, win 14480, options [mss 1460,sackOK,TS val 38287565 ecr 1052151283,nop,wscale 4], length 0

09:01:57.814298 IP my_client.46516 > my_server.8140: Flags [.], ack 1, win 913, options [nop,nop,TS val 1052151283 ecr 38287565], length 0

09:01:57.814686 IP my_client.46516 > my_server.8140: Flags [P.], seq 1:175, ack 1, win 913, options [nop,nop,TS val 1052151283 ecr 38287565], length 174

09:01:57.814715 IP my_server.8140 > my_client.46516: Flags [.], ack 175, win 972, options [nop,nop,TS val 38287566 ecr 1052151283], length 0

09:01:57.815226 IP my_server.8140 > my_client.46516: Flags [P.], seq 1:8, ack 175, win 972, options [nop,nop,TS val 38287566 ecr 1052151283], length 7

09:01:57.815378 IP my_server.8140 > my_client.46516: Flags [F.], seq 8, ack 175, win 972, options [nop,nop,TS val 38287566 ecr 1052151283], length 0

09:01:57.816686 IP my_client.46516 > my_server.8140: Flags [.], ack 8, win 913, options [nop,nop,TS val 1052151284 ecr 38287566], length 0

09:01:57.816884 IP my_client.46516 > my_server.8140: Flags [F.], seq 175, ack 9, win 913, options [nop,nop,TS val 1052151284 ecr 38287566], length 0

09:01:57.816894 IP my_server.8140 > my_client.46516: Flags [.], ack 176, win 972, options [nop,nop,TS val 38287566 ecr 1052151284], length 0

As an additional note, when I stop apache and start puppetmaster with its inbuilt web server, then these 3 clients are happy.

Martinus.

[Martin Alfke]

On 06.07.2012, at 11:09, Martinus wrote:

There is nothing to clean, as "puppet cert --list" or "puppet cert --list --all" does not have an entry for those 3 particular servers.

Deleting the client side ssl* makes no difference either.  The client will recreate the ssl (good) and the same error pops up, without anything showing up on the master (puppet cert --list).

And that is why I thought there is a communication problem.  But here is the tcpdump output to show that they are talking:

09:01:57.812646 IP my_client.46516 > my_server.8140: Flags [S], seq 1288389639, win 14600, options [mss 1460,sackOK,TS val 1052151283 ecr 0,nop,wscale 4], length 0

09:01:57.812700 IP my_server.8140 > my_client.46516: Flags [S.], seq 300735116, ack 1288389640, win 14480, options [mss 1460,sackOK,TS val 38287565 ecr 1052151283,nop,wscale 4], length 0

09:01:57.814298 IP my_client.46516 > my_server.8140: Flags [.], ack 1, win 913, options [nop,nop,TS val 1052151283 ecr 38287565], length 0

09:01:57.814686 IP my_client.46516 > my_server.8140: Flags [P.], seq 1:175, ack 1, win 913, options [nop,nop,TS val 1052151283 ecr 38287565], length 174

09:01:57.814715 IP my_server.8140 > my_client.46516: Flags [.], ack 175, win 972, options [nop,nop,TS val 38287566 ecr 1052151283], length 0

09:01:57.815226 IP my_server.8140 > my_client.46516: Flags [P.], seq 1:8, ack 175, win 972, options [nop,nop,TS val 38287566 ecr 1052151283], length 7

09:01:57.815378 IP my_server.8140 > my_client.46516: Flags [F.], seq 8, ack 175, win 972, options [nop,nop,TS val 38287566 ecr 1052151283], length 0

09:01:57.816686 IP my_client.46516 > my_server.8140: Flags [.], ack 8, win 913, options [nop,nop,TS val 1052151284 ecr 38287566], length 0

09:01:57.816884 IP my_client.46516 > my_server.8140: Flags [F.], seq 175, ack 9, win 913, options [nop,nop,TS val 1052151284 ecr 38287566], length 0

09:01:57.816894 IP my_server.8140 > my_client.46516: Flags [.], ack 176, win 972, options [nop,nop,TS val 38287566 ecr 1052151284], length 0

As an additional note, when I stop apache and start puppetmaster with its inbuilt web server, then these 3 clients are happy.

Are the client working after you have enabled them using webrick puppetmaster?

We are working with nginx and passenger and we needed the following in puppet configuration [master]:

ssl_client_header = HTTP_X_CLIENT_DN 

ssl_client_verify_header = HTTP_X_CLIENT_VERIFY

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/NsecfOnGBsgJ.

To post to this group, send email to puppet...@googlegroups.com.

To unsubscribe from this group, send email to puppet-users...@googlegroups.com.

[Martinus]

Martin,

No, the clients fail again with exactly the same error once I switch apache back on.  Your configuration is slightly different than what I have:

ssl_client_header = SSL_CLIENT_S_DN

ssl_client_verify_header = SSL_CLIENT_VERIFY

Now lets see what happens if I use your example ...

Nope, those changes makes no difference.

Martinus.

[Matthew Burgess]

> As an additional note, when I stop apache and start puppetmaster with its
> inbuilt web server, then these 3 clients are happy.

Ah, that triggered a memory!

http://projects.puppetlabs.com/projects/1/wiki/Using_Passenger has an
example Apache config stanza for the puppetmaster virtualhost. In it
are the following couple of lines:

# CRL checking should be enabled; if you have problems with Apache
complaining about the CRL, disable the next line
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem

I know it won't help understanding *why* your 3 nodes are misbehaving,
but it may help workaround it.

Regards,

Matt.

[Martinus]

Martin,

Everything is worth a try !

But it did not work :(  

I commented out that line (SSLCARevocationFile) and restarted apache.  No change on the working servers, good.  No change on the broken servers, bad.

Martinus.

[Martinus]

It would also help if I call people by their right name, sorry Matt :)

On Friday, 6 July 2012 11:02:10 UTC+1, Matthew Burgess wrote:

[Martin Alfke]

From http://projects.puppetlabs.com/projects/1/wiki/Certificates_And_Security

Check certificate and validity:

openssl x509 -text -noout -in /var/lib/puppet/ssl/certs/hostname.tld.pem

How do you specifiy the  puppetmaster on the clients?

Do you have a server= line in puppet.conf?

How do the three clients resolv the puppetmaster?

Check certificate on master (take care on AltDNS Names

openssl x509 -text -noout -in /etc/puppet/ssl/certs/master.example.com.pem

Check ca on master:

openssl x509 -text -noout -in /etc/puppet/ssl/certs/ca.pem

Simulate a SSL connection:

openssl s_client -host puppet -port 8140 -cert /path/to/ssl/certs/node.domain.com.pem -key /path/to/ssl/private_keys/node.domain.com.pem -CAfile /path/to/ssl/certs/ca.pem

(from http://www.masterzen.fr/2010/11/14/puppet-ssl-explained/)

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/SJL2yF2M0xoJ.

To post to this group, send email to puppet...@googlegroups.com.

To unsubscribe from this group, send email to puppet-users...@googlegroups.com.

[Martinus]

Right, so here is lots of interesting things now !

The puppetmaster is resolved via /etc/hosts and is set with "server=" in the [main] section.

Trying to connect with the openssl command from a working server is just fine of course.  From one of the broken servers the following error shows up:

CONNECTED(00000003)

3073738376:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:724:

---

no peer certificate available

---

No client certificate CA names sent

---

SSL handshake has read 7 bytes and written 174 bytes

---

New, (NONE), Cipher is (NONE)

Secure Renegotiation IS NOT supported

Compression: NONE

Expansion: NONE

---

I can check the certificate on the client with openssl x509 -text -noout -in /var/lib/puppet/ssl/certs/FQDN.pem.  There are some clear differences between the working and broken systems.  For example:

BROKEN

 Public-Key: (1024 bit)

 Modulus:

...

 X509v3 Key Usage: critical

   Digital Signature, Key Encipherment

...

 X509v3 Subject Key Identifier: 

   CC:67:4F:45:C8:26:34:3A:22:66:E4:16:7C:81:7E:42:B8:CA:55:24

 X509v3 Basic Constraints: critical

   CA:FALSE

WORKING

 RSA Public Key: (1024 bit)

 Modulus (1024 bit):

...

 X509v3 Basic Constraints: critical

   CA:FALSE

...

 X509v3 Key Usage: critical

   Digital Signature, Key Encipherment

 X509v3 Subject Key Identifier: 

   CC:67:4F:45:C8:26:34:3A:22:66:E4:16:7C:81:7E:42:B8:CA:55:24

Unfortunately I do not understand these differences.  Also, I am not sure if those differences is a red herring, as those lines are different again compared to another working system.

On the master certificate, under alternative name, both "puppet.tld" and "mymaster.tld" is listed.  All my clients points to "mymaster.tld" as the server.

Martinus.

To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.