SSL issues arising from cloning environment

[Roger Sherman]

I'm in the process of setting up a staging environment for the company I work for. To do this, we've cloned our production environment (vmware), changed the hostnames of the nodes, re-IP'd the nodes, and since that point, I've been trying to get the environment to the point where I can do puppet runs. This is proving difficult, because of a couple SSL issues that have arisen.

We are using Puppet 2.7 on Debian 6 (one of the plans for the staging environment in the first quarter of next year is going to be testing a lot of upgrades). Right now, I'm just focusing on two nodes - the puppet master and a client node, simply trying to get a successful run done on the client node. For that matter, a successful --noop run. What happens now is the following:

# puppet agent -tv --noop

err: Could not request certificate: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol

Exiting; failed to retrieve certificate and waitforcert is disabled

I can ping the puppet master successfully, there is no issue with node to node connectivity. Some googling suggested that time sync could be an issue, and initially it was, but they're synced up now, to no effect. I've of course blown out the certs on the client, since there was a hostname change.

I actually just tried doing a --noop run on the puppet master itself, just to see what would happen. I didn't expect it to be successful...due to the way it's set up (long story short, I inherited this system, and would not have set it up this way), I've never been able to do a successful puppet run on it. This time, though, not only wasn't successful, but I'm getting similar SSL errors:

...

err: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol

...

err: /File[/var/lib/puppet/lib]: Could not evaluate: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol Could not retrieve file metadata for puppet://puppet/plugins: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol

...

err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol

...

err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol

with a bunch of ruby noise in between each error.

I have blown out all the certs on the puppet master, and regenerated, as detailed in step 1:

https://docs.puppetlabs.com/puppet/latest/reference/ssl_regenerate_certificates.html

For a brief, shining moment, I thought that was going to do the job, but then it didn't generate a couple certs that apache needed to restart, and when I tried to generate them manually, puppet failed to sign them. 

Any thoughts on where to begin? I'll be happy to provide any more information deemed necessary. I'm at the point where I'm just going to start making changes to nodes in the environment manually...honestly, I'm hoping I don't have to do that, so very much hoping someone here can help me through this.

Thanks very much in advance,

Rog

[Neil]

Hello

2.7 is pretty old. IIRC noop stops the agent writing anything including the certificate stuff. You can either
Drop the noop
Upgrade puppet on client
Do the certificate work manually
Specify the certname in the config and use the old cert assuming your cloned master still has that

Neil

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/17d2a513-2d44-4996-9ed1-d4b54d81241f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Roger Sherman]

Hi Neil, thank you for the response.

Certificate requests are sent out during a noop run, which are then signed on the PM and then a subsequent noop run puts the cert in place. For some reason, (I think) the PM is unable to sign them. At least, that's what seems to be the case.

Thank you,

Rog

[jcbollinger]

On Monday, November 17, 2014 10:26:41 PM UTC-6, Roger Sherman wrote:

I'm in the process of setting up a staging environment for the company I work for. To do this, we've cloned our production environment (vmware), changed the hostnames of the nodes, re-IP'd the nodes, and since that point, I've been trying to get the environment to the point where I can do puppet runs. This is proving difficult, because of a couple SSL issues that have arisen.

We are using Puppet 2.7 on Debian 6 (one of the plans for the staging environment in the first quarter of next year is going to be testing a lot of upgrades). Right now, I'm just focusing on two nodes - the puppet master and a client node, simply trying to get a successful run done on the client node. For that matter, a successful --noop run. What happens now is the following:

# puppet agent -tv --noop

err: Could not request certificate: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol

Exiting; failed to retrieve certificate and waitforcert is disabled

I can ping the puppet master successfully, there is no issue with node to node connectivity. Some googling suggested that time sync could be an issue, and initially it was, but they're synced up now, to no effect. I've of course blown out the certs on the client, since there was a hostname change.

Check the client's puppet.conf to see by what name the client is trying to contact the server (parameter 'server').  If no server is specified then the default is 'puppet'.  Ensure that the server name the client is using resolves to the master you want it to contact, and that the puppetmaster process is in fact running on that machine.  Also check the master's firewall settings, which conceivably are incorrect for the cloned environment.


John


 

[jcbollinger]

On Tuesday, November 18, 2014 7:57:44 AM UTC-6, Roger Sherman wrote:

For some reason, (I think) the PM is unable to sign them. At least, that's what seems to be the case.

Well yes, sort of.  It appears that the PM is unable to sign the requests because the client is unable to establish a secure connection over which to issue the request in the first place.  (The client doesn't need its own cert for that.  The client cert is for the client to prove its identity to the master, which it doesn't need to do to request cert signing.)


John

[Roger Sherman]

Hi John,

I've already run through those steps, and everything is communicating as expected. 

Thank you,

Rog

[Roger Sherman]

Right - and on that note, I think I've made a little bit of progress, but I'm still not there yet.

I looked at the apache vhost file for the puppetmaster, and found the following:

# you probably want to tune these settings

PassengerHighPerformance on

PassengerMaxPoolSize 12

PassengerPoolIdleTime 1000

# PassengerMaxRequests 1000

PassengerStatThrottleRate 120

RackAutoDetect Off

RailsAutoDetect Off

Listen 8140

NameVirtualHost 10.60.0.100:8140

<VirtualHost 10.60.0.100:8140>

#       LogLevel debug

        ServerName puppet.nyc.viddler.com

        SSLEngine on

        SSLProtocol -ALL +SSLv3 +TLSv1

        SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

        SSLCertificateFile /var/lib/puppet/ssl/certs/puppet.domain.com.pem

        SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/puppet.domain.com.pem

        SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem

        SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem

        # If Apache complains about invalid signatures on the CRL, you can try disabling

        # CRL checking by commenting the next line, but this is not recommended.

        SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem

So "domain" is our old domain, and 10.60 needs to be changed as well.

I'll report back if this fixes the issue or not.

[Roger Sherman]

Turns out this was the problem - thanks for the help, guys, as always, talking it out helped point me down the right path.

Thanks, 

Rog

[Roger Sherman]

Er ah, to be more specific, I had to list the correct .pem files in the puppetmaster vhost, and change the server IP.