"SSLv3 read server certificate B: certificate verify failed." -- Not time related

[Jon Davis]

I recently built, added to puppet and then nuked a server.  Before I re-added the machine (after I rebuilt it, with the same name), I went to the puppet server and ran `puppet cert revoke dev-8.company.com` and `puppet cert clean dev-8.company.com`.  Now when puppet runs on ANY server in my environment, they get the following error:

info: Caching certificate for dev-8.company.com

err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client

warning: Not using cache on failed catalog

err: Could not retrieve catalog; skipping run

err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client

Now I know for a fact that it isn't a time issue because the puppet server is on NTP as are the clients.  The new machine is also within 1-2 seconds of server time.  All of the clients are configured to run (via Cron) `/usr/sbin/puppetd --onetime --no-daemonize --logdest syslog --server puppet.company.com`.  The server is named puppet-1.company.com but puppet. is a valid cname.  I've tried rebooting the puppet server, I've tried upgrading it, just about anything I can think of.  

Any help would be greatly appreciated.

-Jon

PS Both clients and server are running Ubuntu:

root@puppet-1:/etc/puppet# cat /etc/lsb-release

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=11.10

DISTRIB_CODENAME=oneiric

DISTRIB_DESCRIPTION="Ubuntu 11.10"

root@puppet-1:/etc/puppet# uname -a

Linux puppet-1 3.0.0-16-server #28-Ubuntu SMP Fri Jan 27 18:03:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

--
Jon
[[User:ShakataGaNai]] / KJ6FNQ
http://snowulf.com/
http://www.linkedin.com/in/shakataganai

[Russell Van Tassell]

Just a couple of issues...

On Tue, Feb 21, 2012 at 4:56 PM, Jon Davis <j...@snowulf.com> wrote:

I recently built, added to puppet and then nuked a server.  Before I re-added the machine (after I rebuilt it, with the same name), I went to the puppet server and ran `puppet cert revoke dev-8.company.com` and `puppet cert clean dev-8.company.com`.  Now when puppet runs on ANY server in my environment, they get the following error:

info: Caching certificate for dev-8.company.com

err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client

warning: Not using cache on failed catalog

err: Could not retrieve catalog; skipping run

err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client

Now I know for a fact that it isn't a time issue because the puppet server is on NTP as are the clients.  The new machine is also within 1-2 seconds of server time.

For "normal" NTP clients, this would imply that your time sync is off by a few factors (ie. your time differences should be mere fractions of seconds off between servers if your NTP setup is working correctly).

 

 All of the clients are configured to run (via Cron) `/usr/sbin/puppetd --onetime --no-daemonize --logdest syslog --server puppet.company.com`.  The server is named puppet-1.company.com but puppet. is a valid cname.  I've tried rebooting the puppet server, I've tried upgrading it, just about anything I can think of.  

If the reverse (IN-ADDR) of your puppet server is going to return puppet.company.com as its name, but you are connecting to foo.company.com, that's pretty much a textbook SSL error (ie. your SSL certificate doesn't match the name it's claiming to be). What happens if you delete the SSL cert on the client, and re-run the CSR by pointing at the real name of the server?

Hope that helps...

Russell

[Jon Davis]

On Tue, Feb 21, 2012 at 17:05, Russell Van Tassell <russ...@gmail.com> wrote:

Just a couple of issues...

On Tue, Feb 21, 2012 at 4:56 PM, Jon Davis <j...@snowulf.com> wrote:

I recently built, added to puppet and then nuked a server.  Before I re-added the machine (after I rebuilt it, with the same name), I went to the puppet server and ran `puppet cert revoke dev-8.company.com` and `puppet cert clean dev-8.company.com`.  Now when puppet runs on ANY server in my environment, they get the following error:

info: Caching certificate for dev-8.company.com

err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client

warning: Not using cache on failed catalog

err: Could not retrieve catalog; skipping run

err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client

Now I know for a fact that it isn't a time issue because the puppet server is on NTP as are the clients.  The new machine is also within 1-2 seconds of server time.

For "normal" NTP clients, this would imply that your time sync is off by a few factors (ie. your time differences should be mere fractions of seconds off between servers if your NTP setup is working correctly).

 

There isn't any time issue, just my typing `date` one one machine to the other.  Everyone is running NTP it's fine.

 

 All of the clients are configured to run (via Cron) `/usr/sbin/puppetd --onetime --no-daemonize --logdest syslog --server puppet.company.com`.  The server is named puppet-1.company.com but puppet. is a valid cname.  I've tried rebooting the puppet server, I've tried upgrading it, just about anything I can think of.  

If the reverse (IN-ADDR) of your puppet server is going to return puppet.company.com as its name, but you are connecting to foo.company.com, that's pretty much a textbook SSL error (ie. your SSL certificate doesn't match the name it's claiming to be). What happens if you delete the SSL cert on the client, and re-run the CSR by pointing at the real name of the server?

Well unfortunately this worked until a few hours ago and I haven't changed anything in the DNS.  There is actually no IN-ADDR record for this server.   When I generated the SSL cert for puppet, I told it to use puppet.company.com (IE in puppet.conf it says certname=puppet.company.com )

I've deleted certs and re-run puppet on the client about a dozen times now.  I've also made sure to revoke/clean on the server between each try.

 

Hope that helps...

Russell

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Jon Davis]

How can I track down where the issue for this is?   I've found some bugs and blog posts that seem to be related [1][2] and I've followed all of the instructions and checked ALL of the versions related.  I'm running Ruby 1.8.7 and Puppet 2.7.9 on both sides of the equation, which appear to be "OK" versions by everyone's posting.  I've got as far as doing a `puppet cert clean --all` and `puppet cert clean puppet.company.com` and regenerating.  Still doesn't work. I've also followed every step on only Puppet Doc's page that I can find related entries on [3]

-Jon

[1]  http://projects.puppetlabs.com/issues/9084 

[2]  http://urgetopunt.com/puppet/2011/09/14/puppet-ruby19.html 

[3]  http://docs.puppetlabs.com/pe/2.0/maint_common_config_errors.html#do-agents-trust-the-masters-certificate 

[Gary Larizza]

On Wed, Feb 22, 2012 at 11:58 AM, Jon Davis <j...@snowulf.com> wrote:

How can I track down where the issue for this is?   I've found some bugs and blog posts that seem to be related [1][2] and I've followed all of the instructions and checked ALL of the versions related.  I'm running Ruby 1.8.7 and Puppet 2.7.9 on both sides of the equation, which appear to be "OK" versions by everyone's posting.  I've got as far as doing a `puppet cert clean --all` and `puppet cert clean puppet.company.com` and regenerating.  Still doesn't work. I've also followed every step on only Puppet Doc's page that I can find related entries on [3]

Hey Jon,

When you cleaned the certs on the SERVER side, did you also clean the $ssldir on the CLIENT side and try to connect to the master again?  Doing a `puppet config print ssldir` will give you the path to your $ssldir.  I would:

1. Clean the cert on the master

2. Clean the ssldir on the client

3. Try running `puppet agent -t` on the client to generate a CSR on the master

4. Sign the cert on the master

5. Try running puppet again on the client.

Does this work for you?

 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--

Gary Larizza

Professional Services Engineer

Puppet Labs

[Jon Davis]

I was cleaning the clients yes.

After I cleaned the puppet server and the client AND still had issues.  I decided to blow away everything in /var/lib/puppet/ssl on the master and rebuild it.  Fortunately I only have a few dozen puppetized machines because... I have to go through and re-cert them all again.  But for now it seems to be working.

Freaking massive headache.

[Mukul Malhotra]

Just remove the certificates from the client & server from /var/lib/puppet/ssl/* & it will be ok.

mukulm

[Felix Frank]

Hi,

On 02/22/2012 08:58 PM, Jon Davis wrote:
> How can I track down where the issue for this is?

it's always troublesome, but the only clean approach I'm aware of is
"openssl s_client" and "openssl x509" to carefully compare what the
master is presenting when the agent connects to whatever the agent is
expecting (i.e. what's cached on agent side).

I'd like to stress this: Not everyone is operating under testing
conditions. Blasting $ssldir is typically not an option. I know people
are trying to be helpful, but puppet authentication is no a good
instance to endorse the KISS strategy.

Cheers,
Felix

[glm]

Hi,

I am having a similar problem but I am trying to run puppetd -t on the server as a client of itself.  This works on our other puppet master.  Like the poster above, I have cleared /var/lib/puppet/ssl a dozen times and time cannot be an issue because client and server are the same machine.  I have tried this with both puppetmasterd and with the apache passenger module, which is what we have running on our other puppet master, which works.

I am using puppet versions

puppet-2.7.9-2.el6.noarch

puppet-server-2.7.9-2.el6.noarch

on top of ruby versions:

ruby-1.8.7.352-4.el6_2.x86_64

rubygems-1.3.7-1.el6.noarch

ruby-libs-1.8.7.352-4.el6_2.x86_64

All of this on CentOS 6.

Any ideas?

Thanks.

Glen

[Kinzel, David]

Take a look at bug 8858 and 9084. But have some suggested "fixes" to see if you are hitting them. If you are running the client and master on the same server thought (and both are using the same cert) this may not be the case.

________________________________

Any ideas?

Thanks.

Glen

http://www.linkedin.com/in/shakataganai <http://www.linkedin.com/in/shakataganai> <http://twitter.com/shakataganai>

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/we1mj3rXSUcJ.

To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

This email communication and any files transmitted with it may contain
confidential and or proprietary information and is provided for the use of the
intended recipient only. Any review, retransmission or dissemination of this
information by anyone other than the intended recipient is prohibited. If you
receive this email in error, please contact the sender and delete this
communication and any copies immediately. Thank you.

http://www.encana.com

[Asher Bond]

I had this problem. So I ran this on the puppetmaster:

puppet cert --list --all

came back with nothing for the puppetmaster itself. I added my.domain to search domains in /etc/resolv.conf

I rebooted the puppetmaster and when I ran

puppet cert --list --all

I saw two certs, the one for my agent and the one for my puppetmaster. Now it works.