SSL Errors - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B

[Romeo Theriault]

Hello, I'm new to puppet and am getting a puppet server setup with
puppet dashboard. I have the puppet server and puppet dashboard
(Apache/Passenger) setup and working well with 60+ test nodes working
as expected. Only problem is that I have this one error in the logs
which I can't figure out.

Jan 26 17:09:41 ppt01 puppet-agent[27357]: Could not retrieve catalog
from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read
server certificate B: certificate verify failed.  This is often
because the time is out of sync on the server or client
Jan 26 17:09:41 ppt01 puppet-agent[27357]: Using cached catalog
Jan 26 17:09:42 ppt01 puppet-agent[27357]:
(/Stage[main]/Puppet/File[run_puppet.sh]) Could not evaluate:
SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B:
certificate verify failed.  This is often because the time is out of
sync on the server or client Could not retrieve file metadata for
puppet:///modules/puppet/run_puppet.sh: SSL_connect returned=1 errno=0
state=SSLv3 read server certificate B: certificate verify failed.
This is often because the time is out of sync on the server or client
at /etc/puppet/modules/puppet/manifests/init.pp:67
Jan 26 17:09:42 ppt01 puppet-agent[27357]:
(/Stage[main]/Puppet/Cron[puppet]) Dependency File[run_puppet.sh] has
failures: true
Jan 26 17:09:42 ppt01 puppet-agent[27357]:
(/Stage[main]/Puppet/Cron[puppet]) Skipping because of failed
dependencies
Jan 26 17:09:42 ppt01 puppet-agent[27357]: Finished catalog run in 0.21 seconds
Jan 26 17:09:42 ppt01 puppet-agent[27357]: Could not send report:
SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B:
certificate verify failed.  This is often because the time is out of
sync on the server or client

These errors are from the puppet agent that is running on the
puppet-master server. The odd thing is if I run it manually everything
works as it should. I also have a cron job that runs it every 30
minutes and this works fine as well. I have no idea how the puppet
agent is getting called during this failed run. It happens reliably
every 30 minutes but outside of the time that my cron job runs...
Does anyone have any idea what might be calling this failed run?
Something with the dashboard I'm guessing but I'm unable to find
anything.

Next odd thing is that this failed run also doesn't appear to be
affecting anything. All the Dashboard (and puppet master)
functionality is working as it should, including reporting,
filebucketing and inventory. All clients are getting their catalogs,
etc... so I'm really not sure where this is originating from.

I should note that I did change the hostname the puppet server is
using but updated every (I think) to reflect the new hostname,
including regenerating the server and client certs.

I've found this page:
http://docs.puppetlabs.com/pe/2.0/maint_common_config_errors.html#do-agents-trust-the-masters-certificate

which covers these errors but they don't seem to be my issue. It's
obviously not a time issue considering the agent that is complaining
in on the master. I've `puppet cert clean`-ed, re-re-created and
re-signed the client certs against the new master certs and the puppet
agent runs are working from my cron calls and when run manually.

Any help in determining where this is getting called from and how I
can clear it up would be greatly appreciated.

Here is my puppet.conf on my master. I'd be happy to provide any other
info that my be helpful.

[agent]
        server = host.pvt.domain.com
        report = true

[master]
    ssldir = $vardir/ssl
    certname = host.pvt.domain.com

    # For the Inventory service
    facts_terminus = inventory_active_record
    dbadapter = mysql
    dbname = puppet_inventory
    dbuser = puppet
    dbpassword = super-secret
    dbserver = localhost
    dbsocket = /var/lib/mysql/mysql.sock

    # For reports
    reports = store, http
    reporturl = http://host.pvt.domain.com/reports/upload

    # For puppet dashboards external node classification.
    node_terminus = exec
    external_nodes = /usr/bin/env
PUPPET_DASHBOARD_URL=http://puppet:80
/usr/share/puppet-dashboard/bin/external_node

Thank you,

--
Romeo

[Felix Frank]

Hi,

concerning your question why everything seems to work pretty well:

On 01/27/2012 04:59 AM, Romeo Theriault wrote:
> Jan 26 17:09:41 ppt01 puppet-agent[27357]: Using cached catalog

Your agent is using a cached catalog.

puppet agent --test should fail. Also, changing the manifest for this
node should not have any effect until you resolve this problem.

My guess is that the agent has an old master certificate stored or
somesuch. For some reason it regards your current master cert as invalid.

The simplest approach may be to scrutinize the local /var/lib/puppet/ssl
for certificates that match your master's FQDN (perhaps "puppet"). If
you find several, use "openssl x509" to find out how they differ.

HTH,
Felix

[Romeo Theriault]

Hi Felix, thanks for your response to my question. It's taken me a
while to get back to this issue but I finally figured it out tonight.
I had a old puppetd process running in the background (I'd since moved
to using cron to call puppet) that must have been holding open it's
old cert files, etc... After I killed the old puppetd process
everyting is working as it should. (i.e. no more errors and the
correct puppet process is still running as it should).

Thanks,

Romeo

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>

--
Romeo

[vinay mandava]

Make sure the time matches on master and agent. 

The issue CRL is not yet valid for indicates that the time between the Puppet-agent and the Puppetmaster is out of sync. Sync the time (NTP). Remove the certificate from the Puppet-agent and Puppetmaster as well and run Puppet on the agent.