Certificate verify fails without indications

[Luigi Martin Petrella]

I have a puppet master on Centos 6.3 connected and working properly with other Centos 6.3 agent. I installed puppet agent via gems on a RED HAT 4 node. This is what happens when I try to sign certificate for the new node:

AGENT


[root@FP2 ~]$ puppet agent -t Info: Creating a new SSL key for fp2 Info: Caching certificate for ca Info: Creating a new SSL certificate request for fp2 Info: Certificate Request fingerprint (SHA1): 35:51:A0:12:CF:2E:F7:73:22:C3:5E:51:DC:03:AF:4C:FC:54:5C:10 Exiting; no certificate found and waitforcert is disabled

MASTER


[root@puppet centos]# puppet cert list "fp2" (SHA1) 35:51:A0:12:CF:2E:F7:73:22:C3:5E:51:DC:03:AF:4C:FC:54:5C:10 [root@puppet centos]# puppet cert sign fp2 Notice: Signed certificate request for fp2 Notice: Removing file Puppet::SSL::CertificateRequest fp2 at '/var/lib/puppet/ssl/ca/requests/fp2.pem'

AGENT


[root@FP2 ~]$ puppet agent -t Info: Caching certificate for fp2 Warning: Unable to fetch my node definition, but the agent run will continue: Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=Puppet CA: master] Info: Retrieving plugin Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=Puppet CA: master] Error: /File[/var/lib/puppet/lib]: Could not evaluate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=Puppet CA: master] Could not retrieve file metadata for puppet://puppet/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=Puppet CA: master] Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=Puppet CA: master] Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=Puppet CA: master]

I tryied several times to clear certificare on master and agent but I have always the same result. To help to understand and debug the issue, here are some other informations:

– clocks are syncronized on server and agent

-I installed puppet agent on Red Hat 4 node using the following procedure:

Install ruby

a. wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p72.tar.gz

b. tar -xzvf ruby-1.8.7.tar.gz

c. cd ruby-1.8.7

d. ./configure

e. make

f. make install

Install rubygems

a. wget http://rubyforge.org/frs/download.php/70696/rubygems-1.3.7.tgz

b. tar xvzf rubygem.tgz

c. cd rubygem

d. ruby setup.rb

Install library openssl-devel (needed to instal openssl support for ruby, otherwise nothing works)

a. wget ftp://ftp.pbone.net/mirror/ftp.wesmo.com/pub/redhat/i386/openssl-devel-0.9.7-1.i386.rpm

b. rpm –i openssl-devel-0.9.7-1.i386.rpm (Note: 0.9.7 is the most updated version of openssl library that can be installed on red hat 4)

Install openssl support for ruby

a. cd /${ruby_src}/ext/openssl

b. ruby extconf.rb

c. make

d. make install

a. Gem install puppet

• puppet.conf is the same on working and non-working agent

I’m afraid this problem is related to openssl… rpm -qa | grep openssl:

On Centos (master and working nodes)


openssl-devel-1.0.0-25.el6_3.1.i686 openssl-1.0.0-25.el6_3.1.i686

on Red Hat 4 agent:

openssl-0.9.7a-43.17.el4_6.1 openssl-devel-0.9.7-1

Hope someone could help..

[Jo Rhett]

Sounds like your puppet master isn't signing the cert with the name that the agent is connecting with?

All cert problems are either time sync or certificate name issues. So it's one of those two.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To post to this group, send email to puppet...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 

Jo Rhett

Net Consonance : net philanthropy to improve open source and internet projects.

[Luigi Martin Petrella]

Jo, I hope that you are right, because probably time or naming problems are solvable, unlike problems with ssl lib...
Let's assume it is a timing problem: I syncronized date and hwclock on agent manually, obtaining an offset of 2 seconds with master. Is it too much? Shall I set up an NTP service on master?
Otherwise, if the problem is related with naming, what kind of checks should I perform?

thanks a lot in advance

[Felix Frank]

On 02/11/2013 10:51 PM, Jo Rhett wrote:
> All cert problems are either time sync or certificate name issues. So
> it's one of those two.

A bold assertion. It may hold true as far as puppet is concerned, though.

I generally advise to take the time and lern about x509 and openssl's
interface, so one can inspect the actual certificates in question.

> Exiting; no certificate found and waitforcert is
> disabled|

Hmm, so did you *ever* use --waitforcert on your agent side?

If you haven't, that's your problem right there.

HTH,
Felix

[Luigi Martin Petrella]

Felix, why do you think the problem is related to the "--waitforcert" option?
I tryied to run "puppet agent -t --waitforcert 100" , and after signing the request on master, on agent I receive this message:

Error: Could not request certificate: Unsupported digest algorithm (SHA256).
Error: Failed to apply catalog: Unsupported digest algorithm (SHA256).
Error: Could not send report: SSL_CTX_use_PrivateKey:: key values mismatch

[Matthew Black]

What is the versions of the puppet are being used on the client and
the server? Assuming master is running on Linux, what distro and
release is the master running on?

I suspect the openssl might be the issue on the client.

[Luigi Martin Petrella]

Master:
Centos 6.3 , Puppet 3.1.0
Ubuntu, Puppet 3.1.0

Agent:
Redhat 4, Puppet 3.1.0

Yesterday something strange happened:
we tryied to connect RedHat agent with a Puppet Enterprise Master on Centos 6.3, and there wasn't any certificate problems and everything worked.
Today we are trying with the same configuratione, but It appeared the same validation errore described before

[jcbollinger]

On Wednesday, February 13, 2013 6:15:09 AM UTC-6, Felix.Frank wrote:

Hmm, so did you *ever* use --waitforcert on your agent side?

If you haven't, that's your problem right there.

I never use --waitforcert.  Instead, I just run the agent twice when I first set up Puppet, signing the cert in between.  Naturally, the agent receives neither certificate nor catalog on the first run, but on the second it receives both.  That's slightly less secure than the client maintaining the connection and receiving a cert via the same network connection over which it requested one, but it's good enough for me.  Most importantly, it works reliably.  I don't understand the basis for claiming that not using --waitforcert would cause issues.


John

[Luigi Martin Petrella]

I have to do an update.
We just configured one RED HAT 4 node as puppet master, and connected another RH4 agent node without any ssl or certificate issue.

So, brief recap:
MASTER Centos 6.3, Puppet 3.0.1 --> Agent RedHat 4, Puppet 3.0.1 = ERROR
MASTER Ubuntu 12.10, puppet 3.0.1 --> Agent RedHat 4, Puppet 3.0.1 = ERROR
(we mean always the same ERROR on certificate validation)
MASTER Centos 6.3, Puppet Enterprise 2.6 , based on Puppet open source 2.7 --> Agent RedHat 4, Puppet 3.0.1 = OK but not always
MASTER  RedHat 4, Puppet 3.0.1  --> Agent RedHat 4, Puppet 3.0.1 = OK

[Felix Frank]

On 02/13/2013 03:32 PM, Luigi Martin Petrella wrote:
> MASTER Centos 6.3, Puppet 3.0.1 --> Agent RedHat 4, Puppet 3.0.1 = ERROR
> MASTER Ubuntu 12.10, puppet 3.0.1 --> Agent RedHat 4, Puppet 3.0.1 =

> ERROR MASTER RedHat 4, Puppet 3.0.1 --> Agent RedHat 4, Puppet 3.0.1 =
> OK

I agree with Matthew that this does smell like a libssl related issue.

Isn't RedHat 4 ancient? Aren't they past 6 or somesuch by now?

> I don't understand the basis for claiming that not using --waitforcert
> would cause issues.

Mea culpa. I seemed to remember an issue with puppet 2.6 not receiving
the signed certificate unless invoking the option. That may have been me
blundering in some other exciting way, though.

Thanks for clearing that up, John!

[Luigi Martin Petrella]

Yes, RED HAT 4 is very old, but we can't update it.

I agree with the idea that the problem could be ssl library.
As I wrote before, on RH4 we have openssl-0.9.7, on the others systems it'1.0.0

Maybe puppet 3.0.1 master force the use of SHA256 for certificate digest, but SHA256 is not supported by openssl-0.9.7?
Is there any way to force master to use SHA1? I already tryied the option "--digest sha1" in
>puppet cert sign --all --digest sha1
but the error remains...

[Matthew Black]

I think this issue is related to your issue since the version
discussed is 0.9.7.

http://projects.puppetlabs.com/issues/17295

What you will need to do is more than likely is update the openssl on
the agent. I dont think it will work too well but you can try to take
the srpm from rhel 5 or 6 and build it for rhel 4


On Wed, Feb 13, 2013 at 8:31 AM, Luigi Martin Petrella

[Luigi Martin Petrella]

Matthew, you are right, this explain ALMOST everything

"Puppet is using the Solaris-provided OpenSSL as part of the Ruby install in this case, which runs version 0.9.7 with patches and doesn’t support sha256. I don’t mind the idea of compiling 1.0.x but the issue still seems to stand that you can’t choose the digest method anymore – there is an apparent use of SHA256 regardless of what option you choose."

But

If I use as master RH4 with openssl-lib 0.9.7 I have no problem connecting the others RH4 nodes. This means tha Puppet don't use always  SHA256, but only If it is available from openssl library. Right?

So, there are two ways (one harder then the other for me) to solve the issue at openssl level:
1. install opensslib rpm for RH5 on RH4 (but there are a lot of missing dependencies)
2. downgrade openssl lib on Centos 6.3 master from 1.0.0 to 0.9.7
???

Since --digest option won't work, is there any other way to force puppet not to use SHA256??

[Matthew Black]

Yes because as part of the fix it checks on the CA, when its signing
the cert, whether it can support 256 or not. If it does not it drops
down to a lower SHA.

If you look at the pull request that is part of the ticket,
specifically the changes. If you scroll down to the
certificate_signer.rb change it will make more sense.

https://github.com/puppetlabs/puppet/pull/1413/files


On Wed, Feb 13, 2013 at 10:37 AM, Luigi Martin Petrella

[Luigi Martin Petrella]

Yes, it is exactly the cause of the problem!
"

certificate_signer.rb

# Take care of signing a certificate in a FIPS 140-2 compliant manner.

#

# @see http://projects.puppetlabs.com/issues/17295

#

# @api private

class Puppet::SSL::CertificateSigner

  def initialize

    if OpenSSL::Digest.const_defined?('SHA256')

      @digest = OpenSSL::Digest::SHA256

    elsif OpenSSL::Digest.const_defined?('SHA1')

      @digest = OpenSSL::Digest::SHA1

    else

      raise Puppet::Error,

        "No FIPS 140-2 compliant digest algorithm in OpenSSL::Digest"

    end

    @digest

  end

 

  def sign(content, key)

    content.sign(key, @digest.new)

  end

end

"

If I switch the order of these checks

    if OpenSSL::Digest.const_defined?('SHA256')

      @digest = OpenSSL::Digest::SHA256

    elsif OpenSSL::Digest.const_defined?('SHA1')

      @digest = OpenSSL::Digest::SHA1

probably it will work

I'll let you know..

[Luigi Martin Petrella]

The trick worked :-)

Thanks to everyone for your contribution!

[binaryred]

Luigi,

I find I'm in a similar situation as you, except I am not running puppet 3 on my client, I am running puppet 2.7.  This change that you made, was it on the client or your puppet master?

Thanks,

Jason

[Luigi Martin Petrella]

Jason,
I did the change on master, Centos 6.3 with Puppet 3.1.0.
This modification can't be applied on Puppet 2.7.x since the class certificate_signer.rb doesn't exist in Puppet 2.7 source code.

What's your configuration on master and agent nodes?
What's the output of "rpm -qa | grep openssl" ?

[binaryred]

Puppet master is running RHEL 6.3 with the following packages:

puppet-3.1.0-1.el6.noarch
puppet-server-3.1.0-1.el6.noarch
openssl-1.0.0-20.el6_2.5.x86_64

Client is running RHEL 4.8 with the following packages:

puppet-2.7.20-1
openssl-0.9.7a-43.17.el4_7.2

After changing the certificate_signer.rb file as you suggested, I rebooted my puppet master and cleared the cert for the client, and then removed /var/lib/puppet/ssl on the client as well.  I then run 'puppet agent -t' on the client and this is what I get:

err: Could not retrieve catalog from remote server: certificate verify failed: [certificate signature failure for /CN=puppetmaster.example.com
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
err: Could not send report: certificate verify failed: [certificate signature failure for /CN=puppetmaster.example.com]

Maybe this isn't an ssl issue, but I'm not sure what else would be wrong.

Jason

[Luigi Martin Petrella]

Your configuration is almost the same as mine.
I'm not 100% sure but I think that after modifying certificate_signer.rb you should re-install puppet, running "ruby install.rb" again.

(in my case, I first downloaded source code, then modified the class and finally ran the install.rb)

[binaryred]

Unfortunately, I am installing my puppet agent and master with RPMs.  When I uninstall and reinstall the puppet agent, it blows away the certificate_signer.rb file and recreates it with the original file.

I have a number of systems  (not all of which I have control over) that I'll need to do this to or have done to them, so the method suggested is not appropriate.

Any other suggestions?

Thanks,

Jason

[Felix Frank]

On 02/14/2013 05:20 PM, binaryred wrote:
> Any other suggestions?

Yeah, actually...

> err: Could not send report: certificate verify failed: [certificate
> signature failure for /CN=puppetmaster.example.com

> <http://puppetmaster.example.com>]

Is the name of your master puppetmaster.example.com?

Are you sure your puppetca is set up properly?

Regards,
Felix

[binaryred]

Yeah, I just replaced my server name with that.  I've got RHEL5 and RHEL6 machines talking to my puppet master just fine.

[binaryred]

On my puppet master, I uninstalled my puppet RPM, downloaded the tarball for puppet 3.1.0, modified the source for the certificate_signer.rb, and ran 'ruby install.db'.  It installed the modified certificate_signer.rb file and runs just fine on the master (as it did before), but my client RHEL4 boxes still don't want to talk to the puppet master server correctly.  I'm still getting the same error.

Jason

[Luigi Martin Petrella]

Jason, you could try to set one Redhat 4 node as master  and verify if it works correctly with another RH4 agent, so you can establish if the problem is about RH4 agents or RH6 master..

--

[binaryred]

Luigi,

Thanks for the suggestion, however I've already done that in some sense.  Here's my FULL situation:

I was running a puppet 2.6.6 master on a RHEL5 machine with lots of RHEL4,5,6 machines (mostly RHEL5) connecting to it.  The clients are all running puppet 0.25.5 and working just fine.

I've built a new puppet server on a RHEL6 machine, running 3.1.0.  I copied over the SSL certs from the old puppet master so that when the clients connect to the new server, they 'just work', and pretty much that has worked great for me.  I certainly plan to upgrade the clients to the latest version of puppet I can, but for now they are working fine.  EXCEPT for the RHEL4 machines.  I tried the version of puppet that was on them first (0.25.5), and when that didn't work, I found some puppet 2.7 packages (and dependencies) to install, but they don't seem to work any better.

So the short story is, that the RHEL 4 clients can talk to my old puppet master, but not the new one, while everything else talks to the new puppet master just fine.

Jason

[Luigi Martin Petrella]

Jason,  for the reasons we wrote before in prevoius messages (especially what Matt Black said), Puppet 3.1.0 will never work with an agent that run openssl library version 0.9.7 (which is the version running on RH4)
Even if you had master with Puppet 2.7.x working correctly with RH4 nodes, it is perfectly clear that upgrading to puppet 3.1.0 (without modifying certificate_signer.rb)  the connection with RH4 agent will fail rising the error you have.

If you correctly modified certificate_signer.rb and re-installed puppet with the modified source, maybe you have ALSO ANOTHER problem somewhere else, but in that case I can't figure where...

[binaryred]

I will try to work with the certificate_signer.rb file and see if I can get it to work.  Thanks for the help!

Jason

[comp...@gmail.com]

This thread was very helpful and got me most of the way there. I started with bare-bones legacy RHEL4 vms and had to add a couple steps to get things working properly. Here's the complete procedure from start to finish:

RHEL4 Client Installation:

Set up some repos:

cd /etc/yum.repos.d

wget http://public-yum.oracle.com/public-yum-el4.repo

Install some packages:

yum install gcc

yum install zlib

yum install zlib-devel

Install Ruby:

cd /opt

wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p72.tar.gz 

tar -xzvf ruby-1.8.7-p72.tar.gz

cd ruby-1.8.7-p72 

./configure

make

make install

Install ruby gems:

cd /opt

wget http://pkgs.fedoraproject.org/repo/pkgs/rubygems/rubygems-1.3.7.tgz/e85cfadd025ff6ab689375adbf344bbe/rubygems-1.3.7.tgz

tar -xvzf rubygems-1.3.7.tgz

cd rubygems-1.3.7

/usr/local/bin/ruby setup.rb

Install/Upgrade openssl and openssl-devel:

yum install openssl

yum install openssl-devel

Install openssl support for ruby:

cd /opt/ruby-1.8.7-p72/ext/openssl

/usr/local/bin/ruby extconf.rb

make 

make install

Install Puppet:

gem install puppet

Configure Puppet:

mkdir /etc/puppet

cp /usr/local/lib/ruby/gems/1.8/gems/puppet-3.7.3/ext/redhat/puppet.conf /etc/puppet/.

Add your changes to puppet.conf

Oracle Enterprise Linux 7 Server Install Changes:

If you did a yum install puppet-server from the puppetlabs repo already:

This procedure won't blow away changes to config files, but let's back them up anyway:

cp -rp /etc/puppet /root/.

Remove the package and clear the ssl directory.

yum remove puppet-server

rm -rf /etc/puppet/ssl/*

Download puppet source tarball and fixup the code:

cd /opt

wget http://downloads.puppetlabs.com/puppet/puppet-3.7.2.tar.gz

tar -xvzf puppet-3.7.2.tar.gz

cd /opt/puppet-3.7.2/lib/puppet/ssl

vi certificate_signer.rb

  swap all instances of SHA256 for SHA1, swap SHA1 for SHA256 see upthread for details

Install Puppet:

cd /opt/puppet-3.7.2

ruby install.rb

Start the Puppet Master and generate new SSL CA certs:

puppet master --verbose --no-daemonize

You should see the ca cert being generated with SHA1 fingerprint like this:

Info: Creating a new SSL key for ca

Info: Creating a new SSL certificate request for ca

Info: Certificate Request fingerprint (SHA1): 33:81:E5:BF:A2:E4:57:86:17:B2:2F:DC:AB:BA:2D:6E:0F:D6:C3:7E

Notice: Signed certificate request for ca

Info: Creating a new certificate revocation list

Info: Creating a new SSL key for puppet.my.domain.com

Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml

Info: Creating a new SSL certificate request for puppet.my.domain.com

Info: Certificate Request fingerprint (SHA1): AB:B1:A7:21:F0:AA:0A:CB:E4:76:2D:5C:B4:87:38:69:BB:70:23:DF

Notice: puppet.my.domain.com has a waiting certificate request

Info: Autosigning puppet.my.domain.com

Notice: Signed certificate request for puppet.my.domain.com

Notice: Removing file Puppet::SSL::CertificateRequest puppet.my.domain.com at '/var/puppet/ssl/ca/requests/puppet.my.domain.com.pem'

Notice: Removing file Puppet::SSL::CertificateRequest puppet.my.domain.com at '/var/puppet/ssl/certificate_requests/puppet.my.domain.pem'

Notice: Starting Puppet master version 3.7.2

^CNotice: Caught INT; calling stop

Hit ctrl-C when you see the line that says: Notice: Starting Puppet master version 3.7.2

You can now start your apache/rack application and you are good to go.

Make sure to get rid of existing certs on any non-RHEL4 clients so they can regenerate them from the altered master.