Fighting with private keys and puppet master on Debian Squeeze

[Brian Troutwine]

Hello, all.

Let's say I authenticate a server against my puppet master at puppet.example.com, we'll call it apt.example.com. All goes well; it's peachy. Then, apt.example.com dies and I have to authenticate a _new_ apt.example.com against puppet master. 

root@apt:~# puppet agent --test --noop

info: Creating a new SSL key for apt.example.com

warning: peer certificate won't be verified in this SSL session

info: Caching certificate for apt.example.com

err: Could not request certificate: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key

Exiting; failed to retrieve certificate and waitforcert is disabled

Not entirely unexpected. So,

root@puppet:~# puppet cert --revoke apt.example.com

notice: Revoked certificate with serial 3

but then,

root@apt:~# puppet agent --test --noop

err: Could not request certificate: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key

Exiting; failed to retrieve certificate and waitforcert is disabled

as such,

root@apt:~# puppet agent --test --noop

info: Creating a new SSL key for apt.example.com

warning: peer certificate won't be verified in this SSL session

info: Caching certificate for ca

warning: peer certificate won't be verified in this SSL session

info: Caching certificate for apt.example.com

err: Could not request certificate: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key

Exiting; failed to retrieve certificate and waitforcert is disabled

Absolute madness. I can delete the contents of puppet master's ssldir and everything works, but that brings my cluster back to null and I cannot do that. How do I actually revoke a faulty certificate?

root@apt:~# cat /etc/debian_version 

6.0.2

root@apt:~# puppet --version

2.7.1

--
Brian L. Troutwine

[Laurence Southon]

On 20/08/11 01:13, Brian Troutwine wrote:
> How do I actually revoke a faulty certificate?

You can remove the client certificate entirely with:

puppetca --clean apt.example.com

A new one will then be generated next time you connect.

LS
--
Laurence Southon
Tiger Computing, Bexley
www.tiger-computing.co.uk

[Brian Troutwine]

On Sat, Aug 20, 2011 at 10:04 AM, Laurence Southon <l...@tiger-computing.co.uk> wrote:

On 20/08/11 01:13, Brian Troutwine wrote:
> How do I actually revoke a faulty certificate?

You can remove the client certificate entirely with:

puppetca --clean apt.example.com

I overlooked that entirely. Thank you.

A new one will then be generated next time you connect.

LS
--
Laurence Southon
Tiger Computing, Bexley
www.tiger-computing.co.uk

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--
Brian L. Troutwine

[Brian Troutwine]

On Sat, Aug 20, 2011 at 12:18 PM, Brian Troutwine <br...@troutwine.us> wrote:

On Sat, Aug 20, 2011 at 10:04 AM, Laurence Southon <l...@tiger-computing.co.uk> wrote:

On 20/08/11 01:13, Brian Troutwine wrote:
> How do I actually revoke a faulty certificate?

You can remove the client certificate entirely with:

puppetca --clean apt.example.com

I overlooked that entirely. Thank you.

This does look like the flag I was looking for, however:

# puppet cert --clean apt.example.com

notice: Revoked certificate with serial # Inventory of signed certificates

# SERIAL NOT_BEFORE NOT_AFTER SUBJECT

0x0001 2011-08-19T18:20:48GMT 2016-08-17T18:20:48GMT /CN=Puppet CA: puppet.example.com

0x0002 2011-08-19T18:20:48GMT 2016-08-17T18:20:48GMT /CN=puppet.example.com

0x0003 2011-08-19T18:21:46GMT 2016-08-17T18:21:46GMT /CN=gateway.example.com

err: Could not call revoke: Cannot convert into OpenSSL::BN

 and on apt.example.com:

# puppet agent --test --noop

info: Creating a new SSL key for apt.example.com

warning: peer certificate won't be verified in this SSL session

warning: peer certificate won't be verified in this SSL session

info: Creating a new SSL certificate request for apt.example.com

info: Certificate Request fingerprint (md5): FB:05:0D:41:C8:46:3C:44:EE:AC:9D:48:4E:4A:CC:FB

warning: peer certificate won't be verified in this SSL session

warning: peer certificate won't be verified in this SSL session

info: Caching certificate for apt.example.com

err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

warning: Not using cache on failed catalog

err: Could not retrieve catalog; skipping run

A new one will then be generated next time you connect.

LS
--
Laurence Southon
Tiger Computing, Bexley
www.tiger-computing.co.uk

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--
Brian L. Troutwine

--
Brian L. Troutwine

[Brian Troutwine]

Similarly, 

# puppet cert --list --all

+ apt.example.com (4C:FB:40:5B:9F:0F:CB:8B:78:57:78:D2:34:3F:8F:9B)

+ puppet.example.com (C5:37:33:6A:1D:AB:60:55:61:05:55:05:03:56:35:45)

# puppet cert --clean apt.example.com

notice: Revoked certificate with serial 3

notice: Removing file Puppet::SSL::Certificate apt.example.com at '/var/lib/puppet/ssl/ca/signed/apt.example.com.pem'

notice: Removing file Puppet::SSL::Certificate apt.example.com at '/var/lib/puppet/ssl/certs/apt.example.com.pem'

but then,

# puppet agent --test --noop

err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

warning: Not using cache on failed catalog

err: Could not retrieve catalog; skipping run

Mind you, apt is a virgin computer each time. Things that the error message don't tell me:

  * Which certificate failed,

  * why it failed in the context of puppet (not raw ssl jargon) and

  * what I should do to remedy the problem.

 

A new one will then be generated next time you connect.

LS
--
Laurence Southon
Tiger Computing, Bexley
www.tiger-computing.co.uk

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--
Brian L. Troutwine

--
Brian L. Troutwine

--
Brian L. Troutwine

[Denmat]

Hi,

Are you calling the puppet run with the '--server <puppetmaster>' parameter? 

With SSL you basically need the following:

 * working DNS

 * clocks in sync

 * correct certnames

To help solve SSL issues also use 'openssl s_client' to test connections, check certnames and other errors.

This is a definitive reference on puppet and SSL.

http://www.masterzen.fr/2010/11/14/puppet-ssl-explained/

There is a newer doc than this on puppet docs site but can't find it at the moment:

http://projects.puppetlabs.com/projects/1/wiki/Certificates_And_Security

Den

[Brian Troutwine]

On Sat, Aug 20, 2011 at 6:18 PM, Denmat <tu2b...@gmail.com> wrote:

Hi,

Are you calling the puppet run with the '--server <puppetmaster>' parameter? 

Default is 'puppet', no? In any event, using --server or not has no effect.

 

With SSL you basically need the following:

 * working DNS

 * clocks in sync

Done.

 

 * correct certnames

Can you elaborate? 

 

To help solve SSL issues also use 'openssl s_client' to test connections, check certnames and other errors.

This is a definitive reference on puppet and SSL.

http://www.masterzen.fr/2010/11/14/puppet-ssl-explained/

There is a newer doc than this on puppet docs site but can't find it at the moment:

http://projects.puppetlabs.com/projects/1/wiki/Certificates_And_Security

The latter is all that turns up in a google search. Here's the fresh apt:

# puppet agent --test --noop

info: Creating a new SSL key for apt.example.com

warning: peer certificate won't be verified in this SSL session

info: Caching certificate for ca

warning: peer certificate won't be verified in this SSL session

info: Caching certificate for apt.example.com

err: Could not request certificate: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key

Exiting; failed to retrieve certificate and waitforcert is disabled

Then on the puppet master:

# puppet cert --clean apt.example.com

notice: Revoked certificate with serial 4

notice: Removing file Puppet::SSL::Certificate apt.example.com at '/var/lib/puppet/ssl/ca/signed/apt.example.com.pem'

notice: Removing file Puppet::SSL::Certificate apt.example.com at '/var/lib/puppet/ssl/certs/apt.example.com.pem'

back to apt:

# puppet agent --test --noop

err: Could not request certificate: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key

Exiting; failed to retrieve certificate and waitforcert is disabled

root@apt:~# openssl s_client -host puppet -port 8139 -cert /var/lib/puppet/ssl/certs/apt.example.com.pem -key /var/lib/puppet/ssl/private_keys/apt.example.com.pem -CAfile /var/lib/puppet/ssl/certs/ca.pem 

error setting private key

732:error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch:x509_cmp.c:406:

Doesn't really tell me much. Then I noticed that puppet master's running 2.7.1 while my apt client machine is on 2.6.2--part of the catalog is an update of the puppet client to Debian backport's latest. I ran that update manually and:

# openssl s_client -host puppet -port 8139 -cert /var/lib/puppet/ssl/certs/apt.example.com.pem -key /var/lib/puppet/ssl/private_keys/apt.example.com.pem -CAfile /var/lib/puppet/ssl/certs/ca.pem 

connect: Connection refused

connect:errno=111

while

# puppet agent --server puppet.example.com --test

ran to completion, with no errors. That leaves me even more confused than before, frankly. It's possible I'm not using openssl s_client correctly, but I think there's sufficient evidence that the puppet master is listening and will push down catalogs. 

--
Brian L. Troutwine

[Denmat]

Hi,

The standard port is 8140, are you using a different port?

You will need to pass --server on the puppet agent. The command line is different from the daemon config.

With certnames, your --servername must match the DNS name of the master. So using openssl s_client --connect <puppetmaster:8140> you should see the certname in the response. If that is different from the dns name then you will have issues (which can be solved through the puppet.conf certname directive).

Lastly it always good to run the same versions on client and server I find. So it's all running now?

Den

[Nicolai]

To make sure you have a clean client and master knows nothing about it, (ssl-wise) do the following:

on master:

puppetca --clean client.example.com

on client:

rm -r /var/lib/puppet/ssl

puppetd --test

back to master:

puppetca --list   (to check for the signing request from client)

puppetca --sign client.example.com

on client:

puppetd --test

and you shouldnt have any issues with ssl-connection. (if time/dns etc is correct).

Nicolai Mollerup

[Brian Troutwine]

On Sat, Aug 20, 2011 at 7:40 PM, Denmat <tu2b...@gmail.com> wrote:

Hi,

The standard port is 8140, are you using a different port?

Didn't know that was standard, but yes.

 

You will need to pass --server on the puppet agent. The command line is different from the daemon config.

In what way?

 

With certnames, your --servername must match the DNS name of the master. So using openssl s_client --connect <puppetmaster:8140> you should see the certname in the response. If that is different from the dns name then you will have issues (which can be solved through the puppet.conf certname directive).

Ah. I'm already using the certname directive and using the fqdn of the master corrects the issue.

 

Lastly it always good to run the same versions on client and server I find. So it's all running now?

It is; looks to be a version incompatibility. In fact, I've heard so often that I should be using the same versions together than I'm rather nonplussed that it's not possible to put the client and master into strict mode, such that they'll _only_ function together if they've got compatible versions. 

--
Brian L. Troutwine

[Brian Troutwine]

On Sun, Aug 21, 2011 at 8:31 AM, Nicolai <nicolai....@gmail.com> wrote:

To make sure you have a clean client and master knows nothing about it, (ssl-wise) do the following:

on master:

puppetca --clean client.example.com

on client:

rm -r /var/lib/puppet/ssl

puppetd --test

The puppetca and puppetd tools are deprecated, no? Also, my client machines do not have the puppet master program installed. I also do not believe it reasonable for me, the end user of puppet, to be forced to be so hands-on with puppet's ssl certificates. I am led to understand that some individuals run their own CA but, in my case, I let puppet generate everything and the puppet tooling _should_ be clever enough to manage its certificates but, given the volume of ssl questions and errors on this mailing list, is not.

 

back to master:

puppetca --list   (to check for the signing request from client)

puppetca --sign client.example.com

on client:

puppetd --test

and you shouldnt have any issues with ssl-connection. (if time/dns etc is correct).

Nicolai Mollerup

Thank you very much; very helpful.

 

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/Ssoxy6kT-f0J.

To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--
Brian L. Troutwine