Puppet 0.25 Client won't start "Retrieved certificate does not match private key"

435 views
Skip to first unread message

Douglas Garstang

unread,
Oct 13, 2009, 2:08:59 PM10/13/09
to puppet...@googlegroups.com
I've been tearing my hair out since 1am this morning trying to get the
puppet server and client to communicate.

The latest chapter in this epic saga has this coming up on the client
each time I run puppetd:

Could not prepare for execution: Retrieved certificate does not match
private key; please remove certificate from server and regenerate it
with the current key

I know it's not a client issue because I've re-imaged the client, and
used a default standard puppet.conf generated with --genconf.

On the server side, I've removed the puppetmaster rpm, cleared all the
directories, reinstalled the rpm and and regenerated a default
puppet.conf with puppetmasterd --genconf. What is quite disconcerting
is that puppet can't create it's own directories in a lot of cases...

/usr/lib/ruby/site_ruby/1.8/puppet/util/pidlock.rb:33:in `initialize':
Permission denied - /var/puppet/run/puppetmasterd.pid (Errno::EACCES)

... which leaves me wondering what else is screwed up. Yes, I am
running as root. Anyway, after manually creating /var/puppet/run and
chowning it to puppet, puppetmaster starts. I don't know where else to
look. As said, cleared all files on server, reinstalled, re-imaged
client. What am I missing? Puppet version is 0.25rc1.

Doug.

Joe McDonagh

unread,
Oct 13, 2009, 2:31:02 PM10/13/09
to puppet...@googlegroups.com
Depending on how you removed the RPM on the master, you may have SSL
certificates still hanging out under /var/lib/puppet/ssl. That's why the
certificate it serves doesn't match the new private key.

--
Joe McDonagh
Operations Engineer
www.colonfail.com

Douglas Garstang

unread,
Oct 13, 2009, 3:24:22 PM10/13/09
to puppet...@googlegroups.com
I removed /var/lib/puppet too.
--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.g...@gmail.com
Cell: +1-805-340-5627

Douglas Garstang

unread,
Oct 13, 2009, 3:33:21 PM10/13/09
to puppet...@googlegroups.com
Oh, I also found that I get this error on the client when the server
isn't even running.
Huh? I mean, I have a cleanly installed system, with a genconf
generated puppet.conf and it complains about server keys being wrong,
when the server isn't up. Something is seriously screwed here.

James Turnbull

unread,
Oct 13, 2009, 5:47:31 PM10/13/09
to puppet...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Douglas

Can you skip generating the puppet.conf with genconfig and just use
the RPM installed file?

Can you also show use a --trace --debug --verbose run from the
server and the client. I'd like to see the permissions error you
showed before and the current server key error. I've got a ticket
we couldn't reproduce that is similar at:

http://projects.reductivelabs.com/issues/2321

Does this seem like the same thing?

Thanks

James Turnbull

- --
Author of:
* Pro Linux Systems Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBStT1cyFa/lDkFHAyAQIAqAf/Sqd9bh3dTuI75xYBYMX4Z2CDnePILFfD
raayzAxWxs+Jse7urKhucpse1uTGnwgAnkcHKbLJnlSZ70YwrO1aVwgohFkeThDx
AvrsK5cySksPBiAGnvnQyjC5LJSztdFR+SJXOx36JlkBz/ee6RHuzMmt/lXnHnDD
dsXZ69c11UU1gAsWqUctwBElrxEe20GipcZOCYp2/oi2AEmrtbktd43CPBN2xz0t
BtAyd6yLDFDdl+Lh0h+EzOmjM688o2oCXGpVKDZBR4zlvuwUWe+s5XI+zczcuDgK
sEduhdUNYH1bjArNQOOr8htICeBG6htzomsKll72E2sAsKZ4TDrUPg==
=FkY5
-----END PGP SIGNATURE-----

Douglas Garstang

unread,
Oct 13, 2009, 6:59:30 PM10/13/09
to puppet...@googlegroups.com
James,

So, I just removed the puppet RPM on the client, blew away
/etc/puppet/ssl, reinstalled it, and ran the client with the default
puppet.conf. The problem did not occur. Thinking that it was the
genconf generated puppet.conf causing the problem, I removed that and
used the standard puppet.conf. The problem still did not occur. I know
I removed /etc/puppet/ssl once before, so I'm at a loss now.

To add to my frustration, I set autosign = true on the server and the
client is still complaining "notice: Did not receive certificate".
*sigh*

Doug.

So, it seems that when I use the default puppet.conf that comes out of
the RPM, this error does not occur.

James Turnbull

unread,
Oct 13, 2009, 7:25:33 PM10/13/09
to puppet...@googlegroups.com
2009/10/14 Douglas Garstang <doug.g...@gmail.com>:

> So, I just removed the puppet RPM on the client, blew away
> /etc/puppet/ssl, reinstalled it, and ran the client with the default
> puppet.conf. The problem did not occur. Thinking that it was the
> genconf generated puppet.conf causing the problem, I removed that and
> used the standard puppet.conf. The problem still did not occur. I know
> I removed /etc/puppet/ssl once before, so I'm at a loss now.

Can you please send me a link to the --genconfig'ed puppet.conf -
pastie or something like that. Maybe an issue there.

Also the puppet.conf bundled with the RPM is written by the downstream
package maintainers (Todd?) - perhaps there is an issue there also?
Can you pastie that also?

Thanks

James Turnbull

Douglas Garstang

unread,
Oct 13, 2009, 7:34:04 PM10/13/09
to puppet...@googlegroups.com
Malcom,

Client puppetd --genconf output is at http://pastebin.com/m35716797 .
Domain names replaced with 'xxx'.

The default puppet.com from the puppet 0.25rc1 RPM is at
http://pastebin.com/m2985dc06 .

Doug.

Todd Zullinger

unread,
Oct 13, 2009, 8:42:51 PM10/13/09
to puppet...@googlegroups.com
James Turnbull wrote:
> Also the puppet.conf bundled with the RPM is written by the
> downstream package maintainers (Todd?) - perhaps there is an issue
> there also? Can you pastie that also?

We ship the puppet.conf file from conf/redhat in the puppet tarball in
the Fedora/EPEL packages, as well as the packages I've put up on my
fedorapeople.org space. So the package should function as well (or as
unwell) as if it was installed from source. :)

It still couldn't hurt to see the puppet.conf to be sure it's alright.

I thought one of the issues reported with 0.25.x involved ssl certs
and setting up puppetmaster and clients from scratch with 0.25. But
that's just based on a hazy memory, so I could easily be way off.

--
Todd OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An idea is not responsible for the people who believe in it.
-- Anonymous

Matt

unread,
Oct 14, 2009, 6:19:33 AM10/14/09
to puppet...@googlegroups.com
FWIW -

I currently see this error on around 30% of our EC2 nodes since moving
to 0.25. We also use the rpm's provided for both client and master.
The fix is for us to log on to the EC2 node, remove the cert, run a
puppetca --clean on the master for the hostname, and then start puppet
again on the client.

I haven't had time to figure out was going on yet, but it feels like
the first poll is somehow generating a bad key.

2009/10/14 Todd Zullinger <t...@pobox.com>:

James Turnbull

unread,
Oct 14, 2009, 4:47:26 PM10/14/09
to puppet...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matt wrote:
> FWIW -
>
> I currently see this error on around 30% of our EC2 nodes since moving
> to 0.25. We also use the rpm's provided for both client and master.
> The fix is for us to log on to the EC2 node, remove the cert, run a
> puppetca --clean on the master for the hostname, and then start puppet
> again on the client.
>
> I haven't had time to figure out was going on yet, but it feels like
> the first poll is somehow generating a bad key.

Have you logged a ticket?

Could I ask you and/or Douglas to please log one with the client and
server logs showing the error (please run Puppet with --trace
- --verbose --debug).

http://projects.reductivelabs.com/projects/puppet/issues/new

Thanks

James Turnbull

- --


Author of:
* Pro Linux Systems Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBStY43iFa/lDkFHAyAQK4DwgAwXb4c2CEQG5iEkdFF+h7vOFK8cg9a0Rx
I1gJYuvCAX2D7ocSqL0keoHUG/3MVsRjICKalnuMg1yWzroGl7Wg66VH67TyawQi
eGGfWGT/6VLFmhsHL3prPc7prSq65yawOKfl2HvuIbmxHK4CR8h3pxVFJ6uDb2Hq
KzuRFSYuJfFCw/f1RduZDRLmPwUbA8xpyPiXfWgsVsL9NDap+5SHYM9x100y5Cs0
KsH5SGaVoOZCy5/1Pgi4SghT2QGUzm0/1ZZiJQJcqr3yT52H+QMS5aQR9EnFIBix
FmPjTMhnS3Ng+WZV+XwWCLFDtZuXB1EWARvJsNkMg9t7XXnpgg8WkA==
=YR9f
-----END PGP SIGNATURE-----

Matt

unread,
Oct 14, 2009, 5:09:07 PM10/14/09
to puppet...@googlegroups.com
2009/10/14 James Turnbull <ja...@lovedthanlost.net>:

>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Matt wrote:
>> FWIW -
>>
>> I currently see this error on around 30% of our EC2 nodes since moving
>> to 0.25.  We also use the rpm's provided for both client and master.
>> The fix is for us to log on to the EC2 node, remove the cert, run a
>> puppetca --clean on the master for the hostname, and then start puppet
>> again on the client.
>>
>> I haven't had time to figure out was going on yet, but it feels like
>> the first poll is somehow generating a bad key.
>
> Have you logged a ticket?
>
> Could I ask you and/or Douglas to please log one with the client and
> server logs showing the error (please run Puppet with --trace
> - --verbose --debug).
>
> http://projects.reductivelabs.com/projects/puppet/issues/new

I don't like logging tickets unless I can for sure say what's going
on. I'll log one tomorrow with the debug and trace info.

Matt

Silviu Paragina

unread,
Nov 27, 2009, 1:54:34 PM11/27/09
to puppet...@googlegroups.com
This issue seems to still manifest itself in 0.25.1. And I think I can
easly reproduce it. Shall I post the details (new ticket on redmine) or
it's already fixed? Redmine doesn't return anything if I search for
"Retrieved certificate does not match private key".
The reason for this is the way a client retrieves its signed certificate
from the server..


Silviu

PS sorry for replying to such an old post, but since it's a bug I think
it's excusable...
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en
> -~----------~----~----~----~------~----~------~--~---
>
>

Reply all
Reply to author
Forward
0 new messages