"The certificate retrieved from the master does not match the agent's private key." The master and agent in this case are on the same machine.

[Dan Pasacrita]

So for whatever reason I've been seeing this error. I'll usually see it when doing puppet commands like "puppet agent -t" or "puppet cert clean [Agent Hostname]".

Error: Could not request certificate: The certificate retrieved from the master does not match the agent's private key.

Certificate fingerprint: 7D:79:1E:C9:61:0B:96:18:DE:FE:9A:82:99:69:83:63:76:66:BA:5E:C7:8E:5F:BF:C5:37:4D:C1:07:36:4A:29

To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.

On the master:

  puppet cert clean [Master Hostname]

On the agent:

  rm -f /home/dpasacrita/.puppet/ssl/certs/[Master Hostname].pem

  puppet agent -t

Okay so I notice here that it tells me how to fix the issue, it normally wouldn't be that big an issue I bet. But here, as you can see, the Master's certificate doesn't match its own private key. I'm not sure what caused this, but I think it might have been me stupidly changing the mode of the Master's home folder to 777. I don't even remember why I did that, but I've since fixed that. I don't really know how to proceed here though, since I can't run puppet cert clean [Master Hostname] without getting this error, and I'm worried about removing the file it tells me to because I know I can't run puppet agent -t on the master right now and I'm worried I'll be worse off if I mess with it. What can I do here?

[Felix Frank]

Hi,

first off, because your post is really, really confusing: Is this the
agent running on the master machine? Or are other agents afflicted as well?

[Dan Pasacrita]

Hey, sorry, I kinda posted this in a hurry since I've been taking a ton of calls during, sorry about that. This error is occurring with every single agent apparently, including the agent on the master machine.

[Felix Frank]

Okay, thanks.

I don't really see how messing up permissions on the master would cause
such behavior.

Generally, the error means that your agent has lost its private key,
generated a new one, yet the master still retains a certificate it has
signed for use with an older private (agent) key.

To fix this, you would actually need to clean each last *agent*
certificate from the master, then sign new ones. Although why it would
suggest to clean the master's certificate is beyond me.

HTH,
Felix

[Dan Pasacrita]

Well my thinking was that since the certificate is stored in the home directory, messing with the home directory somehow changed the masters certificate, which is why none of the agent's keys match it. I really don't know though, I'm kinda new to puppet and linux. In any case, I can't clean the agent certificates from the master since I can't run the command to do so, unless there's a way to do it manually.

[Felix Frank]

What do you mean by "home directory"?

To clean an agent's certificate, use puppet cert clean, but with with
the agent's FQDN as its argument, not the master's FQDN.

[Dan Pasacrita]

By "home directory" I mean /home, where the directory /home/dpasacrita, and where (importantly) /home/dpasacrita/.puppet is. This is where the certificates are stored if I'm understanding this right, and this issue is apparently with the master's certificate.

I cannot run any puppet cert clean operation, it will give the same error as before.

$ puppet cert clean [Agent FQDN]

Error: The certificate retrieved from the master does not match the agent's private key.

Certificate fingerprint: 07:A8:41:FA:6D:00:3D:93:A3:74:CA:74:A3:6B:16:26:0A:A8:81:26:24:10:D7:D1:C4:70:60:AE:A5:68:D2:B0

To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.

On the master:

  puppet cert clean master.crownawards.com

On the agent:

  rm -f /home/dpasacrita/.puppet/ssl/certs/master.crownawards.com.pem

  puppet agent -t

[Felix Frank]

Ah, so you're running the master from an unprivileged user. That makes a
lot of sense now.

And yes, this is most disturbing.

Would it be feasible to renew all your certificates? Because the current
state looks messed up beyond reason :-)

Cheers,
Felix

[jcbollinger]

On Monday, May 5, 2014 10:25:50 AM UTC-5, Felix.Frank wrote:

Ah, so you're running the master from an unprivileged user. That makes a
lot of sense now.

Maybe.  It is normal to run the master as an unprivileged user.  The master does not store anything in its home directory in that case, unless its home directory is set to something like /var/lib/puppet.  Or am I stuck in the past?

It is abnormal to run the agent as an unprivileged user, because that prevents the agent from applying most of the kinds of system changes that Puppet is typically wanted for.  It is perhaps more common to run 'puppet apply' as a normal user, but the same limitations apply.  Under these circumstances, however, it is my understanding that agent or apply will use $HOME/.puppet.

 

And yes, this is most disturbing.

Would it be feasible to renew all your certificates? Because the current
state looks messed up beyond reason :-)

Again, maybe.  If the agents have in the past run successfully as root (as is more usual), then they will have certificates signed by the master and living in Puppet's normal system directory.  If one then tries to run the agent on one or more of those machines, however, then just such a problem as is reported might arise.

Specifically, when running as an ordinary user, the agent cannot use the certificate that puppet-as-root uses, because it cannot access the corresponding private key.  Instead, it creates its own key pair, and issues a certificate request to the master based on that public key.  The master, on the other hand, already has a signed certificate for the node, which it returns to the agent (this is a feature).  That certificate is indeed not based on the same key pair as the cert request, so the agent would emit exactly the message described.


John

[Felix Frank]

On 05/06/2014 04:56 PM, jcbollinger wrote:
> Ah, so you're running the master from an unprivileged user. That
> makes a
> lot of sense now.
>
>
>

> Maybe. It is normal to run the /master/ as an unprivileged user. The

> master does not store anything in its home directory in that case,
> unless its home directory is set to something like /var/lib/puppet. Or
> am I stuck in the past?

More or less. Hence my awkward wording of "running *from* an
unprivileged user". Sure, you normally want the master to drop root
privileges after initialization, and run as the puppet user.

Since Puppet 3 it is surpassingly easy to run independent master
processes with webrick, though, by just exclaiming `puppet master` from
a terminal, which will (just as the agent) use ~/.puppet as its base of
operation/configuration. (Of course, care must be taken if the master
port is already in use, but that should be it.)

> It is /ab/normal to run the agent as an unprivileged user, because that

> prevents the agent from applying most of the kinds of system changes
> that Puppet is typically wanted for. It is perhaps more common to run
> 'puppet apply' as a normal user, but the same limitations apply. Under
> these circumstances, however, it is my understanding that agent or apply
> will use $HOME/.puppet.

Right.

> Again, maybe. If the agents have in the past run successfully as root
> (as is more usual), then they will have certificates signed by the
> master and living in Puppet's normal system directory. If one then
> tries to run the agent on one or more of those machines, however, then
> just such a problem as is reported might arise.
>
> Specifically, when running as an ordinary user, the agent cannot use the
> certificate that puppet-as-root uses, because it cannot access the
> corresponding private key. Instead, it creates its own key pair, and
> issues a certificate request to the master based on that public key.
> The master, on the other hand, already has a signed certificate for the
> node, which it returns to the agent (this is a feature). That
> certificate is indeed not based on the same key pair as the cert
> request, so the agent would emit exactly the message described.

Very true. Specifically, I have found during last night's debugging
session that the $ssldir contents are very prone to fall off balance if
they are ever used by an agent that has the wrong permissions.

Sadly, bottom line likely remains: If an intact copy of the certs and
keys cannot be secured, it will be necessary to certify from scratch.

Regards,
Felix