PUPPET 6.0 : CSR from master does not match the agent public key

[Andy Hall]

Just deployed a new puppet 6.0 client / server setup and getting the classic CSR signing issue (see details below). Please help clarify my understanding so I can troubleshoot this (I'm sure there's a quick fix for this) :

N.B. The usual "remove the SSL dir on the client and clean the cert on the server" is NOT working.

So I think this is what happens :

1. The agent creates an SSL cert and sends if to the master to be signed - a Certificate Signing Request (CSR).

2. The master signs the cert with its own CA and the key of the agent.

3. The signed cert is returned to the agent which compares the keys to ensure they match.

It would seem that somehow the key being returned is mangled and doesn't match so is rejected by the agent.

This happens from the very first attempt to join an agent to the master and I am at a loss of how to fix this.

Here's is the request from the agent to the master :

==> /var/log/puppetlabs/puppetserver/puppetserver-access.log <==
10.2.73.60 - - [28/Sep/2018:18:34:07 +0100] "GET /puppet-ca/v1/certificate/andy-puppet6-test.london.company.com HTTP/1.1" 404 65 "-" "Puppet/6.0.0 Ruby/2.5.1-p57 (x86_64-linux)" 3
10.2.73.60 - - [28/Sep/2018:18:34:07 +0100] "GET /puppet-ca/v1/certificate_request/andy-puppet6-test.london.company.com HTTP/1.1" 200 1622 "-" "Puppet/6.0.0 Ruby/2.5.1-p57 (x86_64-linux)" 3

And here is the output from the agent :

# puppet agent --test --noop
Info: Creating a new SSL key for andy-puppet6-test.london.company.com
Info: Downloaded certificate for ca from puppet
Error: Could not request certificate: The CSR retrieved from the master does not match the agent's public key.
CSR fingerprint: 9A:16:DA:95:9C:FB:90:89:78:EB:01:86:21:B0:24:E1:B0:66:80:43:ED:58:0B:A5:08:9C:24:60:C8:DE:F5:13
CSR public key: Public-Key: (4096 bit)
Modulus:
    00:9c:ba:32:5e:c9:e9:72:7b:36:17:9a:aa:f6:8e:
    e2:a4:73:0a:95:4d:ae:ca:81:96:1c:02:f3:45:e5:
    6e:13:70:e1:dc:83:dc:88:96:4c:5e:40:d1:eb:c4:
    62:81:8b:9f:25:96:1a:56:1d:ba:cd:25:a8:b2:21:
    72:e6:ef:f3:63:b1:02:65:19:4d:e8:28:9e:bf:40:
    04:c7:77:21:2f:5c:d8:20:07:63:60:c9:ac:75:44:
    34:d0:bd:cf:8c:ae:31:37:8a:16:f3:08:92:a4:c1:
    66:54:53:03:be:b4:02:17:52:93:c2:eb:42:82:90:
    5d:db:b6:92:b1:ae:21:f0:e0:a6:9e:04:4e:0f:eb:
    39:2f:17:f6:89:41:3a:08:b0:13:18:ff:82:2e:20:
    cc:83:d6:67:f6:24:97:a2:8b:72:6d:c6:9c:99:cb:
    70:9d:2b:7b:bd:0a:21:0d:9d:51:7c:22:f8:d0:e3:
    cc:f7:2a:d9:e0:09:8c:1b:f5:7a:6c:69:88:5b:d2:
    32:c2:c5:d7:b3:1d:c0:8f:23:a9:50:ab:1e:9b:4a:
    cf:1e:f7:b3:de:7e:b6:b7:1e:ce:63:fd:ee:10:55:
    48:32:8c:46:65:c2:46:43:90:49:2a:d8:b0:02:96:
    19:71:e8:25:18:5f:c6:8f:79:67:36:da:03:04:83:
    e1:06:6b:29:43:51:76:52:05:c9:22:d0:39:94:0b:
    3b:07:62:66:79:d4:5a:36:af:c4:a3:2f:e1:f9:7b:
    60:1b:55:33:31:52:87:87:53:41:85:86:58:64:ef:
    32:77:8e:33:8c:8d:b3:f5:82:e2:16:a4:6c:65:f0:
    f0:10:71:98:f5:da:ae:c0:df:5f:fa:8a:58:8f:7d:
    69:4f:ea:8f:c7:36:22:f2:9f:85:30:c5:49:c6:ab:
    f4:63:16:bd:ba:5d:a2:c1:06:8a:f9:6a:9b:bc:6a:
    ee:01:2b:d2:75:cd:91:ad:a7:d1:45:e8:b6:a7:45:
    51:0b:20:3b:05:c6:0d:06:17:2d:44:a9:33:2e:51:
    b8:0b:ce:d4:db:f2:33:b9:42:3d:2b:22:1a:1e:f8:
    09:14:43:9e:f0:82:8f:c8:71:74:8d:b2:ee:37:52:
    0b:af:5c:4d:94:48:b2:94:81:32:03:fc:b5:6a:a6:
    f2:c5:59:3c:09:44:f3:57:2f:3e:11:3b:6e:6f:36:
    af:66:a6:10:e0:c7:4f:6a:74:5a:aa:48:51:62:e9:
    cd:1d:72:43:20:7a:8b:80:c9:0f:1c:14:a8:87:15:
    ee:93:95:55:9e:ae:48:4c:e0:4b:63:0b:88:00:fd:
    1f:f1:30:a7:8b:d2:42:6a:1b:89:74:eb:46:67:c8:
    32:d9:e1
Exponent: 65537 (0x10001)

Agent public key: Public-Key: (4096 bit)
Modulus:
    00:cd:0a:ab:52:c8:34:62:3c:86:49:f5:18:7c:3c:
    96:90:3d:0b:53:f9:5c:48:a6:38:e4:2c:84:4a:af:
    5a:b7:1f:93:a7:4c:e5:dd:f3:a2:52:9d:b2:39:f4:
    d3:2b:f0:8a:06:fd:f2:52:40:ec:9f:42:ed:b6:89:
    63:b0:ed:62:cf:77:91:87:27:e1:f9:0b:a5:b8:d1:
    a6:96:96:24:db:43:9f:5b:bd:8f:d5:29:d8:2b:f1:
    57:2a:46:93:ce:cc:12:d4:e9:0d:24:fc:ef:42:11:
    b8:db:a2:a3:51:23:bb:d4:97:18:a1:50:7a:7f:27:
    70:cb:95:24:3c:31:35:90:77:35:68:eb:4c:41:0b:
    1b:b3:1e:7b:2c:86:fa:72:27:3d:27:4c:71:07:13:
    6d:58:ed:95:04:69:15:4c:5b:f2:7e:8e:73:21:65:
    6e:eb:f1:64:ab:bc:67:55:1b:32:b9:1c:2c:c2:71:
    9f:06:fa:a2:61:b7:03:ec:69:f7:9b:64:21:d1:af:
    8a:ea:7b:99:48:7f:a0:27:f3:93:20:54:24:db:26:
    b0:e7:38:24:fe:52:71:3c:79:f7:62:cf:97:e1:56:
    16:35:90:2d:9e:69:c0:b7:ca:31:45:64:d7:44:16:
    8c:1c:c2:a8:11:34:a4:ce:1e:37:61:c7:bb:94:16:
    b1:e5:d7:74:70:67:56:e8:20:59:a5:12:39:01:95:
    c2:ca:09:59:0d:a3:58:0a:1a:83:27:80:55:46:26:
    46:9b:9d:69:57:42:97:b1:7d:cb:1e:a7:65:99:47:
    f4:e8:ae:72:0b:a4:10:32:68:46:8b:77:19:6a:7a:
    fa:32:3c:f8:2d:ff:cf:55:c3:43:64:3f:56:eb:e2:
    8f:be:2d:d3:ec:55:d9:df:a4:c0:f4:ca:f7:44:38:
    71:3e:1f:29:c9:b1:dc:bb:04:a1:90:ab:d9:ce:2f:
    8b:77:87:ef:fa:47:c4:8c:ce:46:60:53:5c:d2:8f:
    7f:4a:ad:ec:54:10:49:18:0f:7e:10:a9:c9:a9:5e:
    8a:ce:2e:9d:55:19:95:fc:15:f2:35:1e:c0:81:f2:
    03:39:4a:11:2c:ab:ba:0e:da:d8:eb:e7:6c:dd:17:
    33:7b:16:c1:3b:ea:99:0f:0d:10:d2:94:0c:ee:0e:
    cb:4f:91:2c:1a:a7:31:c5:23:f2:3d:13:45:a5:ba:
    66:b0:76:58:8e:4a:6d:18:66:5f:4d:d7:6d:30:5d:
    39:ef:5d:6f:1f:ab:61:68:3b:9a:80:3c:2b:8d:0f:
    84:1d:3b:4e:2c:d4:48:0f:52:c5:13:12:69:ad:0f:
    cc:78:6b:01:8e:27:10:29:81:3d:a0:5e:ec:d9:d8:
    32:f0:cd
Exponent: 65537 (0x10001)

To fix this, remove the CSR from both the master and the agent and then start a puppet run, which will automatically regenerate a CSR.
On the master:
  puppet cert clean andy-puppet6-test.london.company.com
On the agent:
  1a. On most platforms: find /etc/puppetlabs/puppet/ssl -name andy-puppet6-test.london.company.com.pem -delete
  1b. On Windows: del "\etc\puppetlabs\puppet\ssl\certs\andy-puppet6-test.london.company.com.pem" /f
  2. puppet agent -t

So the big question is this : what exactly is the CSR public key and what exactly is the agent public key and why should they match ?

Any help would be most greatly appreciated !

Thanks very much.

[Maggie Dreyer]

Hello!

For cleaning the cert on the master, are you trying to use `puppet cert clean`? This error message needs to be updated to instead say "On the master: use `puppetserver ca clean --certname <agent-cert-name>`". The `puppet cert` command was removed in 6.0.0, see https://puppet.com/docs/puppet/6.0/release_notes.html#puppet-600 and https://puppet.com/docs/puppetserver/6.0/subcommands.html#ca. But due to https://tickets.puppetlabs.com/browse/PUP-9155, it doesn't always correctly report its failure. Starting in 6.0.1, `puppet cert` will always error helpfully information about the new alternative commands. Please let me know if you are still having issues after trying `puppetserver ca clean`.

And regardless of whether this fixes your issue, we really appreciate your letting us know when our errors and/or docs are less than helpful. Thanks!

Maggie

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7189ba10-f48c-46b6-8670-65861f4d9e3f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andy Hall]

Hi Maggie - thanks for the reply.

When I run the new command this is what we get :

# puppetserver ca clean --certname andy-puppet6-test.london.company.com
Error:
    When attempting to revoke certificate 'andy-puppet6-test.london.company.com', received:
      code: 403
      body: Forbidden request: /puppet-ca/v1/certificate_status/andy-puppet6-test.london.company.com (method :put). Please see the server logs for details.

And here are the server logs :

==> /var/log/puppetlabs/puppetserver/puppetserver.log <==
2018-10-01T17:48:10.393+01:00 ERROR [qtp1126042654-72] [p.t.a.rules] Forbidden request: ldn1-puppet5.london.company.com(10.2.69.190) access to /puppet-ca/v1/certificate_status/andy-puppet6-test.london.company.com (method :put) (authenticated: true) denied by rule 'puppetlabs cert status'.

==> /var/log/puppetlabs/puppetserver/puppetserver-access.log <==
10.2.69.190 - - [01/Oct/2018:17:48:10 +0100] "PUT /puppet-ca/v1/certificate_status/andy-puppet6-test.london.company.com HTTP/1.1" 403 145 "-" "PuppetserverCaCli" 6

Any thoughts on why the perms seem broke for this ?

Big fan of puppet and we really want to migrate away from 3.x so hopefully we can get this working !

Thanks, Andy.

[Maggie Dreyer]

The new `puppetserver ca` tool makes requests to the CA API (specifically the `certificate_status(es)` endpoints), using the master's host cert for authorization. The master's cert is created with a special extension authorizing it to talk to those endpoints, allowed via a rule in `auth.conf`. A permissions issue would be expected in an upgrade scenario, when the CA files and master cert were created by an old Puppet Server, that did not add the new auth extension to the cert. However, I think you said this was a new install?

Some things to check:

• That these two rules exist in your `auth.conf` file (located by default at `/etc/puppetlabs/puppetserver/conf.d/auth.conf`): https://github.com/puppetlabs/puppetserver/blob/master/ezbake/config/conf.d/auth.conf#L49-L77
• If they don't exist, you can add them.
  
• That your master cert has the required extension. To check this, run `openssl x509 -noout -text -in /etc/puppetlabs/puppet/ssl/certs/<master_name>.pem`. In the output, look for these two lines:

1.3.6.1.4.1.34380.1.3.39:
                ..true 

If the cert doesn't have the extension, your best bet may be to

1) stop puppetserver

2) delete `/etc/puppetlabs/puppet/ssl/*` on your master (assuming this is still a test setup and no agents are relying on the existing CA setup)

3) run `puppetserver ca setup`. This will generate a brand new CA with all the certs in the expected state for a Puppet 6 Server.

4) restart puppetserver

From here, the command should be usable, assuming your `auth.conf` is correct. You will have to also delete the SSL files on your agent node(s) in order to get it to talk to the new CA that you just regenerated.

All of this should have been correct out of the box, but if you've been doing anything odd in your SSL directory trying to get this to work, (doing an agent run on the master after deleting some certs comes to mind) there's always a chance something got messed up along the way.

Please let me know if none of this fixes it for you. I have filed PUP-9187 to fix up the error messaging that initially led you in the wrong direction.

Thanks,

Maggie

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/9de260a1-cbea-469c-b859-ec9af2f1ada7%40googlegroups.com.

[Andy Hall]

Again great response thankyou for this. It is actually an upgrade from 5.x but I will try out the proposed solutions and let you know how I get on. Andy.

[Martin Alfke]

Hi Maggie,

> On 1. Oct 2018, at 19:24, Maggie Dreyer <mag...@puppet.com> wrote:

ack. we already saw this when using puppet 5.5

>
> The new `puppetserver ca` tool makes requests to the CA API (specifically the `certificate_status(es)` endpoints), using the master's host cert for authorization. The master's cert is created with a special extension authorizing it to talk to those endpoints, allowed via a rule in `auth.conf`. A permissions issue would be expected in an upgrade scenario, when the CA files and master cert were created by an old Puppet Server, that did not add the new auth extension to the cert. However, I think you said this was a new install?
>
> Some things to check:
> • That these two rules exist in your `auth.conf` file (located by default at `/etc/puppetlabs/puppetserver/conf.d/auth.conf`): https://github.com/puppetlabs/puppetserver/blob/master/ezbake/config/conf.d/auth.conf#L49-L77
> • If they don't exist, you can add them.
> • That your master cert has the required extension. To check this, run `openssl x509 -noout -text -in /etc/puppetlabs/puppet/ssl/certs/<master_name>.pem`. In the output, look for these two lines:
> 1.3.6.1.4.1.34380.1.3.39:
> ..true
>
> If the cert doesn't have the extension, your best bet may be to
> 1) stop puppetserver
> 2) delete `/etc/puppetlabs/puppet/ssl/*` on your master (assuming this is still a test setup and no agents are relying on the existing CA setup)
> 3) run `puppetserver ca setup`. This will generate a brand new CA with all the certs in the expected state for a Puppet 6 Server.
> 4) restart puppetserver
> From here, the command should be usable, assuming your `auth.conf` is correct. You will have to also delete the SSL files on your agent node(s) in order to get it to talk to the new CA that you just regenerated.
>

What is the proposed way if you run an update on an existing Puppet Master, switching from 5.5 to 6.0?
Should we really delete all signed certificates and the ca?

Best,
Martin

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAMstjg0JV8yWm8Btpxrdnp-PjP3r-hHqv6YROvHi-QddnY5OUw%40mail.gmail.com.

[Andy Hall]

Thanks Maggie these instructions were perfect. The cert didn't have the extension you refer to so i recreated the ca setup on the master and then tried again with good results this time :

AGENT:

# puppet agent --test --noop
Info: Creating a new SSL key for andy-puppet6-test.london.company.com

Info: Downloaded certificate for ca from ldn1-puppet5.london.company.com
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for andy-puppet6-test.london.company.com
Info: Certificate Request fingerprint (SHA256): 28:81:65:77:76:3A:7C:53:D7:D6:63:0E:27:0B:8C:74:DF:10:7C:53:99:3B:7D:42:EC:E1:48:FC:9B:91:70:A7
Exiting; no certificate found and waitforcert is disabled

MASTER:

# puppetserver ca sign --certname andy-puppet6-test.london.company.com
Successfully signed certificate request for andy-puppet6-test.london.flextrade.com

Thanks again for you help - we can begin the migration of agents to 6.x now :-)

On Monday, 1 October 2018 18:24:50 UTC+1, Maggie Dreyer wrote: