single CA with multiple masters - agent getting: sslv3 alert certificate unknown

[Aaron Nicoli]

Hi all,

I have the following puppet layout:

```

Server A - Puppetserver (CA)

  Server B - Puppetserver

    Server C - Agent

```

With the agent (server C) having it's cert signed by the CA (server A) however pointed to (server B) as it's master.

The issue I'm having is that when running `puppet agent -t` on the agent, I can create a request to the CA and have the CA sign it, but then when I go to run again:

```

Error: Connection to https://hostname-of-server-B-puppetserver:8140/puppet/v3 failed, trying next route: Request to https://hostname-of-server-B-puppetserver:8140/puppet/v3 failed after 0.094 seconds: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown

```

Now on the agent, I can see that the Server-B cert is signed by the CA Server-A and it's available on the agent (server C) under: `/etc/puppetlabs/puppet/ssl/certs/ca.pem`

But yet - I still get the error... where is the puppet agent trying to find the `ca.pem` cert to verify the masters certificate?!?

Any ideas?

[warron.french]

This, if I remember correctly, looks like a certificate chain issue.  Your Puppet Architecture is a "Master of Masters" architecture.

Cert for Server B is signed by Cert for Server A?  Correct?

Is the cert for Server C (the agent) signed by the CA certificate chain?

Try executing:  puppet cert list and confirm that all certificates for all three servers are listed.  

--------------------------
Warron French

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/58b21386-50b8-42b2-b903-8db68933e491n%40googlegroups.com.

[Aaron Nicoli]

G'day Warron,

So, doing some ca/ssl info gathering (note puppet cert not being a thing anymore on 7x that I'm running):

On server-A (CA & master 1):
puppetserver ca list --all
server-A (alt names: DNS:puppet, DNS:server-A)
server-B (alt names: DNS:server-B)
server-C (alt names: DNS:server-C)

puppet ssl show
```
...
Issuer: CN=Puppet CA: server-A
...
Subject: CN=server-A
...
```

On server-B (master 2):
puppetserver ca list --all
```

   Error: Failed connecting to https://server-A:8140/puppet-ca/v1/certificate_statuses/any_key

  Root cause: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get certificate CRL)
```

puppet ssl show
```
Error: Could not run: The CRL issued by 'CN=Puppet CA: server-A' is missing
```

I also note that today, puppet agent -t, is now also failing with:
```

Error: The CRL issued by 'CN=Puppet CA: server-A' is missing

Error: Could not run: The CRL issued by 'CN=Puppet CA: server-A' is missing

```

On server-C (agent):

puppet ssl show:
```
...
Issuer: CN=Puppet CA:server-A
...
Subject: CN=server-B
...
```

Hope this helps my case!

[warron.french]

@Aaron, good evening from the east coast of the USA.

It looks like the first break in the chain is on ServerB, if that was not also obvious to you.  To be clear, I have not worked with puppet since version 4, and in my current professional role we don't use Puppet at all (makes me sad actually).

Anyway, I think you need to find The certificate for ServerB and confirm the details about it's PEM file.

Use openssl x509  -noout  -text     ServerB.pem    (the server's PEM file is a placeholder)

Can you provide a list of fullpath/file.pem's back to me?   This is not a production system correct?

--------------------------
Warron French

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/803a0f0a-01c9-4001-9bb6-c8d8a26c783an%40googlegroups.com.

[Aaron Nicoli]

Cheers Warron,

It was pretty obvious something on B is broken, but what who knows...

I've just deleted everything related to B and C (A is prod, B and C are new dev) [rm -rf /etc/puppetlabs/puppet/ssl; rm -rf /etc/puppetlabs/puppetserver/ca; puppetserver ca clean x].

After doing this, will both puppet and puppetserver services stopped on B, I've ran puppet agent -t...
Then signed the request on server-A

Everything working on B (as an agent)...

Then again everything stopped and deleted on C (agent)...
puppet agent -t, signed on A...

Can't connect to the puppetmaster (server-B), because I haven't started the service, but other than that everything perfect.

Start the puppetserver service on server-B... and then run puppet agent -t on server-C again...
```

Error: Connection to https://server-B:8140/puppet/v3 failed, trying next route: Request to https://server-B:8140/puppet/v3 failed after 0.103 seconds: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown

Wrapped exception:

SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown

```

Pretty much back to where we started...

Then I go back to server-B and run puppet agent -t... broken:

```

Error: The CRL issued by 'CN=Puppet CA: server-A' is missing

Error: Could not run: The CRL issued by 'CN=Puppet CA: server-A' is missing

```

From what I can tell, the puppetserver service (upon startup on server-B) is "blatting" the correct certs on server-B...

That's the best conclusion I can come to at this point.

[Aaron Nicoli]

So, I'm thinking this is the issue I'm running into, but still not sure how to resolve it:

https://www.atcomputing.nl/blog/certificate-authority-and-puppet-6/

Need to run puppetserver ca import - before starting the puppetserver.

However, it seems `--private-key` `--crl-chain` and `--cert-bundle` are mandatory.

What should I use for what?
```

/etc/puppetlabs/puppet/ssl/certs/server-b.pem

/etc/puppetlabs/puppet/ssl/certs/ca.pem

/etc/puppetlabs/puppet/ssl/private_keys/server-b.pem

/etc/puppetlabs/puppet/ssl/crl.pem

```

I've tried:
```
puppetserver ca import --config /etc/puppetlabs/puppet/puppet.conf --private-key /etc/puppetlabs/puppet/ssl/private_keys/server-b.pem --crl-chain /etc/puppetlabs/puppet/ssl/crl.pem --cert-bundle /etc/puppetlabs/puppet/ssl/certs/ca.pem
```

But, it gives me the error of:
```

Error:

Could not find certificate matching private key

```

[Aaron Nicoli]

OMG I got it working...

I don't even really know how yet, but I'll reverse engineer what I did and advise back here.

[Aaron Nicoli]

So, this is the best I can do:

Ok, here we go, how to configure an "in the middle" puppetmaster.

Eg.

```

Puppetmaster (CA) - server-A

  Puppetmaster & Agent (to server-A) - server B

    Agent (to server-B) - server-C

```

On server-B:

`service puppet stop`

`service puppetserver stop`

`vi /etc/puppetlabs/puppet/ssl/crls.pem` - new file:

```

-----BEGIN X509 CRL-----

Content from /etc/puppetlabs/puppet/ssl/crl.pem - on server-b

-----END X509 CRL-----

-----BEGIN X509 CRL-----

Content from /etc/puppetlabs/puppetserver/ca/ca_crl.pem - on server-a

-----END X509 CRL-----

```

`vi /etc/puppetlabs/puppet/ssl/certs/ca_bundle.pem` - new file:

```

-----BEGIN CERTIFICATE-----

Content from /etc/puppetlabs/puppet/ssl/certs/ca.pem - on server-b

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

Content from /etc/puppetlabs/puppetserver/ca/ca_crt.pem - on server-a

-----END CERTIFICATE-----

-----BEGIN PUBLIC KEY-----

Content from /etc/puppetlabs/puppetserver/ca/ca_pub.pem - on server-a

-----END PUBLIC KEY-----

```

`cp /etc/puppetlabs/puppet/ssl/certs/server-b.pem /etc/puppetlabs/puppet/ssl/public_keys/server-b.pem`

```

Note. not sure if that last cp is right, and not sure if you also need to copy:

/etc/puppetlabs/puppetserver/ca/ca_key.pem - from server-a

to

/etc/puppetlabs/puppet/ssl/private_keys/server-a.pem - on server-b

```

Then run:

`mv /etc/puppetlabs/puppet/ssl/certs/ca.pem /etc/puppetlabs/puppet/ssl/certs/ca.pem.bak`

`mv /etc/puppetlabs/puppet/ssl/crl.pem /etc/puppetlabs/puppet/ssl/crl.pem.bak`

`mv /etc/puppetlabs/puppet/ssl/certs/server-b.pem /etc/puppetlabs/puppet/ssl/certs/server-b.pem.bak`

And then the import:

```

puppetserver ca import --config /etc/puppetlabs/puppet/puppet.conf --private-key /etc/puppetlabs/puppet/ssl/private_keys/server-b.pem --crl-chain /etc/puppetlabs/puppet/ssl/crls.pem --cert-bundle /etc/puppetlabs/puppet/ssl/certs/ca_bundle.pem

```

Then copy back the originals:

`mv /etc/puppetlabs/puppet/ssl/certs/ca.pem.bak /etc/puppetlabs/puppet/ssl/certs/ca.pem`

`mv /etc/puppetlabs/puppet/ssl/crl.pem.bak /etc/puppetlabs/puppet/ssl/crl.pem`

`mv /etc/puppetlabs/puppet/ssl/certs/server-b.pem.bak /etc/puppetlabs/puppet/ssl/certs/server-b.pem`

You should then be able to successfully run:

`puppet agent -t`

You can then start the puppetserver:

`service puppetserver start`

And should be able to again run:

`puppet agent -t`

You "sub"-agents (eg. server-c) should now "just work" - provided they are signed against the ca of server-a...

Regardless, just run from server-c:

`puppet agent -t`