Can puppetdb use an SSL cert signed by CA that differs from the puppetmasters?

[Hans Lellelid]

Well, the gist of the question is in the title.  Basically we want to use our exiting browser SSL CA infrastructure for the puppetdb box.  We have many puppetmasters which each have their own CAs and our intent is to have a single puppetdb box that all of these talk to.  Currently we are getting "SSL_connect returned1 .... certificate verify failed" errors.  I would like to simply add the CA to some sort of "trust store" for the puppetmasters so that we can talk to a puppetdb that is signed by a CA that differs from the CA puppetmaster is using to sign node certs.

Hope this makes sense!  I see a few different ca.pem files, but am unsure if I can just start concatenating stuff onto these and whether that will break puppetmaster's signing, etc.

Thanks in advance!

Hans

[Ken Barber]

> Well, the gist of the question is in the title. Basically we want to use
> our exiting browser SSL CA infrastructure for the puppetdb box. We have
> many puppetmasters which each have their own CAs and our intent is to have a
> single puppetdb box that all of these talk to. Currently we are getting
> "SSL_connect returned1 .... certificate verify failed" errors. I would like
> to simply add the CA to some sort of "trust store" for the puppetmasters so
> that we can talk to a puppetdb that is signed by a CA that differs from the
> CA puppetmaster is using to sign node certs.

So we have recently implemented the pem based storage options, but in
the passed we pushed people to use the truststore configuration
instead:

http://docs.puppetlabs.com/puppetdb/1.5/configure.html#truststore

This forced a user to create a traditional JKS store and put their CA
certificates in that. I haven't tested it, but this probably accepts
multiple CA certificates. Have you tried this yet? I would need to run
up a test myself to do this, if this sounds sort of like what you are
after happy to help. I'm not positive it works - but this feels like
the way to do it for now.

If it doesn't work, we can work towards solving it properly, but going
forward, the ability to specify multiple ca certificates for the
ssl-ca-cert setting sounds like another viable option:
http://docs.puppetlabs.com/puppetdb/1.5/configure.html#ssl-ca-cert

> Hope this makes sense! I see a few different ca.pem files, but am unsure if
> I can just start concatenating stuff onto these and whether that will break
> puppetmaster's signing, etc.

Hmm. Not sure either, probably won't work.

ken.

[Hans Lellelid]

Thanks for the response! (inline)

<snip>

So we have recently implemented the pem based storage options, but in
the passed we pushed people to use the truststore configuration
instead:

http://docs.puppetlabs.com/puppetdb/1.5/configure.html#truststore

This forced a user to create a traditional JKS store and put their CA
certificates in that. I haven't tested it, but this probably accepts
multiple CA certificates. Have you tried this yet? I would need to run
up a test myself to do this, if this sounds sort of like what you are
after happy to help. I'm not positive it works - but this feels like
the way to do it for now.

If it doesn't work, we can work towards solving it properly, but going
forward, the ability to specify multiple ca certificates for the
ssl-ca-cert setting sounds like another viable option:
http://docs.puppetlabs.com/puppetdb/1.5/configure.html#ssl-ca-cert

So, I didn't have any problems setting up the puppetdb to use the CA we want to use (and we have puppet defines that take care of requesting the certs from certmaster, installing them, etc.).  I'm actually using puppetdb behind apache (reverse proxy to non-ssl port) since I prefer not to mess with keystores when I can help it, but I am pretty confident we'd have no problem using the JKS approach either if we were serving the SSL directly from Jetty.  (We have lots of puppet manifest to help with managing JKS stores for other Java apps we run.)

My problem is with puppetmaster trying to talk to puppetdb.  The cert verify is failed.  Trying to grep through puppet code, it looks like it adds the $ssldir/certs/ca.pem file as the [only] valid authority for the https connections.  I tried pasting more certs into that file -- looking at Net::HTTP ruby module for reference it says ca_file can include multiple certs.  But still getting the permission denied error.  Maybe it is using the /var/lib/puppet/ssl/certs/ca.pem instead of /var/lib/puppetmaster/ssl/certs/ca.pem?  (This is a puppet-managed puppetmaster so we have two trees.)  Anyway, I will continue poking at it.  I have no idea if changing the contents of certs/ca.pem is going to break signing.

 

> Hope this makes sense!  I see a few different ca.pem files, but am unsure if
> I can just start concatenating stuff onto these and whether that will break
> puppetmaster's signing, etc.

Hmm. Not sure either, probably won't work.

Yeah, we'll see.  This doesn't look promising.

It would be really nice if there was a way to say "hey, puppet[master] for your outbound https connections, here's a ca-bundle.pem you can use to verify servers".  The idea that we'd use the puppetmaster CA *seems* wrong here.  Especially in an environment like ours where we have many (dozens) of puppetmasters that each manage their own little ecosystems.  (And we don't really want to force a single CA.)

Maybe I've misunderstood something, though.

Thanks again!

Hans

[Ken Barber]

So I think I understand, what you are saying is that the client
refuses to accept the PuppetDB's server certificate, because its not
signed by its own CA? This is not a client certificate thing, my point
is - purely server side certs. Right? Because your PuppetDB's server
cert is signed with a more global different CA, it doesn't work yeah?

> Yeah, we'll see. This doesn't look promising.
>
> It would be really nice if there was a way to say "hey, puppet[master] for
> your outbound https connections, here's a ca-bundle.pem you can use to
> verify servers". The idea that we'd use the puppetmaster CA *seems* wrong
> here. Especially in an environment like ours where we have many (dozens) of
> puppetmasters that each manage their own little ecosystems. (And we don't
> really want to force a single CA.)

Yeah, its wrong in your multi-CA scenario for sure. We totally just
use the HTTP libraries Puppet provides us and all its cert
assumptions, for simplicity I guess. At least, it works for single-CA
scenarios just fine.

We could look into providing a feature for this probably. The change
would probably need to occur in the puppetdb-terminus code and an
option added to /etc/puppet/puppetdb.conf most probably.

Although if Puppet itself supported multiple CA bundles, this wouldn't
happen - I think the fix is more elegant in Puppet as all
clients/termini/plugins using their HTTP libraries would just work for
multi-ca without each one having to do its own thing.

ken.

[Ken Barber]

> Although if Puppet itself supported multiple CA bundles, this wouldn't
> happen - I think the fix is more elegant in Puppet as all
> clients/termini/plugins using their HTTP libraries would just work for
> multi-ca without each one having to do its own thing.

Specifically, if this supported a colon separated list of CA's for
example, or we added a cacert_secondaries option for CA's the
puppetmaster doesn't manage:

http://docs.puppetlabs.com/references/latest/configuration.html#cacert

Probably needs a bit more architecture and design though, but
something like this anyway :-).

ken.

[Hans Lellelid]

<snip>

So I think I understand, what you are saying is that the client
refuses to accept the PuppetDB's server certificate, because its not
signed by its own CA? This is not a client certificate thing, my point
is - purely server side certs. Right? Because your PuppetDB's server
cert is signed with a more global different CA, it doesn't work yeah?

That's exactly right, yeah.  No client certs, just an issue with different CA being used for the puppetdb server cert.  (I probably could have put this more concisely from the beginning!)

 

> It would be really nice if there was a way to say "hey, puppet[master] for
> your outbound https connections, here's a ca-bundle.pem you can use to
> verify servers".  The idea that we'd use the puppetmaster CA *seems* wrong
> here.  Especially in an environment like ours where we have many (dozens) of
> puppetmasters that each manage their own little ecosystems.  (And we don't
> really want to force a single CA.)

Yeah, its wrong in your multi-CA scenario for sure. We totally just
use the HTTP libraries Puppet provides us and all its cert
assumptions, for simplicity I guess. At least, it works for single-CA
scenarios just fine.

We could look into providing a feature for this probably. The change
would probably need to occur in the puppetdb-terminus code and an
option added to /etc/puppet/puppetdb.conf most probably.

Although if Puppet itself supported multiple CA bundles, this wouldn't
happen - I think the fix is more elegant in Puppet as all
clients/termini/plugins using their HTTP libraries would just work for
multi-ca without each one having to do its own thing.

What you're saying makes perfect sense -- regarding this being something that the puppetdb-terminus stuff does and making it an option in puppetdb.conf, etc. 

Yeah, I'm looking through the code (doesn't help that my knowledge of Ruby is very limited) and I see that the http_pool.rb configures the ssl stuff setting ca_cert = Puppet[:localcacert].  That defaults to $certdir/certs/ca.pem.  I've tried also explicitly specifying it in the config file, but to no avail. I still get the verify failure, although doing this manually in irb is working:

require 'net/https'

http = Net::HTTP.new('puppetdb.<domain>', 443)
http.use_ssl = true

http.ca_file = '/var/lib/puppetmaster/ssl/certs/ca.pem'

# (I concatenated the global CA onto the end of ca.pem)
http.verify_mode = OpenSSL::SSL::VERIFY_PEER

http.send('get', '/')

#<Net::HTTPFound 302 Found readbody=true>

I validated that pointing at the original ca.pem file fails as expected.  So I'm a little confused as to why puppetmaster isn't succeeding after I modified that file.  Perhaps there's some additional verification happening.  I'll probably give up soon :)

Thanks again-

Hans

[Ken Barber]

> What you're saying makes perfect sense -- regarding this being something
> that the puppetdb-terminus stuff does and making it an option in
> puppetdb.conf, etc.
>
> Yeah, I'm looking through the code (doesn't help that my knowledge of Ruby
> is very limited) and I see that the http_pool.rb configures the ssl stuff
> setting ca_cert = Puppet[:localcacert]. That defaults to
> $certdir/certs/ca.pem. I've tried also explicitly specifying it in the
> config file, but to no avail. I still get the verify failure, although doing
> this manually in irb is working:
>
> require 'net/https'
> http = Net::HTTP.new('puppetdb.<domain>', 443)
> http.use_ssl = true
> http.ca_file = '/var/lib/puppetmaster/ssl/certs/ca.pem'
> # (I concatenated the global CA onto the end of ca.pem)
> http.verify_mode = OpenSSL::SSL::VERIFY_PEER
> http.send('get', '/')
> #<Net::HTTPFound 302 Found readbody=true>
>
> I validated that pointing at the original ca.pem file fails as expected. So
> I'm a little confused as to why puppetmaster isn't succeeding after I
> modified that file. Perhaps there's some additional verification happening.
> I'll probably give up soon :)

Can you capture your feature requirements in redmine for us?

http://projects.puppetlabs.com/projects/puppetdb

ken.

[Hans Lellelid]

Yeah, I see a difference here between the CA that puppet uses to sign things (which I believe that cacert config is referencing?) vs. the certificate authorities that puppet users to validate SSL connections to peers.  Obviously in the case where a puppet node is just talking to a puppetmaster, we would expect these CAs to be the same.  And if they weren't we would probably want an error (which we get now) since this is likely a misconfiguration. 

The idea of puppet communicating with other servers, though, brings up the issue of puppet trusting the SSL certificates those other servers are offering up.  For Puppet Dashboard we just don't validate the peer (which probably isn't great) -- I think that was just following recommended install instructions.  Obviously for puppetdb it is validating the peer, since the indirector/rest.rb stuff uses a configured http client that is set to care about SSL.

Just thinking out loud, perhaps the architectural change here would be to change rest.rb to allow users of that class to specify additional CA certificates.  Then puppetdb itself could be configured to use a different ca_file from the one that puppetmaster would use by default.

I'm definitely not sure how this should work, but it does seem important to differentiate between these different types/roles of CA for puppetmaster and to support the concept that other servers that simply need to serve stuff over SSL (and not sign node client certs, etc.) will use different CAs.

Thanks again -

Hans

[Hans Lellelid]

Will do.

Thanks! 

[Hans Lellelid]

Following up: http://projects.puppetlabs.com/issues/23180

Thanks again.  I think I spent enough time to figure out how I can hack this in the short term, though would love to see this accepted!  I'm happy to provide the hack diff that we'll use, though I am positive that you'd not want to just merge that.  This is probably my second time writing any Ruby code :)

Thanks,

Hans

[Ken Barber]

> Following up: http://projects.puppetlabs.com/issues/23180

So someone else was discussing this on IRC, was it you? I'm just
asking because I was trying to remember if there was another user
trying to do a similar thing.

> Thanks again. I think I spent enough time to figure out how I can hack this
> in the short term, though would love to see this accepted! I'm happy to
> provide the hack diff that we'll use, though I am positive that you'd not
> want to just merge that. This is probably my second time writing any Ruby
> code :)

My local git branches are full of hacky patches, that upon scrutiny
aren't the 'final solution' ... lol :-). But workarounds are _always_
welcome, so at least users aren't completely road-blocked when they
find the bug :-). So please, add whatever you find to the ticket, it
helps.

ken.

[Hans Lellelid]

On Mon, Nov 18, 2013 at 2:57 AM, Ken Barber <k...@puppetlabs.com> wrote:

> Following up: http://projects.puppetlabs.com/issues/23180

So someone else was discussing this on IRC, was it you? I'm just
asking because I was trying to remember if there was another user
trying to do a similar thing.

No, that wasn't me.

 

> Thanks again.  I think I spent enough time to figure out how I can hack this
> in the short term, though would love to see this accepted!  I'm happy to
> provide the hack diff that we'll use, though I am positive that you'd not
> want to just merge that.  This is probably my second time writing any Ruby
> code :)

My local git branches are full of hacky patches, that upon scrutiny
aren't the 'final solution' ... lol :-). But workarounds are _always_
welcome, so at least users aren't completely road-blocked when they
find the bug :-). So please, add whatever you find to the ticket, it
helps.

Yeah, I was able to patch out the code that writes stuff to puppetdb (the Command class), but  got stuck trying to fix the other classes that extent Indirector::Rest.  I must be misunderstanding inheritance in Ruby, since I overrode the network() method but am still getting a cert validation error.  -- And I attempted to add a bunch of logging which I don't see, so obviously I'm doing something wrong :)  

If I can get a full working hack, I'd be happy to post a diff.

Thanks again -

Hans