| Prior to PuppetDB 6.8.0, PuppetDB could use the javax.net.ssl.trustStore JVM
property to configure which CA certificates to use when validating database
connections that use SSL. This is the first configuration method recommended by PuppetDB
documentation: https://puppet.com/docs/puppetdb/latest/postgres_ssl.html#using-puppet-certificates-with-the-java-keystore This works in PuppetDB 6.7.3. After upgrading to 6.8.0, it fails silently and
leaves the service in an unresponsive state. Reproduction Case
- Install Puppet Server 6.x on a CentOS 7 VM:
yum install -y http://yum.puppetlabs.com/puppet6-release-el-7.noarch.rpm |
yum install -y puppetserver
|
- Bootstrap the Puppet CA and install the puppetlabs-puppetdb,
puppetlabs-java_ks, and puppetlabs-concat modules:
source /etc/profile.d/puppet-agent.sh |
puppetserver ca setup |
|
puppet module install puppetlabs-puppetdb |
puppet module install puppetlabs-java_ks |
puppet module install puppetlabs-concat
|
- Install PuppetDB 6.7.3 and configure it to use SSL:
puppet apply <<'EOF' |
class { 'puppetdb::globals': |
version => '6.7.3', |
} |
|
$puppetdb_truststore = '/etc/puppetlabs/puppetdb/ssl/puppet_ca.jks' |
|
class { 'puppetdb': |
database_host => $trusted['certname'], |
database_listen_address => '*', |
jdbc_ssl_properties => '?ssl=true', |
java_args => {'-Djavax.net.ssl.trustStore' => "=${puppetdb_truststore}", |
'-Djavax.net.ssl.trustStorePassword' => '=puppetdb'}, |
} |
|
java_ks { 'puppetdb truststore': |
ensure => present, |
certificate => '/etc/puppetlabs/puppet/ssl/certs/ca.pem', |
target => $puppetdb_truststore, |
password => 'puppetdb', |
trustcacerts => true, |
require => Package[$puppetdb::params::puppetdb_package], |
before => Service[$puppetdb::params::puppetdb_service], |
} |
|
file {'postgres private key': |
ensure => present, |
path => "${postgresql::params::datadir}/server.key", |
source => "file:///etc/puppetlabs/puppet/ssl/private_keys/${trusted['certname']}.pem", |
owner => 'postgres', |
mode => '0600', |
require => Package['postgresql-server'], |
} |
|
concat {'postgres cert bundle': |
ensure => present, |
path => "${postgresql::params::datadir}/server.crt", |
owner => 'postgres', |
require => Package['postgresql-server'], |
} |
|
concat::fragment {'agent cert': |
target => 'postgres cert bundle', |
source => "file:///etc/puppetlabs/puppet/ssl/certs/${trusted['certname']}.pem", |
order => '1', |
} |
|
concat::fragment {'CA bundle': |
target => 'postgres cert bundle', |
source => 'file:///etc/puppetlabs/puppet/ssl/certs/ca.pem', |
order => '2', |
} |
|
postgresql::server::config_entry {'ssl': |
ensure => present, |
value => 'on', |
require => [File['postgres private key'], Concat['postgres cert bundle']], |
} |
EOF
|
- Upgrade PuppetDB: yum upgrade -y puppetdb
Outcome The upgrade operation hangs when package scripts re-start PuppetDB. The
PuppetDB log file has a very vague message about DB connections not being
available:
2020-03-25T00:58:54.288Z INFO [o.e.j.s.Server] Started @8184ms |
2020-03-25T00:58:54.290Z INFO [p.p.dashboard] Redirecting / to the PuppetDB dashboard |
2020-03-25T00:58:54.291Z INFO [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@713002f7{/,null,AVAILABLE} |
2020-03-25T00:58:54.294Z INFO [p.t.s.s.status-core] Starting background monitoring of cpu usage metrics |
2020-03-25T00:58:54.305Z INFO [p.t.s.s.status-service] Registering status callback function for service 'status-service', version 1.1.0 |
2020-03-25T00:58:54.305Z INFO [p.t.s.s.status-service] Registering status service HTTP API at /status |
2020-03-25T00:58:54.318Z INFO [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@6abb359b{/status,null,AVAILABLE} |
2020-03-25T00:58:54.329Z INFO [c.z.h.HikariDataSource] PDBWritePool - Starting... |
2020-03-25T00:58:54.374Z INFO [c.z.h.HikariDataSource] PDBWritePool - Start completed. |
2020-03-25T00:58:54.375Z INFO [c.z.h.HikariDataSource] PDBReadPool - Starting... |
2020-03-25T00:58:54.378Z INFO [c.z.h.HikariDataSource] PDBReadPool - Start completed. |
2020-03-25T00:58:54.379Z INFO [p.p.c.services] PuppetDB version 6.9.1 |
2020-03-25T00:58:54.379Z INFO [c.z.h.HikariDataSource] PDBMigrationsPool - Starting... |
2020-03-25T00:58:54.381Z INFO [c.z.h.HikariDataSource] PDBMigrationsPool - Start completed. |
2020-03-25T00:58:57.390Z ERROR [p.p.c.services] Will retry database connection after temporary failure: java.sql.SQLTransientConnectionException: PDBMigrationsPool - Connection is not available, request timed out after 3001ms. |
2020-03-25T00:59:00.391Z ERROR [p.p.c.services] Will retry database connection after temporary failure: java.sql.SQLTransientConnectionException: PDBMigrationsPool - Connection is not available, request timed out after 3000ms.
|
tcpdump shows PuppetDB connection to Postgres, but traffic stops after
the database responds to the "SSL Request" message. The client should initiate
a TLS handshake at this point. Expected Outcome The upgrade completes successfully. Any error with loading TLS configrion should result in a descriptive log message. Recommended Workaround Configure PuppetDB to load CA certificates using the sslrootcert parameter
in the subname settings of database.ini and read_database.ini instead
of using javax.net.ssl.trustStore as described here: https://puppet.com/docs/puppetdb/latest/postgres_ssl.html#using-your-own-self-signed-ca For the reproduction case above, this could be configured by:
puppet apply <<'EOF' |
class { 'puppetdb': |
database_host => $trusted['certname'], |
database_listen_address => '*', |
jdbc_ssl_properties => '?ssl=true&sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem', |
} |
EOF
|
|
