Puppet Certificate Issues

[Rohit]

 Hello, we currently have a puppet docker container setup and are experiencing certificate issues. Basically, in our docker setup (on our main server) I had generated and signed new certificates, but the puppet_db container keeps restarting. Here are logs from the puppet_db container:

    ‘Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=our.puppet.domain]

    Error: Could not retrieve catalog; skipping run

    Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=our.puppet.domain]’

I have tried series of steps to solve this problem as it looks like Puppet is not functioning correctly as our servers are not properly listening to the host server. Any idea what I can do to solve this problem? For reference, we are running Puppet_DB version 4.2 and Puppet Server version 2.7.2, all of which is set up on a docker container environment on one server.

[Morgan Rhodes]

Hi Rohit,

Is the hostname from `/CN=our.puppet.domain` showing up in your puppetserver's certificate? You can verify that with `puppet cert list --all` on the puppetserver container. This looks like a DNS issue.

[Rohit]

Hello Morgan,

If you are refferring to the cert being in the conf/ssl/certs folder, then yes, our.puppet.domain.pem is in the folder. When running the 'puppet cert list --all' I see three certificates (in the SHA256 format):

• computername.our.puppet.domain
• our.puppet.domain
• servername.our.puppet.domain

If it is a DNS issue, do I have to likely change something from the docker-compose side?

[Morgan Rhodes]

A few things to verify:

1) what hostname is your puppetdb container trying to connect to puppetserver at?

    a) This should be in your docker-entrypoint.sh script in the puppetdb container. Likely either 'puppet' or '$PUPPETSERVER_HOSTNAME' depending on what variables you have set in your compose file and what version of the puppetdb container you have.

2) Is the hostname your puppetdb container is trying to connect to listed as one of the certificate names for your puppet server's cert?

    a) For example, in my puppetserver container when I run `puppet cert list --all` I see:

+ "testserver" (SHA256) F0:31:6D:1D:03:82:C0:84:0D:FA:2B:28:5B:52:CB:18:88:87:61:5F:5A:F5:7E:AB:A2:73:29:44:BC:57:D0:99 (alt names: "DNS:testserver", "DNS:foo")

  if my puppetdb container tries to connect to that host over any names other than 'testserver' or 'foo' I get a certificate verify failed error.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/899d2bf7-ceed-4d9e-bd24-c4ba2cc93928%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Morgan Rhodes

mor...@puppet.com

Release Engineer

[Rohit]

1. puppet_db is trying to connect our.puppet.domain, there is no docker-entrypoint.sh script that I was able to find.
1. For reference, this is the docker-compose.yml:

2. puppetdb:

      container_name: puppet_db

      hostname: puppetdb.peninsula.wednet.edu

      dns:

        - 10.0.0.7

      image: puppet/puppetdb:latest

      ports:

        - 8087:8080

        - 8088:8081

      depends_on:

        - puppet

      links:

        - puppet:puppet4.psd401.net

        - puppetdbpostgres:postgres

      volumes:

        - ./puppet-client.conf:/etc/puppetlabs/puppet/puppet.conf

        - ./puppetdb_conf:/etc/puppetlabs/puppetdb/conf.d

        - ./puppetdb_ssl:/etc/puppetlabs/puppet/ssl/

      networks:

        puppet:

          ipv4_address: 172.19.0.4

      restart: always

   
   
2. The hostname that the puppetdb container is trying to connect to is indeed the one listed on the certificate name on the puppet servers cert.

[Morgan Rhodes]

When you look at the output of `puppet cert list all` does the certificate for your puppetmaster also include the alt name 'puppet'? (Something like 'alt names: "DNS:puppet", "DNS:testpuppet"'). If not, I'm guessing that's your problem.

You mentioned in your earlier email that you were using puppetdb 4.2.0. I'm assuming you're running the puppet/puppetdb:4.2.0 container. To get the container entrypoint, I start the container manually with a custom entrypoint so I can look around, there should be a file 'docker-entrypoint.sh' in the root directory of the container.

$ docker run --rm -it --entrypoint /bin/bash puppet/puppetdb:4.2.0

root@e09f677618d7:/# ls

Dockerfile  bin  boot  dev  docker-entrypoint.sh  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

root@e09f677618d7:/# cat docker-entrypoint.sh

#!/bin/bash

if [ ! -d "/etc/puppetlabs/puppetdb/ssl" ]; then

  while ! nc -z puppet 8140; do

    sleep 1

  done

  set -e

  /opt/puppetlabs/bin/puppet agent --verbose --onetime --no-daemonize --waitforcert 120

  /opt/puppetlabs/server/bin/puppetdb ssl-setup -f

fi

exec /opt/puppetlabs/server/bin/puppetdb "$@"

root@e09f677618d7:/#

The docker-entrypoint.sh script in that version of the container doesn't have any logic for a puppetserver with a non-default name, which means when it runs `puppet agent --verbose --onetime --no-daemonize --waitforcert 120` it will connect to the host named 'puppet'. From the link you have set up in your docker-compose.yml, I'm assuming your puppetserver container name is 'puppet' with the hostname 'puppet4.psd401.net'. Since the container name is 'puppet', the puppetdb container is able to resolve 'puppet' as 'puppet4....', so when it runs puppet agent -t it can connect to the host, but certificate validation will fail if puppet isn't listed as one of the valid altnames for the puppet container. 

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/fcf0c6da-82dd-4970-ab81-a60131b291f1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Rohit]

Hello Morgan,

Apologies for the late response here, some of our Puppet services had started working but it looks like the same issue has arised and I am not entirely sure why. I did check the docker-entrypoint.sh file and indeed see the very exact response as you posted. However my question is for the "altname" that you suggested, would I change this in the docker-compose.yml file? I also realize the full docker-compose.yml did not show up in my previous post but have attached it again in a separate file.

[Morgan Rhodes]

Hi Rohit,

No, unfortunately, it's not just a change in your docker-compose.yml. When you're generating the certs for your puppetserver, you'll want to make sure you're passing the `--dns_alt_names=<altnames>`, so it would be something like:

puppet cert generate puppet4.psd401.net --dns_alt_names=puppet,puppet.psd401.net

Afterwards, you can confirm that your certificate has all of the altnames with `puppet cert list --all`, you should see something like:

$ puppet cert list --all

+ "puppet4.psd401.net" (SHA256) <fingerprint> (alt names: "DNS:puppet", "DNS:puppet4.psd401.net")

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1015311c-0e2f-44f0-a096-6c5015d00d98%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Rohit]

Hello Morgan

I was able to generate a new certificate with the alt name, and when doing a 'puppet cert list --all' I see the following: 

+ "puppet4.psd401.net" (SHA256) 1D:16:67:30:0D:62:CE:6C:2A:80:11:7E:C7:79:BA:4F:25:C6:0E:E6:90:9D:4D:9F:86:4B:5C:42:A1:6D:09:96 (alt names: "DNS:puppet", "DNS:puppet4.psd401.net")

But when doing a docker logs on puppet_db, it still says:

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet4.psd401.net]

Info: Retrieving pluginfacts

Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet4.psd401.net]

Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet4.psd401.net]

Info: Retrieving plugin

Error: /File[/opt/puppetlabs/puppet/cache/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet4.psd401.net]

Error: /File[/opt/puppetlabs/puppet/cache/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet4.psd401.net]

Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet4.psd401.net]

Error: Could not retrieve catalog; skipping run

Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet4.psd401.net]

Not entirely sure what the problem here still can be... I did clean the certs as well. Realizing this is a pretty old version of Puppet, would it perhaps be better to do a clean install of Puppet in a non-docker environment?

[John Gelnaw]

I had difficulties with the stock puppetdb entrypoint script.  I wound up changing it thusly:

#!/bin/bash
 
if [ ! -d "/etc/puppetlabs/puppetdb/ssl" ]; then

    set -e
    /opt/puppetlabs/bin/puppet config set certname ${HOSTNAME}
    if [ ! -f "/etc/puppetlabs/puppet/ssl/certs/ca.pem" ]; then
        while ! nc -z puppet 8140; do
            sleep 1
        done
        /opt/puppetlabs/bin/puppet agent --verbose --onetime --no-daemonize --waitforcert 120
    fi
    /opt/puppetlabs/server/bin/puppetdb ssl-setup -f
fi

exec /opt/puppetlabs/server/bin/puppetdb "$@"

And in case it helps, here's the docker-compose stanza for puppetdb:

  puppetdb:
    hostname: puppetdb
#    image: puppet/puppetdb:4.4.0
    build: builds/puppetdb
    ports:
      - 8080
      - 8081
    volumes:
      - ./puppetdb/ssl:/etc/puppetlabs/puppet/ssl/

Note that I'm using a local build (I did the same for puppet itself, but that's because we have a number of local customizations) instead of an official image.

And the Dockerfile I used to build puppetdb:

FROM puppet/puppetdb:4.4.0
 
EXPOSE 8080
EXPOSE 8081
 
COPY docker-entrypoint.sh /
 
VOLUME /etc/puppetlabs/puppet/ssl
VOLUME /etc/puppetlabs/puppetdb
 
ENTRYPOINT ["/docker-entrypoint.sh", "foreground"]

So basically, I'm using the official image, but I'm overwriting the docker-entrypoint.sh with my own version.

The important part is definitely the puppet config line to set the hostname to match the container.  

The filetest for ca.pem was something I put in to prevent a certain condition that may have been unique to my environment-- apparently it was possible to have a local certificate already, but not a (persistent) puppetdb ssl configuration.

[Rohit]

Thanks for the response, I did try those changes to see if it helps but unfortunately the issue still exists

[Rohit]

Any idea if there are other steps I can consider? If not, should I simply rebuild the system? If I do go this route, is there a way to backup all the Puppet configurations set for servers and services that can be reimported in a fresh install? Would it also be suggested to go a non-Docker route due to stability?

[Morgan Rhodes]

Hi Rohit,

I don't have great ideas about what's going on in your environment. Are you using custom built containers or the puppet namespaced containers from hub.docker.com. You could try applying this patch (https://github.com/puppetlabs/puppetdb/commit/a1ab2f50598f12ac51acb21f256232143891dbc1) and setting PUPPETSERVER_HOSTNAME in your docker-compose.yml to puppet4.psd401.net.

In one of your earlier emails you mentioned you were using puppetserver 2.7, but it looks like in the compose file you attached you're using the puppet/puppetserver:latest, which is puppetserver 6.x, just want to make sure I know what versions of things you're running with here.

The containers for puppetserver 2.7 / puppetdb 4.2 are definitely old, and with recent efforts we have made a number of changes and improvements to the containers and the compose stack (https://github.com/puppetlabs/pupperware), but that work has been for puppetserver and puppetdb 5+.

If you run `puppet agent -t` on your puppetserver container does it succeed? (docker-compose exec puppet puppet agent -t)

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/9d319859-8f13-4e6e-97c9-c2366152a4e3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.