Troubleshooting domain name and certificate problems

[Bogdan Bivolaru]

I've had a problem with configuring the puppet client on the same
machine as the puppetmaster (hostname: artbazaar, DNS names:
artbazaar.example.com, puppet.example.com). The problem is now solved,
this is just a description of my solution and a discussion on how to
solve it easier. Please comment on it.
In the following I describe the problem at the present tense because I
started composing the message before solving it myself. At the end of
the post I have made a small suggestion to improve troubleshooting
problems related to the domain names.

PROBLEM:
No matter if I start puppetd with "sudo puppetd --evaltrace --test -l /
home/bogdanbiv/pd.log --certname {artbazaar, puppet}.example.com --
fqdn {artbazaar, puppet}.example.com --server {artbazaar,
puppet}.example.com",
I get this:

info: Retrieving plugins
warning: Certificate validation failed; considering using the certname
configuration option
err: /File[/var/lib/puppet/lib]: Failed to generate additional
resources during transaction: Certificates were not trusted: hostname
was not match with the server certificate
warning: Certificate validation failed; considering using the certname
configuration option
err: /File[/var/lib/puppet/lib]: Failed to retrieve current state of
resource: Certificates were not trusted: hostname was not match with
the server certificate Could not describe /plugins: Certificates were
not trusted: hostname was not match with the server certificate
warning: Certificate validation failed; considering using the certname
configuration option
err: Could not retrieve catalog: Certificates were not trusted:
hostname was not match with the server certificate
warning: Not using cache on failed catalog

Since this doesn't tell me what hostname and certname were used/
detected, I decided to dig deeper, maybe I can find some useful info
(openssl client info from http://alittlestupid.com/2005/11/23/view-ssl-cert-from-terminal/):
bogdanbiv@artbazaar:~$ openssl s_client -connect puppet.gitmusic.net:
8140 -showcerts -showcerts
CONNECTED(00000003)
depth=0 /CN=artbazaar
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /CN=artbazaar
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/CN=artbazaar
i:/CN=artbazaar
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=artbazaar
issuer=/CN=artbazaar
---
No client certificate CA names sent
---
SSL handshake has read 1155 bytes and written 328 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
....
Verify return code: 21 (unable to verify the first certificate)

So the problem is that the certificate was for artbazaar and not the
FQDN of my host artbazaar.example.com. I think the PuppetCA generated
the cert with this name due to a misconfiguration in the DNS service.

SOLUTION: Clean the certificate on the puppetmaster.
1. I stopped the puppet client with "sudo /etc/init.d/puppet stop"
2. The command "sudo puppetca -c -all" reported that no certificates
have been deleted, so I stopped the puppetmaster too with "sudo /etc/
init.d/puppetmaster stop" (maybe I should have done this before
puppetca -c --all ??).

3. I deleted everything inside the /var/lib/puppet/ssl to make sure
that the bad CA certificate is gone. (sudo rm -R /var/lib/puppet/ssl/
*)
Sidenote: make sure you're not deleting -R ./.* because that includes
the parent dir ./.. and that would remove every parent until the
filesystem is broken.

4. I ran sudo puppetca -g artbazaar.example.com puppet.example.com
5. I ran sudo puppetmasterd --genconfig to regenerate a good
configuration
5bis. puppetmasterd did not made the master run as a daemon, it exited
immediately, so I issued sudo /etc/init.d/puppetmasterd start
6.$ I ran openssl client again:
openssl s_client -connect puppet.gitmusic.net:8140 -showcerts
CONNECTED(00000003)
depth=0 /CN=artbazaar.gitmusic.net
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /CN=artbazaar.gitmusic.net
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/CN=artbazaar.gitmusic.net
i:/CN=artbazaar.gitmusic.net
Eeverything was fine,
7. Run puppet client "sudo puppetd --evaltrace -l /home/bogdanbiv/
pd.log --test" which resulted to:
info: No classes to store
info: Caching catalog at /var/lib/puppet/state/localconfig.yaml
notice: Starting catalog run
notice: //File[/tmp/foobar.txt]/ensure: changed file contents from
{md5}b10a8db164e0754105b7a99be72e3fe5 to {md5}
b10a8db164e0754105b7a99be72e3fe5
info: //File[/tmp/foobar.txt]: Evaluated in 0.06 seconds

Everything was fine with my certificates and it applied my changes

7bis I ran "sudo /etc/init.d/puppet start" to make it daemon

SUGGESTION:
Could someone make puppet log what hostname and certname it uses at
the current run? Could it be logged both on the puppet master and on
the client?