puppet agent running on same host as puppet server, possible cert issues

[aaron....@gmail.com]

I'm running into an issue on one of our puppet4 servers where the agent only works when run as 'puppet agent -t' but fails when run via the puppet service 'systemctl restart puppet' results in errors as follows: 

Jun 15 14:49:45 puppet21b puppet-agent[29620]: Starting Puppet client version 4.8.1

Jun 15 14:49:45 puppet21b puppet-agent[29622]: Unable to fetch my node definition, but the agent run will continue:

Jun 15 14:49:45 puppet21b puppet-agent[29622]: getaddrinfo: Name or service not known

I've seen similar behavior when running the first puppet run via puppet agent -t (not waiting for the service to start and do the run itself) and the solution has been to remove all certs in /etc/puppetlabs/puppet/ssl/ and running puppet cert clean hostname.fqdn on the puppet master, however since this is the puppet master also, I don't want to lose all the client certs for other puppet clients. 

What's the proper procedure to clean the agent certs if the master is on the same system?  

I'm running puppet 4.8.1 on CentOS  7.3.1611 

/etc/puppetlabs/puppet/puppet.conf: 

[master]

certname = puppet4.fqdn

dns_alt_names = puppet.fqdn, puppet, puppet4

vardir = /opt/puppetlabs/server/data/puppetserver

logdir = /var/log/puppetlabs/puppetserver

rundir = /var/run/puppetlabs/puppetserver

pidfile = /var/run/puppetlabs/puppetserver/puppetserver.pid

codedir = /etc/puppetlabs/code

autosign = true

[main]

environmentpath = $confdir/environments

manifestdir = /etc/puppetlabs/puppet/environments/$environment/manifests

pluginsync = true

[agent]

server = puppet4

[Peter Faller]

You can specify which client certificate(s) to clean by means of:

# puppet cert clean <certname>

On Friday, 16 June 2017 00:31:57 UTC+2, aaron....@gmail.com wrote:

I'm running into an issue on one of our puppet4 servers where the agent only works when run as 'puppet agent -t' but fails when run via the puppet service 'systemctl restart puppet' results in errors as follows: 

<snip>

[aaron....@gmail.com]

Thanks Peter, I may be running into a different problem now, 

in /var/log/messages I get the error: 

puppet-agent[808]: Could not request certificate: getaddrinfo: Name or service not known

which seems to indicate issues resolving my master address, but I have no trouble resolving the hostname puppet4 either via ping or nslookup and I've tried explicitly setting the hostnames puppet puppet4 and puppet4.<fqdn>  in /etc/hosts both to the interface IP and 127.0.0.1  and I still get the same error.  

if I run 'puppet agent -t' the cert generates and everything checks in just fine, but running the puppet service still has issues:

Jul  3 09:47:46 puppet21b puppet-agent[1429]: Unable to fetch my node definition, but the agent run will continue:

Jul  3 09:47:46 puppet21b puppet-agent[1429]: getaddrinfo: Name or service not known

[Peter Faller]

Is the puppet agent running as the same user when run as a daemon and when run via 'puppet agent' from the command line? I've seen that make a difference.

On Monday, 3 July 2017 15:49:06 UTC+2, aaron....@gmail.com wrote:

[aaron....@gmail.com]

That might be getting closer to the issue, the agent runs as the user 'puppet' but I'm running the agent manually as root...our other systems work fine in this config though, I'm looking for what files might have the wrong permissions for this but so far not finding any differences. 

[aaron....@gmail.com]

I found the issue, digging more through the files that the puppet client service was accessing, I found that /etc/sysconfig/puppet was being referenced, 

PUPPET_EXTRA_OPTS="--server @@PUPPET4_SERVER@@"

I checked our other servers that were working and found that those referenced the fqdn of the server.  Changing that line to replace @@PUPPET4_SERVER@@ with our actual fqdn resolved this issue.   I believe this may have been misconfigured somehow during the early setup on this system.