Re: on puppet master server , puppet agent can't connect to itself

[Corey Hammerton]

How does your auth.conf file look?

On Wednesday, July 4, 2012 4:56:10 PM UTC-4, Clay wrote:

on my puppet master server (v 2.7.17 , both server and client version) ,   the puppet agent can't connect to itself. other clients connected to this puppet server are working fine.
the hostname is puppet.domain.com

[root@puppet /]# cat /etc/puppet/puppet.conf
[main]
    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet

    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet

    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.

    certname = puppet.domain.com
    reports = store, http ,foreman
    reporturl = http://puppet.domain.com:3000/reports/upload
    modulepath = $confdir/modules
    manifest = $confdir/manifests/site.pp
    http_proxy_host = proxy.domain.com
    http_proxy_port = 8080

[dev]
   modulepath = $confdir/env/dev/modules
   manifest = $confdir/env/dev/manifests/site.pp

[testing]
   modulepath = $confdir/env/testing/modules
   manifest = $confdir/env/testing/manifests/site.pp


[agent]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuratiion.  Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$confdir/classes.txt'.
    classfile = $vardir/classes.txt

    # Where puppetd caches the local configuration.  An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig

puppet agent will get a 403 "Forbidden" error,     anyone have any suggestion what to look ?  
[root@puppet ]# puppet agent --test --debug
debug: Puppet::Type::User::ProviderPw: file pw does not exist
debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not exist
debug: Failed to load library 'ldap' for feature 'ldap'
debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dscl does not exist
debug: /File[/etc/puppet/ssl/certificate_requests]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/ssl/certs/ca.pem]: Autorequiring File[/etc/puppet/ssl/certs]
debug: /File[/var/lib/puppet/state/graphs]: Autorequiring File[/var/lib/puppet/state]
debug: /File[/etc/puppet/ssl/certs/puppet.domain.com.pem]: Autorequiring File[/etc/puppet/ssl/certs]
debug: /File[/var/lib/puppet/client_yaml]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/clientbucket]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state/last_run_report.yaml]: Autorequiring File[/var/lib/puppet/state]
debug: /File[/etc/puppet/ssl/private_keys/puppet.domain.com.pem]: Autorequiring File[/etc/puppet/ssl/private_keys]
debug: /File[/var/lib/puppet/client_data]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/classes.txt]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
debug: /File[/etc/puppet/ssl/public_keys/puppet.domain.com.pem]: Autorequiring File[/etc/puppet/ssl/public_keys]
debug: /File[/etc/puppet/ssl]: Autorequiring File[/etc/puppet]
debug: /File[/etc/puppet/ssl/public_keys]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/ssl/private_keys]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/var/lib/puppet/state/resources.txt]: Autorequiring File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/etc/puppet/ssl/certs]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/ssl/private]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/var/lib/puppet/state/last_run_summary.yaml]: Autorequiring File[/var/lib/puppet/state]
debug: /File[/etc/puppet/ssl/crl.pem]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/puppet.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/state/state.yaml]: Autorequiring File[/var/lib/puppet/state]
debug: Finishing transaction 69951197233260
debug: /File[/etc/puppet/ssl/crl.pem]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/ssl/certificate_requests]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/ssl/certs/ca.pem]: Autorequiring File[/etc/puppet/ssl/certs]
debug: /File[/etc/puppet/ssl/certs/puppet.domain.com.pem]: Autorequiring File[/etc/puppet/ssl/certs]
debug: /File[/etc/puppet/ssl/certs]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/ssl/private_keys/puppet.domain.com.pem]: Autorequiring File[/etc/puppet/ssl/private_keys]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/etc/puppet/ssl/private_keys]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/ssl/public_keys]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
debug: /File[/etc/puppet/ssl/public_keys/puppet.domain.com.pem]: Autorequiring File[/etc/puppet/ssl/public_keys]
debug: /File[/etc/puppet/ssl/private]: Autorequiring File[/etc/puppet/ssl]
debug: /File[/etc/puppet/ssl]: Autorequiring File[/etc/puppet]
debug: Finishing transaction 69951196018460
debug: Using cached certificate for ca
debug: Using cached certificate for puppet.domain.com
debug: Finishing transaction 69951197856120
debug: Loaded state in 0.00 seconds
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; using pson
debug: Using cached certificate for ca
debug: Using cached certificate for puppet.domain.com
debug: Using cached certificate_revocation_list for ca
err: Could not retrieve catalog from remote server: 403 "Forbidden"
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
debug: Value of 'preferred_serialization_format' (pson) is invalid for report, using default (yaml)
debug: report supports formats: b64_zlib_yaml raw yaml; using yaml
err: Could not send report: 403 "Forbidden"

[Clay]

thanks for the reply. I think auth.conf is the default one.

[root@puppet ]# grep -v ^# /etc/puppet/auth.conf

path ~ ^/catalog/([^/]+)$
method find
allow $1

path ~ ^/node/([^/]+)$
method find
allow $1

path /certificate_revocation_list/ca
method find
allow *

path /report
method save
allow *

path /file
allow *

path /modules
allow *

path /certificate/ca
auth any
method find
allow *

path /certificate/
auth any
method find
allow *

path /certificate_request
auth any
method find, save
allow *

path /
auth any

[Clay]

I don't have to have the puppet agent on the puppet server up , but when setting up  puppetdb ,  I got  this error from clients:

# puppet agent --test
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for client1.domain.com to PuppetDB at puppet.domain.com:8081: 403 "Forbidden"

from puppetdb document , seems need to get puppet agent on puppet server working first,  not sure it's it's related.  

[Deepak Giridharagopal]

Indeed, PuppetDB  requires that SSL is working between agent and master (at least, the default setup scripts invoked when using pre-built packages assume SSL works). The error you're seeing from PuppetDB is another, separate manifestation of what I think is the same underlying problem. It appears that your master is not trusting the certificate your agent is presenting?

For a conclusive test, though, it may help to temporarily disable PuppetDB and retry. If agent/master communication works, then we know the issue is between the master node and the puppetdb node.

deepak

--

Deepak Giridharagopal / Puppet Labs / grim_radical

[Clay]

Thanks. I already disabled puppetdb and still got the above 403 "Forbidden" error,  also tried remove /etc/puppet/ssl and restarted  puppet master, same error.  

[Felix Frank]

This may be a case of apache refusing to talk to clients on the local
host. Have you checked apache's logs?

Furthermore, what I like trying is starting puppet master with
--no-daemonize and -dv, also --masterport so your regular agents won't
interfere with debugging. Then run the faulty agent against that port
and see what happens.

If this *does* work, it's even more likely you're looking at an
Apache/Passenger problem. You may want to try and enable debugging on
your productive master then.

HTH,
Felix

[Clay]

I find the problem is this in the puppet.conf ,

    http_proxy_host = proxy.domain.com
    http_proxy_port = 8080

after I removed these lines , puppet agent on the master server cant connect to itself now.   I added these because we are behind proxy and need to use it for "puppet module" to connect to internet, now I just set the proxy in http_proxy environment variable.

But during trail and error,  I had some certificate issue,  I ended up re-building the puppet master server. 

Thanks.