puppet master REST API returns 403 when running under passenger works when running from command line

[Anadi Misra]

Hi! Everyone,

puppet agent is not able to fetch any files, plugins or post catalog, reports to the master. both puppet agent and master are on version 3.0.l, passenger version 3.0.18 ,

nginx version: nginx/1.3.9
built by gcc 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/apps/nginx --conf-path=/apps/nginx/nginx.conf --pid-path=/apps/nginx/run/nginx.pid --error-log-path=/apps/nginx/logs/error.log --http-log-path=/apps/nginx/logs/access.log --with-http_ssl_module --with-http_gzip_static_module --add-module=/usr/lib/ruby/gems/1.8/gems/passenger-3.0.18/ext/nginx --add-module=/apps/Downloads/nginx/nginx-auth-ldap-master/

the agent command shows this output

[amisr1@blramisr195602 ~]$ sudo puppet agent --no-daemonize --verbose --server bangvmpllda02.XXXXXX.com
Starting Puppet client version 3.0.1
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31) access to /certificate_revocation_list/ca [find] at :106
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [search] at :106
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find] at :106 Could not retrieve file metadata for puppet://bangvmpllda02.XXXXXX.com/plugins: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find] at :106
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31) access to /catalog/blramisr195602.XXXXXX.com [find] at :106
Using cached catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31) access to /report/blramisr195602.XXXXXX.com [save] at :106

and on master logs I see

[amisr1@blramisr195602 ~]$ sudo puppet agent --no-daemonize --verbose --server bangvmpllda02.XXXXXX.com
Starting Puppet client version 3.0.1
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31) access to /certificate_revocation_list/ca [find] at :106
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [search] at :106
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find] at :106 Could not retrieve file metadata for puppet://bangvmpllda02.XXXXXX.com/plugins: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31) access to /file_metadata/plugins [find] at :106
Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31) access to /catalog/blramisr195602.XXXXXX.com [find] at :106
Using cached catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 403 on SERVER: Forbidden request: 10.209.47.31(10.209.47.31) access to /report/blramisr195602.XXXXXX.com [save] at :106


I am not sure why is it evaluating things on IP?

I also changed agent setup to following

[main]
    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet

    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet

    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = $vardir/ssl
    report = true
    pluginsync = true
    server = devops.XXXXXX.com
    certname = blramisr195602.XXXXXX.com
    dns_alt_names = 10.209.47.31
    modulepath = /etc/puppet/modules

and resigned certifcates on master after clean up, but the puppet master still blocks it. However If I run through puppet master daemon (without nginx + passenger) all requests go through.

Is there any specific configuration for Nginx host header etc or in passenger that I am missing?

BR/
Anadi Misra.

[Felipe Salum]

On Apache/Passenger I have set a few headers:

  RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e 
  RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e 
  RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

And then updated puppet.conf as below:

[master]

ssl_client_header = HTTP_X_SSL_SUBJECT

Does it fail if you use --server devops.XXXXX.com ? 
You should use the --server hostname as the same certname name used on the puppetmaster.

Regards,
Felipe

[Anadi Misra]

Thanks!

I compared it with a similar setup we had done in the past and noticed that this one had both

ssl_client_header = SSL_CLIENT_S_D
ssl_client_verify_header = SSL_CLIENT_VERIFY

in pupet.conf and 

	passenger_set_cgi_param    HTTP_X_CLIENT_DN $ssl_client_s_dn; 
	passenger_set_cgi_param    HTTP_X_CLIENT_VERIFY $ssl_client_verify; 

in the nginx.conf; disabled it from puppet master and it works now.

BR/
Anadi.