Help with configuring Puppet Proxies using Apache

[kschafer2598]

Hi All,

I have an environment with multiple network environments separated by firewalls. I would like to proxy Puppet traffic from these network environments to a central Puppet master server.

I've had a great experience with Puppet overall, though it has been rather trying to find documentation on the subject of proxying puppet traffic.

Can anyone point me in the right direction for how to approach this? I'm assuming I would do this with Apache but I'm open to other methods as well.

Any help is appreciated.

Regards,

Karl

[Felipe Salum]

This is how I do it here

https://github.com/fsalum/vagrant-puppet/blob/master/puppetmaster/templates/etc/apache2/sites-available/puppetmaster_balancer.erb
https://github.com/fsalum/vagrant-puppet/blob/master/puppetmaster/files/etc/apache2/sites-available/puppetmaster_ca
https://github.com/fsalum/vagrant-puppet/blob/master/puppetmaster/files/etc/apache2/sites-available/puppetmaster_worker

Felipe

[pdpinfo]

Hi Karl,

this topic has been discussed many times, particularly in respect of "large scale" and "distributed".
There are many possible setups/solutions.
I try to  add my 2cents, firstly pointing out main issues.
Cannot say if this setup can be recommended, but it works well for us.

1) how large is the base ? You may need a central balanced "puppet farm" to manage the load
    --> manage balancing and CA certificates
    --> manage high performance setup: involves PASSENGER/Apache
    --> manage reports
    --> manage node classification
    --> manage versioning and deploy

2) do you need direct and proxy access at the same time ?
    --> you need an apache configuration supporting both

At the moment the setup I use is:

- A "repository server" with git+gitolite + scripts/hooks for deploying on puppet servers
- Many puppet environments and branches for each one
- A "clustered" active/standby Puppet Server (Puppet 2.7, Passenger, Foreman, Mysql, DRBD+Heartbeat). Supports direct and proxied access.
- Two other "single" puppet servers (contents are deployed by the same gitrepo)
- Three "puppet proxy" servers (Apache + mod_proxy) pointing to the respective puppet server

At the moment this infrastructure (servers are VMs) is supporting almost 500 clients, while a lot are still linked on a legacy puppet.

The whole setup involves many steps, starting points are:

a) manage certificates for many puppet server (here we're using Puppet CA):
    --> basically generate one time a certificate including as DNS alternate names all the DNS names used by puppet servers, including virtual IP/alias
    --> share the server SSLDIR (could be clustered, or networked). Ensure only one CA is used (or 2 servers, acting as the same CA). Provided the CA remains the same, the servers certificate can be freely updated in the future.

b) generate with Puppet CA an SSL server for SSL termination points (proxies): should contain all DNS names used to reach the server (including DNS aliases). Proxy trust Puppet CA.

c) generate certificate for Foreman-proxy, if used

d) setup virtual host for proxy (must extract SSL information for request) and pass them to Puppet Server

e) setup multiple virtual host on Apache on Puppet servers:
    --> VH for proxies: communication with proxies must be secured, because SSL client authentication happens on Proxies
    --> VH for puppet client direct access: "standard" SSL puppet client authentication

f) setup puppet.conf of servers, to manage headers passed by apache, i.e  ssl_client_header, ssl_client_verify_header

So, what is less confusing :-) ?

Regard

Paolo

[kschafer2598]

Paolo, thank you so much for the info. It is a bit confusing, so I've got a bit of a ways to go, but it is helpful in designing a comprehensive puppet infrastructure.

Can you provide any more details on your "puppet proxy" server configuration? I do not have much experience dealing with Apache proxies and any config details would be helpful. That is the item I'm stuck on right now.

Best regards,

Karl

[pdpinfo]

Hi Karl,

here following are apache conf that work, afaik (any comment is welcomed):
- puppetserver: direct and indirect access
- proxy server

You can have direct and proxied clients:

clients 
   |
tcp/8140
   |
Puppet Server
   |
tcp/8141
-----------firewall
   |
  RP
   |
tcp/8140                  
   |             
"remote" clients 

Please note: (disclaimer) this setup, intended for internal networks, does not have imho evident security issues, however you have to understand what issues could arise if you do not manage a "trust chain", that is ensure security on certificates, ssl, network communication, puppetserver access.
More:
- To operate this setup you must already have certificates generated by Puppet CA.
- Certificates must contain all relevant DNS names used by servers, and correct CN.
- Pay attention on header variables and tcp/8141 access restriction, to be not vulnerable to "man-in-the-middle attacks".
- You should update CRL on proxy.
- (This setup does not have SSL client validation for RP when connecting to puppetserver; SSLVerifyClient on VH 8141 recommended.)

Verify you have in your server's puppet.conf:
    ssl_client_header = HTTP_X_PUPPET_CLIENT_DN
    ssl_client_verify_header = HTTP_X_PUPPET_CLIENT_VERIFY

(Change servernames and ACL as requested)
#------------Puppet server-----------
Listen 8141
<VirtualHost *:8141>
    ServerName my_puppet_servername
    ServerAlias my_puppet_servername
    SSLEngine on
    SSLProtocol -ALL +SSLv3 +TLSv1
    SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
    SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/my_puppet_servername.pem
    SSLCertificateFile /var/lib/puppet/ssl/certs/my_puppet_servername.pem
    SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem

    # Passenger options that can be set in a virtual host configuration block.
    PassengerHighPerformance on
    PassengerStatThrottleRate 120
    PassengerUseGlobalQueue on
    RackAutoDetect Off
    RailsAutoDetect Off
    RackBaseURI /

    # X-Client variables required to verify client authentication
    # Values are coming from (trusted) Reverse Proxy that verifies client certificate
    # For correct CA emission, and CRL status
    SetEnvIf X-RP-Client-DN "(.*)" HTTP_X_PUPPET_CLIENT_DN=$1
    SetEnvIf X-RP-Client-Verify "(.*)" HTTP_X_PUPPET_CLIENT_VERIFY=$1
    SetEnvIf X-Forwarded-For "(.*)" REMOTE_ADDR=$1

    DocumentRoot /etc/puppet/rack/public
    <Location />
        Options None
        Order deny,allow
        # List IP address of your proxy
        Allow from my_proxy_IP_address
        Deny from all
    </Location>
</VirtualHost>

Listen 8140
<VirtualHost *:8140>
    SSLEngine on
    SSLProtocol -ALL +SSLv3 +TLSv1
    SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
    SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/my_puppet_servername.pem
    SSLCertificateFile /var/lib/puppet/ssl/certs/my_puppet_servername.pem
    SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
    SSLVerifyClient optional
    SSLVerifyDepth 1
    SSLOptions +StdEnvVars

    # Passenger options that can be set in a virtual host configuration block.
    PassengerHighPerformance on
    PassengerStatThrottleRate 120
    PassengerUseGlobalQueue on

    RackAutoDetect Off
    RailsAutoDetect Off
    RackBaseURI /

    RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
    RequestHeader set X-PUPPET-Client-DN %{SSL_CLIENT_S_DN}e
    RequestHeader set X-PUPPET-Client-Verify %{SSL_CLIENT_VERIFY}e

    DocumentRoot /etc/puppet/rack/public
    <Directory /etc/puppet/rack/>
        Options None
        AllowOverride None
        Order allow,deny
        Allow from all
    </Directory>
</VirtualHost>
#---------------END Puppet Server-----------------


#----------------RP---------------------
Listen 8140

<VirtualHost *:8140>
    ServerName my_RP_servername:8140
    SSLEngine on
    SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA
    SSLProtocol -ALL +SSLv3 +TLSv1
    SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
    SSLCertificateFile /var/lib/puppet/ssl/certs/my_RP_servername.pem
    SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/my_RP_servername.pem
    SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
    SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
    SSLVerifyClient optional
    SSLVerifyDepth 1
    SSLOptions +StdEnvVars

    ErrorLog logs/error_puppet_rp_log
    TransferLog logs/access_puppet_rp_log
    LogLevel warn
    CustomLog logs/ssl_request_puppet_rp_log  "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

    RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

    RewriteEngine On
    TraceEnable Off
    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
    RewriteRule .* - [F]

    SSLProxyEngine on
    SSLProxyVerify require
    SSLProxyCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLProxyCheckPeerCN on
    # SSLProxyMachineCertificateFile /var/lib/puppet/ssl/certs/my_RP_servername_pub_and_key.pem

    ProxyPass / https://my_puppetserver_servername:8141/
    ProxyPassReverse / https://my_puppetserver_servername:8141/
    ProxyPreserveHost On

    <Location />
        Order deny,allow
        allow from my_client_IP_network
        deny from all
   </Location>

</VirtualHost>
#------------END RP--------------------

Regards

Paolo

[kschafer2598]

Very cool, thank you so much! I'll be reviewing this and will give it a try as soon as I can.

Karl

[kschafer2598]

Hi Paolo,

It took me a while but I finally got my puppetmaster setup to use passenger and apache.

I'm working on the remote proxy but had a question about the Puppet cert configuration. How are you keeping the SSL certs used on the RP in sync with the puppetmaster? Or are they a seperate set?

Regards,

Karl



On Wednesday, November 20, 2013 6:41:17 AM UTC-5, pdpinfo wrote:

[kschafer2598]

I finally got it. Lots of searching, poking and twiddling with apache services :) Thank you so much!

Here are my config files. Remote proxy is SLES11, puppetmaster is RHEL5 with EPEL and Puppet repos. Note that SSLProxyMachineCertificateFile /var/lib/puppet/ssl/certs/remote_proxy_host.combined is the private key and cert for the remote proxy put in a single file (/var/lib/puppet/ssl/certs/puppetmaster_host.pem and /var/lib/puppet/ssl/private_keys/puppetmaster_host.pem)

Apache conf.d/puppetmaster.conf:

PassengerMaxPoolSize 4
PassengerMaxRequests 1000
PassengerPoolIdleTime 600

Listen 8141
<VirtualHost *:8141>

    SSLEngine on
    SSLProtocol             All -SSLv2
    SSLCipherSuite          HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP
    SSLCertificateFile      /var/lib/puppet/ssl/certs/puppetmaster_host.pem
    SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/puppetmaster_host.pem
    SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem

    SSLCACertificateFile    /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
    SSLVerifyClient         optional
    SSLVerifyDepth          1

    SSLOptions              +StdEnvVars +ExportCertData

    PassengerStatThrottleRate 120
    DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
    <Directory /usr/share/puppet/rack/puppetmasterd/>
        Options None
        AllowOverride None
        Order Allow,Deny
        Allow from All
    </Directory>

</VirtualHost>

Listen 8140
<VirtualHost *:8140>

    SSLEngine On
    SSLProtocol             All -SSLv2
    SSLCipherSuite          HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP
    SSLCertificateFile      /var/lib/puppet/ssl/certs/puppetmaster_host.pem
    SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/puppetmaster_host.pem
    SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem

    SSLCACertificateFile    /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
    SSLVerifyClient         optional
    SSLVerifyDepth          1

    SSLOptions              +StdEnvVars +ExportCertData

    RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
    RequestHeader set X-PUPPET-Client-DN %{SSL_CLIENT_S_DN}e
    RequestHeader set X-PUPPET-Client-Verify %{SSL_CLIENT_VERIFY}e

    PassengerStatThrottleRate 120
    DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
    <Directory /usr/share/puppet/rack/puppetmasterd/>
        Options None
        AllowOverride None
        Order Allow,Deny
        Allow from All
    </Directory>
</VirtualHost>

/etc/puppet.conf:

    logdir = /var/log/puppet
    rundir = /var/run/puppet
    ssldir = $vardir/ssl
    server = puppetmaster_host
    pluginsync = true
[agent]
    classfile = $vardir/classes.txt
    localconfig = $vardir/localconfig
    report = true
    environment = production
[master]
    reports = store,http,foreman,log
    reporturl = http://puppetmaster_host:3000/reports/upload
    storeconfigs = true
    #async_storeconfigs = true
    dbadapter = mysql
    dbuser = puppet
    dbpassword = f1rmwar3
    dbserver = localhost
    dbsocket = /var/lib/mysql/mysql.sock
    dbconnections = 10
    node_terminus = exec
    facts_terminus = yaml
    external_nodes = /usr/share/puppet/ext/susemanager_enc.rb

    ssl_client_header = HTTP_X_PUPPET_CLIENT_DN
    ssl_client_verify_header = HTTP_X_PUPPET_CLIENT_VERIFY

Remote Proxy

LoadModule headers_module /usr/lib64/apache2/mod_headers.so

Listen 8140

<VirtualHost *:8140>

    ServerName remote_proxy_host:8140
    SSLEngine on
    SSLProtocol             All -SSLv2
    SSLCipherSuite          HIGH:!ADH:RC4+RSA:-MEDIUM:-LOW:-EXP
    SSLCertificateFile /var/lib/puppet/ssl/certs/remote_proxy_host.pem
    SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/remote_proxy_host.pem

    SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
    SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
    SSLVerifyClient optional
    SSLVerifyDepth 1
    SSLOptions +StdEnvVars

    LogLevel warn

    RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
    RequestHeader set X-PUPPET-Client-DN %{SSL_CLIENT_S_DN}e
    RequestHeader set X-PUPPET-Client-Verify %{SSL_CLIENT_VERIFY}e

    RewriteEngine On
    TraceEnable Off
    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
    RewriteRule .* - [F]

    SSLProxyEngine on
    SSLProxyVerify require
    SSLProxyCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLProxyCheckPeerCN on

    SSLProxyMachineCertificateFile /var/lib/puppet/ssl/certs/remote_proxy_host.combined

    ProxyPass / https://puppetmaster_host:8141/
    ProxyPassReverse / https://puppetmaster_host:8141/

    ProxyPreserveHost On

    <Location />
        Order deny,allow

        allow from All

[Ruth]

Hi Karl,

I tried to copy your setup but I still get SSL errors in my puppet clients. I am not sure about the certificates, did you use the same certificates on Puppet Master and RP ? I would be grateful if you could share the results of your work,

Regards,
Marcella

[pdpinfo]

Hi,

I'm glad to hear good news,
and congrats because the setup is a bit tricky.
I noticed you enabled "SSLProxyMachineCertificateFile". I think that now the next step would be to enable "mandatory certificate checking" on puppetmaster_host of remote_proxy_host certificate.
I will try this configuration in the next future.
I guess it will need:

SSLVerifyClient require


and some variable checking;i.e. a compound expression, maybe working as the following:
SSLRequire (( ( %{SSL_CLIENT_S_DN_Email} in {"al...@example.com"} ) or ( %{SSL_CLIENT_S_DN_Email} in {"ali...@example.com"}) ) and ( %{SSL_CLIENT_V_REMAIN} > 0 ) and (( %{SSL_CLIENT_I_DN_CN} in {"CA Cert Signing Authority"}) or ( %{SSL_CLIENT_I_DN_CN} in {"CAcert Class 3 Root"}) ))
or at a minimum checking the client CN.

Let me know,

regards

Paolo

[kschafer2598]

Thanks - I turned it on for each vhost and it doesn't appear to cause any issue. From what you included about SSLVerify, does it mean that option is required for SSLVerifyClient to work properly?

Karl

[kschafer2598]

Hi There,

Yes I did use the same certificates on both. I also used an alternate DNS name for the RP so I created the cert with an alt_dns_name:

puppet cert generate remote_puppet_host --dns_alt_names remote_puppet_host_alt

That cert and private_key gets copied from puppetmaster cert directory to the RP, and used as the SSLCertificateFile and SSLCertificateKeyFile.

Otherwise all the other certs are from the puppetmaster's cert directory and copied over.

I also had lots of trouble with the header (RequestHeader) settings. The config I have here is what ultimately worked for me. Also ensure that the header settings in your puppet.conf is correct as well. on the puppetmaster.

Karl