how to scale puppet with F5 load balancer?

[Hai Tao]

How can I configure F5 load balancer to be infront of multiple puppet masters?

The SSL will break as the server name if different, hostname of the
VIP on the LB vs hostnames of each masters, right?

Can you shed some light?

Thanks.

--
Hai Tao

[Matthew Black]

The best I was able to do is when you generate the certificate for the
puppet master you use dns alt names flag to specify the alternative
names for the puppet master. That is the only way I was able to make
it work in my environment with the F5.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>

[Alan Evans]

From what I can tell there is no need to use alternate names. You can
make the F5 appear to the clients to be the puppetmaster by leveraging
the F5 to do SSL offloading and part of the certificate verification
taking some load off your puppet masters. Even more though, since the
puppet environments and other calls use pretty consistently organized
URI paths, you can do some really neat stuff with F5 HTTP Class
profiles to delegate certain requests to certain servers.

For example we have one server that acts as a CA, all signing requests
go there. Then for normal puppetmaster tasks requests are assigned to
a pool of several servers. Further as part of our development setup
we have separate environments for our main puppet module developers
but these only exist on one puppetmaster so we use the F5 to pick out
those requests and direct them to the correct server.

In general its a lot like configuring a reverse proxy (Apache, Nginx,
etc) in front of puppet.

Our setup is quite extensive but I didn't do most of it so I don't
want to lead anyone astray with an incomplete explanation. Let me see
if I can get some of the details together and make a new wiki page on
projects.puppetlabs.com or perhaps expand on the existing page.
(linked below)

Here's a starting point.
http://projects.puppetlabs.com/projects/puppet/wiki/Load_Balancing_F5

Some examples for reverse proxy w/ Apache
http://www.puppetmanaged.org/documentation/Reference_Guide-Appendices-Example_SSL_Frontend_Reverse_Proxy_Load_Balancer_Configuration.html
http://www.masterzen.fr/2010/03/21/more-puppet-offloading/

-Alan

[Paul.S...@barcap.com]

On Monday, 9 July 2012 06:44:16 UTC+1, Alan Evans wrote:

From what I can tell there is no need to use alternate names.  You can
make the F5 appear to the clients to be the puppetmaster by leveraging
the F5 to do SSL offloading and part of the certificate verification
taking some load off your puppet masters.  Even more though, since the
puppet environments and other calls use pretty consistently organized
URI paths, you can do some really neat stuff with F5 HTTP Class
profiles to delegate certain requests to certain servers.

+1 on Alan's post. This is exactly how we do it where I currently am.

Allows us to scale out the Puppet Master pool horizontally based on demand

and geographical location to keep hops to the minimum.

Using AltDNSNames would not make this flexible at all as you would need

to re-gen the cert each time.

This was all the clients have a single entry point:- puppet.<domain> and

the F5 takes the strain and sends them to the appropriate server and

routes around accordingly.

The only minor downside we have we this is that sometimes debugging the

route from the client to the eventual master means we have to go through

the logs on the potential masters to track down where it went. I do not have

direct access to the F5's.

 

[Matt]

The only issue is there is no real good guide on how to do this. Is there any more information that can be provided? What would really help is to see the F5 VS, F5 SSL Profile, and how the ssl key was generated.

[Matt]

Is there no one who has done this that can provide examples of how they did it?

[Paul.S...@barcap.com]

On Tuesday, 17 July 2012 16:14:05 UTC+1, Matt wrote:

Is there no one who has done this that can provide examples of how they did it?

 

I would if I could. Company policy and all that.

 

Generated the SSL certificate with the single F5 DNS address that all the clients globally will hit puppet.<domain> using puppet cert generate...

 

Put these on the F5 and have SSL terminate there and pass on anything /certificate/ to the pool of CA servers so they can be revoked/stored and what not - although we never will. The same certs are installed on these.

 

Anything without the /certificate/ regex will be passed from the F5 down to the puppet master pool which will contact a central ENC and do whatever it needs to do.

[Doug Chapman]

Someone from Puppetlabs should reach out to F5 and have them create a deployment guide.

http://www.f5.com/solutions/resources/deployment-guides.html

(selfishly interested in seeing this as well).

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/cAYOzqHDhNAJ.

[Eric Sorenson]

Great idea, thanks - I've mailed the in...@f5.com address to start this process but if anybody on list has a more specific contact at F5 I'd happily take this to them directly.

Eric Sorenson - eric.s...@puppetlabs.com

[Matt]

The issue I ran into that caused problems was the following

Puppet 2.7.14 for some reason on my environment sets the keylength by default to 4096. F5 LTM on at least 10.1 version can not support anything more than 2048 for both the cert on the F5 and the client cert for authentication the LTM will allow you to add the certificate but not apply to the SSL profile. The client ssl cert that each puppet agent sends if its greater than 2048 will instantly receive a TCP RST, the request to the puppet master will be still sent for catalog compile. More detail here http://support.f5.com/kb/en-us/solutions/public/12000/100/sol12147.html on the SSL key issue and what is affected and not.

Additional changes were required but this is what my non-ssl (what the F5 is proxying requests) looks like:

# Apache Configuration

<VirtualHost *:18140>

        ServerName puppetmaster1.example.com

        DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/

        RackBaseURI /

        <Directory /usr/share/puppet/rack/puppetmasterd/public/>

                Options None

                AllowOverride None

                Order allow,deny

                allow from all

        </Directory>

        SetEnvIf X-SSL-Subject "(.*)" SSL_CLIENT_S_DN=$1

        SetEnvIf X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1

        SetEnvIf X-Forwarded-For "(.*)" REMOTE_ADDR=$1

        SetEnvIf X-Forwarded-Proto "https" HTTPS=1

        LogLevel error

        ErrorLog "|/usr/sbin/cronolog /var/log/httpd/puppetmaster_error_log.%Y%m%d -l /var/log/httpd/puppetmaster_error_log"

        CustomLog "|/usr/sbin/cronolog /var/log/httpd/puppetmaster_access_log.%Y%m%d -l /var/log/httpd/puppetmaster_access_log" combined

</VirtualHost>

The SSL port (8140) is following the standard guide for apache passenger but with this three lines (like the non-ssl)

        SetEnvIf X-SSL-Subject "(.*)" SSL_CLIENT_S_DN=$1

        SetEnvIf X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1

        SetEnvIf X-Forwarded-For "(.*)" REMOTE_ADDR=$1

Here is what the F5 specific configuration looks like, I substituted the IP addresses and some of the names because of my environment.

pool puppet {

   lb method member least conn

   monitor all gateway_icmp

   members 192.168.1.10:18140 {}

}

virtual puppet {

   snat automap

   pool puppet

   destination 192.168.1.9:8140

   ip protocol tcp

   rules R_PUPPETMASTERS

   profiles {

      http {}

      puppet {

         clientside

      }

      tcp {}

   }

}

profile clientssl puppet {

   defaults from clientssl

   key "puppet.key"

   cert "puppet.crt"

   chain "puppetca.crt"

   ca file "puppetca.crt"

   client cert ca "puppetca.crt"

   renegotiate enable

   peer cert mode require

   authenticate always

}

# Slightly modified iRule based off of http://projects.puppetlabs.com/projects/puppet/wiki/Load_Balancing_F5

rule R_PUPPETMASTERS {

   when HTTP_REQUEST {

  HTTP::header insert "X-Forwarded-Proto" "https"

  set cert_request 0

  set path2 [URI::path [HTTP::uri] 2 2 ]

  if { $path2 == "/certificate/" || $path2 == "/certificate_request/" } {

    set cert_request 1

  }

}

when HTTP_REQUEST_SEND {

  if { $cert_request == 0}{

   clientside {

     if {[SSL::verify_result] == 0} {

        HTTP::header insert "X-Client-Verify" "SUCCESS"

     }

     HTTP::header insert "X-Client-DN" /[X509::subject [SSL::cert 0]]

     HTTP::header insert "X-SSL-Subject" /[X509::subject [SSL::cert 0]]

   }

  }

}

}

# end of F5 configuration

Hopefully this helps people who had similar issues that I had.

To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.

[Nan Liu]

On Tue, Jul 24, 2012 at 10:19 PM, Matt <mjb...@gmail.com> wrote:
> The issue I ran into that caused problems was the following
>
> Puppet 2.7.14 for some reason on my environment sets the keylength by
> default to 4096. F5 LTM on at least 10.1 version can not support anything
> more than 2048 for both the cert on the F5 and the client cert for
> authentication the LTM will allow you to add the certificate but not apply
> to the SSL profile. The client ssl cert that each puppet agent sends if its
> greater than 2048 will instantly receive a TCP RST, the request to the
> puppet master will be still sent for catalog compile. More detail here
> http://support.f5.com/kb/en-us/solutions/public/12000/100/sol12147.html on
> the SSL key issue and what is affected and not.

This was done because of #6663 security concerns, I think you can
modify the puppet keylength settings when generating keys.

This might be a bit meta, but would anyone be interested testing a
deployment using puppetlabs-f5 module so puppet can setup
load-balancing on F5? So far what your example doesn't seem too
complicated, and I put together a example gist (untested) and I think
we can get a puppet module to deploy loadbalancing for puppet masters:

https://gist.github.com/3174705

I know there was a suggestion on producing a deployment doc, but I
would be all for a puppet module instead.

Thanks,

Nan

[Matt]

 

This was done because of #6663 security concerns, I think you can
modify the puppet keylength settings when generating keys.

Hi Nan, I was just highlighting the limitation of the F5 LTM in versions prior to 10.2 since the issue is annoying hard to troubleshoot and was the source of my frustration. Part of the reason it was frustrating is that it allows you to upload certs that are greater than 2048 without error. This caveat with the SSL certs and the F5 LTM probably should be put up on the F5 load balance page in case someone else runs into the issue. I would not mind creating an example f5 configuration utilizing the puppetlabs-f5 module. 

The other apache changes are required to make it work correctly though, especially changing the REMOTE_ADDR environmental variable, I was not able to find an alternative and this was the quickest solution to the problem because the puppet master is verifying the CN of the cert against the IP of the client, which on the F5 is the F5 IP.

[Matt]

Oh yeah one more thing, when generating the ssl certs it has to be on the puppet.conf at the [main] block because when the client generates its SSL cert it will generate the key at the default keylength as well which as stated causes a problem with client authentication because of the F5 limitation.