Pulling my hair out with CA proxying

[Pete Hartman]

I am trying to establish what looks like a common pattern for scaling puppet. My main departure is that I'm using an F5 rather than an apache load balancer.  Namely, I want to have my puppet agents go through the F5 to a pool of "master only" systems, and any Certificate activity to get proxied by those masters through to one single Certificate Authority.  That CA system is not part of the F5 pool, it's role is to provide CA, Puppetdb and Postgresql.  It is configured as a master because that was the easiest way to get a CA stood up, but I don't intend to use it as a master in normal operation (and in fact I don't plan to have it hosting any modules).

I'm using RHEL 6, Apache, and Passenger, and Open Source Puppet.

I initially set up passenger using puppetlabs/passenger from the Forge, (which got me most of the way there but not fully configured).  All of these steps worked fine for the CA system to configure it as a working master (I have tested by registering systems with it, but then done puppet cert clean and wiped the test systems' ssl directories).

I then set up my first master-only system the same way, except I didn't actually start the master service (as the docs say) until after I had set ca = false and ca_server = $MY_CA_SERVER into /etc/puppet.conf.  I also made the necessary changes listed at http://docs.puppetlabs.com/guides/scaling_multiple_masters.html, including the certificate access on the CA system, the SSLProxyEngine on and ProxyPassMatch lines in the VHost definition in /etc/httpd/conf.d/puppetmaster.conf.  I'm positive I followed all the steps in the docs in order, but I'm not having any luck with external agents.

If I run puppet agent -t on the master-only system (with it's "server" in puppet.conf set to itself) it works fine--it can talk to the CA and talk to itself, and all is right with the world.

If I run puppet agent -t on a client host, pointing at the load balancer's address (or even pointing direclty at the master-only system's real hostname), I get:

[root@elmer ~]# puppet agent -t
Info: Creating a new SSL key for elmer.allstate.com
Error: Could not request certificate: Error 400 on SERVER: this master is not a CA
Exiting; failed to retrieve certificate and waitforcert is disabled


I've looked at the logs, enabled debug logging in the webserver with LogLevel, dug around everywhere I can think of, and I see no sign of any actual proxying going on.  tcpdump certainly shows no attempt by the master-only system to contact the CA.

What it LOOKS like is happening is that apache is not actually proxying anything, the request gets passed to the puppet master app running under passenger, and it (rightly) says "I'm not a CA" because /etc/puppet/puppet.conf says so.

I do not see any errors in the logs about proxy attempts failing for this agent.  I do see workers being attached for proxy purposes:

[Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1833): proxy: grabbed scoreboard slot 0 in child 27434 for worker https://caserver.allstate.com:8140/$1
[Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1852): proxy: worker https://caserver.allstate.com:8140/$1 already initialized
[Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1949): proxy: initialized single connection worker 0 in child 27434 for caserver.allstate.com)


I've repeatedly re-checked the settings in /etc/puppet.conf /etc/httpd/conf.d/passenger.conf, /et/chttpd/conf.d/puppetmaster.conf etc against the documentation and I am not seeing any errors.

This seems like I have to be overlooking something really basic, and I'm going to feel stupid when I find it, but it's right in my critical path right now and I can't see it.  Anyone have any suggestions?  I can provide config files and log files if need be, but I'm trying to avoid all the redacting I'd need to do (my server is not literally named "caserver" etc).

Thanks

Pete

[Pete Hartman]

I have to do more testing to determine for certain, but it appears to
have been some combination of
1) the order in which modules were loaded, and
2) not having mod_proxy_http loaded.

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/xY5xnOU09Qg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users...@googlegroups.com.
> To post to this group, send email to puppet...@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.

[Gavin Williams]

Pete

I've not done this before, however am familiar with Puppet, and know a lot more about F5s...

I note that you say that you're expecting apache on the masters to proxy onto the CA server.
Is there any reason you couldn't use the F5 to select the CA server for any CA requests?
Should be a fairly straight forward iRule to do pool selection based on the URI.

Thoughts?

Gav

[Pete Hartman]

I do not have responsibility for the F5's and I'm not sure what my
networking team would be willing to do in terms of custom rules no
matter how simple.

The use of the apache proxy service on the masters is a configuration
documented and recommended (at least as one alternative) by
PuppetLabs; now that I have found what I was missing, I plan to stick
with that.

[Felipe Salum]

Can you paste your /etc/httpd/conf.d/puppetmaster.conf ?

[Pete Hartman]

I tried to update this, but apparently failed.

Problem was my own misunderstanding of apache.

1) the passenger module was loaded before the proxy module, so the app
was responding before apache could proxy the request
2) I didn't recognize this as a working fix at first because I also
omitted mod_proxy_http which was needed in addition to mod_proxy


Thanks...

Pete