A Scalable HA Setup with on 2 configs, check this out !

[Ivan Arjune]

Did i figure out something new here, because I've been digging at this for a week and don't see anyone doing it like this. 

What i'm doing is running multiple puppetmasters behind haproxy.  Each puppetmaster is an active ca server and share a common certificate.  It works like a charm, in a lab.   

Step 1. created a common certificate that all the puppetservers will share.
Step 2. point webserver.conf to the shared certs.
Not a step 3. hit the masters through haproxy

I posted this up on ask.puppet.com a few days ago and nobody seems interest in it.  Either it's a stale forum, which i believe is true, or they think i'm crazy.  Maybe you do to, ugg....

Here is the orig. post with details on the setup.
Puppet CA Shared Certificate Guide: Scalable Puppet?

I'm looking to put this into production on an infra. with around 200 nodes.  I think it's a good idea, but can't figure out why I don't see anyone doing it like this yet.

Million dollar question:
Why must i use a centralized the ca server?

[Peter Kristolaitis]

Serial numbers on SSL certificates are important, and your setup will generate many duplicate serial numbers.  Ergo, this is bad.

Related problem:  Did you test revoking a client certificate?  I suspect not, because the above issue will bite you.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/6dcd4a20-909c-4373-892f-0f7a3e69d19d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Neil - Puppet List]

Hello

I've run multiple puppet masters behind ha proxy for a few years now. I have multiple masters, with haproxy rules directing some clients to particular masters. I only have one puppet master as CA. I've about 600 clients.

Initially I was concerned about only having one CA. But all it does is sign new clients and revoke old. Haproxy trusts the clients based on this CA and a revoke list from the CA.

If the CA went down all existing clients would are fine, I've tested that. I can't sign new clients or revoke existing until I recover the CA but in my environment that's no big deal. I have backups of the CA and a new one would not take long to spin up.

So I wonder why you want multiple CA. What benefits would it bring?

Happy to share example haproxy config etc if you are interested.

Cheers,

Neil

On 12 Sep 2016 16:07, "Peter Kristolaitis" <alt...@alter3d.ca> wrote:

Serial numbers on SSL certificates are important, and your setup will generate many duplicate serial numbers.  Ergo, this is bad.

Related problem:  Did you test revoking a client certificate?  I suspect not, because the above issue will bite you.

On 2016-09-12 12:48 AM, Ivan Arjune wrote:

Did i figure out something new here, because I've been digging at this for a week and don't see anyone doing it like this. 

What i'm doing is running multiple puppetmasters behind haproxy.  Each puppetmaster is an active ca server and share a common certificate.  It works like a charm, in a lab.   

Step 1. created a common certificate that all the puppetservers will share.
Step 2. point webserver.conf to the shared certs.
Not a step 3. hit the masters through haproxy

I posted this up on ask.puppet.com a few days ago and nobody seems interest in it.  Either it's a stale forum, which i believe is true, or they think i'm crazy.  Maybe you do to, ugg....

Here is the orig. post with details on the setup.
Puppet CA Shared Certificate Guide: Scalable Puppet?

I'm looking to put this into production on an infra. with around 200 nodes.  I think it's a good idea, but can't figure out why I don't see anyone doing it like this yet.

Million dollar question:
Why must i use a centralized the ca server?

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/6dcd4a20-909c-4373-892f-0f7a3e69d19d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/c5dbbb59-4de7-720f-3424-3135db424522%40alter3d.ca.

[Gareth Rushgrove]

On 17 September 2016 at 15:06, Neil - Puppet List

<maillis...@iamafreeman.com> wrote:
> Hello
>
> I've run multiple puppet masters behind ha proxy for a few years now. I have
> multiple masters, with haproxy rules directing some clients to particular
> masters. I only have one puppet master as CA. I've about 600 clients.
>
> Initially I was concerned about only having one CA. But all it does is sign
> new clients and revoke old. Haproxy trusts the clients based on this CA and
> a revoke list from the CA.
>
> If the CA went down all existing clients would are fine, I've tested that. I
> can't sign new clients or revoke existing until I recover the CA but in my
> environment that's no big deal. I have backups of the CA and a new one would
> not take long to spin up.
>
> So I wonder why you want multiple CA. What benefits would it bring?
>
> Happy to share example haproxy config etc if you are interested.
>

I'd certainly be interested in the example if you don't mind.

Gareth

>> email to puppet-users...@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-users/6dcd4a20-909c-4373-892f-0f7a3e69d19d%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to puppet-users...@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-users/c5dbbb59-4de7-720f-3424-3135db424522%40alter3d.ca.
>> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/CAAohVBfGJx14uUqocAXPw7oJvBdVsenQhE4rjDSNCXCwjM94Vg%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.



--

Gareth Rushgrove
@garethr

devopsweekly.com
morethanseven.net
garethrushgrove.com

[Neil - Puppet List]

Hello

Below is a slightly edited version of the haproxy.cfg

All the backends except the ca require a valid client cert 'http-request  deny unless { ssl_c_verify 0 }'

global

  chroot  /var/lib/haproxy

  daemon

  group  haproxy

  log  127.0.0.1 local4

  log  127.0.0.1 local5 notice

  maxconn  20000

  pidfile  /var/run/haproxy.pid

  stats  socket /var/run/haproxy.stat mode 600

  tune.ssl.default-dh-param  2048

  user  haproxy

defaults

  log  global

  maxconn  20000

  option  redispatch

  retries  3

  timeout  http-request 10s

  timeout  queue 1m

  timeout  connect 10s

  timeout  client 1m

  timeout  server 1m

  timeout  check 10s

frontend hastats

  bind 0.0.0.0:443 ssl no-sslv3 crt /etc/ssl/private/puppet.lse.ac.uk.pem no-sslv3 ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:AES:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK

  default_backend  ibe_hastats

  mode  http

  option  httplog

  rspadd  Strict-Transport-Security:\ max-age=31536000

frontend puppet

  bind 0.0.0.0:8140 ssl no-sslv3 crt /etc/ssl/private/puppet.lse.ac.uk.pem ca-file /etc/haproxy/ca_crt.pem verify optional crl-file /etc/haproxy/ca_crl.pem ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:AES:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK

  acl  use_ca path_reg ^/[a-z0-9\-\.]*/certificate/

  acl  use_ca path_reg ^/[a-z0-9\-\.]*/certificate_request/

  acl  use_dev ssl_c_s_dn(cn) -m sub -- -dev

  acl  use_foreman ssl_c_s_dn(cn) -m beg testforemanclient

  acl  environment_production path_beg /production/catalog

  default_backend  be_puppet_stable

  http-request  set-header X-SSL                   %[ssl_fc]

  http-request  set-header X-SSL-Client-Verify     %[ssl_c_verify]

  http-request  set-header X-SSL-Client-SHA1       %{+Q}[ssl_c_sha1]

  http-request  set-header X-SSL-Client-DN         %{+Q}[ssl_c_s_dn]

  http-request  set-header X-SSL-Client-CN         %{+Q}[ssl_c_s_dn(cn)]

  http-request  set-header X-SSL-Issuer            %{+Q}[ssl_c_i_dn]

  http-request  set-header X-SSL-Client-Not-Before %{+Q}[ssl_c_notbefore]

  http-request  set-header X-SSL-Client-Not-After  %{+Q}[ssl_c_notafter]

  mode  http

  option  forwardfor

  option  httplog

  use_backend  be_puppet_ca if use_ca

  use_backend  be_puppet_dev if use_dev

  use_backend  be_puppet_foreman if use_foreman

backend be_puppet_ca

  mode  http

  server  sys-puppet-ca-prod-0 sys-puppet-ca-prod-0:8140 check inter 15s rise 2 fall 2

backend be_puppet_dev

  balance  source

  hash-type  map-based

  http-request  deny unless { ssl_c_verify 0 }

  mode  http

  server  sys-puppet-app-prod-0 sys-puppet-app-prod-0:8140 check inter 15s rise 2 fall 2

backend be_puppet_foreman

  balance  source

  hash-type  map-based

  http-request  deny unless { ssl_c_verify 0 }

  mode  http

  server  sys-puppet-app-prod-1 sys-puppet-app-prod-1:8140 check inter 15s rise 2 fall 2

backend be_puppet_stable

  balance  source

  hash-type  map-based

  http-request  deny unless { ssl_c_verify 0 }

  mode  http

  server  sys-puppet-app-prod-2 sys-puppet-app-prod-2:8140 check inter 15s rise 2 fall 2

backend ibe_hastats

  mode  http

  stats  uri /hastats/

  stats  realm HAStatistics

  stats  auth admin:PASSWORDFORADMINACCESSTOSTATSPAGE

  stats  admin if TRUE

On 18 September 2016 at 09:10, Gareth Rushgrove <gar...@morethanseven.net> wrote:

On 17 September 2016 at 15:06, Neil - Puppet List

>> email to puppet-users+unsubscribe@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-users/6dcd4a20-909c-4373-892f-0f7a3e69d19d%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to puppet-users+unsubscribe@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-users/c5dbbb59-4de7-720f-3424-3135db424522%40alter3d.ca.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to puppet-users+unsubscribe@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/CAAohVBfGJx14uUqocAXPw7oJvBdVsenQhE4rjDSNCXCwjM94Vg%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.



--
Gareth Rushgrove
@garethr

devopsweekly.com
morethanseven.net
garethrushgrove.com

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAFi_6yLdmbE9QZ2%2BZ-OpuFGyt-mNRF5n2NuQ21SFiNTyTEbA0g%40mail.gmail.com.

[Neil - Puppet List]

Hello

One extra thing to mention is I have got into issues with configuring the loadbal itself through puppet, as broken loadbal config breaks the puppet service which means the loadbal can;t be fixed via puppet, so admin login is required on these servers.

Thanks

Neil

[Trevor Vaughan]

Hi Neil,

Thanks for sharing that config, it's quite useful.

Did you see any large benefit of this versus using DNS SRV records (yes, I understand the actual load balancing implications).

I'm curious if the extra infrastructure was worth the effort.

I'm partial to a fan-out DNS SRV structure, but that doesn't really help with load unless your servers are active rejecting above a given connection load.

Thanks,

Trevor

On Mon, Sep 19, 2016 at 5:09 AM, Neil - Puppet List <maillis...@iamafreeman.com> wrote:

Hello

One extra thing to mention is I have got into issues with configuring the loadbal itself through puppet, as broken loadbal config breaks the puppet service which means the loadbal can;t be fixed via puppet, so admin login is required on these servers.

Thanks

Neil

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAAohVBfC-E%2BXp5_td2j0n9EW4rcKPcx47kxYJ4tT8sjpN%3DSBkw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

[Neil - Puppet List]

Hello Trevor,

I put this in when we did a fairly big puppet upgrade. This meant I could direct a few clients to the upgraded server upgrade the agents see how that went then do all dev service before moving others.

I guess we could have done that in a number of other ways but this worked well for me and I like not configuring the clients differently, by changing their server setting, as I did the upgrade

In a similar manner I've used it to experient with different node classifiers not something I think DNS records would allow.

Before I used environments I used it for other changes like upgrading modules such as puppet labs Apache by having the newer version only on one puppet server and directing some clients there.

So in general the benefit is when you do not want the same puppet version or code on all puppet servers.

Overhead of running a pair of tiny vms for the loadbal is tiny for me as we run a dozen or so other loadbal pairs.

Neil

On 19 Sep 2016 21:28, "Trevor Vaughan" <tvau...@onyxpoint.com> wrote:

Hi Neil,

Thanks for sharing that config, it's quite useful.

Did you see any large benefit of this versus using DNS SRV records (yes, I understand the actual load balancing implications).

I'm curious if the extra infrastructure was worth the effort.

I'm partial to a fan-out DNS SRV structure, but that doesn't really help with load unless your servers are active rejecting above a given connection load.

Thanks,

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CANs%2BFoX3jnRD6O%2BNUKNyrmiN9vMeQNt986oeV6DqRCxGoS9uNA%40mail.gmail.com.

[Ivan Arjune]

I did some testing and yes if you reinstall a host and don't perform puppet cert clean on all the ca servers you will run into issue.  But other than this fact I haven't seen any issues with masters verifying a signed certificate.  Since running puppet cert clean is standard procedure I don't see this as a complicated task to distribute when re-provisioning .

[Ivan Arjune]

Neil,

I agree one CA is more than capable and the load balancing point here is pretty much moot.  I as well will have many nodes dispersed world wide within DC's and with Hosting providers, like AWS and DO.  Having a flexible and simple setup which can operate independent of other sites is a requirement.  We will be building and tearing down nodes frequently so having zero downtime with the provisioning and CM services is also a requirement here.   Aside from the simple puppetserver / ca config the haproxy setup i'm running is very straight forward.  With the shared certificate I can call all masters with the same name and the puppet web server points to the correct cert. 

I honestly haven't encountered the problem everyone says exists with active/active CA's.

A CA's job is to sign new certificates.  When a node is toredown the cert will be wiped at all CA's so in the event the hostname is reused there shouldn't be a problem. 

Where does the problem arise with serial number conflicts?  How can i reproduce this issue?

Also is the traffic from your haproxy to the masters not using ssl?

[Ivan Arjune]

This is what my haproxy cfg is like.  I'm doing tcp passthough.

global
  chroot  /var/lib/haproxy
  daemon 
  group  haproxy

  log  10.0.2.15 local0
  maxconn  4000
  pidfile  /var/run/haproxy.pid
  stats  socket /var/lib/haproxy/stats

  user  haproxy


defaults
  log  global

  maxconn  8000
  option  redispatch
  retries  3
  stats  enable

  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

listen puppetserver
  bind 192.168.56.110:8140
  mode tcp
  balance roundrobin
  option tcplog
  server puppetserver-02.vm 192.168.56.109:8140
  server puppetserver-01.vm 192.168.56.111:8140