SSL Makes My Brain Bleed

[Paul Lathrop]

Hi guys,

Really could use some help with the way Puppet uses SSL. In my
environment, I need to have 2 puppetmasters. One of them is
responsible for passing out configurations to production machines, the
second one is responsible for passing out configurations to
development machines. When new systems are built, they have a default
puppet.conf which points them at the production puppetmaster. They
retrieve their config, which includes a puppet.conf that makes sure
they are pointed at the appropriate puppetmaster. I have 3
requirements in this setup.

1) I cannot be generating SSL certs by hand. Autosign is a must.
2) The development puppetmaster has to function as a client of the
production puppetmaster.
3) Nodes which switch from the production puppetmaster to the
development puppetmaster must continue functioning without SSL
complaining.

In 0.24.8 I did this, but it was basically opaque magic to me. I
simply copied over /var/lib/puppet/ssl from the one puppetmaster to
the other before starting any of the puppet daemons, and this worked
out happily.

Now, as I try to set up new 0.25.1 puppetmasters, this is no longer working.

I copied the /var/lib/puppet/ssl/ca directory from my original 0.24.8
puppetmaster, and all my 0.24.8 clients happily switched to the new
0.25.1 puppetmaster. That part was easy. However, nothing I can do
will convince the second 0.25.1 puppetmaster to work as outlined in
the requirements list above.

I've consulted the IRC channel, and learned much about SSL, all to no avail.

Happy to provide config files as needed to help me figure this out.

--Paul

P.S. I would love to buy support for this, but that decision isn't up
to me at the moment.

[Douglas Garstang]

Seems like there are LOTS and LOTS of problems with puppet 0.25.
I eventually gave up. I could simply NOT get it to work.

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.g...@gmail.com
Cell: +1-805-340-5627

[Avi Miller]

Hey,

Douglas Garstang wrote:
> Seems like there are LOTS and LOTS of problems with puppet 0.25.
> I eventually gave up. I could simply NOT get it to work.

Just as another statistical point: I have a 0.25.1 Puppet Master running
on Passenger 2.2.2 with 140 0.24.8 and about 40 0.25.1 clients and they
all work swimmingly. In fact, with 0.25.1, our compile times with
thin_storeconfigs is down to subseconds again, where before with 0.24.8
it would take a good 10-20 seconds.

We now also have Foreman running via Passenger, so I've even fixed the 8
odd clients that had been broken by errant sysadmins.

So, I'm not sure there are LOTS and LOTS of problems.

cYa,
Avi

[Dan Bode]

Hi Paul,,

I just want to share how I have done this before.

1. Production server is the only certificate authority.
2. Development server sets ca_server = false
3. Development server calls puppetd --server production.server
4. Development server now gets a copy of the production servers certificate (ca.pem)
5. Other machines must get signed by the prod server before they can call the dev server (there is a ca_server command line argument)

the puppet dev server ensures that any calling machines have been signed by the production server (its ca).

Can you try this setup and see if it resolves your issue?

There is another thread of people discussing passenger issues. I will go ahead and stage the passenger config with 25.1 this weekend. I will make a post outlining my findings.

hope this helps,

Dan

[James Turnbull]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Douglas Garstang wrote:
> Seems like there are LOTS and LOTS of problems with puppet 0.25.
> I eventually gave up. I could simply NOT get it to work.

Douglas

Can you list out the issues you had? Did you log tickets for them?
If not, could you please do so and adding platform and log/trace
output to the tickets that'd also greatly help.

I'm aware of a number of sites running 0.25.1 without issue and I am
sure we can get you there too!

Regards

James Turnbull

- --
Author of:
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBSvUSHyFa/lDkFHAyAQJlFgf+J925aoJP7IT2sksLo3Gf/BLyN0t/jXY1
kYuhbQ6kOwsVc+cZJl/SQ6biw6nrI43D38audxMywrwTU0XADgVDBDApQ/gzX56E
+T8ah4N5DHseb1P73ZRddrq+NfA+F6Khdjri38waSMLDqigqWUcyy78Cu3uwtnNO
6fJgIoA79JXeojNyTHYrV3ogTqKdh4IjKE7e2dlJ+1DtvGcitb8v+Vw8oMzJ9P6A
UhFgPpCyCR4T+I3ds1nWWKM7RtbECEixGfHGh/graRoTf5CWBfYhfTkkS3mBclOF
HhCG5Mp6IEpYnoEeVE0L4NOdVicFaUn/8L0ToD7sK0L6AwrThd38pA==
=txRC
-----END PGP SIGNATURE-----

[Ohad Levy]

+1 for this setup for your scenario.

I ended removing all of the ca functionality from all of the dev severs, it makes things much simplified.

additionally you need to sign only in one place, which makes it easy to automate it with a common build infrastructure.

Cheers,
Ohad

[Douglas Garstang]

Sorry James... by the time I tossed in the towel I was a quivering sweaty blob.

Doug.

[Paul Lathrop]

Dan,

This looked like exactly what I needed, but I couldn't get this to
work either. I'm at my wits end and have given up completely on the
upgrade to 0.25.1 at this point.

I will now go pester my boss to buy support.

--Paul

[Dan Bode]

 Hi Paul,

I am currently (but unfortunately  slowly) staging this, so far I have done the following:

1. installed .24.8 with passenger, tested
2, Upgraded to .25.1(head from git)
3. Reconfigured passenger
4. Tested client/server on one machine
5. Upgraded additional client machine to.25.1
6. Verified that client can get ssl keys and run puppetd against the server

Can you get this far? Can you upgrade the CA server and connect client?

My next step is to stage all of this with 2 puppetmasters using the methods explained above.

I will make a post once I get it working (its hard to find the time though, this will take some effort).


regards,

Dan

[Paul Lathrop]

Hi Dan,

I'm not upgrading, I'm installing a server from scratch. I want to
upgrade just my puppetmaster and leave the clients alone for now,
because Puppet is a huge part of our infrastructure I can't do it all
in one pass.

I can get a single puppetmaster up and running. I cannot then get a
second puppetmaster to be a client of the first puppetmaster.

I also, at this point, can't roll back because SSL is totally screwed
in my infrastructure now. So, I'm in fire-fighting mode.

Thanks for your help, but I'm definitely done with this.

--Paul

[Mark Christian]

I am keen to get this to work, but can't seem to. Will this work with
Mongrel and Apache as described at http://reductivelabs.com/trac/puppet/wiki/UsingMongrel
? Im using the EPEL puppet package versions 24.8-4 and simply can't
get the client to retrieve the catalog from the "Development" server.

Could not retrieve catalog: Certificates were not trusted: tlsv1 alert
unknown ca

The clients have all been signed by the "Production" server, I've put
ca_server = myProduction.Server.com in the client's puppet.conf.

Is there any more detail to add to step 5. below?

Thank you.

[Hubert Krause]

Hello Mark Christian,

Am Thursday 19 November 2009 03:10:38 schrieb Mark Christian:
> I am keen to get this to work, but can't seem to. Will this work with
> Mongrel and Apache as described at
> http://reductivelabs.com/trac/puppet/wiki/UsingMongrel ? Im using the EPEL
> puppet package versions 24.8-4 and simply can't get the client to retrieve
> the catalog from the "Development" server.
>
> Could not retrieve catalog: Certificates were not trusted: tlsv1 alert
> unknown ca

I' ve run into ssl Problems too a few days before. Check the following:

"hostname -f" on the client and "host <client-ip-adress>" on the server are
the same?

"openssl s_client -showcerts -connect myProduction.Server.com:8140" shows a
line like:

"subject=/CN=myProduction.Server.com"

If there is some mismatch, configure hostnames that they will work and
recreate ssl certs by removing /var/lib/puppet/ssl/* and run puppetmaster (or
puppetd on the client) to recreate certs. (at first the server of course).

If this test show no Problems I have no idea...

Hope that helps,

Hubert

--
Hubert Krause
Risk & Fraud Division
INFORM GmbH, Pascalstraße 23, 52076 Aachen, Germany
Phone: +49 24 08 - 94 56 188
E-Mail: hubert...@inform-ac.com, Web: http://www.inform-ac.com
INFORM Institut fuer Operations Research und Management GmbH
Registered AmtsG Aachen HRB1144 Gfhr. Adrian Weiler

[Atha]

Hi Mark,
I can understand your frustration. We have been struggling with Puppet
and SSL a lot lately. Our setup is similar but a bit more complicated
so your scenario shouldn't pose any issues. Try this (assuming you are
starting from scratch):

1. Start the production puppet master as usual. This will be your CA.
2. In your development puppet master, set ca = false and ca_server =
production.hostname in puppet.conf in the puppetmasterd section. Also
set server = production.hostname in the puppetd section.
3. In your development puppet master, run puppetd first! This is
needed to generate the certificates and request the CA to sign them.
If you start puppetmasterd first it will fail.
4. Sign the development puppet master certificate on your production
puppet master or set autosign.
5. Once the certificate is signed, re-run puppetd to verify.
6. Start the development puppet master.
7. In your development clients, set ca_server = production.hostname to
have the production puppet master sign their certificates since its
your only CA.

Cheers,
Atha

On Nov 18, 11:10 pm, Mark Christian <supertr...@gmail.com> wrote:
> I am keen to get this to work, but can't seem to.  Will this work with

> Mongrel and Apache as described athttp://reductivelabs.com/trac/puppet/wiki/UsingMongrel

[Peter Meier]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi

> 1. Start the production puppet master as usual. This will be your CA.
> 2. In your development puppet master, set ca = false and ca_server =
> production.hostname in puppet.conf in the puppetmasterd section. Also
> set server = production.hostname in the puppetd section.
> 3. In your development puppet master, run puppetd first! This is
> needed to generate the certificates and request the CA to sign them.
> If you start puppetmasterd first it will fail.
> 4. Sign the development puppet master certificate on your production
> puppet master or set autosign.
> 5. Once the certificate is signed, re-run puppetd to verify.
> 6. Start the development puppet master.
> 7. In your development clients, set ca_server = production.hostname to
> have the production puppet master sign their certificates since its
> your only CA.

would be awesome if you could document that on the wiki.

cheers pete
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksK+QAACgkQbwltcAfKi3/qiACgioznQvrbmf6jbhJKajqaaLOx
p9kAnjLvuFNhG5jbcEShnjUiCjCuINyt
=Jf/T
-----END PGP SIGNATURE-----

[Atha Kouroussis]

Hi Pete,
I was thinking of doing that since all the entries in the wiki addressing Puppet Scalability deal with multiple CAs which in my opinion overly complicates things. I am also waiting on a resolution for #2848 which arose from this kind of setup.

Cheers,
Atha

> --
>
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=.
>
>

[Ohad Levy]

there are a few very good SSL recipe wiki pages - one of them that I wrote about SSL authentication chains - http://reductivelabs.com/trac/puppet/wiki/PuppetScalability under the section Centralized Puppet Infrastructure

Ohad

[Paul Lathrop]

On Fri, Nov 6, 2009 at 4:49 PM, Paul Lathrop <pa...@tertiusfamily.net> wrote:

I just wanted to follow up and let you guys know how I handled this
(with help from Luke).

First, I set up the production puppetmaster. Once the production
puppetmaster was running correctly and serving clients, I used
puppetca --certdnsnames "<insert names here>" --generate to generate
the signed cert/key files for the development puppetmaster. These
ended up under $ssldir for me, so I had to copy them out of there to
the development puppetmaster.

On the development puppetmaster I started with a clean $ssldir with
nothing in it except the CA cert and the signed cert/key files I
generated above (in their proper subdirectories). I put ca = false in
puppet.conf, and started the development puppetmaster.

Finally, on clients, I set ca_server = <production_puppetmaster>

This works well for me.

--Paul