what's the google's pupetmaster structure ?

[huangmingyou]

hi,guys
google use pupet manage 6000 puppet clients. so ,I think the
puppetmaster will be a cluster or other someting ?
who knows ? share it ,I think not just only I interest
it.

huang mingyou

[Nigel Kersten]

We're not the only company on this list trying to strike an appropriate balance between wanting to be good members of the community and yet needing to also do the right thing by our company in terms of releasing appropriate info :)

Look for something to be published relatively soon, I've had preliminary clearance from our relevant people, we just need to write it for final approval and you'll see something on the wiki.

If you have specific questions about scaling puppet, I'm sure that a few of us in the community can help answer them.

--
Nigel Kersten
Systems Administrator
MacOps

[huangmingyou]

: D waiting for you update the wiki.

now, in the wiki I only found this, http://www.reductivelabs.com/trac/puppet/wiki/PuppetScalability
and it's not so complete !

On Apr 29, 11:19 am, "Nigel Kersten" <nig...@google.com> wrote:

[Ohad Levy]

if you have any questions in the meanwhile - feel free to ask!

I guess the hardest part about scaling up would be the certificate managements.
if you have a good version control system enabled, all the rest should not be a big deal....

Ohad

[huangmingyou]

yes ,the certfificate managements is a hard work.
and desgin a struct like this
http://www.flickr.com/photos/26121547@N07/2451219297/
but not yet have time to test it. I want to know some other pepole's
good idea :>

include pupet+mongrel+apache group, mysql cluster, cluster
filesystem, and use svk+svn version control.

On Apr 29, 8:13 pm, "Ohad Levy" <ohadl...@gmail.com> wrote:
> if you have any questions in the meanwhile - feel free to ask!
>
> I guess the hardest part about scaling up would be the certificate
> managements.
> if you have a good version control system enabled, all the rest should not
> be a big deal....
>
> Ohad
>

[Nigel Kersten]

Exactly. And remember you can separate the server from the ca_server... and then there really isn't much to keep in sync between the non-ca servers,

I'm not sure of the current state of certificates when automatically set up by puppet. We ended up rolling them by hand, but all you have to do is make sure all the certs are signed by the same CA, and if you stick with one ca_server then you have a single authoritative store for certificates and don't have to worry about certname namespace collisions.

Otherwise you're going to have to either put certificates on a shared filesystem or somehow keep them in sync and work out how to handle duplicate certnames.

This is on the wiki, but Pound works better under heavy load than Apache does in my experience.

Serving large files via puppet:/// urls will have an impact upon performance as they need to be escaped before sending  to the clients.

[huangmingyou]

good, if no certificates sync ,it will be make this structure to be
simple . yes, use puppet:/// share most file or large file is not so
good. so I need find a good idea to distribute large or most file to
puppet client . but I had not find yet :( ,

if can't find good solution ,I think I will test the puppet fileserver
group (cluster)+ dns round loop.

hmy

On Apr 29, 10:21 pm, "Nigel Kersten" <nig...@google.com> wrote:
> Exactly. And remember you can separate the server from the ca_server... and
> then there really isn't much to keep in sync between the non-ca servers,
>
> I'm not sure of the current state of certificates when automatically set up
> by puppet. We ended up rolling them by hand, but all you have to do is make
> sure all the certs are signed by the same CA, and if you stick with one
> ca_server then you have a single authoritative store for certificates and
> don't have to worry about certname namespace collisions.
>
> Otherwise you're going to have to either put certificates on a shared
> filesystem or somehow keep them in sync and work out how to handle duplicate
> certnames.
>
> This is on the wiki, but Pound works better under heavy load than Apache
> does in my experience.
>
> Serving large files via puppet:/// urls will have an impact upon performance
> as they need to be escaped before sending to the clients.
>
>
>
> On Tue, Apr 29, 2008 at 5:13 AM, Ohad Levy <ohadl...@gmail.com> wrote:
> > if you have any questions in the meanwhile - feel free to ask!
>
> > I guess the hardest part about scaling up would be the certificate
> > managements.
> > if you have a good version control system enabled, all the rest should not
> > be a big deal....
>
> > Ohad
>

[Nigel Kersten]

DNS round robin is certainly an option.

Remember if you have something like Pound in front, and your concerns are more related to load distribution than redundancy, you can always have a single Pound instance that load balances across multiple Puppet servers. You just need to bind the mongrel instances to the public IP rather than localhost (which may or may not be a security risk in your environment)

[huangmingyou]

thank you very much ,Nigel Kersten. I'll redesign the structure
then test it and share it :)

[Joel Wood]

On the DNS round robin bit, if you also want some redundancy with your
load distribution you could always use wackamole. It allows you to share
a pool of virtual ips between a number of hosts. You can use it with DNS
round robining to give you failover of sorts.

http://www.backhand.org/wackamole/

-Joel

[Ohad Levy]

Hi,

Are you using the same methods for multi site setup, or is it mainly focused on a single big site architecture (or low latency etc..)

I'm asking, because I found that having a remote puppet master in an 100+ms environment is not that good of a solution...

We handled the certificate management in a different way, since all of our puppet masters are also puppet clients, there is a hierarchy of certificates, but each puppet master can still sign the clients below, and as its trusted, each client can connect to any other puppet master.

Ohad

[huangmingyou]

all puppet clients in muti data center, even cross china and us. so
it's a challenge for me :)

[Evan Hisey]

>
> We handled the certificate management in a different way, since all of our
> puppet masters are also puppet clients, there is a hierarchy of
> certificates, but each puppet master can still sign the clients below, and
> as its trusted, each client can connect to any other puppet master.
>
> Ohad
>

This is interesting. Can you give a bit more detail on this setup?

Evan

[Paul Lathrop]

On Tue, Apr 29, 2008 at 5:41 PM, Ohad Levy <ohad...@gmail.com> wrote:
> We handled the certificate management in a different way, since all of our
> puppet masters are also puppet clients, there is a hierarchy of
> certificates, but each puppet master can still sign the clients below, and
> as its trusted, each client can connect to any other puppet master.

That sounds like a really interesting and useful setup. Would you be
willing to share more specifics? How did you set this up?

Regards,
Paul

[Jeremy Pruitt]

I would also love to get more details on this setup. I have tried
following the Multiple Certificate Authorities page on the wiki but to
no success.

- Jeremy

On Apr 30, 10:08 am, "Paul Lathrop" <p...@tertiusfamily.net> wrote: