a complete solution for puppet

[Hai Tao]

Hi,

I notice that many components of puppet do not scale well and are not
intended for large environment. For example, stored config and
inventory service. In order to scale, we need to use puppetDB, right?
Another example is the webrick, and which should be replaced by a
decent web server such as apache. All these need a lot of new
installation of pieces of software and configurations.

My question is why the designer of puppet did not consider this and
integrate everything into a complete solution at the beginning, rather
than having us have to reconfigure everything by hand. Who will use
puppet if he has only 50 nodes?

--
Hai Tao

[Christopher Wood]

Sounds like you should be talking to your managers about buying Puppet Enterprise.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
>

[Hai Tao]

I see. so it is on purpose to make it not easy to use so the
enterprise can be sold? :)

--
Hai Tao

[Stefan Schulte]

On Wed, Jul 25, 2012 at 02:00:37PM -0700, Hai Tao wrote:
> Hi,
>

[...]

>
> My question is why the designer of puppet did not consider this and
> integrate everything into a complete solution at the beginning, rather
> than having us have to reconfigure everything by hand. Who will use
> puppet if he has only 50 nodes?
>

You probably want Puppet Enterprise as it comes with one installer for
all these dependencies

http://puppetlabs.com/puppet/faq/

-Stefan

[Christopher Wood]

On Wed, Jul 25, 2012 at 02:20:17PM -0700, Hai Tao wrote:
> I see. so it is on purpose to make it not easy to use so the
> enterprise can be sold? :)

There are different skill levels at different tasks in the enterprise space, and it is legitimate that some organizations are better off with a prefabbed installer for a configuration management system.

I've created a puppet installation of reasonable complexity without puppet enterprise, but that is possibly just me:

$ cd files/puppet/svn/prod/trunk
$ ls manifests/nodes | wc -l
43
$ find modules -name "*pp" | wc -l
174

That's not to say I don't salivate a bit at the thought of Puppet Enterprise, but my budget of $0 doesn't help there. Or perhaps a career-long $0 budget has helped, in that I'm more used to building from components instead of buying the package. People who are more used to buying than building may be better off with a different situation than mine.

[Jakov Sosic]

On 07/25/2012 11:20 PM, Hai Tao wrote:
> I see. so it is on purpose to make it not easy to use so the
> enterprise can be sold? :)

If you don't know how to migrate puppet to passenger (either on nginx or
apache), then I don't understand how come that you are able to configure
your machines through puppet manifests?!? It ain't that hard.



--
Jakov Sosic
www.srce.unizg.hr

[Jakov Sosic]

On 07/25/2012 11:25 PM, Stefan Schulte wrote:

> You probably want Puppet Enterprise as it comes with one installer for
> all these dependencies
>
> http://puppetlabs.com/puppet/faq/

Is it a bundle with everything prepackaged? Or does it use system's
httpd for example?

I hate bundles :D


--
Jakov Sosic
www.srce.unizg.hr

[Ohad Levy]

You could try using foreman installer(aka as plain puppet modules)[1], which does take care for the following

1. apache configuration + passenger

2. puppet master (with optional dynamic git branches / environments)

3. foreman stuff

if you don't need storeconfigs, then you don't have to use it.

if you need to scale to more than one master, my guess is that the main thing to overcome is the CA setup (single CA server etc) and thats the kind of knowledge you need to have regardless of an installer.

Ohad

[1] https://github.com/theforeman/foreman-installer#readme

[James A. Peltier]

I'm sure that the designers of puppet made every effort to make puppet as usable and scalable as they could when it was being designed. This is proven time and time again but the sites that currently have puppet deployed with tens of thousands of nodes. You should take care when making such bold statements, but I assume of course that it was not you intention to insult anyone.

We originally rolled puppet out with 10 machines. Managing 10 machines without puppet was 10 times more work than managing 10 machines with puppet. We had to log into each host and apply updates as appropriately. We had to develop management scripts to manage them as well. Puppet helped unify this and ensure less drift between machines so any more than one machine and puppet can be really handy.

Now the other thing is that much of Puppet's configuration options can be swapped out for better scalability. For example the default storage backend could be MySQL, PostgreSQL or PuppetDB. The built in webrick server swapped for {Apache,cherokee,nginx}+passenger. The management from CLI to something like Puppet Dashboard, The Foreman or the Puppet Enterprise solution. It's completely up to you the route you take.

So does puppet scale well for large environments by default no, but that's ok, swap out a component that's the bottleneck and move on. It seems that your complaint is that there isn't a single "all-in-one" solution for you to choose from, when in fact there is, Puppet Enterprise. This comes with all the tools you'd need to scale puppet in a "black box" style. If you're too cheap and don't want to pay for that you can try The Foreman installer which does a pretty good job of installing the toolchain to scale puppet to larger environments using Apache and Passenger and you don't really need to know squat about how the components go together. You could find a "How To" or tutorial online and just follow it line by line if you want to! Go for it!

At any rate, you're going to have to learn about the toolchain associated with scaling puppet anyway. You're going to need to learn how to monitor and tweak Apache and Passenger if you need to scale the web components. You're going to need to know how to tweak MySQL, PostgreSQL or Puppet DB for stored configs. For PuppetDB you're also going to have to learn a bit about tuning Java to make it scale too!

Learn the tools or pay for a tuned black box. You can't have it both ways. If you want a rather simple way to deploy Puppet+Apache+Passenger+The Foreman, use The Foreman installer which works quite well. Have fun!

--
James A. Peltier
Manager, IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone : 778-782-6573
Fax : 778-782-3045
E-Mail : jpel...@sfu.ca
Website : http://www.sfu.ca/itservices
http://blogs.sfu.ca/people/jpeltier

Success is to be measured not so much by the position that one has reached
in life but as by the obstacles they have overcome. - Booker T. Washington

[Ashley Penney]

I have heard from various other puppet users of Open Puppet
installations of tens of thousands of nodes (at least last I heard,
there's probably people with more) so it definitely scales that high.
Generally they've had to restrict themselves on some features or
sometimes use Puppet in slightly strange ways to improve scaling, but
people are definitely doing it today.

The flash you get from PE just makes things easier. Most of us
started with a fairly small Puppet trial and over time developed
modules to build up an entire puppet infrastructure. I have a fairly
(badly) written module that installs puppetmasters, sets up repos,
hiera, databases, everything needed to fully deploy a relatively
scalable Puppet infrastructure. I think there's a lot of us out
there, maybe a few of us should get together, share what we have, and
try to build a combined module that people can use to help bootstrap
and scale.


On Wed, Jul 25, 2012 at 7:34 PM, Stuart Cracraft <smcra...@me.com> wrote:
> Hey, Chris: so that begs the question, do you think you have some "secret"
> or are just
> happier with fewer flashy gui's, more install/deployment scripts, and so
> forth.
>
> In other words, do you think the scaling of Open Puppet is adequate to scale
> much larger
> without the flash?
>
> Or, is there something fundamentally holding back Open Puppet from handling
> thousands, tens of thousands, or hundreds of thousands of nodes, in your
> opinion?
>
> Cheers,
> Stuart

> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/MW0Ok3Eent8J.

[Christopher Wood]

On Wed, Jul 25, 2012 at 04:34:34PM -0700, Stuart Cracraft wrote:
> Hey, Chris: so that begs the question, do you think you have some "secret"
> or are just
> happier with fewer flashy gui's, more install/deployment scripts, and so
> forth.

No actual secret. I'm happier with the command line and text configuration files (though a GUI absolutely has its place). I prefer interpreted (or shell) based installers so that I can tell what's going on, or going wrong. Of course, this may have as much to do with how my mind works as the amount of practice I've had in different computing ecosystems (because, again, $0 budget). Just because these things are good for me doesn't automatically mean they're good for other people with other talents.

> In other words, do you think the scaling of Open Puppet is adequate to
> scale much larger
> without the flash?

Yes, I do.

> Or, is there something fundamentally holding back Open Puppet from
> handling
> thousands, tens of thousands, or hundreds of thousands of nodes, in your
> opinion?

But then, it's not just puppet that people are scaling. If somebody thinks that they're going to point 100k nodes at a single virtual machine running puppet and have everything work at 99.999% uptime, they're making a mistake. (The same mistake that we've all seen for mail, radius, dns, et cetera.) Even if that works, are they willing to lose a cluster's configuration management if a single VM goes down? So now we have multiple puppet servers. Unix-like directories get slow when we add hundreds of thousands of directory entries. Let's stop tossing our node definitions in a single directory. How do we keep the certificates in sync? Now we have a system to sync certs. Can our switches handle the load of all that network traffic? Let's make sure we have redundant switches in our network core. Do we really want every server to depend on one set of puppetmasters? Let's break things out into pods. Can't keep the pods in sync? Maybe centralized is the way to go. Your whole datacentre checks into the puppetmaster at the same time every hour? It's time to spread thousands of requests out over the 3600 seconds you have in each hour, or add more backend puppetmasters or check in less often. There's a ton more of these scaling items.

In short: scaling puppet is about more than puppet. The puppet component is ready to compile a catalog from your manifests and send it to the node, yes. Every other layer has to be ready to scale up in support of that goal.

> Cheers,
> Stuart
> On Wednesday, July 25, 2012 2:52:00 PM UTC-7, Christopher Wood wrote:
>

> [2]puppet...@googlegroups.com.

> > >> To unsubscribe from this group, send email to

> [3]puppet-users...@googlegroups.com.

> > >> For more options, visit this group at

> [4]http://groups.google.com/group/puppet-users?hl=en.

> > >>
> > >>
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> > > To post to this group, send email to

> [5]puppet...@googlegroups.com.

> > > To unsubscribe from this group, send email to

> [6]puppet-users...@googlegroups.com.

> > > For more options, visit this group at

> [7]http://groups.google.com/group/puppet-users?hl=en.

> > >
> >
> >
> >
> > --
> > Hai Tao
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.

> > To post to this group, send email to [8]puppet...@googlegroups.com.

> > To unsubscribe from this group, send email to

> [9]puppet-users...@googlegroups.com.

> > For more options, visit this group at

> [10]http://groups.google.com/group/puppet-users?hl=en.

> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.

> To view this discussion on the web visit

> [11]https://groups.google.com/d/msg/puppet-users/-/MW0Ok3Eent8J.

> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>

> References
>
> Visible links
> 1. mailto:christop...@pobox.com
> 2. mailto:puppet...@googlegroups.com
> 3. mailto:puppet-users%2Bunsu...@googlegroups.com
> 4. http://groups.google.com/group/puppet-users?hl=en
> 5. mailto:puppet...@googlegroups.com
> 6. mailto:puppet-users%2Bunsu...@googlegroups.com
> 7. http://groups.google.com/group/puppet-users?hl=en
> 8. mailto:puppet...@googlegroups.com
> 9. mailto:puppet-users%2Bunsu...@googlegroups.com
> 10. http://groups.google.com/group/puppet-users?hl=en
> 11. https://groups.google.com/d/msg/puppet-users/-/MW0Ok3Eent8J

[Trevor Vaughan]

Best.....Post.....Ever

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Ken Barber]

+1

[Stuart Cracraft]

I decided to go with Puppet Enterprise and will be upgrading this weekend or possibly today.

--Stuart

Via Apple iPhone 4S on the AT&T Wireless Network

[James Turnbull]

On Friday, July 27, 2012 8:10:05 AM UTC-7, Trevor Vaughan wrote:

Best.....Post.....Ever

Seconded. Scaling is not a stand-alone problem. Indeed beyond the infrastructure components you also need to consider how you scale risk and availability across multiple combinations of architecture and stack, your Puppet configuration, your module style, ENC/Hiera design and how you handle commonalities and exceptions. The "default" setup and configuration for any application or stack generally has a sweet spot. And generally that sweet spot isn't at 1 machine or 100,000 machines but tries to strike a balance in the middle. You will almost always need to tune the application or stack to your sweet spot.

Regards

James