scaling puppet, skipping puppetmaster?

[Philip Brown]

I've heard that after (some # of machines) x (some size of manifests)  puppet does not scale well, due to bottlenecking on the puppetmaster.

Anyone doing large scale use by some other methods?
For example, running puppet on each machine individually using cron or something? With the manifest distributed....
(via NFS? http? something else?)

I'd be interested to hear alternatives.

[Aaron Grewell]

Note that using multiple masters is one way to solve this.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/ucp6vuDFbjYJ.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Ryan Bowlby]

Currently we have two puppetmaster servers with ALL requests being
load balanced. I use unison to keep the ssl directory in sync between
hosts. Each server runs keepalived and requests go to a VIP that
exists on one of the servers. The server with the VIP load balances
the requests (mod_proxy) between both servers.

It's working relatively fine, though it would be ideal to have the
agents connect at semi-random intervals in order to reduce "thundering
herd" issues. We are over 500 without any real issues. Also, the
decentralized approach works fine but there are caveats related to the
use of custom functions that rely on a central server, virtual
resources(?), etc. I would try to scale your masters as it's not that
hard.

-Ryan

[Luke Bigum]

I use Puppet Commander, an MCollective tool:
http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/ToolPuppetcommander

Rather than have my Puppet Agents check in, no Puppet service runs on
any server, instead MCollective is running. Puppet Commander uses the
MCollective framework to schedule Puppet runs. I decide how many
should run at once and what their run interval should be based on the
load of the Puppet Master.

When I hit the upper limit of the VM running the Puppet Master but
want to increase the run speed, I'll put in load balanced Puppet like
others have mentioned.

Masterless Puppet is an option, however files copied from the Puppet
Master using Puppet's in built file transfer (source =>
"puppet:///...") are not included in the catalog, so each host needs
to be able to request files from the Puppet Master which may re-
introduce your bottleneck depending on how many there are. You can put
file _contents_ into a catalog, but this blows out the size and you'll
be re-transferring every file with each catalog (plus any security
concerns of having this file content stored on disk in many places).

From your other post if we manage 15 config files and services on 200
servers with Puppet - that should be nothing in terms of load for one
Apache/Passenger driven Puppet Master. A few CPUs and 4GiB RAM, tops.

[Philip Brown]

On Sat, Apr 28, 2012 at 2:12 AM, Luke Bigum <Luke....@lmax.com> wrote:
> I use Puppet Commander, an MCollective tool:
> http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/ToolPuppetcommander
>
> Rather than have my Puppet Agents check in, no Puppet service runs on
> any server, instead MCollective is running. Puppet Commander uses the
> MCollective framework to schedule Puppet runs. I decide how many
> should run at once and what their run interval should be based on the
> load of the Puppet Master.

Huh. Sounds like central scheduled rdist, in a way. The main
difference being, your scheduler tells clients, "okay get files from
me now", whereas rdist tells clients, "HERE are your files, take em!"
:)

Thanks, I'll look into mcollective.