[Puppet Users] mcollective scalability

[linuxdatacenter]

I've been thinking a lot about running mcollective on my production
servers recently, especially about its nice integration with puppet
(using facts and so on). However I've got some concern about its
scalability - how publish/subscribe middleware scales in terms of
speed and flooding the network. I run about 2000 servers. Has anyone
got any experience running mcollective on similar scale?


-Regards

--linuxdatacenter.blogspot.com---

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[R.I.Pienaar]

hello,

Probably best to send this kind of thing to mcollective's mailing list rather than puppets :)

----- "linuxdatacenter" <linuxda...@gmail.com> wrote:

> I've been thinking a lot about running mcollective on my production
> servers recently, especially about its nice integration with puppet
> (using facts and so on). However I've got some concern about its
> scalability - how publish/subscribe middleware scales in terms of
> speed and flooding the network. I run about 2000 servers. Has anyone
> got any experience running mcollective on similar scale?

2000 nodes certainly is within my goals with the design, that said there has not been such a big deploy.

If any problems at all it will be related to the fast producer/slow consumer style problems. In MC the consumer is very fast (it does very little) but still, should there be a problem it will be a fairly simple matter to convert the replies to go over a queue rather than a topic. That should resolve any issue with it.

You'd probably want a few activemq instances in a cluster in such a setup, though I've heard of much larger client counts on activemq.

--
R.I.Pienaar

[donavan]

> 2000 nodes certainly is within my goals with the design, that said there has not been such a big deploy.

We've got 500+ nodes on mcollective currently. Nothing special as far
as setup, a few of the contrib agents and few more in house agents.
Nodes are primarily split between two buildings connected by gigE. 95%
response from mc-ping is 300ms, average 200ms.

I'm expecting at least 1000 nodes in three geo areas by the end of the
year. My todo list for this summer has federating some more activemqs
into the setup.

> You'd probably want a few activemq instances in a cluster in such a setup, though I've heard of much larger client counts on activemq.

The above all runs off of a single activemq server currently. Quad
core xeon w/ 3gb ram sitting at 99% idle.

[Matt Wallace]

On Wed, 2010-05-05 at 23:31 -0700, donavan wrote:
> > 2000 nodes certainly is within my goals with the design, that said there has not been such a big deploy.
>
> We've got 500+ nodes on mcollective currently. Nothing special as far
> as setup, a few of the contrib agents and few more in house agents.
> Nodes are primarily split between two buildings connected by gigE. 95%
> response from mc-ping is 300ms, average 200ms.
>
> I'm expecting at least 1000 nodes in three geo areas by the end of the
> year. My todo list for this summer has federating some more activemqs
> into the setup.
>
> > You'd probably want a few activemq instances in a cluster in such a setup, though I've heard of much larger client counts on activemq.
>
> The above all runs off of a single activemq server currently. Quad
> core xeon w/ 3gb ram sitting at 99% idle.
>

For those of you that are interested, I've put up a few blogposts about
configuring ActiveMQ and Mcollective on both Centos and Ubuntu at
http://www.threedrunkensysadsonthe.net/

Kind regards,

Matt

[Eric Shamow]

Matt,

Really great posts -- you've inspired me to take a crack at MCollective.

-Eric

[Matt Wallace]

On Thu, 2010-05-06 at 20:29 -0400, Eric Shamow wrote:
> Matt,
>
> Really great posts -- you've inspired me to take a crack at MCollective.

Thanks Eric,

Let us know how you get on!

M.