Mongrel Memory Usage

[Trevor Vaughan]

All,

I'm trying to figure out the best way to estimate the amount of memory
usage that will be taken up by Mongrel and I'm hoping that you have
some estimates that can help me.

I'm currently estimating the following:

Memory Required = Number of running puppetmasters * size of manifests
* number of clients managed

The mongrel model doesn't seem to *ever* reclaim memory.

Suggestions on reclaiming memory welcome though I'm sure most people
will just urge me to move to Passenger.

Thanks,

Trevor

[Larry Ludwig]

Passenger is slowly becoming the 'official' way to run puppetmaster,
and is recommended by Reductive Labs.

As far as memory, there are many dependancies like 32-bit over 64-bit,
the size, amount of files transfered through puppet, your manifests,
variables, clients used, etc. Some is also dependent upon the version
ruby itself and it's optimizations.

Out of all of these variables what I CAN say are most important:
- files and the size of files transfered though Puppet
- 32 bit over 64 bit.

0.25 will definitely help with the memory consumption with file usage.

-L

--
Larry Ludwig
Reductive Labs

[Mark Plaksin]

Trevor Vaughan <peiri...@gmail.com> writes:

You're right--use Passenger :)

When we were running Mongrel out puppetmasterd processes started out at
100MB and grew to 1G in a couple of weeks. With Passenger processes
don't stick around as long so if nothing else you dodge the memory
leaks. And we've heard that some of the memory leak problems are in
Ruby itself--not Puppet.

[Brice Figureau]

On 13/08/09 22:36, Trevor Vaughan wrote:
> All,
>
> I'm trying to figure out the best way to estimate the amount of memory
> usage that will be taken up by Mongrel and I'm hoping that you have
> some estimates that can help me.
>
> I'm currently estimating the following:
>
> Memory Required = Number of running puppetmasters * size of manifests
> * number of clients managed

In 0.24, what cost the most is the file transfer, hopefully 0.25 is
waaay better in this respect.

> The mongrel model doesn't seem to *ever* reclaim memory.

My master don't grow, but they're not really under load.
One of the issue with MRI, is that if your master are constantly under
load the ruby garbage collector never really start, and thus memory is
not reclaimed.

> Suggestions on reclaiming memory welcome though I'm sure most people
> will just urge me to move to Passenger.

Make sure you're using a recent ruby. It is known that some older ruby
version are leaking memory.

I can't comment on Passenger since I never tried it. But in the near
future (understand when I'll have some spare cycle) you'll have another
promising alternative: Puppet on JRuby on Glassfish.
I was able to run this stack in the lab, and limited testing show that
it was at least 30% faster than using mongrel. For the memory part, I
didn't check, but the JVM has a superior garbage collector than the ruby
one. I really expect some memory gains, just because you'll be running
only one process...

--
Brice Figureau
My Blog: http://www.masterzen.fr/

[Trevor Vaughan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Interesting, thanks for the information.

My only concern with a 32 bit architecture, is that my memory usage may
legitimately grow above 2G which would toast the puppetmaster process
pretty quickly.

Thanks,

Trevor

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkqEtk8ACgkQyjMdFR1108BxlQCaA10cbRfhT8TpSRaI2jOlDb6r
TkIAn0Cv09xARvbAyLZBrFbs+rx2/3Tr
=JftR
-----END PGP SIGNATURE-----

[Larry Ludwig]

On Aug 13, 2009, at 8:56 PM, Trevor Vaughan wrote:

>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Interesting, thanks for the information.
>
> My only concern with a 32 bit architecture, is that my memory usage
> may
> legitimately grow above 2G which would toast the puppetmaster process
> pretty quickly.
>

The rub is Ruby itself is not optimized for 64 bit. an int for
example is double in size.

[Ohad Levy]

When I was using mongrels (yes also switched to passenger) I had large memory abuse per mongrel.
I was using monit to control the size of the mongrels (e.g. restart a certain mongrel process when its too big), maybe this will be useful for you:
http://github.com/ohadlevy/puppet-multipuppetmaster/blob/755dd82b45295db104a15290840eddf8686afda5/templates/monit.erb
(note that if you browse the repo, a lot of stuff are already removed - e.g. all the mongrel part can only be found at the repo older tree).

Ohad

[Trevor Vaughan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thanks Had.

I had been looking at doing something like this and this will save some
time.

Trevor

iEYEARECAAYFAkqFMj8ACgkQyjMdFR1108CERwCgjgRiVRHrC6TgdI1NAGTlg3PZ
mfEAnAr0tT5OWV47HE9cEqq69bnJZYqI
=SdR9
-----END PGP SIGNATURE-----