Extreme memory leak

[Radamanthus Batnag]

Hi folks,

I have a production setup that isn't live yet (<500 visitors per day)
but the mongrels are already misbehaving.
monit has been configured to restart mongrels that are consuming more
than 170MB. What I observe is that the mongrels quickly hit this
limit, typically within 10-20 minutes of starting up. I relaxed the
memory cap for one mongrel and set it to 500MB but the result is still
the same - it still hits the monit memory limit almost as fast as the
other mongrels.

log rotation is in place and the logs are being truncated when they
hit more than 10MB.
gettext is being used heavily in almost all pages.
page caching and page fragment caching is used almost everywhere it
can be used.

On my local development machine, I find that I can easily hit 200MB
with less than 50 clicks. I typically keep my mongrel running all the
time during development. As of today my dev mongrel has been up since
yesterday and I just found out that it is now at 800MB, with just me
poking at it over the last 36 hours or so.

I (well, me and my teammates) are looking at the following steps to
see what's going on:
1. run bleak_house to see where the leaks are coming from
2. run on JRuby and see if we'll get the same memory leak patterns. If
we do then definitely it's our code that's suspect here. If we don't
then maybe it's with one of the gems or libraries we're using.

I am also hoping that these too will help:
3. install the latest ruby 1.8.6 from source (patch level 114?)
4. install the latest postgres gem (0.7.9?)

I'd very much appreciate any pointers on other possible actions to
take.

Thanks!
rad

Here are more details on the production server:
Dual Intel Xeon 2GHz
4GB RAM
64-bit CentOS 5.1
ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]
rubygems 1.1.1

gems:
actionmailer (2.0.2, 1.3.6, 1.3.3)
actionpack (2.0.2, 1.13.6, 1.13.3)
actionwebservice (1.2.6, 1.2.3)
activerecord (2.0.2, 1.15.6, 1.15.3)
activeresource (2.0.2)
activesupport (2.0.2, 1.4.4, 1.4.2)
ar_mailer (1.3.1)
ar_mailer_generator (1.0.0)
bleak_house (4.0)
BlueCloth (1.0.0)
capistrano (2.2.0)
cgi_multipart_eof_fix (2.5.0)
columnize (0.1)
daemons (1.0.10)
eventmachine (0.10.0)
fastthread (1.0.1)
gem_plugin (0.2.3)
gettext (1.10.0)
highline (1.4.0)
hoe (1.5.1)
hpricot (0.6)
htmlentities (4.0.0)
linecache (0.42)
mocha (0.5.6)
mongrel (1.1.4)
mongrel_cluster (1.0.5)
needle (1.3.0)
net-sftp (1.1.1)
net-ssh (1.1.2)
production_log_analyzer (1.5.0)
rails (2.0.2, 1.2.6, 1.2.3)
rails_analyzer_tools (1.4.0)
railsmachine (1.0.0)
rake (0.8.1)
RedCloth (3.0.4)
rmagick (2.3.0)
ruby-debug (0.10.1)
ruby-debug-base (0.10.1)
ruby-postgres (0.7.1.2006.04.06)
ruby-prof (0.6.0)
rubyforge (0.4.5)
RubyInline (3.6.7)
rubypants (0.2.0)
swiftiply (0.6.1.1)
SyslogLogger (1.4.0)
ZenTest (3.9.2)

[Mark John Buenconsejo]

Hi, we are also struggling with the memory consumption of the mongrel
nodes. Ours go about 200MB to 300MB a piece, and have monit restart
them if they go beyond 300MB.

However, we made some testing even with a base Rails installation, and
it looks like Rails is not at fault with the memory leaks. We also
tried to figure out what went wrong, and until now, it's an on-going
investigation. It's hard to pin-point where the leak is coming from,
since it's a really big app.

Based on our investigation one particular common mistake that leads to
leaks is when we manipulate instantiated objects within the request
(controller action) and pass them over to a global method (class
method). Like for example, keeping a global list of some object that
is appended in the request. The mistake doesn't have to be this
obvious, but can be in a similar situation where locally instantiated
objects are being pointed to in the global context. You might want to
keep this in mind when you look for that leak.

Hope that helps!

Mark

--
------------------------------------
Mark John S. Buenconsejo
http://www.simpleteq.com

[David Beckwith]

If possible step backwards in time through your SVN commits in a divide-and-conquer manner (halve the version number each time).  Deploy, run, check monit, deploy, run, check monit, until you find the SVN version that causes the memory leak.  I heard Rmagick leaks, and you have that listed up there.

As Mark mentioned the base installation presumable has no memory leak. 

David :)

--
SMS/Text: +63 905 220 2207
Skype: dbit_solutions

[Radamanthus Batnag]

Thanks for the quick reply guys.

It's a good thing we have tagged releases, so I can go back to SVN one
release at a time. I'll look for the code patterns Mark mentioned; it
seems to me that our permissions code which is called almost
everywhere use that pattern.

rad

[j4s0n]

i'm thinking that it's ruby GC acting weird given that the server has
4GB of memory. The larger the memory is, the longer it takes for the
ruby GC to pass on it. It was said that don't spoil your mongrels too
much. But then, I maybe wrong because you've mentioned that it's
acting the same on your dev machine. Just some thoughts lang. :)

On May 14, 12:48 pm, "Radamanthus Batnag" <radamant...@gmail.com>
wrote:

> Thanks for the quick reply guys.
>
> It's a good thing we have tagged releases, so I can go back to SVN one
> release at a time. I'll look for the code patterns Mark mentioned; it
> seems to me that our permissions code which is called almost
> everywhere use that pattern.
>
> rad
>

> On Tue, May 13, 2008 at 9:49 PM, David Beckwith <dbitsoluti...@gmail.com> wrote:
> > If possible step backwards in time through your SVN commits in a
> > divide-and-conquer manner (halve the version number each time).  Deploy,
> > run, check monit, deploy, run, check monit, until you find the SVN version
> > that causes the memory leak.  I heard Rmagick leaks, and you have that
> > listed up there.
>
> > As Mark mentioned the base installation presumable has no memory leak.
>
> > David :)
>

> > On Tue, May 13, 2008 at 4:33 AM, Radamanthus Batnag <radamant...@gmail.com>

[David Beckwith]

btw, make sure you don't have

require 'ruby-dbug'
Debugger.start in there

http://codehappy.wordpress.com/2007/01/17/tracking-a-memory-leak-in-rails/