I have a production setup that isn't live yet (<500 visitors per day)
but the mongrels are already misbehaving.
monit has been configured to restart mongrels that are consuming more
than 170MB. What I observe is that the mongrels quickly hit this
limit, typically within 10-20 minutes of starting up. I relaxed the
memory cap for one mongrel and set it to 500MB but the result is still
the same - it still hits the monit memory limit almost as fast as the
other mongrels.
log rotation is in place and the logs are being truncated when they
hit more than 10MB.
gettext is being used heavily in almost all pages.
page caching and page fragment caching is used almost everywhere it
can be used.
On my local development machine, I find that I can easily hit 200MB
with less than 50 clicks. I typically keep my mongrel running all the
time during development. As of today my dev mongrel has been up since
yesterday and I just found out that it is now at 800MB, with just me
poking at it over the last 36 hours or so.
I (well, me and my teammates) are looking at the following steps to
see what's going on:
1. run bleak_house to see where the leaks are coming from
2. run on JRuby and see if we'll get the same memory leak patterns. If
we do then definitely it's our code that's suspect here. If we don't
then maybe it's with one of the gems or libraries we're using.
I am also hoping that these too will help:
3. install the latest ruby 1.8.6 from source (patch level 114?)
4. install the latest postgres gem (0.7.9?)
I'd very much appreciate any pointers on other possible actions to
take.
Thanks!
rad
Here are more details on the production server:
Dual Intel Xeon 2GHz
4GB RAM
64-bit CentOS 5.1
ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]
rubygems 1.1.1
Hi, we are also struggling with the memory consumption of the mongrel nodes. Ours go about 200MB to 300MB a piece, and have monit restart them if they go beyond 300MB.
However, we made some testing even with a base Rails installation, and it looks like Rails is not at fault with the memory leaks. We also tried to figure out what went wrong, and until now, it's an on-going investigation. It's hard to pin-point where the leak is coming from, since it's a really big app.
Based on our investigation one particular common mistake that leads to leaks is when we manipulate instantiated objects within the request (controller action) and pass them over to a global method (class method). Like for example, keeping a global list of some object that is appended in the request. The mistake doesn't have to be this obvious, but can be in a similar situation where locally instantiated objects are being pointed to in the global context. You might want to keep this in mind when you look for that leak.
Hope that helps!
Mark
On Tue, May 13, 2008 at 7:33 PM, Radamanthus Batnag
> I have a production setup that isn't live yet (<500 visitors per day) > but the mongrels are already misbehaving. > monit has been configured to restart mongrels that are consuming more > than 170MB. What I observe is that the mongrels quickly hit this > limit, typically within 10-20 minutes of starting up. I relaxed the > memory cap for one mongrel and set it to 500MB but the result is still > the same - it still hits the monit memory limit almost as fast as the > other mongrels.
> log rotation is in place and the logs are being truncated when they > hit more than 10MB. > gettext is being used heavily in almost all pages. > page caching and page fragment caching is used almost everywhere it > can be used.
> On my local development machine, I find that I can easily hit 200MB > with less than 50 clicks. I typically keep my mongrel running all the > time during development. As of today my dev mongrel has been up since > yesterday and I just found out that it is now at 800MB, with just me > poking at it over the last 36 hours or so.
> I (well, me and my teammates) are looking at the following steps to > see what's going on: > 1. run bleak_house to see where the leaks are coming from > 2. run on JRuby and see if we'll get the same memory leak patterns. If > we do then definitely it's our code that's suspect here. If we don't > then maybe it's with one of the gems or libraries we're using.
> I am also hoping that these too will help: > 3. install the latest ruby 1.8.6 from source (patch level 114?) > 4. install the latest postgres gem (0.7.9?)
> I'd very much appreciate any pointers on other possible actions to > take.
> Thanks! > rad
> Here are more details on the production server: > Dual Intel Xeon 2GHz > 4GB RAM > 64-bit CentOS 5.1 > ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux] > rubygems 1.1.1
If possible step backwards in time through your SVN commits in a divide-and-conquer manner (halve the version number each time). Deploy, run, check monit, deploy, run, check monit, until you find the SVN version that causes the memory leak. I heard Rmagick leaks, and you have that listed up there.
As Mark mentioned the base installation presumable has no memory leak.
David :)
On Tue, May 13, 2008 at 4:33 AM, Radamanthus Batnag <radamant...@gmail.com> wrote:
> I have a production setup that isn't live yet (<500 visitors per day) > but the mongrels are already misbehaving. > monit has been configured to restart mongrels that are consuming more > than 170MB. What I observe is that the mongrels quickly hit this > limit, typically within 10-20 minutes of starting up. I relaxed the > memory cap for one mongrel and set it to 500MB but the result is still > the same - it still hits the monit memory limit almost as fast as the > other mongrels.
> log rotation is in place and the logs are being truncated when they > hit more than 10MB. > gettext is being used heavily in almost all pages. > page caching and page fragment caching is used almost everywhere it > can be used.
> On my local development machine, I find that I can easily hit 200MB > with less than 50 clicks. I typically keep my mongrel running all the > time during development. As of today my dev mongrel has been up since > yesterday and I just found out that it is now at 800MB, with just me > poking at it over the last 36 hours or so.
> I (well, me and my teammates) are looking at the following steps to > see what's going on: > 1. run bleak_house to see where the leaks are coming from > 2. run on JRuby and see if we'll get the same memory leak patterns. If > we do then definitely it's our code that's suspect here. If we don't > then maybe it's with one of the gems or libraries we're using.
> I am also hoping that these too will help: > 3. install the latest ruby 1.8.6 from source (patch level 114?) > 4. install the latest postgres gem (0.7.9?)
> I'd very much appreciate any pointers on other possible actions to > take.
> Thanks! > rad
> Here are more details on the production server: > Dual Intel Xeon 2GHz > 4GB RAM > 64-bit CentOS 5.1 > ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux] > rubygems 1.1.1
It's a good thing we have tagged releases, so I can go back to SVN one release at a time. I'll look for the code patterns Mark mentioned; it seems to me that our permissions code which is called almost everywhere use that pattern.
On Tue, May 13, 2008 at 9:49 PM, David Beckwith <dbitsoluti...@gmail.com> wrote: > If possible step backwards in time through your SVN commits in a > divide-and-conquer manner (halve the version number each time). Deploy, > run, check monit, deploy, run, check monit, until you find the SVN version > that causes the memory leak. I heard Rmagick leaks, and you have that > listed up there.
> As Mark mentioned the base installation presumable has no memory leak.
> David :)
> On Tue, May 13, 2008 at 4:33 AM, Radamanthus Batnag <radamant...@gmail.com> > wrote:
> > Hi folks,
> > I have a production setup that isn't live yet (<500 visitors per day) > > but the mongrels are already misbehaving. > > monit has been configured to restart mongrels that are consuming more > > than 170MB. What I observe is that the mongrels quickly hit this > > limit, typically within 10-20 minutes of starting up. I relaxed the > > memory cap for one mongrel and set it to 500MB but the result is still > > the same - it still hits the monit memory limit almost as fast as the > > other mongrels.
> > log rotation is in place and the logs are being truncated when they > > hit more than 10MB. > > gettext is being used heavily in almost all pages. > > page caching and page fragment caching is used almost everywhere it > > can be used.
> > On my local development machine, I find that I can easily hit 200MB > > with less than 50 clicks. I typically keep my mongrel running all the > > time during development. As of today my dev mongrel has been up since > > yesterday and I just found out that it is now at 800MB, with just me > > poking at it over the last 36 hours or so.
> > I (well, me and my teammates) are looking at the following steps to > > see what's going on: > > 1. run bleak_house to see where the leaks are coming from > > 2. run on JRuby and see if we'll get the same memory leak patterns. If > > we do then definitely it's our code that's suspect here. If we don't > > then maybe it's with one of the gems or libraries we're using.
> > I am also hoping that these too will help: > > 3. install the latest ruby 1.8.6 from source (patch level 114?) > > 4. install the latest postgres gem (0.7.9?)
> > I'd very much appreciate any pointers on other possible actions to > > take.
> > Thanks! > > rad
> > Here are more details on the production server: > > Dual Intel Xeon 2GHz > > 4GB RAM > > 64-bit CentOS 5.1 > > ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux] > > rubygems 1.1.1
i'm thinking that it's ruby GC acting weird given that the server has
4GB of memory. The larger the memory is, the longer it takes for the
ruby GC to pass on it. It was said that don't spoil your mongrels too
much. But then, I maybe wrong because you've mentioned that it's
acting the same on your dev machine. Just some thoughts lang. :)
On May 14, 12:48 pm, "Radamanthus Batnag" <radamant...@gmail.com>
wrote:
> It's a good thing we have tagged releases, so I can go back to SVN one
> release at a time. I'll look for the code patterns Mark mentioned; it
> seems to me that our permissions code which is called almost
> everywhere use that pattern.
> rad
> On Tue, May 13, 2008 at 9:49 PM, David Beckwith <dbitsoluti...@gmail.com> wrote:
> > If possible step backwards in time through your SVN commits in a
> > divide-and-conquer manner (halve the version number each time). Deploy,
> > run, check monit, deploy, run, check monit, until you find the SVN version
> > that causes the memory leak. I heard Rmagick leaks, and you have that
> > listed up there.
> > As Mark mentioned the base installation presumable has no memory leak.
> > David :)
> > On Tue, May 13, 2008 at 4:33 AM, Radamanthus Batnag <radamant...@gmail.com>
> > wrote:
> > > Hi folks,
> > > I have a production setup that isn't live yet (<500 visitors per day)
> > > but the mongrels are already misbehaving.
> > > monit has been configured to restart mongrels that are consuming more
> > > than 170MB. What I observe is that the mongrels quickly hit this
> > > limit, typically within 10-20 minutes of starting up. I relaxed the
> > > memory cap for one mongrel and set it to 500MB but the result is still
> > > the same - it still hits the monit memory limit almost as fast as the
> > > other mongrels.
> > > log rotation is in place and the logs are being truncated when they
> > > hit more than 10MB.
> > > gettext is being used heavily in almost all pages.
> > > page caching and page fragment caching is used almost everywhere it
> > > can be used.
> > > On my local development machine, I find that I can easily hit 200MB
> > > with less than 50 clicks. I typically keep my mongrel running all the
> > > time during development. As of today my dev mongrel has been up since
> > > yesterday and I just found out that it is now at 800MB, with just me
> > > poking at it over the last 36 hours or so.
> > > I (well, me and my teammates) are looking at the following steps to
> > > see what's going on:
> > > 1. run bleak_house to see where the leaks are coming from
> > > 2. run on JRuby and see if we'll get the same memory leak patterns. If
> > > we do then definitely it's our code that's suspect here. If we don't
> > > then maybe it's with one of the gems or libraries we're using.
> > > I am also hoping that these too will help:
> > > 3. install the latest ruby 1.8.6 from source (patch level 114?)
> > > 4. install the latest postgres gem (0.7.9?)
> > > I'd very much appreciate any pointers on other possible actions to
> > > take.
> > > Thanks!
> > > rad
> > > Here are more details on the production server:
> > > Dual Intel Xeon 2GHz
> > > 4GB RAM
> > > 64-bit CentOS 5.1
> > > ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]
> > > rubygems 1.1.1
On Tue, May 13, 2008 at 10:43 PM, j4s0n <jason.e.tor...@gmail.com> wrote:
> i'm thinking that it's ruby GC acting weird given that the server has > 4GB of memory. The larger the memory is, the longer it takes for the > ruby GC to pass on it. It was said that don't spoil your mongrels too > much. But then, I maybe wrong because you've mentioned that it's > acting the same on your dev machine. Just some thoughts lang. :)
> On May 14, 12:48 pm, "Radamanthus Batnag" <radamant...@gmail.com> > wrote: > > Thanks for the quick reply guys.
> > It's a good thing we have tagged releases, so I can go back to SVN one > > release at a time. I'll look for the code patterns Mark mentioned; it > > seems to me that our permissions code which is called almost > > everywhere use that pattern.
> > rad
> > On Tue, May 13, 2008 at 9:49 PM, David Beckwith <dbitsoluti...@gmail.com> > wrote: > > > If possible step backwards in time through your SVN commits in a > > > divide-and-conquer manner (halve the version number each time). > Deploy, > > > run, check monit, deploy, run, check monit, until you find the SVN > version > > > that causes the memory leak. I heard Rmagick leaks, and you have that > > > listed up there.
> > > As Mark mentioned the base installation presumable has no memory leak.
> > > David :)
> > > On Tue, May 13, 2008 at 4:33 AM, Radamanthus Batnag < > radamant...@gmail.com> > > > wrote:
> > > > Hi folks,
> > > > I have a production setup that isn't live yet (<500 visitors per > day) > > > > but the mongrels are already misbehaving. > > > > monit has been configured to restart mongrels that are consuming > more > > > > than 170MB. What I observe is that the mongrels quickly hit this > > > > limit, typically within 10-20 minutes of starting up. I relaxed the > > > > memory cap for one mongrel and set it to 500MB but the result is > still > > > > the same - it still hits the monit memory limit almost as fast as > the > > > > other mongrels.
> > > > log rotation is in place and the logs are being truncated when they > > > > hit more than 10MB. > > > > gettext is being used heavily in almost all pages. > > > > page caching and page fragment caching is used almost everywhere it > > > > can be used.
> > > > On my local development machine, I find that I can easily hit 200MB > > > > with less than 50 clicks. I typically keep my mongrel running all > the > > > > time during development. As of today my dev mongrel has been up > since > > > > yesterday and I just found out that it is now at 800MB, with just me > > > > poking at it over the last 36 hours or so.
> > > > I (well, me and my teammates) are looking at the following steps to > > > > see what's going on: > > > > 1. run bleak_house to see where the leaks are coming from > > > > 2. run on JRuby and see if we'll get the same memory leak patterns. > If > > > > we do then definitely it's our code that's suspect here. If we don't > > > > then maybe it's with one of the gems or libraries we're using.
> > > > I am also hoping that these too will help: > > > > 3. install the latest ruby 1.8.6 from source (patch level 114?) > > > > 4. install the latest postgres gem (0.7.9?)
> > > > I'd very much appreciate any pointers on other possible actions to > > > > take.
> > > > Thanks! > > > > rad
> > > > Here are more details on the production server: > > > > Dual Intel Xeon 2GHz > > > > 4GB RAM > > > > 64-bit CentOS 5.1 > > > > ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux] > > > > rubygems 1.1.1
