memory leak ... and why doesn't a process get smaller?
Stephen Bannasch
application and in doing the investigation I created a very simple
Rails 2.02 application for testing.
FYI: the testing script and all the testing results are available here:
I create a simple model that has two attributes:
rails leaktest
cd leaktest
script/generate scaffold comment name:string body:text
RAILS_ENV=production rake db:migrate
My test consists of measuring the real memory usage after each
request and a test run consists of accessing the default rails page
four times and then the index page for the model four times.
I was able to measure the small memory leak throughout these tests
(approximately 8-20 bytes per request) that Ara Howard tried and
failed to track down last October.
I then create 100 copies of the model each with about 1k data and
duplicate the test.
I subsequently ran the test with 1000 and 10000 model objects.
Between 0 and 1000 copies of the model object the process size varied
between 55 and 70M however when I increased the memory size to 10000
model objects the process size ballooned to about 220M.
The database is SQLite and the actual file is only 11MB when filled
with 10000 model objects. I was surprised that instantiating an SQL
object of around 1k bytes results in an increase of memory in the
process space of around 15k.
Change in AR object count Change in process size
0 => 1000 20MB
1000 => 10000 150 MB
I then deleted all the model objects and ran the tests again and the
process size stayed at 218k.
I was surprised because I thought the objects would have been garbage
collected and the process size shrunk. I have not yet tried
instrumenting this process with Dike or Bleakhouse.
Does the fact that the process size stay at 220k mean that there are
unused objects that can no longer be garbage collected?
Is there a way to have the process size shrink back down?
Piers Cawley
Yup, it will do, ruby doesn't give any memory it's allocated back to
the system. Try creating a bunch more objects - the process size won't
grow until the new objects are taking up more space than the old ones
because ruby will simply reuse the memory it's already allocated.
Getting memory from the OS is expensive, ruby (and perl and python and
almost every other interpreted language you can think of) assume that,
because the program gets that big once, it's likely to get that big
again, so it's a good idea to keep the memory hanging around ready for
that eventuality.
This probably isn't suitable subject matter for rubyonrails-core though. Ho hum.
Stephen Bannasch
>Yup, it will do, ruby doesn't give any memory it's allocated back to
>the system. Try creating a bunch more objects - the process size won't
>grow until the new objects are taking up more space than the old ones
>because ruby will simply reuse the memory it's already allocated.
>Getting memory from the OS is expensive, ruby (and perl and python and
>almost every other interpreted language you can think of) assume that,
>because the program gets that big once, it's likely to get that big
>again, so it's a good idea to keep the memory hanging around ready for
>that eventuality.
>
>This probably isn't suitable subject matter for rubyonrails-core though. Ho hum.
Piers,
Thanks for your information on process size functionality.
Actually the process size does get smaller -- it changes size by about 17% on alternating requests when the AR collection of 10,000 objects is being rendered -- so a process can shrink -- but I don't know why.
I'd like to know more about what it would take to change the functionality of how a process works so that it was able to shrink in response to further requests which didn't uses as much memory. While this may be functionality that is not part of core Rails someone on this list might be able to point me towards work others are doing in this area.
The problem I am investigating is an app that uses much more memory when I create a instance variable with a collection of AR objects in the controller and use that in the view than if I do not create that instance variable in the controller and instead just create the AR collection in the view.
This may be due to a bug in rails but I need to continue to develop a simple test case so our information is useful for me to be able to progress towards either finding a bug in Rails or in my app. I wasn't planning on sharing more details about this issue with this list until I had determined it was a bug in rails. I did however think it was useful to describe the tests and instrumentation process I was using. I assume that folks on this list may have similar or more involved instrumentation strategies.
I think it is useful to report to the core list that the leak Ara wasn't able to track down last October is still present and to document the test that shows this result.
I'm curious about why instantiating an AR model object takes approximately 15x the size of the object when stored in the db and think this is also a question best directed to the core mailing list.
Does anybody have suggestions about better venues for raising these questions?
Michael Koziarski
I don't know if these two are related, ara's leak was related to
session handling wasn't it?
>
> I'm curious about why instantiating an AR model object takes approximately 15x the size of the object when stored in the db and think this is also a question best directed to the core mailing list.
The fact that it uses more memory than the database uses on disk
shouldn't really come as a surprise. As far as the specific multiple
it could almost certainly be improved with some judicious memory
profling. Alexander Dymo's work has yielded some good payoffs lately,
if you're looking for tooling suggestions he's the guy to listen to:
http://blog.pluron.com/2008/02/memory-profilin.html
>
> Does anybody have suggestions about better venues for raising these questions?
This is the best venue to discuss this stuff, at least unless the bug
is tracked down and shown to be some other problem :)
--
Cheers
Koz