memory leak - model remains in memory after requests

[David Zejda]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

My web2py instance gradually eats memory, during day the consumption
grows up to several gigs, so I have to restart often. According to guppy
most of memory is occupied by gluon.dal.Field and other classes of dal:

Partition of a set of 3231760 objects. Total size = 443724152 bytes.
Index Count % Size % Cumulative % Kind
0 113419 4 189636568 43 189636568 43 dict of gluon.dal.Field
1 1324208 41 80561096 18 270197664 61 str
2 328642 10 15982732 4 286180396 64 tuple
3 26637 1 13851240 3 300031636 68 dict of
gluon.validators.IS_IN_DB
4 98796 3 13436256 3 313467892 71 dict of gluon.dal.Set
5 20042 1 13344464 3 326812356 74 dict (no owner)
6 8199 0 11860464 3 338672820 76 gluon.dal.Row
7 16615 1 11482224 3 350155044 79 gluon.dal.Table
8 63682 2 8660752 2 358815796 81 dict of gluon.dal.Query
9 137779 4 7363776 2 366179572 83 list
<2282 more rows. Type e.g. '_.more' to view.>

The proportion is relatively stable. It seems that model definition
remains in memory after each request. It is probably caused by a weird
reference, but I'm not sure how to track it. Please do you have any ideas?

Thanks :)
David
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAk0VN9gACgkQ3oCkkciamVFHHwCfWiIkmrH9buBYA/7HvgIbz+mR
ei0AniZ0UYwZtj9zagp2sx/IawmBE2iA
=9cqX
-----END PGP SIGNATURE-----

[Thadeus Burgess]

This is due to the built in rocket server (it is not ment for production). If you use Apache with mod_wsgi this will not happen.

--
Thadeus

2010/12/24 David Zejda <dv...@atlas.cz>

[Jonathan Lundell]

On Dec 24, 2010, at 4:20 PM, Thadeus Burgess wrote:

This is due to the built in rocket server (it is not ment for production). If you use Apache with mod_wsgi this will not happen.

Do we understand why? Seems like it ought to be fixable.

[mdipierro]

Are we sure it is not a cache issue? Caching selects or actions with
arguments eats memory.

On Dec 24, 6:26 pm, Jonathan Lundell <jlund...@pobox.com> wrote:
> On Dec 24, 2010, at 4:20 PM, Thadeus Burgess wrote:
>
> > This is due to the built in rocket server (it is not ment for production). If you use Apache with mod_wsgi this will not happen.
>
> Do we understand why? Seems like it ought to be fixable.
>
>
>
> > --
> > Thadeus
>

> > 2010/12/24 David Zejda <d...@atlas.cz>

> > Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org

[Timbo]

Thadeus,

You seem to have more knowledge about this problem. Can you file a
bug report? Did you know that Rocket was recently updated fixing
several bugs (and creating one that has already be addressed). I'm
not denying the possibility, but let's be a good open source
community.

David,

If your environment allows it, please replace rocket.py line 1071
"break" with "return". Note that this will put a hard limit on the
number of requests/second rocket can serve to the number of
min_threads set. If the problem remains after that, then rocket is
not the issue.

I'm tending to side with Massimo. Caching issue?

-tim

On Dec 24, 6:20 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> This is due to the built in rocket server (it is not ment for production).
> If you use Apache with mod_wsgi this will not happen.
>
> --
> Thadeus
>

> 2010/12/24 David Zejda <d...@atlas.cz>

> > Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org

[mdipierro]

There is an easy way to check this: run web2py with any other web
server using the new web2py/anyserver.py script.

[ron_m]

I used this tool to track down memory leaks in a Python gstreamer program I wrote

http://mg.pov.lt/objgraph/

provides object graphs of the heap. Also the web page contains good docs on tracking down memory problems. It produces PNG files if graphviz is available or from the command line use xdot. I have not tried this on web2py but it might be interesting to try and add this to admin by putting objgraph.py into modules and then building a controller with the possible functions for forward and backward references and a view to show the png product.

This project also references Dozer which is a WSGI wrapper that might not be as pretty but a more direct approach - have not used this but looks simple.

http://pypi.python.org/pypi/Dozer

Ron

[David Zejda]

Thanks for the all replies!

I do not directly cache DAL objects, but yes, caching may be behind
the leak
through some references between objects. Maybe I'll examine it with
objgraph,
and I agree, it would be nice to have the function at hand in the
admin interface.

Cherrypy & anyserver brings no significant change.

With Cherokee & uWSGI the problem has gone. Moreover the new setup is
subjectively
faster to the previous one (Rocket behind Apache proxy). :)

David

[Timbo]

So you do use caching? Is it RAM caching or disk caching? If RAM
caching, it could be that running under Cherokee and uWSGI is deleting
the environment that web2py is run in after a certain number of
requests. This would reduce the usefulness of a RAM cache but would
also produce the results you're seeing. Whereas a Python-based server
(Rocket or Cherrypy) would not do this and a RAM cache would persist
as long as the server process (or until otherwise deleted).

Please let us know when you look into the possibilities of caching
being the issue.

@ron_m: Awesome. Great suggestions!

-tim

[ron_m]

I monitored my own app running under Rocket in development mode with Ubuntu System Monitor and the RSS memory size started at about 26 MB. After pushing every link in the interface it had grown to 38.5 MB and then stopped at that size. It has been running for over a day now with no further increase in RSS memory.

If you are using ram cache in a db().select() statement then the memory build up could be because every query selects a different set of records due to different query conditions resulting in every query adding a new cached item to the ram cache.

Is there any chance you have code in the modules directory that is creating new global objects on every invocation of the model which is per request? The module code is imported, the model code is run by exec so that might account for a difference. With modules unless the reload=True parameter is supplied on local_import they cannot be edited without a server restart so they behave like a long life addition to the server.

Ron

[David Zejda]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I think you are right. It is highly probable that my app is leaking
somehow. I do not cache dal objects directly and I use finite set of
indentifiers for cached objects, but it is possible that some of my
objects reference dal objects in an unwanted way, leading to some kind
of circular references which prevent garbage collection. It is not easy
to identify where exactly is the problem rooted, because my model is
quite complex (more than 100 tables, highly interlinked, etc.) and I
cache business objects built above DAL a lot. Maybe I'll use some tool
to examine the references between objects in memory (objgraph?).

So, I agree that there is probably no leak directly in web2py or Rocket
and uWSGI does not bring cure for the memory leak disease causation, but
it removes the symptoms, which makes me satisfied for now. The new setup
is also more stable. With Rocket my web2py freezed every couple of hours
(http://groups.google.com/group/web2py/browse_thread/thread/bd4f6e9f20d1a5aa)
and I had to monitor its state and force restarts whenever it happened.
Now it either freezes no more or Cherokee restarts uwsgi behind the
scenes whenever necessary. :)

David

- --
David Zejda, Open-IT cz
web development & services
http://www.o-it.info

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAk0YZeMACgkQ3oCkkciamVFgdgCfczag6uRCsadFl+TtTBj+SDgV
BokAoLfKc4Caslc0QISWt1fXf6lAhCBv
=+XpP
-----END PGP SIGNATURE-----

[MarkEdson AtWork]

I found a similar issue with a db.py module with code like this in it...

from gluon.packages.dal.pydal import DAL, Field

    db = DAL("sqlite://storage.sqlite")

    db.define_table(

        "test",

        Field("myid", default=DEFAULT_VALUE, length=15),

        Field("name", default=DEFAULT_VALUE, length=15),

        Field("code", default=DEFAULT_VALUE, length=2),

     )

every time the application was accessed the memory grew a little bit as if these objects were never garbage collected after the http response was sent.  Is my assumption that this should be garbage collected incorrect?  If I let it run long enough I did run out of memory so it was never cleaned up.

I am relatively new to web2py, I feel I must be missing something obvious or this would be reported often.

[MarkEdson AtWork]

Update, 

I ran the sample code I placed in this post over the weekend and it ended up consuming 1.8GB before python stopped functioning.

I am running pyDAL (17.1) with a stable web2py release.

Is this the built-in Rocket server issue I have run into?

On Friday, December 24, 2010 at 4:16:24 PM UTC-8, David Zejda wrote:

[MarkEdson AtWork]

I should also mention that I recently updated my code to use multiple controllers.

[Dave S]

On Monday, February 6, 2017 at 9:00:21 AM UTC-8, MarkEdson AtWork wrote:

I should also mention that I recently updated my code to use multiple controllers.

On Monday, February 6, 2017 at 7:53:20 AM UTC-8, MarkEdson AtWork wrote:

Update, 

I ran the sample code I placed in this post over the weekend and it ended up consuming 1.8GB before python stopped functioning.

I am running pyDAL (17.1) with a stable web2py release.

Is this the built-in Rocket server issue I have run into?

I run web2py with Rocket in a variety of environments.  The one on AWS Linux, according to top, scoots up to 3.4% mem occasionally; there's a main process and a scheduler running.  It's been running for at least a month since the last restart, but doesn't have a heavy load.  My dev environment is a Centos machine, with slightly more load but still not anything serious.  I've never run out of memory on it.  The "toy" environment on Windows doesn't generally run for more than a couple weeks at a time, but behaves itself during its uptime.

/dps

[Anthony]

On Friday, February 3, 2017 at 4:05:55 PM UTC-5, MarkEdson AtWork wrote:

I found a similar issue with a db.py module with code like this in it...

from gluon.packages.dal.pydal import DAL, Field

    db = DAL("sqlite://storage.sqlite")

    db.define_table(

        "test",

        Field("myid", default=DEFAULT_VALUE, length=15),

        Field("name", default=DEFAULT_VALUE, length=15),

        Field("code", default=DEFAULT_VALUE, length=2),

     )

How are you using that module within your app?

Most likely, you should not be defining the db object at the top level of your module. If you do that, you will have a single db object that lives across all requests. The problem with that is if you dynamically change any attributes of any models in some requests (e.g., temporarily changing the default value or validator of a field), those changes will persist across all requests (until changed again somewhere else).

Instead, you should define the db object and its models for each request. If you want to keep the code in a module, put it inside of functions or classes and call those functions/classes from your app's model or controller code.

Anthony

[MarkEdson AtWork]

Thank you for the replies,

Just to be sure I understand you, are you suggesting doing something like the following instead?

>>default.py

def entry_point()

    db = get_db()

    # use database

>>db.py

from gluon.packages.dal.pydal import DAL, Field

def get_db()

    db = DAL("sqlite://storage.sqlite")

    db.define_table(

        "test",

        Field("myid", default=DEFAULT_VALUE, length=15),

        Field("name", default=DEFAULT_VALUE, length=15),

        Field("code", default=DEFAULT_VALUE, length=2),

     )

     return db