Memory leak in standalone DAL (issue #731), can you please help test?

[nick name]

I can reproduce this problem on both Linux and Windows (have no access to a Mac), and Massimo cannot reproduce this on his Mac. Perhaps something is borked about all my python installations (some site-packages I use or something). Can you help test? Just go into the web2py directory, and start python with the following script (or just start python and paste the script into the python console)

#!/usr/bin/python
#encoding: utf-8

# get a web2py database environment
import sys
sys.path.append('../../')
from gluon.dal import DAL

while 1:
    mydal = DAL('sqlite:memory')
    sys.stdout.write('.')
    sys.stdout.flush()
    mydal._adapter.close()

For me: memory usage grows at about 20MB/sec, measured with htop on linux and TaskManager on windows. If I wait enough time, all the memory is eaten up. If you drop the last line (mydal._adapter.close()) the memory will be eaten much more quickly for me.

Thanks.
Issue reference: 731: Standalone DAL is leaking memory+resources (don't know whether or not inside web2py)

[Anthony]

I can confirm the same behavior with this script on both Ubuntu 12.04 and Windows 7 -- both running Python 2.7 and web2py trunk.

Anthony

[Massimo Di Pierro]

In order to isolate the problem, let's check if this is a sqlite:memory issue. Can you reproduce the problem with sqlite://storage.db ?

[nick name]

On Wednesday, April 18, 2012 10:26:32 PM UTC-4, Massimo Di Pierro wrote:

In order to isolate the problem, let's check if this is a sqlite:memory issue. Can you reproduce the problem with sqlite://storage.db ?

Yes, same result exactly. Note that the 'storage.db' is empty, but I'm seeing this in production, with a real database full of data. (This short script was my result of trying to track this down from a real world problem)

Perhaps it happens more slowly on the Mac, but I'm surprised it doesn't happen. Massimo, did you try to leave it running for (e.g.) 10 minutes and check the memory consumption? Or maybe you have a VPS with python on to try?

[Wikus van de Merwe]

I've tested in on Ubuntu 11.04, with web2py 1.99.2 and trunk and in both cases after 35s the memory use is 1GB and growing.
Same results with DAL("sqlite://storage.db").

[Ron McOuat]

I just tried on OSX Lion and the problem is reproduced there. The RSIZE in top increases to about 1GB in less than a minute. The Python version on Lion is 2.7.1

I can also reproduce on Ubuntu 10.04.4 which is Python 2.6.5

Ron

[Ricardo Pedroso]

On Wed, Apr 18, 2012 at 11:41 PM, nick name
<i.like.pr...@gmail.com> wrote:

> Issue reference: 731: Standalone DAL is leaking memory+resources (don't know
> whether or not inside web2py)

I post a comment on this issue:
http://code.google.com/p/web2py/issues/detail?id=731#c4

I think this is not a bug but an incorrect use of the dal api.

Ricardo

[nick name]

On Thursday, April 19, 2012 7:08:06 PM UTC-4, Ricardo Pedroso wrote:

I post a comment on this issue:
http://code.google.com/p/web2py/issues/detail?id=731#c4

I think this is not a bug but an incorrect use of the dal api.

Ricardo, thanks! That is indeed the problem. Whether or not it is a misuse of the API, the API that I actually need (cleanly close only SOME dal/adapter instances and not ALL dal/adapter instances) does not currently exist, and I need to to reach into the underlying implementation to do what I want.

For massimo: something like the following might be useful (note: untested):

(added to ConnectionPool class)


    def close_or_recycle(self, action):
        """ to close cleanly in a multithreaded environment """
        if self in thread.get(instances, []):
            if action:
                if callable(action):
                    action(self)
                else:
                    getattr(self, action)()
            # ## if you want pools, recycle this connection
            really = True
            if self.pool_size:
                sql_locker.acquire()
                pool = ConnectionPool.pools[self.uri]
                if len(pool) < self.pool_size:
                    pool.append(self.connection)
                    really = False
                sql_locker.release()
            if really:
                self.close()
           
        if callable(action):
            action(None)
        return

and rewriting close_all_instances to use this while popping thread.instances on one hand, and having a DAL.close() method which calls self._adapter.close_or_recycle() (possibly also from DAL.__del__)

Thanks!