web2py DAL, requests, threads and memory

[Gleb]

Hi All,

I'm building an app on Pyramid framework and would like to use the
web2py-DAL with it. Pyramid makes every request in a single thread, so
how should I manage the DAL object?

Should it be created once and then passed to every thread? If I do so
and some frequent queries are passed to the DAL (simple reads by
SELECT), I get errors like "invalid cursor state", "cannot close a
closed cursor" and so on. That is because a single cursor is accessed
form different threads, so one thread closes the cursor while the
other assumes it to be open.

Should the DAL object be created on each request(like root =
DAL("sqlite://blah"))? If I do so, the cursor state errors go, but a
memory problem appears. Each request allocates memory for a new DAL
object and that memory is never released. After some dozens requests
I've got some hundredths MB of wasted memory.

How is the thread per request - DAL access managed by the web2py?

Thanks!

[Mengu]

you can create a new request class extending from the original one and
via using config.set_request_factory(YourReqClass)and make dal an
attribute of it. whenever the request ends, you can close the
connection.

[Vasile Ermicioi]

I think DAL connection  was  designed with web2py design in mind:

on each request a controller is executed and also are executed all models files including database connection, 

and after request memory is freed and connection close,

I used it successfully in a long running app with a few hundreads of greenthreads (gevent), but I forced a close after each db query

db._adapter.connection.close()

and app always consumes less that 70Mb of RAM

[Gleb]

Thank you!
That was a point - I should close the connection manually. Now it works great.

[Gleb]

Goot idea! At the end I did it like that, but a little other way. There is a
request.add_finished_callback
property of the request object, so I extended the DAL object:

class Root(DAL):

    def __init__(self, request, uri):

        DAL.__init__(self, uri, pool_size=0)

        

        request.add_finished_callback(self._close)

    def _close(self, request):

        self._adapter.close()

[Mengu]

glad you sorted it out. :)

On Apr 6, 8:57 pm, Gleb <gleb.sternh...@googlemail.com> wrote:
> Goot idea! At the end I did it like that, but a little other way. There is a

> *request.add_finished_callback*