Views and garbage collection

[Arndt Droullier]

Hi,

I am trying to get to the bottom of some leaking pyramid application.

Now, what I found out is the following simple example where the request never get

cleaned up: 

-------------------------------------------------------------------------------------------------------------------

from pyramid.config import Configurator

from pyramid.httpexceptions import HTTPFound

from pyramid.response import Response

import weakref

class Class1(object):

    def __init__(self):

        self.alot="------------------ ".join([str(a) for a in range(0,10000)])

    def __del__(self):

        print "del object"

class View(object):

    def __init__(self, context, request):

        self.request = request

        self.context = context

    def __del__(self):

        print "del view"

    def test1(self):

        url = self.request.url

        return Response(body='hello world!', content_type='text/plain')

class ViewWeakref(object):

    def __init__(self, context, request):

        self._request = weakref.ref(request)

        self.context = context

    @property

    def request(self):

        return self._request()

   

    def __del__(self):

        print "del view"

    def test1(self):

        url = self.request.url

        return Response(body='hello world!', content_type='text/plain')

class Root(object):

    def __getitem__(self, name):

        return Class1()

def getRoot(request):

    return Root()

def main(global_config, **settings):

    """ This function returns a Pyramid WSGI application.

    """

    config = Configurator(root_factory=getRoot, settings=settings)

    config.add_view(view=View, name="test1", attr="test1")

    return config.make_wsgi_app()

-------------------------------------------------------------------------------------------------------------------

The application traverses the root object ("Root"), generates a context ("Class1") and 

calls a view ("View.test1") for the context. 

So the url to call this was for example "http://127.0.0.1:6543/class/test1"

What happens is __del__ functions are never called and and the application consumes 

more and more memory.

I think the important part is that the view is a "class" and it stores the request as

class attribute. And in fact with a few modifications to store the request as a weakref attribute

(class ViewWeakref) the application works allright and everything gets cleaned up.

Now, I don't think this happens every time a view class is involved. My original application

for example stores the request as view class atrribute (without using weakref) and works allright.

But I suppose there are a few rare cases depending on how views are invoked by pyramid 

making it easy to run into leaking applications. 

Has someone else run into this? Or do you explicitly cleanup after requests?

Arndt.

--
Arndt Droullier / DV Electric / www.dvelectric.de

[Marius Gedminas]

Hi,

How significant is the memory growth? Some actual numbers would help.

Does calling gc.collect() help?

Does removing the __del__ methods and then calling gc.collect() help?

Do you see a constant increase in len(gc.get_objects())? It could be
just memory fragmentation.

> I think the important part is that the view is a "class" and it stores
> the request as class attribute. And in fact with a few modifications
> to store the request as a weakref attribute (class ViewWeakref) the
> application works allright and everything gets cleaned up.

I see no class attributes in your code, only instance attributes.

The fact that using a weakref helps is a hint that there's normally a
reference cycle, which means reference counting isn't going to free the
objects normally, and you have to wait for the next scheduled garbage
collection to happen (or call gc.collect() explicitly) for objects to be
freed.

Note that the amount of memory used by the Python process is unlikely to
go down, due to memory fragmentation.

Also note that Python cannot free objects participating in a reference
cycle if they have __del__ methods on objects.

> Now, I don't think this happens every time a view class is involved.
> My original application for example stores the request as view class
> atrribute (without using weakref) and works allright. But I suppose
> there are a few rare cases depending on how views are invoked by
> pyramid making it easy to run into leaking applications.

I wrote http://pypi.python.org/pypi/objgraph when I was debugging a
memory leak in an app's test suite. It may help you narrow down the
leak, if there is one.

Marius Gedminas
--
To express oneself
In seventeen syllables
Is very diffic
-- John Cooper Clark.

[Arndt Droullier]

In fact I had two problems and thought they were related somehow. Well, they were not.

The one described in my previous mail just happened in the test code I wrote. I never had it in a 'real' application.

So maybe you're right this would be solved by the gc at some point.

The second is actually related to zope.interfaces and not pyramid. Though I've posted an issue with details on github (https://github.com/Pylons/pyramid/issues/688). Looks like it's somehow related to the c code or caching.

 

Also note that Python cannot free objects participating in a reference
cycle if they have __del__ methods on objects.

No? I never thought of that.

 

I wrote http://pypi.python.org/pypi/objgraph when I was debugging a
memory leak in an app's test suite.  It may help you narrow down the
leak, if there is one.

Thanks. I'll try it. Maybe it gives some more details. 

Arndt.

--
DV Electric / Arndt Droullier / www.dvelectric.de