Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Views and garbage collection
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  3 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Arndt Droullier  
View profile  
 More options Sep 12 2012, 8:49 am
From: Arndt Droullier <ar...@dvelectric.de>
Date: Wed, 12 Sep 2012 14:49:13 +0200
Local: Wed, Sep 12 2012 8:49 am
Subject: Views and garbage collection

Hi,

I am trying to get to the bottom of some leaking pyramid application.

Now, what I found out is the following simple example where the request
never get
cleaned up:

--------------------------------------------------------------------------- ----------------------------------------
from pyramid.config import Configurator

from pyramid.httpexceptions import HTTPFound
from pyramid.response import Response
import weakref

class Class1(object):
    def __init__(self):
        self.alot="------------------ ".join([str(a) for a in
range(0,10000)])

    def __del__(self):
        print "del object"

class View(object):
    def __init__(self, context, request):
        self.request = request
        self.context = context

    def __del__(self):
        print "del view"

    def test1(self):
        url = self.request.url
        return Response(body='hello world!', content_type='text/plain')

class ViewWeakref(object):
    def __init__(self, context, request):
        self._request = weakref.ref(request)
        self.context = context

    @property
    def request(self):
        return self._request()

    def __del__(self):
        print "del view"

    def test1(self):
        url = self.request.url
        return Response(body='hello world!', content_type='text/plain')

class Root(object):
    def __getitem__(self, name):
        return Class1()

def getRoot(request):
    return Root()

def main(global_config, **settings):
    """ This function returns a Pyramid WSGI application.
    """
    config = Configurator(root_factory=getRoot, settings=settings)
    config.add_view(view=View, name="test1", attr="test1")
    return config.make_wsgi_app()

--------------------------------------------------------------------------- ----------------------------------------

The application traverses the root object ("Root"), generates a context
("Class1") and
calls a view ("View.test1") for the context.
So the url to call this was for example "http://127.0.0.1:6543/class/test1"

What happens is __del__ functions are never called and and the application
consumes
more and more memory.

I think the important part is that the view is a "class" and it stores the
request as
class attribute. And in fact with a few modifications to store the request
as a weakref attribute
(class ViewWeakref) the application works allright and everything gets
cleaned up.

Now, I don't think this happens every time a view class is involved. My
original application
for example stores the request as view class atrribute (without using
weakref) and works allright.
But I suppose there are a few rare cases depending on how views are invoked
by pyramid
making it easy to run into leaking applications.

Has someone else run into this? Or do you explicitly cleanup after requests?

Arndt.

--
Arndt Droullier / DV Electric / www.dvelectric.de


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Marius Gedminas  
View profile  
 More options Sep 16 2012, 2:02 am
From: Marius Gedminas <mar...@gedmin.as>
Date: Sun, 16 Sep 2012 09:02:41 +0300
Local: Sun, Sep 16 2012 2:02 am
Subject: Re: Views and garbage collection

Hi,

How significant is the memory growth?  Some actual numbers would help.

Does calling gc.collect() help?

Does removing the __del__ methods and then calling gc.collect() help?

Do you see a constant increase in len(gc.get_objects())?  It could be
just memory fragmentation.

> I think the important part is that the view is a "class" and it stores
> the request as class attribute. And in fact with a few modifications
> to store the request as a weakref attribute (class ViewWeakref) the
> application works allright and everything gets cleaned up.

I see no class attributes in your code, only instance attributes.

The fact that using a weakref helps is a hint that there's normally a
reference cycle, which means reference counting isn't going to free the
objects normally, and you have to wait for the next scheduled garbage
collection to happen (or call gc.collect() explicitly) for objects to be
freed.

Note that the amount of memory used by the Python process is unlikely to
go down, due to memory fragmentation.

Also note that Python cannot free objects participating in a reference
cycle if they have __del__ methods on objects.

> Now, I don't think this happens every time a view class is involved.
> My original application for example stores the request as view class
> atrribute (without using weakref) and works allright.  But I suppose
> there are a few rare cases depending on how views are invoked by
> pyramid making it easy to run into leaking applications.

I wrote http://pypi.python.org/pypi/objgraph when I was debugging a
memory leak in an app's test suite.  It may help you narrow down the
leak, if there is one.

Marius Gedminas
--
  To express oneself
  In seventeen syllables
  Is very diffic
                -- John Cooper Clark.

  signature.asc
< 1K Download

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Arndt Droullier  
View profile  
 More options Sep 17 2012, 12:39 pm
From: Arndt Droullier <ar...@dvelectric.de>
Date: Mon, 17 Sep 2012 18:39:15 +0200
Local: Mon, Sep 17 2012 12:39 pm
Subject: Re: Views and garbage collection

In fact I had two problems and thought they were related somehow. Well,
they were not.
The one described in my previous mail just happened in the test code I
wrote. I never had it in a 'real' application.
So maybe you're right this would be solved by the gc at some point.

The second is actually related to zope.interfaces and not pyramid. Though
I've posted an issue with details on github (
https://github.com/Pylons/pyramid/issues/688). Looks like it's somehow
related to the c code or caching.

> Also note that Python cannot free objects participating in a reference
> cycle if they have __del__ methods on objects.

> No? I never thought of that.
> I wrote http://pypi.python.org/pypi/objgraph when I was debugging a
> memory leak in an app's test suite.  It may help you narrow down the
> leak, if there is one.

Thanks. I'll try it. Maybe it gives some more details.

Arndt.

--
DV Electric / Arndt Droullier / www.dvelectric.de


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »