Reference cycles in Pympler

[Ludwig Hähne]

Hi guys,

the last couple of hours I've tried to track down the cause of reference
cycles generated while using Pympler and came up with a number of fixes
(please see my latest commit). Some explanations:

When using asizeof(all=True), the stack frame objects are fetched and
stored as local variables during sizing. Referencing the current stack
frame from the list of local variables builds a reference cycle (because
the locals are referenced by the frame). Therefore, the current frame
and the caller (to which the references are passed) are excluded from
the list. This should also be correct, assuming profiling overhead
should not be included in the size computation.

Muppy get_objects was more tricky. One problem was again due to the
infamous frame objects. (With the current fix, it will still create
ref-cycles if the returned object list is passed up to the caller of the
caller of get_objects - an option would be to ignore all frame objects.)

Another problem which took quite some time to recognize is that the
locals (i.e. 'res') are also returned by gc.get_objects. Problem: The
objects (including res) were appended to the already existing list res
creating a self reference --- in effect building a huge reference cycle
that comprised all known objects. I think the latter might have led to
the problem Brian has encountered.

Please check if you see any problem with the changes I've made. A side
effect is that a Muppy test fails now (**), but I'm not sure whether the
code or the test needs to be fixed. Robert, can you take a look?

Cheers,
Ludwig

(**)
Traceback (most recent call last):
File "test/muppy/test_tracker.py", line 109, in
test_stracker_create_summary
self.assert_(len(tmp) != 0)
AssertionError

[Robert Schuppenies]

Hi Ludwig.

Thanks for taking on this problem.

> When using asizeof(all=True), the stack frame objects are fetched and
> stored as local variables during sizing. Referencing the current stack
> frame from the list of local variables builds a reference cycle (because
> the locals are referenced by the frame). Therefore, the current frame
> and the caller (to which the references are passed) are excluded from
> the list. This should also be correct, assuming profiling overhead
> should not be included in the size computation.

This is an old problem and I have fallen for it :| For example see here:
http://docs.python.org/library/inspect.html#the-interpreter-stack

> Muppy get_objects was more tricky. One problem was again due to the
> infamous frame objects. (With the current fix, it will still create
> ref-cycles if the returned object list is passed up to the caller of the
> caller of get_objects - an option would be to ignore all frame objects.)

I am not sure whether ignoring all frame objects is the best solution in terms
of tracking memory issues. But maybe we should settle for this for now, as it
may be the cleanest. I fought this problem before and one exception leads to
another one which leads to another one.

> Another problem which took quite some time to recognize is that the
> locals (i.e. 'res') are also returned by gc.get_objects. Problem: The
> objects (including res) were appended to the already existing list res
> creating a self reference --- in effect building a huge reference cycle
> that comprised all known objects. I think the latter might have led to
> the problem Brian has encountered.

Good find.

> Please check if you see any problem with the changes I've made. A side
> effect is that a Muppy test fails now (**), but I'm not sure whether the
> code or the test needs to be fixed. Robert, can you take a look?

I'll look into this.

[Jean Brouwers]

It looks like stack frames create sizing problems.  It may take a while to get those to work right.  Perhaps, we should not size stack frames at all.  If stack frames do need to be sized, the caller can pass those explicitly.  Or we could add an option to in- or exclude stack frames.

/Jean

[Ludwig Hähne]

Hi Jean,

I have not encountered any problems sizing (caller) stack frames -
what kind of problems do you have in mind?

The only problem concerning frames I can see is keeping and passing
around references to frame objects. This does not seem to be a problem
in asizeof as references are not leaked to the caller. In muppy
(get_objects), however, it may make sense to ignore frame objects or
converting the frame references to some other proxy object (e.g. the
string representation).

Ludwig

2009/9/8 Jean Brouwers <mrj...@gmail.com>:

[Jean Brouwers]

The current asizeof 5.10 in Pympler does not use  get_objects for all=True.  But later versions will (at least that's what I had planned), see for example 5.12 at <http://code.activestate.com/recipes/546530/>.  And in that case, asizeof will have the same issues as muppy.

/Jean

[Robert Schuppenies]

I was also hoping to consolidate the retrieval of objects into one place
in the code :) So we have to things here: Gathering objects (and keeping
them for analysis) and the analysis of such objects. asizeof allows this
analysis (passing objects to asizeof) so I assume we are fine on this
side. As for the gathering of objects, I think we should ignore frame
objects by default.

cheers,
robert

Jean Brouwers wrote:
> The current asizeof 5.10 in Pympler does not use get_objects for
> all=True. But later versions will (at least that's what I had planned),
> see for example 5.12 at <http://code.activestate.com/recipes/546530/>.
> And in that case, asizeof will have the same issues as muppy.
>
> /Jean
>
>
> On Tue, Sep 8, 2009 at 7:18 AM, Ludwig Hähne <lha...@googlemail.com
> <mailto:lha...@googlemail.com>> wrote:
>
>
> Hi Jean,
>
> I have not encountered any problems sizing (caller) stack frames -
> what kind of problems do you have in mind?
>
> The only problem concerning frames I can see is keeping and passing
> around references to frame objects. This does not seem to be a problem
> in asizeof as references are not leaked to the caller. In muppy
> (get_objects), however, it may make sense to ignore frame objects or
> converting the frame references to some other proxy object (e.g. the
> string representation).
>
> Ludwig
>

> 2009/9/8 Jean Brouwers <mrj...@gmail.com <mailto:mrj...@gmail.com>>:

[Ludwig Hähne]

+1 for having a unique get_objects function in Pympler which ignores
frame objects.

If one still needs to include outer frame objects for sizing one could
still do something like that:

>>> all = pympler.get_objects().extend(inspect.stack()[offset:])

With an offset governed by the number of profiling overhead frames.

2009/9/8 Robert Schuppenies <robert.sc...@gmail.com>:

[Robert Schuppenies]

Ludwig Hähne wrote:
> +1 for having a unique get_objects function in Pympler which ignores
> frame objects.
>
> If one still needs to include outer frame objects for sizing one could
> still do something like that:
>
>>>> all = pympler.get_objects().extend(inspect.stack()[offset:])

Could you file an issue so we can track this? And maybe also mention
this nifty code snippet to be put in the docs. I really like that :)

[Ludwig Hähne]

> Could you file an issue so we can track this? And maybe also mention
> this nifty code snippet to be put in the docs. I really like that :)

Done (see issue 29). Please add any information that might be relevant.
Thanks, Ludwig

[Robert Schuppenies]

Thanks a a lot.

[Jean Brouwers]

That is the way yo go, no question.

/Jean