finding memory leak in edgewall trac 0.11
rupert.thurner
edgewall trac uses excessive memory. see
http://groups.google.com/group/trac-dev/browse_thread/thread/116e519da54f16b
for some details, where jonas suggested
http://wingolog.org/archives/2007/11/27/reducing-the-footprint-of-python-applications
as reading.
tiran already gave some hints on http://bugs.python.org/issue1871, but
also suggested to ask the general mailing list:
Do you have classes with a __del__ method which may create reference
cycles? The GC can't break cycles when a __del__ method is involved.
Are you keeping references to tracebacks, exception objects (except
Exception, err) or frames (sys._getframe())
many thanks,
rupert.
Christian Heimes
I forgot one important point in my reply. The GC module contains some
useful methods for debugging. Check gc.garbage. It should be empty.
http://docs.python.org/lib/module-gc.html
Christian
Christian Heimes
I forgot one important point in my reply. The GC module contains some
Jeroen Ruigrok van der Werven
-On [20080119 16:16], Christian Heimes (li...@cheimes.de) wrote:
>I forgot one important point in my reply. The GC module contains some
>useful methods for debugging. Check gc.garbage. It should be empty.
Yeah, we're messing around with that stuff as well as many other ways of
trying to track issues, but it can really be looking for a needle in a
haystack to be honest.
There's so much output that, I guess, make sense only when you're semi-deep
into the Python internals to even make heads or tails out of it. =\
And even third-party code is not helping much to reduce the clutter and
provide insight.
--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
イェルーン ラウフロック ヴァン デル ウェルヴェン
http://www.in-nomine.org/ | http://www.rangaku.org/
Life is not a problem to be solved but a reality to be experienced...
Christian Heimes
> Hi Christian,
>
> -On [20080119 16:16], Christian Heimes (li...@cheimes.de) wrote:
>> I forgot one important point in my reply. The GC module contains some
>> useful methods for debugging. Check gc.garbage. It should be empty.
>
> Yeah, we're messing around with that stuff as well as many other ways of
> trying to track issues, but it can really be looking for a needle in a
> haystack to be honest.
> There's so much output that, I guess, make sense only when you're semi-deep
> into the Python internals to even make heads or tails out of it. =\
> And even third-party code is not helping much to reduce the clutter and
> provide insight.
Under normal circumstances gc.garbage should be an empty list. In
general it's a bad sign if gc.garbage contains lots of objects.
I found several potential leaks in trac:
$ find -name \*.py | xargs grep __del__
./trac/versioncontrol/svn_fs.py: def __del__(self):
./trac/versioncontrol/svn_fs.py: def __del__(self):
./trac/db/pool.py: def __del__(self):
$ find -name \*.py | xargs grep frame
./trac/web/main.py:
[...]
./trac/core.py: frame = sys._getframe(1)
./trac/core.py: locals_ = frame.f_locals
I recommend that you either replace __del__ with a weak reference
callback or to remove it. Referencing a frame, traceback or f_locals is
going to leak, too. You *must* explicitly del every frame and locals
variable.
Christian
Christian Heimes
rupert.thurner
many thanks! as the main change was replacing clearsilver with genshi,
this means one could do the same thing with genshi, http://genshi.edgewall.org/?
$ find -name \*.py | xargs grep frame
./genshi/filters/html.py: 'dir', 'disabled', 'enctype', 'for',
'frame', 'headers', 'height',
./genshi/input.py: _EMPTY_ELEMS = frozenset(['area', 'base',
'basefont', 'br', 'col', 'frame',
./genshi/output.py: 'http://www.w3.org/TR/html4/frameset.dtd'
./genshi/output.py: 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-
frameset.dtd'
./genshi/output.py: * "html-transitional" for the HTML 4.01
frameset DTD
./genshi/output.py: * "xhtml-frameset" for the XHTML 1.0
frameset DTD
./genshi/output.py: 'html-frameset': DocType.HTML_FRAMESET,
./genshi/output.py: 'xhtml-frameset': cls.XHTML_FRAMESET,
./genshi/output.py: _EMPTY_ELEMS = frozenset(['area', 'base',
'basefont', 'br', 'col', 'frame',
./genshi/template/base.py: _ctxt2dict = lambda ctxt: ctxt.frames[0]
./genshi/template/base.py: self.frames = deque([data])
./genshi/template/base.py: self.pop = self.frames.popleft
./genshi/template/base.py: self.push = self.frames.appendleft
./genshi/template/base.py: return repr(list(self.frames))
./genshi/template/base.py: for frame in self.frames:
./genshi/template/base.py: if key in frame:
./genshi/template/base.py: del frame[key]
./genshi/template/base.py: value, frame = self._find(key)
./genshi/template/base.py: if frame is None:
./genshi/template/base.py: self.frames[0][key] = value
./genshi/template/base.py: """Retrieve a given variable's value
and the frame it was found in.
./genshi/template/base.py: for frame in self.frames:
./genshi/template/base.py: if key in frame:
./genshi/template/base.py: return frame[key], frame
./genshi/template/base.py: for frame in self.frames:
./genshi/template/base.py: if key in frame:
./genshi/template/base.py: return frame[key]
./genshi/template/base.py: for frame in self.frames:
./genshi/template/base.py: keys += [key for key in frame if
key not in keys]
./genshi/template/directives.py: # Store the function reference
in the bottom context frame so that it
./genshi/template/directives.py: ctxt.frames[-1][self.name] =
function
./genshi/template/directives.py: frame = {}
./genshi/template/directives.py: ctxt.push(frame)
./genshi/template/tests/directives.py: frame =
exc_traceback.tb_next
./genshi/template/tests/directives.py: frames = []
./genshi/template/tests/directives.py: while frame.tb_next:
./genshi/template/tests/directives.py: frame =
frame.tb_next
./genshi/template/tests/directives.py:
frames.append(frame)
./genshi/template/tests/directives.py:
frames[-1].tb_frame.f_code.co_name)
./genshi/template/tests/directives.py:
frames[-1].tb_frame.f_code.co_filename)
./genshi/template/tests/directives.py: self.assertEqual(2,
frames[-1].tb_lineno)
./genshi/template/tests/eval.py: frame =
exc_traceback.tb_next
./genshi/template/tests/eval.py: frames = []
./genshi/template/tests/eval.py: while frame.tb_next:
./genshi/template/tests/eval.py: frame = frame.tb_next
./genshi/template/tests/eval.py: frames.append(frame)
./genshi/template/tests/eval.py:
frames[-3].tb_frame.f_code.co_name)
./genshi/template/tests/eval.py:
frames[-3].tb_frame.f_code.co_filename)
./genshi/template/tests/eval.py: self.assertEqual(50,
frames[-3].tb_lineno)
./genshi/template/tests/eval.py: frame =
exc_traceback.tb_next
./genshi/template/tests/eval.py: while frame.tb_next:
./genshi/template/tests/eval.py: frame = frame.tb_next
./genshi/template/tests/eval.py: code =
frame.tb_frame.f_code
./genshi/template/tests/eval.py: self.fail("never found
the frame I was looking for")
./genshi/template/tests/eval.py: self.assertEqual(50,
frame.tb_lineno)
./genshi/template/tests/eval.py: frame =
exc_traceback.tb_next
./genshi/template/tests/eval.py: while frame.tb_next:
./genshi/template/tests/eval.py: frame = frame.tb_next
./genshi/template/tests/eval.py: code =
frame.tb_frame.f_code
./genshi/template/tests/eval.py: self.fail("never found
the frame I was looking for")
./genshi/template/tests/eval.py: self.assertEqual(50,
frame.tb_lineno)
rupert
rupert.thurner
wrote:
>
> $ find -name \*.py | xargs grep frame
> ./genshi/filters/html.py: 'dir', 'disabled', 'enctype', 'for',
>
> - Show quoted text -
i forgot to mention that i cannot see any explicit sys._getframe(), or
__del__ in the genshi code, while the ones in trac-core seemed to be
there in 0.10.4.
rupert
Christian Heimes
> i forgot to mention that i cannot see any explicit sys._getframe(), or
> __del__ in the genshi code, while the ones in trac-core seemed to be
> there in 0.10.4.
Does the code keep a reference to a traceback object or an attribute of
a traceback object?
Christian
rupert.thurner
> rupert.thurner wrote:
> > i forgot to mention that i cannot see any explicit sys._getframe(), or
> > there in 0.10.4.
>
> Does the code keep a reference to a traceback object or an attribute of
> a traceback object?
>
> Christian
if there is no other possibility but doing it with sys.exc_traceback,
sys.last_traceback, sys.exc_info, the answer is no for genshi.
rupert.thurner
> rupert.thurner wrote:
> > i forgot to mention that i cannot see any explicit sys._getframe(), or
> > there in 0.10.4.
>
> Does the code keep a reference to a traceback object or an attribute of
> a traceback object?
>
> Christian
could you have one look at http://genshi.edgewall.org/browser/trunk/genshi/_speedups.c
?
rupert.