Memory leak in my Tk app
EKB
I have written an IDE in Tcl/Tk (for a language other than Tcl - in
case anyone's interested, it's hosted at
http://www.ipat-s.kb-creative.net). But I am getting a serious memory
leak.
The language supported by the IDE allows several files to be in a
project. The user can open different files in a project on different
tabs. Memory expands when tabs are opened *and* when tabs are closed.
It almost never shrinks. It also expands during normal operation on a
single file. (I'm measuring memory using the "VM Size" on Windows Task
Manager, which is supposed to be better than the wildly inaccurate "Mem
Usage".)
Each of the files is opened in its own text widget and color-coded.
Each widget has its own bindings and tags (for color-coding). Regexps
are used (over and over again) for the color-coding. When a file is
closed, the text widget is destroyed. (Actually, only the containing
frame is explicitly destroyed, but that should destroy the child
windows, right?)
I am wondering: are tags, bindings, compiled regexps not removed from
memory? And more importantly, if they aren't, what can I do about it?
(I've already gotten some feedback that the bindings ought to be
destroyed, but not the tags - is that right?)
A related question: if the compiled regexps aren't destroyed, is there
any way I can get access to them to re-use them on repeated calls to a
proc where they appear? I haven't found a way to do that.
Thanks,
Eric
Gerald W. Lester
> Hi,
>
> I have written an IDE in Tcl/Tk (for a language other than Tcl - in
> case anyone's interested, it's hosted at
> http://www.ipat-s.kb-creative.net). But I am getting a serious memory
> leak.
>
> The language supported by the IDE allows several files to be in a
> project. The user can open different files in a project on different
> tabs. Memory expands when tabs are opened *and* when tabs are closed.
> It almost never shrinks. It also expands during normal operation on a
> single file. (I'm measuring memory using the "VM Size" on Windows Task
> Manager, which is supposed to be better than the wildly inaccurate "Mem
> Usage".)
>
> Each of the files is opened in its own text widget and color-coded.
> Each widget has its own bindings and tags (for color-coding). Regexps
> are used (over and over again) for the color-coding. When a file is
> closed, the text widget is destroyed. (Actually, only the containing
> frame is explicitly destroyed, but that should destroy the child
> windows, right?)
>
> I am wondering: are tags, bindings, compiled regexps not removed from
> memory? And more importantly, if they aren't, what can I do about it?
> (I've already gotten some feedback that the bindings ought to be
> destroyed, but not the tags - is that right?)
I'm a little surprised that your memory *ever* shrinks! Tcl does not
release its memory back to the OS, rather it releases it to an internal pool
which it goes to first when it needs memory for something.
--
+--------------------------------+---------------------------------------+
| Gerald W. Lester |
|"The man who fights for his ideals is the man who is alive." - Cervantes|
+------------------------------------------------------------------------+
Jeff Hobbs
> The language supported by the IDE allows several files to be in a
> project. The user can open different files in a project on different
> tabs. Memory expands when tabs are opened *and* when tabs are closed.
> It almost never shrinks. It also expands during normal operation on a
> single file. (I'm measuring memory using the "VM Size" on Windows Task
> Manager, which is supposed to be better than the wildly inaccurate "Mem
> Usage".)
FWIW, the Task Manager isn't nearly as good as 'top' on unix
for monitoring mem usage of any type.
> Each of the files is opened in its own text widget and color-coded.
> Each widget has its own bindings and tags (for color-coding). Regexps
> are used (over and over again) for the color-coding. When a file is
> closed, the text widget is destroyed. (Actually, only the containing
> frame is explicitly destroyed, but that should destroy the child
> windows, right?)
>
> I am wondering: are tags, bindings, compiled regexps not removed from
> memory? And more importantly, if they aren't, what can I do about it?
> (I've already gotten some feedback that the bindings ought to be
> destroyed, but not the tags - is that right?)
As with all major dynamic languges, the standard malloc behavior
of Tcl is high-water mark allocation. It does not explicit free
all memory back to the OS that was previously requested. It moves
it to an unused internal pool for reuse. In general though, the
example you cite above should provide for reuse of memory.
> A related question: if the compiled regexps aren't destroyed, is there
> any way I can get access to them to re-use them on repeated calls to a
> proc where they appear? I haven't found a way to do that.
In the RE case, Tcl keeps a compiled rep cache of the 30 most
recent REs used for performance reasons.
--
Jeff Hobbs, The Tcl Guy
http://www.ActiveState.com/, Dynamic Tools for Dynamic Languages
EKB
workaround, or some way of organizing my app so that I don't keep
adding memory. Here's the technical problem: I need to open several
files at once, and they need to be color-coded (which means bindings,
regexps, and tags). I need to do this without continually increasing
the memory. (Relatively high but stable memory is OK: continually
increasing is not.)
Is there some way to do this? I don't want to abandon Tcl/Tk as the
language for my app, but the expanding memory usage is really
unacceptable.
Thanks!
Donal K. Fellows
> FWIW, the Task Manager isn't nearly as good as 'top' on unix
> for monitoring mem usage of any type.
The 'VM Size" column (as opposed to the "Memory Usage" column which I
think just lists the amount of physical memory in use by the process) is
much more relevant. A pity that it is not displayed by default.
Donal.
Gerald W. Lester
> Thanks for both these replies. If this is the case, I need a
> workaround, or some way of organizing my app so that I don't keep
> adding memory. Here's the technical problem: I need to open several
> files at once, and they need to be color-coded (which means bindings,
> regexps, and tags). I need to do this without continually increasing
> the memory. (Relatively high but stable memory is OK: continually
> increasing is not.)
You should reach a "relatively high but stable memory" situation if you app
is pure Tcl/Tk. (assuming you close files and unset any arrays/variables you
use to store info about the files).
If you are using any extensions, you may want to carefully check them. It
has been my experience that an extension is much more likely to leak than Tcl.
Jeff Hobbs
> Thanks for both these replies. If this is the case, I need a
> workaround, or some way of organizing my app so that I don't keep
> adding memory. Here's the technical problem: I need to open several
> files at once, and they need to be color-coded (which means bindings,
> regexps, and tags). I need to do this without continually increasing
> the memory. (Relatively high but stable memory is OK: continually
> increasing is not.)
I'm a bit dubious that you would see continuously increasing
memory without a stable state. Are you sure that you are cleaning
up and destroying the widgets?
Donal K. Fellows
> I'm a bit dubious that you would see continuously increasing
> memory without a stable state. Are you sure that you are cleaning
> up and destroying the widgets?
Alas, there's quite possibly a real problem. If he's generating widget
names or tag names using a counter, he has a memory leak (due to Tk_Uid
instances). The only way to avoid this is to maintain a pool of unused
names and reuse those whenever possible.
On a related note, if you're doing very high performance canvas
graphics, pooling the items themselves is the best approach, as it is
much quicker to fetch an existing item out of a pool than to create a
new one. I've used this in the past to (approximately) double the speed
of an animation involving very large numbers of items... :-)
Donal.
EKB
I re-use the same names on different widgets. For example, I'll have
several text widgets open, each of which has a "keyword" tag. But if I
repeatedly open and close the same file, every time I open *and* close
it, the app takes up more memory.
Does this sound like I'm not really destroying the widgets? Or does
each widget instance take up (and retain) additional memory for the
tags?
Thanks,
Eric
Donal K. Fellows
> I re-use the same names on different widgets. For example, I'll have
> several text widgets open, each of which has a "keyword" tag. But if I
> repeatedly open and close the same file, every time I open *and* close
> it, the app takes up more memory.
Hmm, nothing that I see there indicates that the app *must* leak. The
problem is that memory leaks depend utterly on the specifics; it could
well be something you've not mentioned (or even that you've overlooked
completely). No, I'm not offering to debug it for you. :-)
> Does this sound like I'm not really destroying the widgets? Or does
> each widget instance take up (and retain) additional memory for the
> tags?
While the widget exists[*], it will take space for each tag. That's sort
of obvious. To know if a widget exists, use [winfo exists $pathname] and
find out if you're leaking in an obvious way that isn't Tk's fault. :-)
Donal.
[* Widgets take more space when displayed, but still take some space
even when not displayed. ]
EKB
But I was at a loss about where to look. I think now I have a clue.
Cameron Laird
.
.
Diagnosis through isolation: narrow the problem down.
Are you saying that a file open-and-close affects memory,
or that a file open-and-close in your code necessarily
involves widget operations?
Shaun Deacon
Donal K. Fellows wrote:
> On a related note, if you're doing very high performance canvas
> graphics, pooling the items themselves is the best approach, as it is
> much quicker to fetch an existing item out of a pool than to create a
> new one. I've used this in the past to (approximately) double the speed
> of an animation involving very large numbers of items... :-)
Donal,
This caught my attention...could you provide some more details, or
perhaps direct me to a relevant/related page in the wiki ?
I draw a large number of items onto a canvas and after a while
refreshing the display starts to become very slow - maybe I could
benefit from this kind of optimization.
Thanks,
Shaun
EKB
close necessarily involves widget operations in my code. It seems to be
the widget operations that take memory.
I've been going over the code with home-made breakpoints (pairs of
tk_messageBoxes bracketing commands) to see where the increase in
memory occurs, as well as doing some other checks (listed below). Using
my tests, I've found a couple of places where I could slow the increase
in memory use, but memory use is still growing at a faster rate than I
would like, and doesn't *seem* to be stabilizing.
I'll keep looking, and trying to narrow things down. I suppose at this
point the main thing that would help is to know where memory problems
usually occur (which I'm thinking of as "not releasing things that
could be released"). Here's my list so far:
- Deleting windows properly [it turned out I was doing this correctly]
- Closing all open file handles [I had one proc that didn't do this
properly, now fixed]
- Not adding global variables (e.g. with a counter) [I was clean on
this one]
- Not adding tags (e.g., with a counter) [I was clean on this one]
What's getting me down is that actually I'm doing pretty well on my
tests so far. I've only made progress by short-cutting some
memory-intensive operations, to slow the memory increase. But I'm
worried that I still have a leak somewhere, and I'm having trouble
spotting it.
Thanks again!
Gerald W. Lester
> ...
> What's getting me down is that actually I'm doing pretty well on my
> tests so far. I've only made progress by short-cutting some
> memory-intensive operations, to slow the memory increase. But I'm
> worried that I still have a leak somewhere, and I'm having trouble
> spotting it.
Are you using an extensions?
EKB
BLT
BWidgets
Incr Widgets
tcllib/struct
Uwe Klein
> Yes, a few:
> BLT
which version ( tcl and blt)
> BWidgets
> Incr Widgets
> tcllib/struct
>
the other version probably as well
how fast does it grow?
continuous or per operation.
uwe
EKB
BLT: 2.4
iwidgets: 4.0.2
tcllib: 1.8
It's growing per operation. I'll give some examples. The app is an IDE
that allows the user to open several color-coded text files on tabs.
So...
- I open a file on a new tab, mem goes from 13.59 MB to 14.10 MB
- I close the same file, mem goes from 14.10 MB to 14.20 MB
- I open another file, mem goes from 14.20 MB to 14.26 MB
- I close that file, mem goes from 14.26 MB to 14.34 MB
Uwe Klein
> Tcl/Tk: ActiveState 8.4.11.2
> BLT: 2.4
>
you are doing this on Windows?
uwe
EKB
And yes, I'm doing this on Windows... I'm using the "VM Size" rather
than "Mem Usage" in the Windows Task Manager. (From what I could find
by Googling, it seems that VM Size is fairly reliable, while Mem Usage
is way off.)
Darren New
> And yes, I'm doing this on Windows... I'm using the "VM Size" rather
> than "Mem Usage" in the Windows Task Manager.
You probably want to use the performance monitor. That's what it's for. :-)
--
Darren New / San Diego, CA, USA (PST)
"I think these anchovies are spoiled.
They're not flat."
EKB
who don't know, either, go to:
Control Panel -> Administrative Tools -> Performance Monitor
Once the perf monitor is running, click the "+" button to add an object
to monitor and choose "Process" from the list of "performance objects".
Then add the process you're interested in and the relevant variables to
the display.
You'll also probably want to change the "scale". Do that by selecting
the item you're monitoring (e.g., wish84), right clicking, and
selecting "Properties" from the menu. Then set the scale. The scale is
a multiplication factor, so, e.g., if your process takes up around 12
MB, then set the scale to 0.000001 (1/1,000,000) so it'll show up as a
"12" on the graph.
EKB
behavior. With the changes I've put in (since trying to track this
problem down) that avoid memory-intensive operations, my application is
certainly better behaved.
I'll spend a while continuing to poke through the code, debugging and
bracketing suspect bits of code, and use the Performance Monitor to
check the results. If I come up with anything interesting, I'll post
them on the Tcler's wiki.
Thanks, everybody!