About memory management from Tcl: A way to force tcl to free memory?
Petasis George
I have a quite large application written in tcl+extensions that
offers the end user a working environment that can do various stuff.
All memory allocation is done through tcl routines (Tcl_Malloc ...)
as this memory is shared between tcl and the extensions.
Depending on the processing you do inside the application, you may
for some period of time require large amounts of memory (typical sizes
may be over 500MB) which are freed after this specific processing
finishes and results are saved on disk. The problem is that this
additional memory space is not returned in the OS, since I don't need
it any more. Under some systems (i.e. linux) if malloc/free was used
instead of Tcl_Malloc/Tcl_Free, this memory would have gone back
to the OS. I know that currently there is no way to force
this memory to be returned to the os, and the only solution is to
restart the application :-)
In my opinion there should be a function (at least at the C level)
that would force tcl to free memory that is not needed any more.
Is this feasible? (I suppose it is:-)) How this sounds?
Also at the tcl level such a command would not be bad, i.e. to
free memory after unsetting an array you know it was large. But
providing such functionality at the C api would be trivial to
wrap it into an extension...
George
Andreas Otto
> Hi all,
>
> I have a quite large application written in tcl+extensions that
> offers the end user a working environment that can do various stuff.
> All memory allocation is done through tcl routines (Tcl_Malloc ...)
> as this memory is shared between tcl and the extensions.
.....
>
Hi,
a way to solve this problem is to start the high-memory-using
process as separat tclsh
there is a short paper about the technology:
http://www.compiler-factory.com/MqCon-README_E.html
mfg
aotto :)
--
================================================================
(C) Compiler-Factory Phone: ++49-(0)8152-399540
Dipl.-Ing Andreas Otto mailto:in...@compiler-factory.com
Business Solutions http://www.compiler-factory.com
Ulmenstrasse 3 => "Compiler", FastWeb, OpMenu
D-34289 Zierenberg => C, C++, Tcl, HTML, database,
=================================================================
Petasis George
>
> Petasis George wrote:
>
> > Hi all,
> >
> > I have a quite large application written in tcl+extensions that
> > offers the end user a working environment that can do various stuff.
> > All memory allocation is done through tcl routines (Tcl_Malloc ...)
> > as this memory is shared between tcl and the extensions.
>
> .....
>
> >
> Hi,
>
> a way to solve this problem is to start the high-memory-using
> process as separat tclsh
>
> there is a short paper about the technology:
>
> http://www.compiler-factory.com/MqCon-README_E.html
>
> mfg
>
> aotto :)
Thanks, but this is not feasible in my case. The processing is highly
integrated with the GUI as you see/modify/export results. I really have to
have access to the data. The time that these data will be disposed is
also under user commands, in the GUI. And putting/retrieving so much
information with pipes won't be efficient at all as we speak about
millions of tcl *unicode* objects. I can't imagine this working under
windows :-) I am using this only when spanning external processes, so as
not my GUI to freeze...
Thanks,
George.
David Gravereaux
You can replace the whole Tcl memory manager entirely at compile-time. I played
around with it for a time until I found my real reason I was leaking so much :)
But I see your point. The method goes like this:
1) add -DUSE_TCLALLOC=0 to the compile flags
*2) use your own implementation of malloc/realloc/free
[*=optional]
If you go with the one in msvcrt, no redefining needed, you'll notice a marked
improvement in memory blocks returned. In theory, it will slow down Tcl, but I
don't have any numbers on that.
--
David Gravereaux <davy...@pobox.com>
Chang LI
Petasis George wrote in message <3B0214F1...@iit.demokritos.gr>...
>
I do not think you need this function. OS will use the page management to
collect the inactive pages. Expand your computer to 512MB your application
will run great.
Chang
Phil Ehrens
>In my opinion there should be a function (at least at the C level)
>that would force tcl to free memory that is not needed any more.
>Is this feasible? (I suppose it is:-)) How this sounds?
>Also at the tcl level such a command would not be bad, i.e. to
>free memory after unsetting an array you know it was large. But
>providing such functionality at the C api would be trivial to
>wrap it into an extension...
This is what dlmalloc does. Linux uses dlmalloc by default, with
Sun, for example, you must preload libdlmalloc:
/usr/bin/env LD_PRELOAD=/usr/local/lib/libdlmalloc.so tclsh
See:
http://g.oswego.edu/dl/html/malloc.html
It works just dandy, that's why it's the default memory allocator
for Linux ;^)
It works best on large single allocations.
Phil
Petasis George
So you imply that under linux you see a tcl script allocating much
memory and then *freeing* it? I mean, you create a huge hash table,
making the process to use much memory (lets say 50 MB), then you free this
table (with array unset) and see the memory the tclsh process is
using to *go less* than 50MB? The strange thing is that I haven't
noticed such behaviour:-) Perhaps I haven't looked carefully enough?
George
Donal K. Fellows
> Phil Ehrens wrote:
>> This is what dlmalloc does. Linux uses dlmalloc by default, with
>> Sun, for example, you must preload libdlmalloc:
> memory and then *freeing* it? I mean, you create a huge hash table,
> making the process to use much memory (lets say 50 MB), then you free this
> table (with array unset) and see the memory the tclsh process is
> using to *go less* than 50MB? The strange thing is that I haven't
> noticed such behaviour:-) Perhaps I haven't looked carefully enough?
Not everything is freed. Tcl_Objs are kept around indefinitely on the
assumption that if you used N before, you might need that many again
(and it makes allocation of these frequently-used constructs *much*
faster.) To see the real effects, try allocating and then destroying
(completely) a 250k-element array or a big image. If that still doesn't
dealloc, someone somewhere is being inaccurate...
Donal.
--
Donal K. Fellows http://www.cs.man.ac.uk/~fellowsd/ fell...@cs.man.ac.uk
-- Always running as a superuser is not a fault, it's an OS preference.
-- <mz...@hotmail.com>
Phil Ehrens
>Petasis George wrote:
>> Phil Ehrens wrote:
>>> This is what dlmalloc does. Linux uses dlmalloc by default, with
>>> Sun, for example, you must preload libdlmalloc:
>[...]
>> So you imply that under linux you see a tcl script allocating much
>> memory and then *freeing* it? I mean, you create a huge hash table,
>> making the process to use much memory (lets say 50 MB), then you free this
>> table (with array unset) and see the memory the tclsh process is
>> using to *go less* than 50MB? The strange thing is that I haven't
>> noticed such behaviour:-) Perhaps I haven't looked carefully enough?
>
>Not everything is freed. Tcl_Objs are kept around indefinitely on the
>assumption that if you used N before, you might need that many again
>(and it makes allocation of these frequently-used constructs *much*
>faster.) To see the real effects, try allocating and then destroying
>(completely) a 250k-element array or a big image. If that still doesn't
>dealloc, someone somewhere is being inaccurate...
>
>Donal.
Better yet, allocate a 500Mb object all in one shot, and then free it.
That's what we do and yes it works. See the source for dlmalloc and
stop implying that I'm loopy. I posted the URL, it's your move George.
Does Tcl ever allocate large chunks all at once? Our number crunching
code sure does, and yes we get it back with dlmalloc.
Frank Pilhofer
>
> Better yet, allocate a 500Mb object all in one shot, and then free it.
> That's what we do and yes it works. See the source for dlmalloc and
> stop implying that I'm loopy. I posted the URL, it's your move George.
> Does Tcl ever allocate large chunks all at once? Our number crunching
> code sure does, and yes we get it back with dlmalloc.
>
Anyway, releasing memory back to the OS is a problematic issue.
All in all, the only cross-platform solution is to use Shared Memory.
Frank
--
Frank Pilhofer ........................................... f...@fpx.de
An argument is two people trying to get the last word in first.
- Alfred E. Neuman
Donal K. Fellows
.ligo.caltech.edu> writes
>Better yet, allocate a 500Mb object all in one shot, and then free it.
>That's what we do and yes it works. See the source for dlmalloc and
>stop implying that I'm loopy. I posted the URL, it's your move George.
>Does Tcl ever allocate large chunks all at once? Our number crunching
>code sure does, and yes we get it back with dlmalloc.
While Tcl definitely allocates fairly large amounts at once, you're not
necessarily going to reap the benefits. Places where you are most
likely to see something are: large strings[*], large arrays[**] and, if
you're using Tk, large images. Large allocation is also used elsewhere,
but reclamation is likely to be problematic due to the way things get
subdivided internally. <shrugs>
Donal.
[* Particularly if they have both UTF8 and UNICODE reps. ]
[** But only if you nuke the entire array, and not just delete all the
elements in it. ]
--
Donal K. Fellows (at home)
--
FOOLED you! Absorb EGO SHATTERING impulse rays, polyester poltroon!!
(WARNING: There is precisely one error in this message.)
Petasis George
know what it means :-)) but I think you have not understand what
I am trying to point out. The problem is not allocating and freing
large objects (like big images) but rather using at some point
million of tiny objects and then freing them. I think the problem
here is what Donal said "Tcl_Objs are kept around indefinitely on the
assumption that if you used N before, you might need that many again..."
Try the following script:
proc CreateHugeArray {} {
global Array
for {set i 0} {$i < 500000} {incr i} {
set Array("string_$i") "This is the value with key $i"
}
}
puts "Filling array..."
CreateHugeArray
puts "Done. Press any return to continue..."
gets stdin
puts "Unsetting array..."
unset Array
puts "Done. Press any return to exit..."
gets stdin
exit
It allocates 76MB on solaris. When the array is freed, nothing is freed.
Using dlmalloc uses 72MB and finally 1MB is freed, but this is not the
point.
It is not a problem with the allocator, but rather a tcl problem. I can
understand
that we want to have some objects already allocated so as to use them,
but shouldn't we also have a function to force releasing these objects?
Resetting this "cache"? What if a script creates a huge array, does
something
and then repeats the same thing after hours? Why not having a way to return
all this unused for hours memory back to other processes, untill we need it?
It is ok to hold a few thousants of objects handy, but we shouldn't keep
millions.
At least to offer the developer a way to free them.
In my opinion such a function is needed. Anybody who needs it should use it.
No changes to the way tcl uses memory now. I don't know if it is
feasible though :-)
George
Donal K. Fellows
> Well sorry if I implied anything (especially "loopy" which I don't
> know what it means :-)) but I think you have not understand what
> I am trying to point out. The problem is not allocating and freing
> large objects (like big images) but rather using at some point
> million of tiny objects and then freing them. I think the problem
> here is what Donal said "Tcl_Objs are kept around indefinitely on the
> assumption that if you used N before, you might need that many again..."
> It is not a problem with the allocator, but rather a tcl problem. I can
> understand that we want to have some objects already allocated so as to
> use them, but shouldn't we also have a function to force releasing these
> objects? Resetting this "cache"?
Difficult to actually do, since memory for Tcl_Objs is allocated in
large chunks and then subdivided. Figuring out if all the objects are
free (the necessary condition for safely releasing the memory back to
the OS) is *interesting*, given that the free list is not kept sorted
(which would be expensive.)
<braindump>
Hmm. A compromise might be to mark each freed object with a special
object type, and to scan (on the release of an object) its block to see
if the underlying block is all free, which would be a signal to reclaim
the memory itself (though this would also force a possibly-expensive
scan of the free list.) Alternatively, you could keep a free-list per
block and use bit-masking to get the location of the pointer to the
start of the per-block free list (which would force the use of a
system-page allocator for this purpose.) You could also mark some
blocks as not deallocatable, allowing you to keep a few blocks so small
scripts would run fast. That would give a per-block overhead of 16
bytes (assuming 32-bit machine) something like this:
struct TclObjBlock {
Tcl_Obj *freeList;
struct TclObjBlock *previousBlock;
struct TclObjBlock *nextBlock;
int flags; /* For marking non-free()able */
Tcl_Obj vector[2]; /* Actual objects, must be last */
}
A quick calculation indicates that this gives 170 objects-per-block:
170 * 24 + 16 = 4096 (the standard page size on 32-bit architectures.)
It would also be possible to overload the flags word with a count of the
number of free objects in the block (low 8 bits for flags would leave
plenty of room for object count on all architectures.)
The only problem would come if there was thrashing of free pages, which
could happen if a program happens to oscillate back and forth across the
boundary between two pages (e.g. if it needed 169 objects, then 171,
then 169, then 171, etc.) But that's trivial to fix by not deallocating
a page when it first becomes free, but only when another page becomes
free after it, so giving a buffer zone of 170 objects which is enough
for most small loops.
</braindump>
> What if a script creates a huge array, does something
> and then repeats the same thing after hours? Why not having a way to return
> all this unused for hours memory back to other processes, untill we need it?
> It is ok to hold a few thousants of objects handy, but we shouldn't keep
> millions. At least to offer the developer a way to free them.
> In my opinion such a function is needed. Anybody who needs it should use it.
> No changes to the way tcl uses memory now. I don't know if it is
> feasible though :-)
You can't reset the cache, but you could easily modify generic/tclObj.c
to not use this allocation scheme at all. Indeed, compile with
TCL_MEM_DEBUG turned on and it will do exactly this (plus a load of other
things that are less desirable in performance terms.) Maybe it would be
worthwhile investigating doing this? (AFAIK, this sort of allocation
cacheing is only done for Tcl_Objs; everything else that is freed should
pass through to a real free()...)
Donal.
--
Donal K. Fellows http://www.cs.man.ac.uk/~fellowsd/ fell...@cs.man.ac.uk
-- Actually, come to think of it, I don't think your opponent, your audience,
or the metropolitan Tokyo area would be in much better shape.
-- Jeff Huo <je...@starfall.com.nospam>
Phil Ehrens
all at once in a GUI?? And if you are really going to do that, I think
you are better off building a big map in memory, blitting hunks of it,
and then freeing it, possibly on GUI inactivity.
Petasis George
Have I ever mentioned GUI in my postings?
I am talking about huge arrays, using tcl objects for storing data,
or processes in tcl that run for long period of time.
How have you got that idea?
I am not talking about some special uses of tcl, where the fact that tcl
objects
are never freed once allocated can lead to unnessesary memory use.
I am not talking about any specific application, or about GUIs or anything.
Just pointing a shortcumming and trying to see if we can do anything to
remove it.
And finally, what loopy means?
George
Phil Ehrens
>Have I ever mentioned GUI in my postings?
>I am talking about huge arrays, using tcl objects for storing data,
>or processes in tcl that run for long period of time.
>How have you got that idea?
>I am not talking about some special uses of tcl, where the fact that tcl
>objects
>are never freed once allocated can lead to unnessesary memory use.
>I am not talking about any specific application, or about GUIs or anything.
>Just pointing a shortcumming and trying to see if we can do anything to
>remove it.
Okay, sorry. In any case, trying to give up anything back to the
OS that is smaller than some threshhold size is never going to
be efficient. dlmalloc returns memory to the OS when it is allocated
in chunks beyond a tunable threshhold, and there is probably no need
for anything else. Why you doubt that dlmalloc works as advertised
is beyond me.
I have Tcl processes that run for months, and they do not leak
memory, and behave exactly like any other kind of long running
processes... They do not grow in size endlessly.
In general, attempts to micro-manage memory result in unexpected
problems. Have you ever seen what happens when a prog that is
very careful about allocating memory allocates the last page on
a system? Much better to try and whack off a big hunk and have
it fail than to run into the oom-killer.
It would be interesting to construct a Tcl that was dlmalloc aware
and have it, say, allocate in big hunks for some definable
namespaces (using the dlmalloc threshhold value), and then when
the namespace is deleted, voila!
Here is something you may find educational:
http://www.ligo.caltech.edu/~pehrens/tcl-dlmalloc.tar.bz2
The package itself is for Solaris, but the Tcl interface
should work with any *NIX.
>And finally, what loopy means?
You know, loopy. Like when I read "objects" as "widgets" in your
lat post...
Phil
Petasis George
> Okay, sorry. In any case, trying to give up anything back to the
> OS that is smaller than some threshhold size is never going to
> be efficient. dlmalloc returns memory to the OS when it is allocated
> in chunks beyond a tunable threshhold, and there is probably no need
> for anything else. Why you doubt that dlmalloc works as advertised
> is beyond me.
I never said that dlmalloc or any other malloc implementation is not
working. My problem is that tcl will never free a TclObj once allocated.
It will never call free on it, not matter if it is unused. Its a tcl
feature, not a bug in malloc/free implementations.
As Donal explained, when new TclObj's are needed, tcl allocates
a memory space that can contain more than one object (arount 170).
One of them is returned. The others are reserved when more new
objects are needed. When an object gets freed, it is simply added
to a list of free objects of this space. But this space is never
freed. If you request 1.000.000 TclObjs, tcl will allocate
the necessary space and return them to you.
Now if you free ALL of them, all allocated memory won't be
given back to the os, simply because tcl will keep this
space and will never call free on it, not because the malloc/free
of each system is not working.
>
> I have Tcl processes that run for months, and they do not leak
> memory, and behave exactly like any other kind of long running
> processes... They do not grow in size endlessly.
I don't say it leaks memory. Yes but if at some point the script
does something that requires much memory, then the memory allocated
by the script won't be given back to the os when the processing is
done and all objects are freed.
>
> In general, attempts to micro-manage memory result in unexpected
> problems. Have you ever seen what happens when a prog that is
> very careful about allocating memory allocates the last page on
> a system? Much better to try and whack off a big hunk and have
> it fail than to run into the oom-killer.
I am not trying to micro-manage memory. What I am trying to do is
first to add a kernel function that will check all these memory spaces
allocated by tcl. If such an area has only unallocated objects, this area
should be freed. This function will *never* be called by the core!
So no overhead. But me, as a programmer of a specific application
*will* know when to call it. And perhaps, after we test it we could
find places in the core that should be called, perhaps when deleteing
arrays that have more than a fixed number of keys, or something similar...
> >And finally, what loopy means?
>
> You know, loopy. Like when I read "objects" as "widgets" in your
> lat post...
I got confused:-) The only meaning I was aware was about loops, iterations
and so. And my electronic english-greek dictionary is not so advanced:-)
George
Petasis George
> Difficult to actually do, since memory for Tcl_Objs is allocated in
> large chunks and then subdivided. Figuring out if all the objects are
> free (the necessary condition for safely releasing the memory back to
> the OS) is *interesting*, given that the free list is not kept sorted
> (which would be expensive.)
>
Note that this shouldn't be done when every object is freed.
My proposition is to simply add a public C function that will do
exactly this, i.e. check all allocated memory chunks and free those
that can be freed. This will never be called from the core.
But we offer the freedom to C extension developers to call it,
since they will know when they freed many objects.
(I already have uses for such a function :-))
And perhaps later we can find potential uses to the core,
for example when we delete an array that has many many keys...
But still I have to more closely examine the code in tclObj.c,
to see if this can be done without adding overhead to the current
implementation. Just waiting the upgrade to our solaris machines to
be completed :-)
George
Volker Hetzer
> I never said that dlmalloc or any other malloc implementation is not
> working. My problem is that tcl will never free a TclObj once allocated.
> It will never call free on it, not matter if it is unused. Its a tcl
> feature, not a bug in malloc/free implementations.
On my platform the tclAlloc.c gets compiled with -DUSE_TCLALLOC=0 by default.
Incidentally, this has no effect on memory usage for arrays. Don't know why yet.
Greetings!
Volker
--
They laughed at Galileo. They laughed at Copernicus. They laughed at
Columbus. But remember, they also laughed at Bozo the Clown.
Donal K. Fellows
> As Donal explained, when new TclObj's are needed, tcl allocates
> a memory space that can contain more than one object (arount 170).
No, Tcl currently allocates 100 at a time. It *should* allocate 170 and
use a paging-based allocator instead of plain-old malloc(), but doesn't.
Donal.
--
Donal K. Fellows http://www.cs.man.ac.uk/~fellowsd/ fell...@cs.man.ac.uk
-- With a complex beast like Swing, it's not just a matter of "What button
should I push", but rather "How do I put myself into a nice metamorphosis
so that I am deemed acceptable by the Swing Gods." -- Anonymous
Donal K. Fellows
> Petasis George wrote:
>> I never said that dlmalloc or any other malloc implementation is not
>> working. My problem is that tcl will never free a TclObj once allocated.
>> It will never call free on it, not matter if it is unused. Its a tcl
>> feature, not a bug in malloc/free implementations.
> What platform are you working on?
Doesn't matter. Try looking at generic/tclObj.c for enlightenment.
Donal.
--
Donal K. Fellows http://www.cs.man.ac.uk/~fellowsd/ fell...@cs.man.ac.uk
Volker Hetzer
>
> Volker Hetzer wrote:
> > Petasis George wrote:
> >> I never said that dlmalloc or any other malloc implementation is not
> >> working. My problem is that tcl will never free a TclObj once allocated.
> >> It will never call free on it, not matter if it is unused. Its a tcl
> >> feature, not a bug in malloc/free implementations.
> > What platform are you working on?
>
> Doesn't matter. Try looking at generic/tclObj.c for enlightenment.
IMHO that should be changeable.
I'll have a look into it but can't guarantee a patch yet.
Volker Hetzer
>
> "Donal K. Fellows" wrote:
> >
> > Volker Hetzer wrote:
> > > Petasis George wrote:
> > >> I never said that dlmalloc or any other malloc implementation is not
> > >> working. My problem is that tcl will never free a TclObj once allocated.
> > >> It will never call free on it, not matter if it is unused. Its a tcl
> > >> feature, not a bug in malloc/free implementations.
> > > What platform are you working on?
> >
> > Doesn't matter. Try looking at generic/tclObj.c for enlightenment.
> I see. TclFreeObj does indeed never free an object.
> IMHO that should be changeable.
> I'll have a look into it but can't guarantee a patch yet.
Here's a piece of code:
void
TclAllocateFreeObjects()
{
Tcl_Obj *prevPtr, *objPtr;
int i;
prevPtr = NULL;
for (i = 0; i < OBJS_TO_ALLOC_EACH_TIME; i++)
{
objPtr=malloc(sizeof(Tcl_Obj)*sizeof(char));
assert(objPtr!=NULL);
objPtr->typePtr=malloc(sizeof(Tcl_ObjType)*sizeof(char));
assert(objPtr->typePtr!=NULL);
objPtr->internalRep.otherValuePtr = (VOID *) prevPtr;
prevPtr = objPtr;
}
tclFreeObjList = prevPtr;
}
void
TclFreeObj(objPtr)
register Tcl_Obj *objPtr; /* The object to be freed. */
{
register Tcl_ObjType *typePtr = objPtr->typePtr;
if ((typePtr != NULL) && (typePtr->freeIntRepProc != NULL)) {
typePtr->freeIntRepProc(objPtr);
}
Tcl_InvalidateStringRep(objPtr);
/*
* If debugging Tcl's memory usage, deallocate the object using ckfree.
* Otherwise, deallocate it by adding it onto the list of free
* Tcl_Obj structs we maintain.
*/
Tcl_MutexLock(&tclObjMutex);
fprintf(stderr,".");
free(typePtr);free(objPtr);
Tcl_MutexUnlock(&tclObjMutex);
}
And this is the test code:
% for {set i 0} {$i<100000} {incr i} {set X$i $i}
.......% for {set i 0} {$i<100000} {incr i} {unset X$i}
.......%
-> The unset doesn't free the objects. Any hints?
Donal K. Fellows
> I see. TclFreeObj does indeed never free an object.
> IMHO that should be changeable.
> I'll have a look into it but can't guarantee a patch yet.
The easiest way to change this is to avoid the use of a free object
list and directly use ckalloc/ckfree each time. But Tcl uses a *lot*
of objects, so you'll take a fair performance hit for doing that...
Volker Hetzer
>
> Volker Hetzer wrote:
> > I see. TclFreeObj does indeed never free an object.
> > IMHO that should be changeable.
> > I'll have a look into it but can't guarantee a patch yet.
>
> The easiest way to change this is to avoid the use of a free object
> list and directly use ckalloc/ckfree each time. But Tcl uses a *lot*
> of objects, so you'll take a fair performance hit for doing that...
It redefines the ckalloc stuff and creates some debugging-Output in tclObj.
It shows that Objects set by the "set" command don't get freed
with the "unset" command.
Example script:
for {set I 0} {$I<100000} {incr I} {set X$I $I}
for {set I 0} {$I<100000} {incr I} {unset X$I}
Also the memory consumtion does not decrease.
Any hints where to look further?
Volker Hetzer
> I've done a little patch for debugging purposes.
> It redefines the ckalloc stuff and creates some debugging-Output in tclObj.
> It shows that Objects set by the "set" command don't get freed
and both get called quite often ('bout ten times per set or unset) but memory
consumption is still high.
Next thing'll be mem_debug.
David Gravereaux
>Petasis George wrote:
>> As Donal explained, when new TclObj's are needed, tcl allocates
>> a memory space that can contain more than one object (arount 170).
>
>No, Tcl currently allocates 100 at a time. It *should* allocate 170 and
>use a paging-based allocator instead of plain-old malloc(), but doesn't.
Donal,
A long time ago I look at this myself. At least tclAlloc.c, and saw it allocs
blocks by about 32-bytes (i don't remember exactly) just over the page boundary
in MoreCore() calling TclpSysAlloc() which happens to be HeapAlloc() on windows.
HeapAlloc() is managed memory. The Tcl allocator is already a manager. [2 sets
of overhead data!]. VirtualAlloc() would be the preferred system allocator in
this case, as VirtualFree() is 100% guaranteed to give the amount back to the
system. If, in fact, that feature for returning a fully free block existed.
[2 cents inserted. Where do i collect my credits?]
--
David Gravereaux <davy...@pobox.com>
Donal K. Fellows
> A long time ago I look at this myself. At least tclAlloc.c, and saw it allocs
> blocks by about 32-bytes (i don't remember exactly) just over the page boundary
> in MoreCore() calling TclpSysAlloc() which happens to be HeapAlloc() on windows.
> HeapAlloc() is managed memory. The Tcl allocator is already a manager. [2 sets
> of overhead data!]. VirtualAlloc() would be the preferred system allocator in
> this case, as VirtualFree() is 100% guaranteed to give the amount back to the
> system. If, in fact, that feature for returning a fully free block existed.
After examination of the code ("helpfully" split across many files) I
see that on some platforms at least, it just falls through to the
standard libc allocator, but not on Windows. I don't really recall
the details of why that is though.
Handing back blocks is nice, but in many applications the cost of
deciding when a block is free is such that it is simpler to just hang
onto everything and give it back at program termination...
> [2 cents inserted. Where do i collect my credits?]
Try a 2-cent-credit dispenser. :^)
Donal.
--
Donal K. Fellows http://www.cs.man.ac.uk/~fellowsd/ fell...@cs.man.ac.uk
-- OK, there is the MFC, but it only makes the chaos object orientated.
-- Thomas Nellessen <nell...@gmx.de>
Jeff Hobbs
> see that on some platforms at least, it just falls through to the
> standard libc allocator, but not on Windows. I don't really recall
> the details of why that is though.
Up until at least VC++5, the mem allocator in Tcl was better than
the standard system one - that's why. This should be reevaluated
with VC++6. It shouldn't be hard - just compile two static builds
of Tcl - one with -DTCL_ALLOC=0 and one with -DTCL_ALLOC=1 and
run them over tclbench. Unfortunately I use my Windows machine
quite a bit to sacrifice the >1hour run you need to leave the
machine alone for.
BTW, one person found the Tcl allocator to be quite useful when
some version of HP had a bug relating to the internal allocator.
So even if we don't use it, I'd still leave the code there.
--
Jeff Hobbs The Tcl Guy
Senior Developer http://www.ActiveState.com/
Andreas Kupries
Jeff Hobbs <Je...@ActiveState.com> writes:
> Donal wrote:
>> After examination of the code ("helpfully" split across many files)
>> I see that on some platforms at least, it just falls through to the
>> standard libc allocator, but not on Windows. I don't really recall
>> the details of why that is though.
> This should be reevaluated with VC++6. It shouldn't be hard - just
> compile two static builds of Tcl - one with -DTCL_ALLOC=0 and one
> with -DTCL_ALLOC=1 and run them over tclbench.
> Unfortunately I use my Windows machine quite a bit to sacrifice the
> >1hour run you need to leave the machine alone for.
How about a cron-job (or how it is called for windows) and doing it
during the night ?
--
Sincerely,
Andreas Kupries <a.ku...@westend.com>
Developer @ <http://www.activestate.com/>
Private <http://www.purl.org/NET/akupries/>
-------------------------------------------------------------------------------
David Gravereaux
>How about a cron-job (or how it is called for windows) and doing it
>during the night ?
Yeah, that's the task scheduler.
Microsoft Windows 2000 [Version 5.00.2195]
(C) Copyright 1985-2000 Microsoft Corp.
C:\WINNT\system32>at
The service has not been started.
C:\WINNT\system32>at /?
The AT command schedules commands and programs to run on a computer at
a specified time and date. The Schedule service must be running to use
the AT command.
AT [\\computername] [ [id] [/DELETE] | /DELETE [/YES]]
AT [\\computername] time [/INTERACTIVE]
[ /EVERY:date[,...] | /NEXT:date[,...]] "command"
\\computername Specifies a remote computer. Commands are scheduled on the
local computer if this parameter is omitted.
id Is an identification number assigned to a scheduled
command.
/delete Cancels a scheduled command. If id is omitted, all the
scheduled commands on the computer are canceled.
/yes Used with cancel all jobs command when no further
confirmation is desired.
time Specifies the time when command is to run.
/interactive Allows the job to interact with the desktop of the user
who is logged on at the time the job runs.
/every:date[,...] Runs the command on each specified day(s) of the week or
month. If date is omitted, the current day of the month
is assumed.
/next:date[,...] Runs the specified command on the next occurrence of the
day (for example, next Thursday). If date is omitted, the
current day of the month is assumed.
"command" Is the Windows NT command, or batch program to be run.
C:\WINNT\system32>
--
David Gravereaux <davy...@pobox.com>
Jeffrey Hobbs
>
> > After examination of the code ("helpfully" split across many files) I
> > see that on some platforms at least, it just falls through to the
> > standard libc allocator, but not on Windows. I don't really recall
> > the details of why that is though.
>
> Up until at least VC++5, the mem allocator in Tcl was better than
> the standard system one - that's why. This should be reevaluated
> with VC++6. It shouldn't be hard - just compile two static builds
> of Tcl - one with -DTCL_ALLOC=0 and one with -DTCL_ALLOC=1 and
> run them over tclbench. Unfortunately I use my Windows machine
OK, I finally got around to this and the results where....
no difference. The Tcl based version ran 3 seconds faster
at 15:12 vs 15:15 for the -DUSE_TCLALLOC=0 version. Almost
all the tests were within 1% variance of each other. That
was using VC++6. I don't have anything else.
--
Jeff Hobbs The Tcl Guy
Senior Developer http://www.ActiveState.com/
Tcl Support and Productivity Solutions
Ludwig Callewaert
Being that person, I would prefer that the code remains in place
as I am still using it. The HP problem does not seem to be solved
so far.
--
Ludwig Callewaert e-mail: ludwig_c...@frontierd.com
Senior Software Engineer Frontier Design, Belgium
__________________________________________________________________
Petasis George
After looking how tcl objects are allocated, it seems
impossible to actually free the unused ones.
There is indeed a list that holds the objects
(unsorted as Donal pointed). When a new block
is allocated, the newly allocated objects
are appended to this list in reverse order, i.e.
the last object in the allocated block will be inserted
first in the list. This list *is not* a tcl list object
(and why should it be?) but rather a simple linked
list where each tcl object points to another one through one
of its available pointer. So, this list does not need
extra memory for itself.
Taking as a starting point the fact that we don't want
to further increase memory consumption (as we already
have complains about memory), I tried to implement this
"freeing" function as follows:
I place all free object pointers in an array and sort this
array with qsort. As I know that always a fixed number of objects
is allocated, I tried to identify areas in this array
where all 100 succesive pointers differ by the same padding.
These 100 objects have high posibility to belong to the same
allocated memory segment, but can I be sure so as to free it?
For example, if I request 200 objects is it possible that
succesive memory segments are allocated, so all 200 pointes
differ by the same padding?
George
Volker Hetzer
>
> Jeff Hobbs wrote:
> > BTW, one person found the Tcl allocator to be quite useful when
> > some version of HP had a bug relating to the internal allocator.
> > So even if we don't use it, I'd still leave the code there.
> >
> > --
> > Jeff Hobbs The Tcl Guy
> > Senior Developer http://www.ActiveState.com/
>
> Being that person, I would prefer that the code remains in place
> as I am still using it. The HP problem does not seem to be solved
> so far.
Ludwig Callewaert
>
...
> I'm using tcl on HP too. What problem are you referring to?
>
> Greetings!
> Volker
> --
> They laughed at Galileo. They laughed at Copernicus. They laughed at
> Columbus. But remember, they also laughed at Bozo the Clown.
Last year I posted a TCL performance problem which was due to a problem
with the HP-UX allocator. This problem was reported to HP but we did get
no reaction so far.
(Just search google groups with: " performance group:comp.lang.tcl author:Callewaert"
and you will find the original posting).
The problem has to do with the text widget and lots of tagged text.
Using the TCL built-in allocator meant 2 seconds opening time of a report
instead of minutes, which was a considerable improvement.
The positive news is that with the latest 8.3.3 version built on
an HP-UX 11 system, the problem does not occur any longer. It still
does occur with our 8.2 version built on an HP-UX 10.20. The difference
most likely is due to the different libc version being used.
Regards,
Ludwig
Jeff Hobbs
...
> I place all free object pointers in an array and sort this
> array with qsort. As I know that always a fixed number of objects
> is allocated, I tried to identify areas in this array
> where all 100 succesive pointers differ by the same padding.
> These 100 objects have high posibility to belong to the same
> allocated memory segment, but can I be sure so as to free it?
> For example, if I request 200 objects is it possible that
> succesive memory segments are allocated, so all 200 pointes
> differ by the same padding?
If you actually do all the checking to make sure that there are
100 successive pointers, you may be able to free those - if you
are starting at the right spot. However, it's highly unlikely
in practical code that the allocated objects are going to be
freed all together. If you start to amass 100s (or 1000s) of
now unused objects, some garbage collection might be appropriate.
It would be interesting to apply some stat information in there
to really see how much waste occurs in non-trivial, practical
applications.
Petasis George
>
> If you actually do all the checking to make sure that there are
> 100 successive pointers, you may be able to free those - if you
> are starting at the right spot.
Well I suppose that is my big problem. Yes, I can easily find 100
successive pointers, but is the address of the first one the right
spot?
> However, it's highly unlikely
> in practical code that the allocated objects are going to be
> freed all together. If you start to amass 100s (or 1000s) of
> now unused objects, some garbage collection might be appropriate.
> It would be interesting to apply some stat information in there
> to really see how much waste occurs in non-trivial, practical
> applications.
I have tried to simply get an image of the number of objects
that are free at any time.
On simple tcl scripts, indeed the number of objects is quite small,
around 50. But the situation changes when you use arrays that then
are freed. You can end with 10.000 unused object with an array with
a few thousant keys. Unless you need again such an array, all these
objects are not going to be needed. After doing something typical
in my application (causing wish to reach about 16 MB ram) I end up
with more than 60.000 freed objects after processing is done.
Since I have already code to examine this free list under my system,
I can collect such statistics on specific tcl scripts (benchmarks?)
if you are interested.
George
Andreas Kupries
Petasis George <pet...@iit.demokritos.gr> writes:
> I have tried to simply get an image of the number of objects that
> are free at any time. On simple tcl scripts, indeed the number of
> objects is quite small, around 50. But the situation changes when
> you use arrays that then are freed. You can end with 10.000 unused
> object with an array with a few thousant keys. Unless you need again
> such an array, all these objects are not going to be needed. After
> doing something typical in my application (causing wish to reach
> about 16 MB ram) I end up with more than 60.000 freed objects after
> processing is done.
> Since I have already code to examine this free list under my system,
> I can collect such statistics on specific tcl scripts (benchmarks?)
> if you are interested.
In terms of benchmarks we currently only have the testsuite
distributed with the sources and the scripts in tclbench. The first is
to ensure correctness of the core, the other to measure its
performance.
Would you mind to start writing benchmarks to measure memory usage ?
I believe that we could integrate this into tclbench, with the numbers
being "used memory" or similar instead of time [x].
I have to admit that while reading this thread I became curious if
this whole thing of allocating blocks of Tcl_Obj's is really
necessary, how much we could gain in memory from dropping it and how
much loss of performance there would be [*]. Right now it looks as if
Tcl does with the blocking something in its library the memory
allocator might be better able to do. Especially optimized allocators
like 'dlmalloc' or 'gnumalloc'.
[x] The main management script (runbench.tcl) just runs the benchmark
scripts gets some numbers back and formats them for output in
various styles. The benchmark library implementing the [bench] cmd
would need some modification/enhancement but not much.
[*] And not allocating blocks also means that the can implement
various strategies for the free-list:
- Infinite size (current mode)
- No list at all.
- List with a maximum size.
- Dynamic maximum size, based on memory load and aging for
returning the free objects ...
- even more complex algorithms ...
Jeffrey Hobbs
> I have to admit that while reading this thread I became curious if
> this whole thing of allocating blocks of Tcl_Obj's is really
> necessary, how much we could gain in memory from dropping it and how
> much loss of performance there would be [*]. Right now it looks as if
I did test the difference between the current behavior (block
allocation with infinite life Tcl_Obj's) and simple alloc/free
(ckalloc for each new obj, and a ckfree on free), and it did
have a noticable effect on speed (in favor of the current
behavior of course).
> [*] And not allocating blocks also means that they can implement
> various strategies for the free-list:
>
> - Infinite size (current mode)
> - No list at all.
Basically the two above.
> - List with a maximum size.
This may be reasoably doable and a good compromise.
> - Dynamic maximum size, based on memory load and aging for
> returning the free objects ...
> - even more complex algorithms ...
This may just get you into trouble for making one use of Tcl
slower than another...
--
Jeff Hobbs The Tcl Guy
Senior Developer http://www.ActiveState.com/
Petasis George
> In terms of benchmarks we currently only have the testsuite
> distributed with the sources and the scripts in tclbench. The first is
> to ensure correctness of the core, the other to measure its
> performance.
>
> Would you mind to start writing benchmarks to measure memory usage ?
>
> I believe that we could integrate this into tclbench, with the numbers
> being "used memory" or similar instead of time [x].
I will try. The only problem is that I haven't even run the benchmark suite
so I have to first get familiar with it. Of course this suite won't be
portable, as not all systems offer the ability to see how many memory
the app occupies (i.e. windows 95/98/Millenium). As a first goal of course
we have to measure the number of free objects that tcl holds after doing
and freeing various stuff... Probably this will require a compile define,
that registers a new "memory" command?
>
> I have to admit that while reading this thread I became curious if
> this whole thing of allocating blocks of Tcl_Obj's is really
> necessary, how much we could gain in memory from dropping it and how
> much loss of performance there would be [*]. Right now it looks as if
> Tcl does with the blocking something in its library the memory
> allocator might be better able to do. Especially optimized allocators
> like 'dlmalloc' or 'gnumalloc'.
Well we have to measure the performance under various systems. I recall
that recently somebody reported that under windows at least the performance
is not affected by removing it. But also some others reported that due to
various problems, this is needed under some systems (HP ?)
> - Infinite size (current mode)
> - No list at all.
> - List with a maximum size.
> - Dynamic maximum size, based on memory load and aging for
> returning the free objects ...
> - even more complex algorithms ...
The nicest thing is that list works without using a single byte more that
the memory required by the objects :-)
George
Donal K. Fellows
> Jeff Hobbs wrote:
>> If you actually do all the checking to make sure that there are
>> 100 successive pointers, you may be able to free those - if you
>> are starting at the right spot.
>
> Well I suppose that is my big problem. Yes, I can easily find 100
> successive pointers, but is the address of the first one the right
> spot?
'Tis quite a difficult task; I've been working on some code that
takes this particular bull by the horns, and it now seems to work
at least somewhat. The following test files fail:
autoMkindex.test
interp.test
pkgMkIndex.test
safe.test
In all cases, the mode of failure is that of apparently hanging.
No idea why as yet. (Warning: the code is *not* portable! Use at
your own risk.)
Donal.
--
Donal K. Fellows http://www.cs.man.ac.uk/~fellowsd/ fell...@cs.man.ac.uk
-- If I teach this course thoroughly enough, none of you will attempt the exam
questions on this topic, and I shall consider this to be a complete success.
-- Arthur Norman
Andreas Kupries
Petasis George <pet...@iit.demokritos.gr> writes:
> Andreas Kupries wrote:
> > In terms of benchmarks we currently only have the testsuite
> > distributed with the sources and the scripts in tclbench. The first is
> > to ensure correctness of the core, the other to measure its
> > performance.
> > Would you mind to start writing benchmarks to measure memory usage ?
> > I believe that we could integrate this into tclbench, with the numbers
> > being "used memory" or similar instead of time [x].
> I will try. The only problem is that I haven't even run the
> benchmark suite so I have to first get familiar with it. Of course
> this suite won't be portable, as not all systems offer the ability
> to see how many memory the app occupies (i.e. windows
> 95/98/Millenium). As a first goal of course we have to measure the
> number of free objects that tcl holds after doing and freeing
> various stuff... Probably this will require a compile define, that
> registers a new "memory" command?
Look into the existing "TCL_MEM_DEBUG" switch and the [memory] command
it activates.