limit on Win32 # threads?
usene...@googlemail.com
The simple script below fails on thread::create right under 12,000
iterations.
Attempts to create any more threads fail in the same shell once a call
to
thread::create fails.
Restarting tclsh and rerunning seems to work fine within the limits
mentioned
above.
Would anyone know the reason this would fail?
---------------
tclsh
package require Thread
set i 1
while { 1 } {
set id [thread::create]
thread::release $id
puts -nonewline " $i "
flush stdout
update
incr i
}
TIA for all help,
-joe
David Gravereaux
Hash: SHA1
You probably want more physical memory to create more threads, is my
guess. But honestly, why do you consider this a problem? Any viable
system that requires 12K threads is going to be a wash in thread context
switches for probably as much as 20% of wasted CPU time... Add in the
tcl sync locks, and you're not getting much work done, unless you have
have a 12K CPU motherboard to go with those threads. <wink>
http://en.wikipedia.org/wiki/Amdahl's_law
http://en.wikipedia.org/wiki/Gustafson%27s_Law
usene...@googlemail.com wrote:
> I am running tcl 8.4 on Windows XP Pro 2002, SP2.
> The simple script below fails on thread::create right under 12,000
> iterations.
>
> Attempts to create any more threads fail in the same shell once a call
> to
> thread::create fails.
> Restarting tclsh and rerunning seems to work fine within the limits
> mentioned
> above.
>
> Would anyone know the reason this would fail?
Yeah, you reached a realistic system limit. Threads aren't "free".
>
>
> ---------------
> tclsh
> package require Thread
> set i 1
> while { 1 } {
> set id [thread::create]
> thread::release $id
> puts -nonewline " $i "
> flush stdout
> update
> incr i
> }
>
> TIA for all help,
> -joe
>
- --
David Gravereaux <davy...@pobox.com>
[species:human; planet:earth,milkyway(western spiral arm),alpha sector]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (MingW32)
iD8DBQFEtA12lZadkQh/RmERAjCmAKDQtNVit92sMdH5aN4VFX7kKY6LIgCg5aZ7
455YENNRFWAlgcHEjIOZtvc=
=oqcE
-----END PGP SIGNATURE-----
usene...@googlemail.com
Thanks for taking the time to reply David.
I don't need 12k threads simultaneously. Just sequentially.
So at any given point in time, there are only two threads active -
the main tclsh and one that my application creates.
And the reason I'd like this is to try and workaround a problem I am
seeing with the tcom library. It doesn't seem to release all resources
when working with Excel files. In spite of being fastidious with the
resource/variable tracking, I can't seem to get it right.
So the thought was that opening a new thread for each excel file
(working on them sequentially - one at a time), I'd be able to work
around
the problem.
I do need new threads to avoid problems with any shared data between
calls
to the dll.
So I anticipate circumventing most of the issues you raise.
-joe
Uwe Klein
>>---------------
>>tclsh
>>package require Thread
>>set i 1
>>while { 1 } {
>> set id [thread::create]
>> thread::release $id
<< thread::release -wait $id
>> puts -nonewline " $i "
>> flush stdout
>> update
>> incr i
>>}
Do you run into the same limit when using
the -wait option to [thread::release]?
uwe
David Gravereaux
Hash: SHA1
joe,
I ran your script this time and I see what's going on :)
The Tclsh process runs out of virtual memory space! Not committed
memory.. True VM allocations. I even tried the old fashioned way with
thread::wait and thread::unwind that I personally know for a fact do not
leak handles.. The issue is not a handle leak. The issue is not
committed memory (apparently), but it appears that for each thread
created, a slice of VM is given to it permanently and that slice isn't
recouped (though it appears to have been uncommitted).
Joe, insert your script as a new bug. There's something going on that's
a bit deeper than proper handle management.
the Old way (that has the same behavior):
package require Thread
set i 1
while { 1 } {
set id [thread::create {thread::wait}]
thread::send -async $id {thread::unwind}
puts -nonewline " $i "
flush stdout
update
incr i
}
- --
David Gravereaux <davy...@pobox.com>
[species:human; planet:earth,milkyway(western spiral arm),alpha sector]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (MingW32)
iD8DBQFEtUq1lZadkQh/RmERAnlTAKDTaCKeb1Nq3ci2/zG8dqIOiiU2CgCfQdb1
CAzSrOWJB4YNUCpzIlrzxuI=
=30UL
-----END PGP SIGNATURE-----
Pat Thoyts
>joe,
>
>I ran your script this time and I see what's going on :)
>
>The Tclsh process runs out of virtual memory space! Not committed
>memory.. True VM allocations. I even tried the old fashioned way with
>thread::wait and thread::unwind that I personally know for a fact do not
>leak handles.. The issue is not a handle leak. The issue is not
>committed memory (apparently), but it appears that for each thread
>created, a slice of VM is given to it permanently and that slice isn't
>recouped (though it appears to have been uncommitted).
>
>Joe, insert your script as a new bug. There's something going on that's
>a bit deeper than proper handle management.
>
>the Old way (that has the same behavior):
>
>package require Thread
>set i 1
>while { 1 } {
> set id [thread::create {thread::wait}]
> thread::send -async $id {thread::unwind}
> puts -nonewline " $i "
> flush stdout
> update
> incr i
>}
>
Heap fragmentation? We ran into this a while back for a different
software setup. Windows provides alternative heap management
algorithms and in our case getting the application to use the low
fragmentation heap solved our problem. I would not really expect Tcl
to suffer this though as it manages a pool for small objects. For
more information on this - research HeapSetInformation You can
illustrate the problem of heap fragmentation by creating a program
that allocates a large number of random-sized chunks and then loops
randomly freeing and reallocating these slots with new random sized
chunks. After a while the virtual memory exceeds the 2GB limit and
your process is out of memory.
--
Pat Thoyts http://www.patthoyts.tk/
To reply, rot13 the return address or read the X-Address header.
PGP fingerprint 2C 6E 98 07 2C 59 C8 97 10 CE 11 E6 04 E0 B9 DD
David Gravereaux
Hash: SHA1
David Gravereaux wrote:
> joe,
>
> I ran your script this time and I see what's going on :)
>
> The Tclsh process runs out of virtual memory space!
BTW, works fine in 8.5, fresh compile off the HEAD w/thread 2.6.2. Goes
bang in 8.4.11
IOW, 8.4.11 has a leak of some sort.
- --
David Gravereaux <davy...@pobox.com>
[species:human; planet:earth,milkyway(western spiral arm),alpha sector]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (MingW32)
iD8DBQFEuY3XlZadkQh/RmERApj/AKCfD0Lt6otZf7LHCU3pZi8tADfkDACgnXEL
Dz4c31MDK36hOp4RKRwBK0A=
=+q1E
-----END PGP SIGNATURE-----
Scott
David Gravereaux wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> David Gravereaux wrote:
> > joe,
> >
> > I ran your script this time and I see what's going on :)
> >
> > The Tclsh process runs out of virtual memory space!
>
> BTW, works fine in 8.5, fresh compile off the HEAD w/thread 2.6.2. Goes
> bang in 8.4.11
>
> IOW, 8.4.11 has a leak of some sort.
The leak also seems to be present in 8.4.13/thread 2.6.3 (ActiveState
ActiveTcl 8.4.13.0).
Scott