It is local to a proc. Not global.
At first when the proc would finish the system would seem to hang for up
to 5 minutes,
during this time "wish" would have 100% of my two Xeon processors (& an
appropriate share of my 2 Gigs of Ram).
I thought it was when the file of this data was written and closed, but
it isn't.
At the end of the procedure , along with some
puts " here I am "
statements to help me track it,
I added an
unset listvarname
So now I have narrowed the "hang" down to the line above when the
variable is "unset" ,
AM I doing something stupid ??
..M'
try:
set listvarname ""
(maybe it is faster, but i don't know. I think much time is used to
decrement the refcount of the 500.000 line objs and the 1.500.000 real
objects in the list)
Using some other data storage for your numbers (for example a Metakit in
memory dataset it's perfect for that kind of stuff) could be a huge win
for you.
Michael
:AM I doing something stupid ??
Here's a similar type thing - except my input data was integers rather
than floats.
$ cat longtime.tcl
#! /usr/tcl84/bin/tclsh
proc inwork {} {
# read the file one line at a time
set fp [open "/tmp/somefile" r]
fconfigure $fp -buffering line
set indata [ read -nonewline $fp]
while {$indata != ""} {
set indata [ split $indata "\n"]
set indata [ read -nonewline $fp]
}
close $fp
unset indata
return
}
puts [time inwork]
$
I fed it 800,000 lines of 3 fields of integers. When I run this, I get:
$ /tmp/longtime.tcl
7346733 microseconds per iteration
I don't see even 3 minutes of wait...
I'm using Tcl 8.4 compiled on a SPARC Solaris 2.6 system and on a single
processor SPARC SunBlade 100.
--
The Tenth Annual Tcl/Tk Conference <URL: http://www.tcl.tk/community/tcl2003 >
Even if explicitly stated to the contrary, nothing in this posting
should be construed as representing my employer's opinions.
<URL: mailto:lvi...@yahoo.com > <URL: http://www.purl.org/NET/lvirden/ >
> According to mitchu <mit...@houston.rr.com>:
> :I have a [list] variable that gets packed with 500,000 lines or so of data,
>
> :AM I doing something stupid ??
>
>
> Here's a similar type thing - except my input data was integers rather
> than floats.
From the description, it's not clear that it's similar.
> while {$indata != ""} {
> set indata [ split $indata "\n"]
> set indata [ read -nonewline $fp]
> }
> close $fp
> unset indata
It might be more like this:
proc listtest {} {
set a [list]
for {set i 0} {$i < 10000000} {incr i} {
lappend a [expr {rand()}]
}
puts "done"
}
puts [time listtest]
Even this doesn't show any horrible properties. It takes a noticeable
time to deallocate the list (i.e., the time between seeing "done" and
the time), but only two or three seconds, certainly not minutes.
[...]
I have finally made a very short standalone script that demonstrates my
problem.
( the progarming problem is over, I am just still wondering what the
heck WISH is doing? )
On my Twin 2.2 Xeons, the follwing script results in the following
table of "Times to unset" a large listvar.
This is the result of [ time { unset varname }]
Lines in LIst | Time per List item to unset | Total
"unset" time
------------------------------------------------------------------------------
100,000 | 33 ms / line |
0.5 seconds
200,000 | 136 ms / line |
4.5 secs
300,000 | 237 ms / line |
11.85
400,000 | 335 ms / line |
--
500,000 | 434 ms / line |
---
600,000 | 523 ms / line |
52 Seconds to run " unset data " !!!
Maybe its just my computer or OS,
you guys try this. And see what you get,
I just don't see why it takes sooo long to just let go.
--------------- snip
----------------------------------------------------------
for { set x 1 } { $x < 7} {incr x } {\
set data [list]
set newcount [expr $x * 100000]
for {set i 0} {$i < $newcount } {incr i} {
lappend data [ list $i.123012345678921
$i.10412345678901 $i.2012345612345612 ]
}
puts "listsize = [llength $data] "
puts "\" unset data \""
set unsetime [ time { unset data } ]
puts " [expr double( [ expr [lindex $unsetime 0 ] / $newcount ])]
ms per list item \n"
}
--------------- snip
----------------------------------------------------------
...M'
Great!
:On my Twin 2.2 Xeons, the follwing script results in the following
:table of "Times to unset" a large listvar.
:
:This is the result of [ time { unset varname }]
Here's the results on my SPARC SunBlade 100, running Solaris 8 and
Tcl/Tk 8.4:
$ /tmp/lt.tcl
listsize = 100000 " unset data " 9.0 ms per list item
listsize = 200000 " unset data " 6.0 ms per list item
listsize = 300000 " unset data " 6.0 ms per list item
listsize = 400000 " unset data " 6.0 ms per list item
listsize = 500000 " unset data " 7.0 ms per list item
listsize = 600000 " unset data " 6.0 ms per list item
and the application quit quickly after the last entry.
...M'
Larry: threaded build? The reason I keep asking this is that threaded
builds use a different memory allocator, and this does look like a
problem with memory allocation/freeing ...
I run it on my linux 600Mhz laptop, and got results very slightly better
than mitchu's, with the same bad asymptotics in a standard build
(19,88,194,...) ms/item; OTOH, a threaded build gave me a constant 3ms/item.
Something to notice though: the threaded allocator is a "high-water mark
allocator", the memory freed is not really returned to the OS. So this
may show a problem with libc's malloc/free? To be continued ...
Miguel
[Note: Larry answered on the Tcl'ers chat: NO THREADS. Solaris seems to
be much better at malloc/free ...]
8.3.2:
listsize = 100000 " unset data " 2.0 ms per list item
listsize = 200000 " unset data " 1.0 ms per list item
listsize = 300000 " unset data " 1.0 ms per list item
listsize = 400000 " unset data " 1.0 ms per list item
listsize = 500000 " unset data " 2.0 ms per list item
listsize = 600000 " unset data " 6.0 ms per list item
8.4.2
listsize = 100000 " unset data " 5.0 ms per list item
listsize = 200000 " unset data " 9.0 ms per list item
listsize = 300000 " unset data " 20.0 ms per list item
listsize = 400000 " unset data " 30.0 ms per list item
listsize = 500000 " unset data " 38.0 ms per list item
listsize = 600000 " unset data " 47.0 ms per list item
Here's the numbers for running tcl 8.4 threaded on the same machine:
$ /tmp/lt.tcl
listsize = 100000 " unset data " 7.0 ms per list item
listsize = 200000 " unset data " 7.0 ms per list item
listsize = 300000 " unset data " 10.0 ms per list item
listsize = 400000 " unset data " 7.0 ms per list item
listsize = 500000 " unset data " 6.0 ms per list item
listsize = 600000 " unset data " 6.0 ms per list item
lwv26awu (6850) $
Have you tried running this script with a tclsh rather than a wish? And
have you tried using 8.4.2 ? It's unlikely someone is going to come up
with patches for a year or two old version of tcl...
Forgot to mention: those numbers are for 8.5a0 tclsh (today's HEAD)
I really hate to upgrade , because I have software in about 15
countries and don't want to break anything thats not
easily in my reach.
Buuuut..... I'll bite the bullet, upgrade one of my local boxes and
report back in.
...M'
...M'
You were right to catch the symptom. It *is* anomalous; some-
thing, somewhere, is wrong. Several people are pitching in to
track it down. We're probably going to ask you again to con-
firm the *exact* platform where you're seeing the symptom.
Incidentally, precisely because so many of us *do* write soft-
ware that stays in the field for years, Tcl has been particularly
productive in technologies such as VFS, Starpack, and so on, that
help "future-proof" deliveries.
--
Cameron Laird <Cam...@Lairds.com>
Business: http://www.Phaseit.net
Personal: http://phaseit.net/claird/home.html
https://sourceforge.net/tracker/index.php?func=detail&aid=735388&group_id=10894&atid=110894
Please report any additional info at the bug tracker. Mitchu: what is
your platform (which I assume to be linux): kernel and glibc versions;
if your tcl/wish is self-compiled, which gcc version?
It looks like this is a linux-only bug, and may be a gcc or glibc bug
(rather than a tcl one).
miguel
I get the same results as before, so my glibc doesn't matter I presume.
slackware linux 2.4.20 smp
twin Xeon-2.2 2.0 Gb Ram
just for th erecord
glibc 2.2.5
gcc version 2.95.3 20010315 (release)
...M'
>
> It looks like this is a linux-only bug, and may be a gcc or glibc bug
> (rather than a tcl one).
Did you see my nearby post about win2k results? It seems the
slow behavior is cross platform. No?
Some of the follow-ups in these threads--mine, in
particular--perhaps strike you as pedantic and
tangential to your real purpose (how *did* the
transatlantic demonstration go?). We're trying
to get things *right*, though, and need to be
careful.
Your glibc DOES matter to execution of "the pre-
compiled distribution available from ActiveState".
AS builds with "dynamic linking" (doesn't it?
Surely somebody'll correct me if this has changed
since the last time I examined it); one specific
consequence in this case is that an error in the
glibc installed on your host will manifest when
you run AS's product, even though you didn't build
it.
I've completely lost track of whether this is still
a problem for any of your customers. If it is, it
could well happen that a "fix" for them is to
install what they'll perceive as a *Linux* patch,
that is, an updated libc. Nothing about *your*
application (Tcl, your scripts, ...) need change.
Mhh ... friday tunnel vision here, sorry :(
Your post shows on win2k:
- linear behaviour in 8.3.2
- non-linear behaviour in 8.4.2 (I missed this part)
So, it seems indeed to be cross-platform in 8.4; in 8.3, apparently not
present in win2k but present in linux. Now, what did we break on win
that was already broken in linux?
The Demo went great, I recoded the problem procesdure to just use
sequential file accesses, since there was really no reason to lump
the whole huge dataset into a list for what I was doing.
However, there are several large lists that are part of the core of
the program , which displays and allows editing and filtering of a
15 to 100 channel
dataset that normaly runs around 600,000 values per channel. To
speed up display, refresh, panning and so on, I keep the whole banana in
a big list of lists, and just commit it to disk as changes are made.
This is running great as far as my customer is concerned. But
most certainly,they will experience the wind in their hair when this
issue gets resolved.
It is no doubt inducing some un-neccesary delay, and more important is
loading up the processors when this happens. On most of my servers I
have 3 to 10
users running at once. I have seen these beasts get SUPER loaded
down when more than one user has their program in this "state" at the
same time.
I am hoping your suggestion about a libc upgrade will just make it go
away.
Unfortunately diddling with my libs,
usually ends up with me in the corner in a fetal position and some
sort of gooey blue substance oozing out of my PC.
I am however willing to try a libc upgrade, if I can a get little
(experienced) advice on how to go about it correctly.
I have been here before and got my self in all kinds of trouble.
...M'
A better future is in store for you. I'm all for simplicity;
if a list of lists works for you, so much the better. HOWEVER,
when you move to Tclkit (and you'll want to), you get Metakit
for free! Metakit is an in-memory "embedded" data manager
that emphasizes performance and aims to eliminate administrative
overhead; for users, Metakit applications can be *easier* than
those that write to external flat files.
When you asked about the long unset times, were you not looking for
a solution? Specific tcl code changes would have the possibility of
breaking things that were not easily in your reach as well.
How do others structure things so that when they have to perform updates,
they can do so in a 'safe' manner?
Initially, of course, you have to have testing plans for unit, regression
and system testing. That provides you some level of assurance that the
code is going to work.
Wait. I've a suggestion before you move on.
First, let me ask - tell us about your appliation. It is Tcl scripts?
What extensions? Any C code in your application?
The reason I ask is that I'd suggest replace use of tclsh with the
appropriate tclkit. That avoids the dynamic libc problem (I think).
However, you _could_ find that the latest tclkit has been statically
linked with the bad library ...
Anyways, if your app makes use of linked extensions, then you'll have
to temporarily modify your code a bit to extend your app_path to
include the right directory.
The Utility itself consists of mostly tcl-tk, and a few perl scripts,
When serious processing and alot of floating point math is done the
utility make calls to some external compiled binary programs written
in C, Pascal & mostly Fortran. (they are called with exec)
All the programs (except some scattered use of ImageMagick), are
part of my distribution tree
The reason I am so protective of the utilitty, is because it has been
growing in size and complexity for almost 2 years.
I am a programmer of more than 10 years , but a TCL-er of only about
20 months.
I am hesitant to jump in and use something like Metakit or Tclkt,
because I don't fully understand what they are. SO far in
17 months of Production Use my program (the utility is called KLOG) has
never lost anyones work or corrupted a data set.
Out of numerous commercial products that hve been used before my
stuff, no one can make that claim. SO am I very
nervous about soiling my record.
I don't look forward to spending a week ( for me maybe two) to dig
down and teach myself MetaKit or Tclkit.
But after all the talk I have heard about performance increases , it
looks like the only way to go.
Its starting to look like the weekend will be spent digging for a
Metkit/Tcl Kit kickstart.
...M
Tclkit is just a single file version of a Tcl interpreter. To use it,
you change
/usr/bin/local/tclsh
to
/usr/bin/local/tclkit
or wherever you store it.
: I don't look forward to spending a week ( for me maybe two) to dig
:down and teach myself MetaKit or Tclkit.
Nothing to teach regarding _using_ of Tclkit.
You deserve to know, though, that concern with reliability,
"stability", data-integrity, and "openness" to a variety of
technologies were primary motivations behind TclKit. More-
over, there does not need to be a "big bang" with TclKit;
you can make a gentle transition into TclKit, unlike some
"all or nothing" technologies.
Start by reading <URL: http://wiki.tcl.tk/tclkit >.
I'll probably start poking it with a stick later this weekend to
see if it bites.
My programs all run under wish. Will that still work ?
...M'
mitchu wrote:
> OK, you and CL have me convinced to look at tclkit.
>
> I'll probably start poking it with a stick later this weekend to
> see if it bites.
>
> My programs all run under wish. Will that still work ?
>
> ....M'
Lets do tclkit ........
I have a binary :
tclkit-linux-x86
Do (under linux) I just make it
"chmod a+rx" ,
and make a link like :
ln -s tclsh tclkit-linux-x86
???
here I go .....
...M'
That last line has args the wrong way around (see "man ln").
Fastest way to get tclkit on Linux:
1. download the binary file from:
http://www.equi4.com/pub/tk/8.4.2/tclkit-linux-x86.upx.bin
2. type the following two commands:
mv tclkit-linux-x86.upx.bin tclkit
chmod +x tclkit
3. If you insist on calling it "tclsh", type:
ln tclkit tclsh
That's it. Happy Tcl/Tk'ing :)
-jcw
no, you're not the only one:
here, 900Mhz Pentium-III, linux (suse7.1), tclkit (8.4.2):
listsize = 100000 " unset data " 29.0 ms per list item
listsize = 200000 " unset data " 145.0 ms per list item
listsize = 300000 " unset data " 309.0 ms per list item
listsize = 400000 " unset data " 466.0 ms per list item
listsize = 500000 " unset data " 630.0 ms per list item
...
By that time, tclkit was using ~130MB already, and the machine with
256MB (and some other big apps in memory) got into swapping, so I aborted
the prog.
--
Nichts ist schlimmer als zu wissen, wie das Unheil sich entwickelt,
und voll Ohnmacht zusehn muessen - das macht mich voellig krank...
-- (Musical Elisabeth; "Die Schatten werden laenger")
> According to mitchu <mit...@houston.rr.com>:
> :My programs all run under wish. Will that still work ?
>
> Yes, they will work just fine. If you test under Windows, make certain
> that you get tclkit and not tclkitsh - the latter is a tcl only (no tk)
> version of the interpreter.
And on Linux you'll need a "package require Tk"
Steve
---
Steve Landers Software Design Solutions
Digital Smarties st...@DigitalSmarties.com
Perth, Western Australia DigitalSmarties.com
The tcl'ers chat bunch has been trying to reproduce this on win2k;
result is:
- most do not see the bug
- pat Thoyts sees the bug on one machine (SP1), but not on another
one (SP2)
So, getting data to try to understand the differences ...
Roy: what service packs do you have installed? Do you see any swapping
when running 8.4.2 and not under 8.3.2? Anything else you suspect might
be relevant?
Thanks
Miguel
This appears to be true for Solaris as well.
Interestingly though, when I run my scripts the normal (to me) way
which is :
----- snip ------
#! /bin/sh
# the next line restarts using wish \
exec wish "$0" "$@"
----- snip ------
I get the error:
-- snip ------
"application-specific initialization failed: unknown encoding "iso8859-1""
----- snip ------
But the program runs like usual otherwise.
If I include the
"package require Tk"
I get the error :
----- snip ------
"can't find package Tk
while executing
"package require Tk" "
----- snip ------
Ok , now you guys are trying confuse me right ? ;^)
...M'
Tclkit does not contain all the encodings that Tcl support on account of
space. I would think that Latin1 is included though. But you can look at
the Tclkit page for more information. If you need to use additional
encodings that can be taken care of, just takes a little tinkering.
> But the program runs like usual otherwise.
>
> If I include the
> "package require Tk"
>
>
> I get the error :
>
> ----- snip ------
> "can't find package Tk
> while executing
> "package require Tk" "
> ----- snip ------
>
If you run on windows, then as I understand it Tclkit really works like
a wish shell. That means you do not "package require" tk. That could be
a possible reason for this.
- Veronica
Any script that uses Tk commands ought to first
package require Tk $minVersion
Either the interpreter already has Tk in it, and this is a harmless
no-op, or it will load Tk, or it will fail, and the user will get a
good error messages about Tk not being available.
--
| Don Porter Mathematical and Computational Sciences Division |
| donald...@nist.gov Information Technology Laboratory |
| http://math.nist.gov/~DPorter/ NIST |
|______________________________________________________________________|
On Linux when I run :
----------- snip ---------------------------
#! /bin/sh
# the next line restarts using wish
exec wish "$0" "$@"
package require Tk $minVersion
----------- snip ---------------------------
When I run that , I get :
----------- snip ---------------------------
bash-2.05a# ./test_require.tcl
Error in startup script: can't read "minVersion": no such variable
while executing
"package require Tk $minVersion"
(file "./test_require.tcl" line 6)
----------- snip ---------------------------
And it doesn't really matter what I have my "tclsh" linked to :
tcl8.4 or tcl_kit .
I get the same result.
...M'
If you invoke the 'wish' command, it performs the package require Tk for
you.
: package require Tk $minVersion
:When I run that , I get :
:
:----------- snip ---------------------------
:bash-2.05a# ./test_require.tcl
:Error in startup script: can't read "minVersion": no such variable
Don was symbolically trying to indicate that a developer who is going to
use Tk should do something like
package require Tk 8.4
or whatever the minimum version of Tk is that they require.
However, the
package require ...
is an idiom characteristic of a newer style in Tcl coding.
It helps improve compatibility for people involved (for
example) with TkCon, Tclkit, "wrapped" Tcl, ... and so on.
You're likely to do those things in the future, and a
(correct) [package require] doesn't hurt now, so Don and
the rest are trying to ease your progress. That's all.
> package require ...
> is an idiom characteristic of a newer style in Tcl coding.
> It helps improve compatibility for people involved (for
> example) with TkCon, Tclkit, "wrapped" Tcl, ... and so on.
If you do use features of recently-released Tcl/Tk, it is
really good to specify a minimum version with package require
(unless you only use the program on your original computer).
Running a program for Tk 8.4 is likely to give an error message
about "invalid option -padx", whereas if you use
package require Tk 8.4
you get a clearer error message.
Donald Arseneau as...@triumf.ca
I would not recommend that. In fact, I'd recommend against it.
Your scripts ought to be using [package require] to record their
own dependencies. What you have above is an attempt to encode Tk's
dependency on the same version of Tcl. That's not your job. Tk can
take care of itself.
I think I had originally picked this up on the "wish application
template" wiki page http://wiki.tcl.tk/8287
(In fact that page is not very helpful as far as this point is concerned.)
Having thought about it, you have a good point. I had thought that this
might be a good idea for testing with different versions of tclsh and
Tk, but it seems that versions of Tk are only compatible with version of
Tcl with the same major and minor version number.
As far as dependencies go, I prefer to use something like:
package require Tk
if { [package vcompare $::tk_patchLevel 8.4.2] < 0 } {
puts "This application requires Tk version 8.4.2 or above"
puts "You have version $::tk_version (patch level $::tk_patchLevel)"
puts "Some features may be disabled, or the application may not start"
exit ; # or whatever
}
or even a dialog box (now that some version of Tk is loaded)
A bit lower on the page are comments about not using that -exact approach -
along with a hope that someone will change the example. Which hasn't
yet been done.
There isn't much on that page yet, is there?