This program just keeps coming back

[mitch]

One of my old customers has asked me to rework an old program of mine to
improve speed anywhere possible.

It deals with large datasets of waveform data stored in integer format
in text files. A typical dataset would be 30 files (one for each
waveform channel), with up to 700,000 values or samples in each file.

In the past I would load up a list of 30 lists, with each of the later
mentioned lists holding one channel of data. This worked great back
when they were typically sampling 5 channels and 100,000 samples.......
things have grown and the huge datasets have really slowed things down.

I would display the data on a window of 30 canvases, and draw the data
in each canvas with the "canvas create line" command. The operator could
zoom and browse the data by using provided controls. When the operator
moves around in the data , the proper data to display is snatched from
the "list of lists".

Can anyone suggest a smarter / faster approach ?

Thanks to anyone who took the time to read this ...!

Best Regards,
Mitch

[Arjen Markus]

mitch schreef:

> One of my old customers has asked me to rework an old program of mine to
> improve speed anywhere possible.
>

...

> Can anyone suggest a smarter / faster approach ?
>
> Thanks to anyone who took the time to read this ...!
>
> Best Regards,
> Mitch

Do you re-create the lines or do you simply change the coordinates?
The latter seems to be faster. But I honestly have no idea if this
makes any difference in your situation.

Another thing: the coordinates are presented as a list, not as
individual
coordinates, like:
.c create line $list_of_coords
versus
.c create line $x1 $y1 $x2 $y2 ...
(or:
eval .c create line $list_of_coords)

Regards,

Arjen

[mitch]

well, what is happening is , lets say the data is 500,000 long, but the
user can only work with about 2000 samples at a time, otehrwise it gets
all scrunched together .
So using provided controls for movement , the operator sort of drives
up and down the data. To get the data set for the display I snatch a
range of data from the list using [lrange ...]. And only do the
[$canvasname create line ...] using the data from the section he wants
to look at.

[mitch]

oh yeah, and yes I do make the call with a list
[$canvasname create line $plotlist ]
...M'

[Uwe Klein]

mitch wrote:
> One of my old customers has asked me to rework an old program of mine to
> improve speed anywhere possible.
>
> It deals with large datasets of waveform data stored in integer
> format in text files. A typical dataset would be 30 files (one for
> each waveform channel), with up to 700,000 values or samples in each file.
>
> In the past I would load up a list of 30 lists, with each of the
> later mentioned lists holding one channel of data. This worked great
> back when they were typically sampling 5 channels and 100,000
> samples....... things have grown and the huge datasets have really
> slowed things down.
>

do you have all 30 canvases visible at the same time?

i would read the data sets into blt_vectors and display them with the
blt_graph widget.

you can easily switch displayed data by changing the associated vectors
have different axes, zooomin , switching single traces on/off
and all that jazz.

I have used it to display 30-40 pairs of set/is traces with >100k samples.

uwe

[mitch]

That sounds like it is worth looking at.
I also need to be able to let the user click and draw boxes on the
data in the window , which I use to get coordinates for the edits the
user wants to perform on the data. Can I get that kind of feedback
from the BLT system ?

...M'

[Gerald W. Lester]

BLT has some charting/graphing packages that will speed thing up. Part of
the problem is that you are drawing a *lot* of virtual pixels that you can
not display, BLT has some particular optimizations for this case.

--
+--------------------------------+---------------------------------------+
| Gerald W. Lester |
|"The man who fights for his ideals is the man who is alive." - Cervantes|
+------------------------------------------------------------------------+

[mitch]

Well, actually I do under sample based on how much canvas is showing, so
I don't draw dots on top of dots.

The display is part of but not all of the problem, the data values are
frequently recalculated based on functions applied to the data by the
operator. Many of the number crunching funtions are not done in tcl, but
written in C or fortran for speed, then called as an
exec $command
so it gets crunched and saved, then needs reloading for display.

I don't know what to call it , but I need to sort of "map" the
display to file data on the disk, so changes in the diskfile
automatically update the data in my "list of lists".
Right now when I fetch the data I do :
set infile [ open $filename r]
set inlist [ split [ read -nonewline $infile] "\n"]
close $infile
----------
when needed I save it using a similar command:
puts -nonewline $outfile [ join $outlist "\n"]
close $outfile

All suggestions are MUCH appreciated
...M'

[Uwe Klein]

mitch wrote:
> That sounds like it is worth looking at.
> I also need to be able to let the user click and draw boxes on the
> data in the window , which I use to get coordinates for the edits the
> user wants to perform on the data. Can I get that kind of feedback
> from the BLT system ?

yes
there is little, very basic example using vector/graph on the wiki:
http://wiki.tcl.tk/15000 : function plotting

see the manpage blt_graph for using vectors, axes, markers ( per dot
or area ) and converting coords .. .

uwe

[Gerald W. Lester]

Again,

Check out BLT -- it has C APIs for its charting/graphing that will take a C
array of integers (it may even be able to take a filename).

It may be this is a time to write a small C extension to read the file in as
a C array and pass it to BLT in C.

This area of BLT was added back in the mid 1990s to handle very large
quantities of data from instruments -- basically to act like a virtual strip
chart or oscilloscope.

Pure Tcl is not always the answer.

[mitch]

Now THAT sounds intersting. I am going to try to get the customer to
pay for a little R&D to see if this approach can work and how much speed
it will yield, Many thanks for pointing me in some sort of direction .
...Mitch

[Ulrich Schöbel]

Hi Mitch,

sounds like a typical job for snack:
http://www.speech.kth.se/snack/download.html

Kind regards

Ulrich