Loading large data sets

[Roger Blum]

Hi everybody,

I need to load large data sets (up to 20 images 9000 by 4500 pixels each) to
display with vtk. I would like to only load a minimal amount of data at
startup to have the GUI respond to user interaction as soon as possible. I
would like to load the rest of the data in the background (without freezing
up the GUI). Do I need the thread extension to do this?
If yes, I have the following questions
- Is there a binary package of the thread extension for Windows (I have
tried to build with cygwin, but can not load the extension)
- How can I pass the data (vtk objects?) from a reader thread to the main
thread?

Is there an other/ better way to load the images without freezing the GUI?

Thanks a lot for your help,

Roger

[Donal K. Fellows]

Roger Blum wrote:
> I need to load large data sets (up to 20 images 9000 by 4500 pixels each) to
> display with vtk.

Ow! Each of those images, if displayed by Tk's photo image system,
would have a *minimum* memory requirement of 4*9000*4500 bytes (==
155MB!) with more memory actually being required to really put an image
on the display. Indeed, just loading that much data at once would
require something over 3GB. As I said, ow!

Since it is unlikely that people will be *really* able to see that much
data at once, can you come up with a way of doing partial loading of
the data? Using a custom image type for this sort of thing is probably
a very good idea.

Once you've stopped trying to load that vast amount of data at once, we
can start thinking about how to speed up such loading as you actually
need to do. (The custom image type is probably also a good idea for if
you're going to work with threads, since Tcl's script-level threading
model is strongly-isolated message-based and so isn't designed for this
sort of thing, where a shared-memory model is better suited.)

Donal.

[Kevin Kenny]

Roger Blum wrote:
>>I need to load large data sets (up to 20 images 9000 by 4500 pixels each) to
>>display with vtk.

Donal K. Fellows wrote:
> Since it is unlikely that people will be *really* able to see that much
> data at once, can you come up with a way of doing partial loading of
> the data? Using a custom image type for this sort of thing is probably
> a very good idea.

Donal,

Several of us ought to get together on this. I've been working with
some commercial JPEG2000 software, but there are open source JP2K
decoder/region-of-interest selectors out there, for instance JasPer
(http://www.ece.uvic.ca/~mdadams/jasper/). JP2K is just about the
only compression format that is going to deal with images that big.
(I use it with gigapixel images; it *does* scale.) Speed is very
important to me, too, to the point that I'm looking at things
like preloading partially-compressed images into OpenGL texture
maps - although I'm hoping against hope that I don't have to push
things that far. JasPer is dog-slow, which is why I'm using a
commercial alternative.

I understand that JasPer is already interfaced to ImageMagick, so
that might be a starting point for further development.

It would be nice if we could at least all agree on a Tcl API
for RoI, resolution and quality selection in large images.

--
73 de ke9tv/2, Kevin

[Darren New]

Roger Blum wrote:
> I need to load large data sets (up to 20 images 9000 by 4500 pixels each) to
> display with vtk. I would like to only load a minimal amount of data at
> startup to have the GUI respond to user interaction as soon as possible.

Consider taking the large images and chopping them into tiles in
advance? cutting them into 1000x1000 images, for example, might make it
pretty easy to scroll around, unloading the images you don't need to
display at any given instant.

--
Darren New / San Diego, CA, USA (PST)
This octopus isn't tasty. Too many
tentacles, not enough chops.

[Donal K. Fellows]

Kevin Kenny wrote:
> Several of us ought to get together on this.

[...]

Good, there has been work done by others on this for us to build on top
of. Given that, you've got my moral support for taking this on. :-)

Donal.

[Roger Blum]

Thanks for your answer, Donal.

Maybe I didn't make my problem clear enough. I am using the Vtk
visualization toolkit to display and manipulate the images. Vtk is quite
efficient in handling this amount of data. But if it needs to load
additional data it freezes up the GUI. I haven't used the thread extension
so far, so my question is: can I use it to create the Vtk objects in a
worker thread and then pass it to the main (GUI) thread?

I have seen some notes about shared-memory on the Thread extension site, but
I am not sure if this really does what I need it for.

Thanks again,
Roger

"Donal K. Fellows" <donal.k...@man.ac.uk> schrieb im Newsbeitrag
news:1156690734.6...@m79g2000cwm.googlegroups.com...

[Sean Woods]

Roger Blum wrote:
> Thanks for your answer, Donal.
>
> Maybe I didn't make my problem clear enough. I am using the Vtk
> visualization toolkit to display and manipulate the images. Vtk is quite
> efficient in handling this amount of data. But if it needs to load
> additional data it freezes up the GUI. I haven't used the thread extension
> so far, so my question is: can I use it to create the Vtk objects in a
> worker thread and then pass it to the main (GUI) thread?

I think we did hear you. And what we told you is that those are HUGE
images
that are going to chew up a HUGE amount of memory. Unless you have a
9000x4500 pixel monitor that is going to render the image natively, you
are
better hacking the picture into screen sized chunks.

No amount of threading, background launching, or whatever is going to
get
around loading 3gb of image data into and out of a video card.

-Sean

[MH]

In article <m0zIg.5120$S_5...@tornado.socal.rr.com>,

Roger Blum <roge...@hawaii.rr.com> wrote:
>Thanks for your answer, Donal.
>
>Maybe I didn't make my problem clear enough. I am using the Vtk
>visualization toolkit to display and manipulate the images. Vtk is quite
>efficient in handling this amount of data. But if it needs to load
>additional data it freezes up the GUI. I haven't used the thread extension
>so far, so my question is: can I use it to create the Vtk objects in a
>worker thread and then pass it to the main (GUI) thread?
>
>I have seen some notes about shared-memory on the Thread extension site, but
>I am not sure if this really does what I need it for.
>
>Thanks again,
>Roger

Not claiming to be an expert, but..

Are you runnning a 64-bit version of Tcl on a BIG machine (ie. 4+ GB RAM)

If the calculations of others are even remotely correct, then using 3GB of
RAM will either hit the swapfile HUGE, or use up ALL your ram in 32-bit
mode (assuming you're running Linux compiled for "BIG_RAM" or whatever it's
called - no idea what Windows or Mac do when hitting 3-4GB with 32-bit
apps).

Will all your users be using similar hardware?

My guess is, you're only going to be displaying one image at a time?

Does it matter if your GUI is directly attached to the window (i.e. all in
one big window), or is it ok if you do something a la ImageMagick, where you
have the window with the image, and another window with the controls?

If the latter's ok, I'd suggest you write a "slave" Tcl app that displays
one image, and maybe responds to direct mouse requests, and communicates
over a socket with a master app.

The main app/gui shows the controls, and sends a "show image xyz.jpg" to the
slave app, and gets a message back when done. That way, the master app is
always responsive, and you don't have to worry about multiple threads,
shared memory, etc.

You can even tell the slave app to "load images 1.jpg 2.jpg 3.jpg" and it
will respond when done. If you get an error due to a bad image, or the slave
crashes, you can easily re-launch it from your main gui with the next image,
etc..

Why do it this way? Avoid threaded mode whenever possible.. :-)

MH

[Roger Blum]

Thanks all for your suggestions. I understand (and did before) that these
files are huge. But this is the way I get the files (only a sample) from my
client. The files are then downsized by filters in vtk (sequentially). My
tests with up to 12 files didn't show any memory problems so far (2.6 GHz
machine with 1 GB memory), besides the GUI freezing, when new data has to be
loaded (and then processed). But I will try to persuade my client to have
the input images preprocessed.

But this still doesn't solve my problem of loading and processing the data
in the background. As I understand all of you, the thread extension is not
the way to do this. Is there an other way to achieve this with tcl?

Thanks again to all of you,
Roger

"Roger Blum" <roge...@hawaii.rr.com> schrieb im Newsbeitrag
news:m0zIg.5120$S_5...@tornado.socal.rr.com...

[pwe...@zonnet.nl]

Although I never worked with VTK combined with threads, you could try to
have VTK in one thread and your GUI in another, and send commands
from your gui to the VTK thread. But if you send a command whilst the
VTK thread is preprocessing you would still have to wait for the
preprocessing to finish, with the possibility that you hang your GUI.

If the preprocessing is a filter that converts the huge data set to a
smaller data set, you could also start this filter as a separate program
with

exec tclsh.exe myfilter.tcl -myoptions mybigfile &

The ampersand will cause exec to return immediately so your gui will
continu to work normally. You will then ofcourse need a method to see if
the preprocessing has finished (for instance look for an file etc).

Regards

Peter Welten.

[Stephan Kuhagen]

Roger Blum wrote:

> Is there an other/ better way to load the images without freezing the GUI?

Some remarks to make sure, I understand your problem correctly:
- You use the Tcl-binding of VTK to load big datasets.
- You want to view the images after loading in a canvas (?)
- While loading your GUI should not freeze.

If you can live with the canvas being a separate toplevel-window, then you
do not need threading. You can create two Tcl-scripts, one containing the
main GUI and functionality, and the other containing just the display, the
image-loader and maybe some other functionality, that must be run
synchronous to the loader.

The main-script then starts the loader-script as an external process using
[open] as a pipe and configuring a fileevent whenever the pipe becomes
readable or writable. You can read the (std|err)output of this second
process and send strings (read: commands) to it asynchronously. Your GUI
stays responsive and the other process reads commands sent to it via its
stdin and loads data. Your loader script then reads commands from its
stdin, such as "load FILE.tif", and executes them. I did something quite
similar for a script that runs several other processes in the background,
catching their output and sending them commands to execute, while my GUI
stays responsive all the time. Using [open] has also the advantage, that
you can "kill" external processes (although this might hurt some peoples
aesthetic feelings) with just closing the opened command pipe from the main
process.

HTH, Regards
Stephan

[Uwe Klein]

pwe...@zonnet.nl wrote:

> exec tclsh.exe myfilter.tcl -myoptions mybigfile &

use bgexec from BLT or the extracted version from the wiki for this.

uwe

[Bruce Hartweg]

Stephan Kuhagen wrote:
> Roger Blum wrote:
>
>> Is there an other/ better way to load the images without freezing the GUI?
>
> Some remarks to make sure, I understand your problem correctly:
> - You use the Tcl-binding of VTK to load big datasets.
> - You want to view the images after loading in a canvas (?)
> - While loading your GUI should not freeze.
>
> If you can live with the canvas being a separate toplevel-window, then you
> do not need threading. You can create two Tcl-scripts, one containing the
> main GUI and functionality, and the other containing just the display, the
> image-loader and maybe some other functionality, that must be run
> synchronous to the loader.

you don't even have to live with he canvas in a separate window
create a frame in your main app with the -container option and
then when you launch the child script use the -use option on wish
to have it use that container.

>
> The main-script then starts the loader-script as an external process using
> [open] as a pipe and configuring a fileevent whenever the pipe becomes
> readable or writable. You can read the (std|err)output of this second
> process and send strings (read: commands) to it asynchronously. Your GUI
> stays responsive and the other process reads commands sent to it via its
> stdin and loads data. Your loader script then reads commands from its
> stdin, such as "load FILE.tif", and executes them. I did something quite
> similar for a script that runs several other processes in the background,
> catching their output and sending them commands to execute, while my GUI
> stays responsive all the time. Using [open] has also the advantage, that
> you can "kill" external processes (although this might hurt some peoples
> aesthetic feelings) with just closing the opened command pipe from the main
> process.
>

Bruce

[Stephan Kuhagen]

Bruce Hartweg wrote:

> you don't even have to live with he canvas in a separate window
> create a frame in your main app with the -container option and
> then when you launch the child script use the -use option on wish
> to have it use that container.

Right, sorry for the misleading info.

Stephan

[Roger Blum]

Thanks for your suggestions. But I think this will not work with vtk. vtk
doesn't use the canvas widget but comes with an own widget to display 3D
stuff. It's not possible to execute a vtk filter with "open ..." or "bgexec
..." as these filters are not stand-alone programs but objects created in a
tcl interpreter. The output of one filter is directly passed to next object
as it's input.

So my idea was to create these objects in a shared memory area, where more
than one thread has access. That way I could create the objects in the main
thread and then execute the time consuming update of these objects from a
second thread. But as far as I can see now, this is not a practicable way to
go.

Thanks for your help anyway,
Roger

"Stephan Kuhagen" <nos...@domain.tld> schrieb im Newsbeitrag
news:ed8cad$1pr$1...@kohl.informatik.uni-bremen.de...

[Donal K. Fellows]

Roger Blum wrote:
> Thanks for your suggestions. But I think this will not work with vtk. vtk
> doesn't use the canvas widget but comes with an own widget to display 3D
> stuff. It's not possible to execute a vtk filter with "open ..." or "bgexec

> ...." as these filters are not stand-alone programs but objects created in a

> tcl interpreter. The output of one filter is directly passed to next object
> as it's input.
>
> So my idea was to create these objects in a shared memory area, where more
> than one thread has access. That way I could create the objects in the main
> thread and then execute the time consuming update of these objects from a
> second thread. But as far as I can see now, this is not a practicable way to
> go.

But you can embed a window created by one wish process in a window
created by another wish process. See the option -container on [frame]s
and the option -use on [toplevel]s (and which can also be an argument to
the wish executable itself). This might be enough, and you can get
started with this with very little extra effort (especially compared
with doing lots of shared memory thread voodoo).

Donal.

[Roger Blum]

Hi Donal,

Thanks for this idea! So I basically would have the tcl/tk stuff in one wish
starting a second wish with the vtk stuff. This would still freeze the vtk
widget when it loads the data, but the menues and other widgets would be
responsive. Sounds really good. I wasn't aware (means have never tried to do
this) before. I suppose for the communication (applying commands from the
main wish on the vtk wish) I use the send command. Is this the best way to
do it?

Thanks again,
Roger

"Donal K. Fellows" <donal.k...@manchester.ac.uk> schrieb im Newsbeitrag
news:ed9bom$1ifd$1...@godfrey.mcc.ac.uk...

[Donal K. Fellows]

Roger Blum wrote:
> I suppose for the communication (applying commands from the
> main wish on the vtk wish) I use the send command. Is this the best way to
> do it?

If you know you're going to be on Unix/X11, [send] is nice and easy.
Otherwise, you're better off using a pipe (create the subprocess using
[open |...] and not [exec ... &]). You could also use the 'comm'
extension, but that's overkill for this I think since we know that one
process is starting the other.

Donal.