Fast Image Loading - Browser temporarily stops on cornerstone.loadImage/displayImage

1,508 views
Skip to first unread message

jamesa...@gmail.com

unread,
May 6, 2016, 11:12:16 PM5/6/16
to cornerstone platform
I created a WADOImageLoader example to test loading an 863 kb DICOM image to the site. When my custom function changeImage( ) is called, it calls cornerstone.loadImage which then calls cornerstone.displayImage to display the image in the element. Before the cornerstone.loadImage is called though, I have a loading.gif that becomes visible. But when cornerstone.loadImage is called (loading the 863kb DICOM file) my browser freezes up for about 1.5s to 2s loading the image (even though I'm using my localhost as server), stopping the loading.gif animation as well. I figure when I have even a larger file than that, and try to load it from a remote server it might take even longer.

Right now I'm trying to figure out what is a good approach to this. How can I keep things running while the image loads?
So far I have looked at:
-imageCache (not sure if this fixes it)
-Web Workers (don't know how to implement this with cornerstone)
-startHandlerLoad from cornerstoneTools (tried, not sure how this works or how to use this)

Thanks for any help guys.

Chris Hafey

unread,
May 6, 2016, 11:41:03 PM5/6/16
to cornerstone platform
JavaScript runs in the main/ui thread unless you explicitly create web workers.  Cornerstone does not utilized web workers so all decompression is done in the main/ui thread.  If you are downloading a compressed image, this would explain the lockup you see (especially if it is JPEG2000 which is computationally heavy).  Your best short term solution is to use an uncompressed transfer syntax or one that is not as heavy computationally.  Alternatively you could try modifying cornerstoneWADOImageLoader to do the decompression in a web worker - this shouldn't be too hard to do as promises are used to return the image object after it is loaded and decompressed.

jamesa...@gmail.com

unread,
May 7, 2016, 1:09:11 AM5/7/16
to cornerstone platform
Trying to find the decompression function in cornerstone.js. Can I use webworker on function loadImageFromImageLoader and achieve same results?

Chris Hafey

unread,
May 7, 2016, 2:28:18 AM5/7/16
to cornerstone platform
Compression is done in the cornerstoneWADOImageLoader library, not cornerstone itself:

jamesa...@gmail.com

unread,
May 7, 2016, 2:32:30 AM5/7/16
to cornerstone platform
Yes I was just looking at the WADOImageLoader Lib, I'm assuming decompression is done by jpxImage.parse(compressedPixelData) and this is what I have to use a webWorker on?

jamesa...@gmail.com

unread,
May 7, 2016, 2:33:20 AM5/7/16
to cornerstone platform
Sorry did not see the link. Will try that. Thanks!


On Saturday, May 7, 2016 at 2:28:18 PM UTC+8, Chris Hafey wrote:

jamesa...@gmail.com

unread,
May 7, 2016, 5:19:01 AM5/7/16
to cornerstone platform
I was able to place a web worker to execute jpeg.parse( frameData ) from the decodeJPEGBaseline function




in my jpeg_parse I simply returned parsed jpeg data.

But I get an error that jpeg is still undefined even though I have added the return statements to the worker message function when it replies. 

 Sorry for the many questions. Not sure on what I'm doing wrong here.

Ghet Ghetolay

unread,
May 8, 2016, 4:33:14 AM5/8/16
to cornerstone platform
Hi,

I have a modified version of cornerstoneWADOImageLoader using web worker but I think it's still too heavily dependent to my own viewer.
If you're not in a hurry I could review the code and create a github fork, maybe next week-end. 

Ghetolay

Chris Hafey

unread,
May 8, 2016, 10:43:31 AM5/8/16
to cornerstone platform
Hi Ghet - it would be great to see how you did this!  I can see how a medical imaging web app would make extensive use of web workers (e.g. decompression, segmentation, calculations, etc).  It might make sense to create (or adopt) a web worker framework where web worker jobs can be queued, prioritized and cancelled based on what the user is doing.  This framework would also prevent too many web workers from being spawned which could impact the interactivity of the application.

Chris

jamesa...@gmail.com

unread,
May 10, 2016, 3:00:34 AM5/10/16
to cornerstone platform
Hi Ghet, I've been trying to time different processes of the WADOImageLoader, trying to figure out the calculation/decompression-heavy processes. I've tried to add a web worker to the jpeg.parse from the decodeJPEGBaseline function in the cornerstoneWADOImageLoader.js file:

function decodeJPEGBaseline(dataSet, frame)
  {
    ...
    var jpeg = new JpegImage();

    var w = new Worker("jpeg_parse.js");
    w.onmessage = function(event) {
      jpeg = event.data;
      if(bitsAllocated === 8) {
      return jpeg.getData(width, height);
    }
    else if(bitsAllocated === 16) {
      return jpeg.getData16(width, height);
    }
    };
    w.postMessage( frameData );
    

   
  }

By passing the frameData, I though I would be able to get a parsed jpeg image out of my jpeg_parse.js file that I made:
importScripts('../jpx.min.js');
var ColorSpace = ...
var JPEGImage = ...
self.addEventListener('message', function(e) {
    var jpeg = new JpegImage();
    jpeg.parse(e.data);
    postMessage(jpeg);
}, false);


But I keep getting an error that says cannot get property 'length' of undefined. I also tried to just return the frameData from jpeg_parse.js and determine whether the frameData sent and the frameData that was returned were equal, which I found out they were not, but had the same type.
I have also tried putting a web worker in the <script> portion of my webpage when the function cornerstone.loadImage( ) is called, which would work except web workers will not let me import jquery which is needed for cornerstone. I have also tried multhread.js in place of web workers, with no luck. I believe there is some error with how I use web workers. The questions I'm plagued with right now is:

Where is the best place (the most computational-heavy) to put web workers in cornerstone WADOImageLoader and how do you use web workers there?

Thanks for any help in advance, I'd really like to see how you used web workers in your modified version, and adapt the method you used.

Ghetolay

unread,
May 10, 2016, 5:13:05 AM5/10/16
to jamesa...@gmail.com, cornerstone platform
Why do you split decodeJPEGBaseline between main thread and web worker. You should do something like that :


var w = new Worker("jpeg_parse.js");
 w.onmessage = function(event) {
    buildImageObjectFromPixelData( event.data );    
 };
 w.postMessage( frameData, [frameData] );

//Make decodeJPEGBaseline to work directly with frameData
function decodeJPEGBaseline(frameData)
  {   
    var jpeg = new JpegImage();
    jpeg.parse( frameData );
    if(imgData.bitsAllocated === 8) {
      return jpeg.getData(imgData.width, imgData.height);
    }
    else if(imgData.bitsAllocated === 16) {
      return jpeg.getData16(imgData.width, imgData.height);
    }
  }


self.addEventListener('message', function(e) {
    var pixelData = decodeJPEGBaseline( e.data, );
    
    postMessage( pixelData, [pixelData] );
}, false);

Also you did not used transferable object, you should totally use it to speed up data transfer between threads.

In my implementation, web workers manage the download and the decompression and return the pixel data then the main thread create a image object from that.
First I did was removing jquery dependency, jquery is only used for Deferred and maybe events. So it's easily replaced with callbacks and/or Promise.

I'm sending you my final WADOImageLoader but it's probably not suitable for you. This only works with one scenario which use wado-rs to retrieve metadata and then pixel data and won't work for multi frame images.
If you are working differently you'll have to adapt. That's why I need to review the code before submitting it.

Here is the 'algo' : 
  • using wado-rs I retrieve metadata of all images of a serie
  • for each image I call a web worker with a list of essential metadata (see below)
    web worker will
    • retrieve pixel data using wado-rs
    • decompress if necessary
    • return a partial image object with just few properties (see below)
  • then with the partial image object and the medata I create the final image object

And the code : (obviously worker.onmessage must be set before worker.postMessage, I’m just keeping the process order here).

var metaData = ...
var image;

worker.postMessage( {
    id:  metaData.x$SOPInstanceUID$,
    url: metaData.$retireveURL$,
    transferSyntax: metaData.x$RransferSyntaxUID$,
    imgData: {
        width: metaData.x$Columns$,
    height: metaData.x$Rows$,

    photometricInterpretation: metaData.x$PhotometricInterpretation$,
    planarConfiguration: metaData.x$PlanarConfiguration$,
    pixelRepresentation: metaData.x$PixelRepresentation$,
    bitsAllocated: metaData.x$BitsAllocated$,
    samplesPerPixel: metaData.x$SamplesPerPixel$   
    }
);

Worker :

onmessage = function(e){

    cornerstoneWADOImageLoader.wadors.loadImage(e.data.url, e.data.imgData, function(img){
        /* img = {
                imageId : ....
                minPixelValue : ....
                maxPixelValue : ....
                pixelData: ...
                color: ....
           }
        */
        var transferable;

        if(img.pixelData instanceof ImageData)
            transferable = [img.pixelData.data.buffer];
        else if(img.pixelData instanceof ArrayBuffer)
            transferable = [img.pixelData];
        else
            //typedarray
            transferable = [img.pixelData.buffer];

        postMessage(img, transferable);

     }, e.data.transferSyntax);
};
    worker.onmessage = function(e){
        image = addImageData(e.data, metaData); //add rows, height, rescale slope and intercept, window width and center etc from metaData
    };

Hope this will help you (sorry for the code I don't know why it wasn't prettify).

Ghetolay.

--
You received this message because you are subscribed to a topic in the Google Groups "cornerstone platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cornerstone-platform/v5boDxIMsig/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cornerstone-plat...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cornerstone-platform/445793df-35a1-44a7-b7a3-de3f2e1bab18%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

cornerstoneWADOImageLoader.js

Ghetolay

unread,
May 10, 2016, 5:27:36 AM5/10/16
to jamesa...@gmail.com, cornerstone platform

Ho just realized imgData is undefined in my first snippet inside decodeJPEGBaseline(). So I guess you have to pass it to the web worker along with the frameData.

Also I forgot to tell you but there is a lot of broken code on the cornerstoneWADOImageLoader file I attached. For example I changed the signature of ```createImageObject()```` function and I found at least 4 calls to it using the old signature.
And even if it still has “ImageLoader” on the name, I don’t use the ImageLoader feature of cornerstone so this is probably broken too.

Really you should just get inspiration from this code it’s far from being stable and usable.

jamesa...@gmail.com

unread,
May 12, 2016, 6:47:08 AM5/12/16
to cornerstone platform, jamesa...@gmail.com
Hi Ghet,

Thanks for the answer! I get the idea of removing the jquery dependency and using js promises.
I have a few questions with assumptions (please correct me if I am wrong)
-The xhrRequest function is what downloads the images and it is asynchronous?

function decodeJPEGBaseline(dataSet, frame)
 
{
   
//frameData gets its value here

   
var jpeg = new JpegImage();

   jpeg
.parse(frameData);      //This  function takes  alot of time
     
if(bitsAllocated === 8) {

     
return jpeg.getData(width, height);  // I could place  deferred/promise here but wouldn't that still make the browser freeze?

   
}
   
else if(bitsAllocated === 16) {
     
return jpeg.getData16(width, height);
   
}
 
}

At this stage, would it be better to put the whole process of getting the imagePromise in a Web Worker function, and just postMessage(imagePromise)  (Since its now jquery independent) so that no part of the loading will freeze the browser?

Gadi Levy

unread,
Aug 7, 2016, 6:14:17 PM8/7/16
to cornerstone platform
Hi,
Were you able to produce a working version of WADOImageLoader using web workers for the decompression? If so, could you please share it?
I get these freeze ups when loading a series compressed with jpeg lossless.
Thanks,

Chris Hafey

unread,
Aug 8, 2016, 8:40:50 AM8/8/16
to cornerstone platform
Yes - I have a WIP version available here:


hoping to finish it up in the next few weeks

Chris

Ghet Ghetolay

unread,
Aug 8, 2016, 9:14:54 AM8/8/16
to cornerstone platform
I also have some kind of demo but it's a bit messy because I've edited the dist file directly that's why it's not online.

But I guess the one done by Chris should be good enough if not even better.

@Chris I didn't know you were working on it, I could have helped.

Chris Hafey

unread,
Aug 8, 2016, 10:46:43 AM8/8/16
to cornerstone platform
@Ghet - I could still use help, here is where I am atL

1) The UI thread code that creates web workers and posts decode requests to them is here:

2) The web worker code receives decode image requests containing the image frame and compressed pixel data from the UI thread webWorkerManager:

3) All of the codecs are compiled into a single JS fie.  Currently they are all in the ~/codecs directly and the gruntfile concats and minifies them.  The web worker loads this file.  This is nice because we don't have to bog down the UI thread with loading or initializing the codecs

 A few things I need to think through still:

1) How to manage decoding request prioritization.  We should prioritize interactive requests over stack prefetch requests
2) How many web workers to spawn.  Right now it is set to 1 by default but can be configured via API call.  Would be nice to do something automatic here
3) Whether or not to move the web worker into cornerstone core so it can be reused for other purposes (like running algorithms)

I did a lot of refactoring before this which I'll create another post on, but that is mostly orthogonal to the web worker changes.  Please review and let me know your thoughts

Chris

Gadi Levy

unread,
Aug 8, 2016, 4:45:29 PM8/8/16
to cornerstone platform
Hi,
Great to hear that this is coming soon!
Meanwhile, the packaged source in dev has two references that are not in the dist.
Are they available somewhere (without building the project?)
Thanks,

var config = {
maxWebWorkers: 1,
webWorkerPath : '../../dist/cornerstoneWADOImageLoaderWebWorker.js',
codecsPath: '../dist/cornerstoneWADOImageLoaderCodecs.js'
};

Chris Hafey

unread,
Aug 9, 2016, 12:09:06 AM8/9/16
to cornerstone platform
Oops - I forgot to commit them, they are there now

Gadi Levy

unread,
Aug 9, 2016, 3:21:02 PM8/9/16
to cornerstone platform
Works like a charm. No more frozen UI while loading. Thanks!

Chris Hafey

unread,
Aug 20, 2016, 5:32:38 PM8/20/16
to cornerston...@googlegroups.com
I just pushed a new version which addresses the outstanding issues - my apologies if anyone started working on these but nobody said anything so I went ahead.  I would appreciate feedback on the design of the web worker framework and testing to identify any bugs.  Once this has settled for a week or so I will push it out.

You can find documentation on the new web worker framework here:


  Thanks

PS - Check out the cool new example that uses the web worker framework to do custom image processing tasks here:

Ghet Ghetolay

unread,
Aug 24, 2016, 4:40:26 AM8/24/16
to cornerstone platform
Hi Chris,

I've been off 2 weeks and I can see you are moving fast.

I didn't had time to run it but here is what I can say from just code review : 

1) The main difference with my implementation is that you are decoding on webworker while I'm loading entirely the image (download, parsing, decoding) on webworker. I think everything that's not directly related to UI and/or is computationally heavy should be on webworkers. I've even tested creating the ImageData object (for recent browser) on webworkers for big images or MIP rendering for example (I've encountered some problem but not the subject here). The only con/question I have about my implementation is that when image is downloading we may have a sleeping webworker while we may also have queued tasks waiting. I couldn't test and benchmark that easily so I'll do that later.
I've seen you have add custom task but this would need some forth and back from webworkers and main thread to chain tasks, we can't chain task inside the same webworker. 
So what do you think ?

2) It seems you are using Transferrable for webworkers results but not when you call a webworker meaning there is a copy of the full uncompressed pixel data between the main thread and webworkers. Is it intended to avoid to "lose" those pixel data in case there is something wrong with the webworker ?

3) Our WebworkerManager are close, I should had post it as a gist or something, could had saved you some time. There is yet 2 things I support on my implementation and could add to yours : 
     - handle webworker path involved with CORS. This is just a matter of a few lines of code : if webworker url involve CORS, create a blob and use an object URL from that blob instead of the real url.
     - Add the possibility to stop/cancel a task based on it's id or group. I'm not totally sure about the group feature, we could totally do it outside the WorkerManager, just keep a list of task id, loop through it and cancel each one. In my case I was working for myself and I needed it so there was no debate.

I can already PR the CORS workaround if you want.
I'll try to play with it soon and provide more feedback (not sure at all).


Ghet`  

Chris Hafey

unread,
Aug 24, 2016, 6:35:51 AM8/24/16
to cornerstone platform
Hi Ghet,

1)
I thought about this but decided against it for the following reasons:
A) It is not necessary to download in the web worker because it is asynchronous (does not tie up the UI thread) and not CPU intensive.  
B) The decode task is a reusable piece of code.  It is currently used by WADO-URI HTTP, DICOM P10 from filesystem, will eventually be used by WADO-RS and could be used by other loaders.  Separating it from the loading makes it more reusable (note that WADO-RS does not involve parsing DICOM p10)
C) I wanted to maximize CPU concurrency and that is easy to do with a pool of web workers that matches the number of cores.  Maximizing download concurrency has nothing to do with CPU cores and could complicate things (I am not sure how we would meet both goals - and your comment about workers sleeping while downloading indicates you don't know either).  

I do think we can move DICOM Parsing to the web worker.  I have a task in the backlog to "Free up DICOM P10 instance after decoding to reduce memory consumption" which would probably need to be done first so we could just discard the dataSet entirely.

I am not sure we need to support chaining of tasks within the web worker itself.  This can be done by the task developer without support from the framework.  I suppose it would be nice for one task to spawn other tasks - I can look at adding this if someone actually needs it.

2) Correct - I would like to ultimately discard the dataSet to keep memory use down "Free up DICOM P10 instance after decoding to reduce memory consumption" but didn't tackle that in this effort.  If someone wants to do this work, I would accept the PR.  Note that some cornerstone tools may depend on dataSets and need t be updated when this is done.

3) I am interested in your CORS hack - can you send a PR or post a gist?  Concurrency managers can get very complex - if we need task cancellation or grouping we can certainly add it.

Thank you for taking the time to review and provide feedback!

Chris 

Gadi Levy

unread,
Sep 6, 2016, 5:54:54 PM9/6/16
to cornerstone platform
Hi,
It seems that the WadoImageLoader with the webworker functionality is not loading images on iOS devices (Chrome and Safari on iOS), although it works great for me on Windows (Chrome and IE) and Android (Chrome). My setup is not ideal for iOS debugging and I'm not sure what the actual problem is. Can anyone confirm this ? According to this webworkers are supported by Safari.
also, the newest version (v0.14.0 - 2016-09-01) throws an Uncaught TypeError: Cannot read property 'addProvider' of undefined on line 1131 upon loading.
G

Chris Hafey

unread,
Sep 7, 2016, 12:11:02 PM9/7/16
to cornerstone platform
Hi James - I will try to look at this next week, thanks for the report

Chris Hafey

unread,
Sep 11, 2016, 9:43:48 AM9/11/16
to cornerstone platform
Hi Gadi,

The examples are working properly on iOS now:


The problem seemed to be related to some redirect rawgit.com was doing, not the code itself.  I did not observe the "Uncaught TypeError: Cannot read property 'addProvider' of undefined on line 1131 upon loading.", did you see this in your code or the examples?  If your code, the bug is in... your code :)

Gadi Levy

unread,
Sep 11, 2016, 5:30:36 PM9/11/16
to cornerstone platform
Hi Chris,
I think there is still an issue, as the following example shows:

which I believe is a valid DICOM P10 instance, with transfer syntax 1.2.840.10008.1.2.4.70 (JPEG lossless),


With Chrome on Windows - the image is loaded and displayed fine.
With IE11 on Windows - the image is not loaded, and there's an exception: SCRIPT0: DataView operation access beyond specified buffer length at cornerstoneWADOImageLoaderCodecs.js line 1151.

Perhaps this is the same issue I'm seeing on iOS Chrome and Safari (although I'm only guessing here).
Can you take a look ?
G

Chris Hafey

unread,
Sep 11, 2016, 9:43:24 PM9/11/16
to cornerstone platform
No this is a different issue and I suspect it is specific to the JPEG Lossless codec.  It is odd that it does not occur in the current master build.  This will require more investigation to understand, very strange.

Chris

Gadi Levy

unread,
Oct 25, 2016, 8:29:05 PM10/25/16
to cornerstone platform
Hi Chris,
I was wondering if there's an update on this issue.
Thanks

Chris Hafey

unread,
Oct 25, 2016, 8:44:46 PM10/25/16
to Gadi Levy, cornerstone platform
No update, would be nice if someone could look into it!

Sent from my iPhone
--
You received this message because you are subscribed to the Google Groups "cornerstone platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cornerstone-plat...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cornerstone-platform/53776991-9a9a-40e2-8d01-9627f30bc7a5%40googlegroups.com.

Nihal Mehta

unread,
Nov 8, 2016, 2:29:09 PM11/8/16
to cornerstone platform
Hi Chris,

I have been working independently trying to get images to download faster and tried to put the cornerstone.loadImage method in the background but am not making much progress.

Firstly, I tried to put the cornerstone.loadImage("wadourl:" + url) .then(function(image) {...}) in the webworker directly. That didn't work as it depends on jquery, and jquery and webworkers don't go together.  So, I created a "fake DOM" that allows jquery code to be used in webworkers. That allowed the image to download but then got into other problems.

I also tried doing the image downloads in iframe, but even that was not successful.

What was confusing me was that even though I'm over a very fast network where "general" 100MB files download in seconds, some studies that are only 20MB would take too long. 

So, that's when I found this group. Going through it, I think I understand why cornerstone.loadImage in webworkers or iframes don't work - the loadImage does the image download as well as the decoding of the dicom image. And the latter takes the bulk of the time. 

However, it's not clear what cornerstone method is for only downloading the image, and then use a different method to do the decoding. 
Could you point me to do that? (I checked the webworker examples but I don't think those illustrate the specific questions I have above.)

Thanks, 
Nihal Mehta
To unsubscribe from this group and stop receiving emails from it, send an email to cornerstone-platform+unsub...@googlegroups.com.

Ghet Ghetolay

unread,
Nov 9, 2016, 5:20:39 AM11/9/16
to cornerstone platform
Hi,

Firstly, I tried to put the cornerstone.loadImage("wadourl:" + url) .then(function(image) {...}) in the webworker directly. That didn't work as it depends on jquery, and jquery and webworkers don't go together.  So, I created a "fake DOM" that allows jquery code to be used in webworkers. That allowed the image to download but then got into other problems.

Cornerstone only uses jQuery for Deferred you could easily get rid of jQuery by replacing Deferred with Promise or even Callback no need for a "fake DOM". I have a cornertstoneWadoImageLodader with no jQuery dependency and were download an decoding is done on WebWorker. But I was playing with it few monts ago and didn't kept it up to date, also I was only using wadoRS for image retrieving never tested other protocols. 

However, it's not clear what cornerstone method is for only downloading the image, and then use a different method to do the decoding. 

What do you have in mind for "use a different method" ? 
I think the real problem is how slow js is to run image decoding algorithm, not sure we can do much about it. If you plan to use your app in a fast network, you may try to download decompressed image directly. I already thought about that : comparing download uncompressed image vs download a compressed image and uncompressed it. Test it with different network speed and see if using uncompressed file isn't a better option in some case.
Last solution I had in mind is to use a compressing algorithm compatible with the browser, so we can let the browser do the uncompressing natively, maybe something like webp. Too bad browsers (except safari)  never wanted to implement j2k.

Chris Hafey

unread,
Nov 9, 2016, 10:59:09 AM11/9/16
to cornerstone platform
In the dev branch of cornerstoneWADOImageLoader, the downloading of the image is done in the UI thread and the decoding is done in a web worker.  Here is where the download is started:


And here is where the web worker decode is started:


Chris

Nihal Mehta

unread,
Nov 9, 2016, 11:01:21 AM11/9/16
to cornerstone platform
Thanks for the quick reply. 

Regarding "Cornerstone only uses jQuery for Deferred you could easily get rid of jQuery by replacing Deferred with Promise," how do I do that?

By a "different method," what I meant was instead of using cornerstone.loadImage which does the downloading and decoding, is there a method to just download the images and a different method to do the decoding. This way, I could download the images (which should be quick), and then do the decoding on a as-needed basis. 

Thanks,
Nihal
Reply all