Calling Matlab from QuPath- Passing Image Data

[Jason DeFuria]

Starting the first of the development of a script to call one of our colorspace functions for color normalization. In QuPath, I have selected the ROI, and would like to send that RGB data to our function. Here's the Script that I have so far. The MATLAB algorithm we are calling is titled "colorassign_manual.m"

/**
 * Script to Pass QuPath to colorassign_manual.m,
 * QuPath detection objects.
 *
 * This requires the MATLAB Engine, available with MATLAB R2016b or later,
 * and setup as described at https://github.com/qupath/qupath/wiki/Working-with-MATLAB
 *
 * @author Jason DeFuria
 */

import qupath.lib.classifiers.PathClassificationLabellingHelper
import qupath.lib.common.ColorTools
import qupath.lib.objects.classes.PathClass
import qupath.lib.objects.classes.PathClassFactory
import qupath.lib.scripting.QP
import qupath.extension.matlab.QuPathMATLABExtension
import com.mathworks.matlab.types.Struct
import qupath.lib.images.tools.BufferedImageTools
import qupath.lib.objects.PathObject
import qupath.lib.objects.PathTileObject
import qupath.lib.regions.RegionRequest
import qupath.lib.roi.PolygonROI
import qupath.lib.scripting.QP
import qupath.extension.matlab.QuPathMATLABExtension

import java.awt.image.BufferedImage

// Import the helper class
QuPathMATLAB = this.class.classLoader.parseClass(QuPathMATLABExtension.getQuPathMATLABScript())

// Get the MATLAB engine
QuPathMATLAB.getEngine(this)
try {

    def imageData = QP.getCurrentImageData()

    double downsample = 8
    BufferedImage img;
    def roi = QP.getSelectedROI()
    if (roi != null) {
        img = imageData.getServer().readBufferedImage(RegionRequest.createInstance(imageData.getServerPath(), downsample, roi))
    } else
        img = imageData.getServer().getBufferedThumbnail(1000, -1, 0)

    // Put RGB version of QuPath image & calibration data into MATLAB workspace
    if (roi == null) {
        double downsampleH = (double)imageData.getServer().getHeight()/(double)img.getHeight()
        double downsampleW = (double)imageData.getServer().getWidth()/(double)img.getWidth()
        double downsampleActual = Math.max(downsampleH, downsampleW)
        QuPathMATLAB.putQuPathImageStruct("img", img, null, 0, 0, downsampleActual, true)
    }
    else {
        def imgMask = BufferedImageTools.createROIMask(img.getWidth(), img.getHeight(), roi, roi.getBoundsX(), roi.getBoundsY(), downsample)
        QuPathMATLAB.putQuPathImageStruct("img", img, imgMask, roi.getBoundsX(), roi.getBoundsY(), downsample, true)
    }
    
    // Take Image and Move Into MATLAB Color Assign Manual function
      QuPathMATLAB.eval("image = imread(img);")
      QuPathMATLAB.eval("colorassign_manual(image)")


    // Print a message so we know we reached the end
    print("Image loaded in MATLAB")
} catch (Exception e) {
    println("Error running script: " + e.getMessage())
} finally {
    QuPathMATLAB.close()
}

I am getting an error on the imread line...

INFO: Error using <a href="matlab:matlab.internal.language.introspective.errorDocCallback('imread>parse_inputs', '/Applications/MATLAB_R2018a.app/toolbox/matlab/imagesci/imread.p', 450)" style="font-weight:bold">imread>parse_inputs</a> (<a href="matlab: opentoline('/Applications/MATLAB_R2018a.app/toolbox/matlab/imagesci/imread.p',450,0)">line 450</a>)
The file name or URL argument must be a character vector.

Error in <a href="matlab:matlab.internal.language.introspective.errorDocCallback('imread', '/Applications/MATLAB_R2018a.app/toolbox/matlab/imagesci/imread.p', 322)" style="font-weight:bold">imread</a> (<a href="matlab: opentoline('/Applications/MATLAB_R2018a.app/toolbox/matlab/imagesci/imread.p',322,0)">line 322</a>)
[filename, fmt_s, extraArgs, was_cached_fmt_used] = parse_inputs(cached_fmt, varargin{:});


INFO: Exception running statement: image = imread(img);
INFO:   Caused by com.mathworks.mvm.exec.MvmRuntimeException: The file name or URL argument must be a character vector.
INFO: Error running script: image = imread(img);

Any thoughts?

[Pete]

I'm restricted to answering on a phone this week, so am lacking an ability to explore much... but it looks like the trouble is that

QuPathMATLAB.eval("image = imread(img);")

expects the path to an image, but in fact you should at that point have a struct that includes the pixels already. You could try something like this:

QuPathMATLAB.eval("image = img.im;")

[Jason DeFuria]

Thanks, Pete! That's exactly the syntax and I was able to get our MATLAB scripts up and running for color normalization! Will share work once I have the okay!

[Jason DeFuria]

Hi Pete,

Working on my next iteration. It seems easier to me to keep all the MATLAB info together, in lieu of starting multiple threads.

pwd in MATLAB points to the correct directory.

In QuPath it outputs to  '/Applications/QuPath.app/Contents/Java' (at least on mac).

So I'm confused the QuPath will call the commands from the pwd, but will not output to the pwd. I'm using the command imwrite in MATLAB

I don't want to hardcode it on my end, although that does work. Do you have any idea on how to output it to the users pwd?

/**
 * Script to Pass QuPath to Color Normalization in Workflow. Must use RGB image at input, but can be of any fileformat that QuPath Supports

 * QuPath detection objects.
 *
 * This requires the MATLAB Engine, available with MATLAB R2016b or later,
 * and setup as described at https://github.com/qupath/qupath/wiki/Working-with-MATLAB
 *

 * NOTE: You will also need the Parallel Computing Toolbox installed for MATLAB, or if you do not have it, comment out the lines in the MATLAB scripts. (AKA- change ('UseParallel',true) to ('UseParallel',false))
 * @author Jason DeFuria
 */

import qupath.lib.classifiers.PathClassificationLabellingHelper
import qupath.lib.common.ColorTools
import qupath.lib.objects.classes.PathClass
import qupath.lib.objects.classes.PathClassFactory
import qupath.lib.scripting.QP
import qupath.extension.matlab.QuPathMATLABExtension
import com.mathworks.matlab.types.Struct
import qupath.lib.images.tools.BufferedImageTools
import qupath.lib.objects.PathObject
import qupath.lib.objects.PathTileObject
import qupath.lib.regions.RegionRequest
import qupath.lib.roi.PolygonROI
import qupath.lib.scripting.QP
import qupath.extension.matlab.QuPathMATLABExtension

import java.awt.image.BufferedImage

// Import the helper class
QuPathMATLAB = this.class.classLoader.parseClass(QuPathMATLABExtension.getQuPathMATLABScript())

// Get the MATLAB engine
QuPathMATLAB.getEngine(this)
try {

    def imageData = QP.getCurrentImageData()

    double downsample = 1

    BufferedImage img;
    def roi = QP.getSelectedROI()
    if (roi != null) {
        img = imageData.getServer().readBufferedImage(RegionRequest.createInstance(imageData.getServerPath(), downsample, roi))
    } else

        img = imageData.getServer().getBufferedThumbnail(1000, -1, -1)

    // Put RGB version of QuPath image & calibration data into MATLAB workspace
    if (roi == null) {
        double downsampleH = (double)imageData.getServer().getHeight()/(double)img.getHeight()
        double downsampleW = (double)imageData.getServer().getWidth()/(double)img.getWidth()
        double downsampleActual = Math.max(downsampleH, downsampleW)
        QuPathMATLAB.putQuPathImageStruct("img", img, null, 0, 0, downsampleActual, true)
    }
    else {
        def imgMask = BufferedImageTools.createROIMask(img.getWidth(), img.getHeight(), roi, roi.getBoundsX(), roi.getBoundsY(), downsample)
        QuPathMATLAB.putQuPathImageStruct("img", img, imgMask, roi.getBoundsX(), roi.getBoundsY(), downsample, true)
    }

   
    // Take Image and Move Into MATLAB Color Assign Manual function. Function should load GUI, and you must select at least 1 value each for Nuclei, Stroma, Lumen, and Cytoplasm.
      QuPathMATLAB.eval("loadedimage = img.im;")
      QuPathMATLAB.eval("[idx, lumen, nuclei, stroma, cytoplasm] = colorassign_manual(loadedimage)")

       // Print a message so we know we reached the end of the first function
    print("Color Assign Manual Complete")

    // Take Image and outputted Lumen, Nuclei, Stroma, and Cytoplasm variables and Move Into MATLAB Train Classifier function.
      QuPathMATLAB.eval("classifier = train_classifier(loadedimage, idx, lumen, nuclei, stroma, cytoplasm)")

       // Print a message so we know we reached the end of the second function
    print("Train Classifier Function Complete")

// Take Image and Move Into MATLAB Color Classifier
QuPathMATLAB.eval("classified = color_classify(loadedimage, classifier)")
       // Print a message so we know we reached the end of the third function
    print("Color Classifier Function Complete")


// Take Image and Move Into MATLAB Color Normalize

       QuPathMATLAB.eval("load('target.mat')")
       QuPathMATLAB.eval("rgb = color_normalize(loadedimage, target, classified)")
          // Print a message so we know we reached the end of the fourth function
    print("Color Normalize Function Complete")
   
// Take Color Normalized image and Save Output
QuPathMATLAB.eval("imwrite(rgb,'Normalized_Image.tif')")

    // Print a message so we know we know the image was saved to the Present Working Directory in MATLAB
    print("Image Saved to MATLAB PWD")

   
    // Print a message so we know we reached the end

    print("Complete Color Classify and Normalize Process Complete")

[Pete]

I suspect everything inside 'QuPath.app' is write-protected, and it could be risky to modify files within the app.

To avoid hard-coding paths, I often get the base directory of the current project* in my Groovy script, and sometimes create a subdirectory inside it:

path = buildFilePath(PROJECT_BASE_DIR, 'matlab')
mkdirs(path)

*-This does assume you are working with a project, and would fail if you just have a single image open on its own...

[Jason DeFuria]

Maybe that's not even what I'm going for.

Can I take the whole image, that has now been color corrected, and send it back to the QuPath application?

[Pete]

Are you working with a whole slide image, and are you able to write it out again as a full image pyramid (in some kind of format) - or are you working with smaller images in this case?

Also, how computationally intensive is the color correction in the end?  After estimating whatever parameters you need, could it conceivably be applied dynamically to the image by QuPath while it is reading directly from the original image file...?  This would be similar to how color deconvolution is applied dynamically when viewing the image (under the Brightness/Contrast dialog).  If the correction in your case is applied pixel by pixel that could work, but if it requires more complex calculations involving larger pixel neighborhoods then it may not.

[Jason DeFuria]

Working with a full slide image, and the image pyramids are the output.

The correction is applied to each pixel, after it has been classified in a GUI (the image is reduced to 10 colors via K-means). Each pixel is assigned one of those 10 values, and then those pixels are "pushed" to a target.

We use a ROI to train, but then apply that to the entire image, to cut down on computations.

[Pete]

If you have already written out the whole slide image, can you already read it back into QuPath again (as an entirely new image) and view it there?

If I understand correctly, this would be the first thing to check.  Thereafter I can think of three options for how to proceed:

• Write the image file, and read it into QuPath as a separate image (as above, but possibly reading it in again automatically)
• Write the image file, and display it on top of the original image as an overlay - but retain the original pixel values underneath (so that the overlay can be toggled on/off, or have its transparency adjusted)
• Do not write the image file, but rather write the transform information and apply it dynamically to the original pixel values (so that the image behaves as if the transformed pixels were the originals)

As mentioned above, the feasibility of the final option depends to some extent on the complexity of the transform that needs to be applied on a per-pixel basis.

The first option is the easiest, the second is possible (I have done something similar, but would need to hunt out the relevant code - such an overlay can be scripting to generate a quick prototype, but a more user-friendly implementation would require the next QuPath release).

[Jason DeFuria]

Hi Pete,

I can currently do #1, I used your Dialog helper code to save, and then I can manually open it.

I am very interested in option #2, if that would be possible! I'd love to see the code for activating an overlay.

Best,

Jay

[Pete]

I couldn't find the code, so rewrote it... you can see it at https://gist.github.com/petebankhead/98d82f5de92abefaf4cfd87ec0479121

It's described in the comments at the top.  I plan for overlays to become much more important (and useful) in the future, and therefore the API will change... which will then necessitate changes in any code you write based on this, for compatibility with later QuPath versions.  But ultimately the changes should not be too painful to make, and result in more useful and maintainable code in the end.

Any new release is still some months away, so I think it's still worthwhile to try this out.

Best wishes,

Pete

[Jason DeFuria]

Great, I will try this.

On a separate MATLAB note:

When each pixel is classified. I get an output. The problem I am encountering is in the very last step. I have a huge file, and it is processed in MATLAB, but it takes a while (i.e. hours) to get put back into QuPath...why would this be the case? Does it parse the text and input it a line at a time? I've tried it in both 0.1.2. and 0.1.3, and it is incredibly slow, even on a relatively decent computer. I see it is done in MATLAB, and the processing has already finished, but that QuPath isn't sure about this because it is slowly reading the data back.

Example (this will likely finish at current pace in about 2 hours, but has been done in MATLAB after 30 seconds)

[Pete]

If an image pyramid is being written, could the output returned to QuPath simply be the path to that image?

If the pixels of the image itself are being returned, that is indeed likely to be slow.  Especially if these are being printed, as the log suggests.  Adding a large amount of text to the log window in QuPath is not very well optimized, and can be very slow.  Even the time taken to return the data is tolerable, printing it won't be.

[Jason DeFuria]

Hmm- is there someway to suppress output in the window, then?

[Pete]

I don't know where the output is coming from in this case; if it's

QuPathMATLAB.eval("colorassign_manual(image)")

then a semicolon at the end of the MATLAB command might do the trick.

QuPathMATLAB.eval("colorassign_manual(image);")

It would be best to try to prevent any attempt to print, rather than try to stop the GUI component displaying the log in QuPath (since potentially different components might be registered to display the printed output).

[Jason DeFuria]

No idea what changed between the semi-colon suppression on Friday and today.

I tried the overlay, and can't figure out the input of the script that you wrote. Is it "path" under this section?

class WholeSlideImageOverlay extends AbstractImageDataOverlay {

    private boolean initialized = false
    private QuPathViewer viewer
    private ImageServer<BufferedImage> server
    private AffineTransform transform
    private boolean bindToObjectDisplay

    public WholeSlideImageOverlay(final QuPathViewer viewer, final String path, boolean bindToObjectDisplay) {
        super(viewer.getOverlayOptions(), viewer.getImageData())
        this.viewer = viewer
        this.server = ImageServerProvider.buildServer(path, BufferedImage.class)
        this.initialized = true
        this.bindToObjectDisplay = bindToObjectDisplay
    }

[Pete]

Yes.  Currently it's just the current image

def path = viewer.getServerPath()

[Jason DeFuria]

So let's say I am trying to add in this corrected file- "/Users/jason/Documents/MATLAB/Normalized_Image723.tif"

I'm just not seeing how to input that file to the original (which I have open in QuPath).

[Pete]

Have you tried this?

def path = "/Users/jason/Documents/MATLAB/Normalized_Image723.tif"

[Pete]

If so, please post any error messages you get.

[Jason DeFuria]

It actually works if I don't uncomment this line:

        // Apply transform... handy if using the same image, just to check *something* happens
       g2d.translate(1000, 1000)