Running QuPath from Python- a sample from a newbie!

[micros...@gmail.com]

Hi all, I have finally gotten around to trying some of the command line stuff I have seen floating around, and set up a simple Python script that handles a simple, mostly QuPath built-in, script that can run on images and export data.
The Python parts are written for Python 3 using PyCharm (Thanks Pete!) and the scripts in IntelliJ IDEA community edition.  Most of what I have built is based off of posts from other users in this forum, feel free to chime in with suggestions or ideas!
All work was done in Win10 and is reflected in the names/file structures.

The two scripts are based on the assumptions that:
1. All files are brightfield and in a specific folder (in this case D:\Images to Process)
2. Results for all image processing will go into an existing folder called D:\Results\Data


The QuPath script is based mostly on the work in these two threads:
https://groups.google.com/forum/#!msg/qupath-users/7cdsBsdy4HQ/faFXwPN3BgAJ
https://groups.google.com/forum/#!topic/qupath-users/6g7sUPvcNQo

And is as follows:

import static qupath.lib.scripting.QP.*;
import static qupath.lib.scripting.QPEx.*;

//Set the base image information, since you are not building off of a .qpdata files, you must include these
//Therefor, this script will currently only work well on one type of known image.
//Future work: consider checking the entire annotation area for estimating stain vectors and accept default result.
setImageType('BRIGHTFIELD_H_DAB');
setColorDeconvolutionStains('{"Name" : "H-DAB default", "Stain 1" : "Hematoxylin", "Values 1" : "0.65111 0.70119 0.29049 ", "Stain 2" : "DAB", "Values 2" : "0.26917 0.56824 0.77759 ", "Background" : " 255 255 255 "}');

//Stuff you want to do here. In this case, I detect tissue at a fairly dark threshold, and then select the annotation
//to run a basic cell detection
runPlugin('qupath.imagej.detect.tissue.SimpleTissueDetection2', '{"threshold": 220, "requestedPixelSizeMicrons": 5.0, "minAreaMicrons": 1000000.0, "maxHoleAreaMicrons": 1000000.0, "darkBackground": false, "smoothImage": true, "medianCleanup": true, "dilateBoundaries": false, "smoothCoordinates": true, "excludeOnBoundary": false, "singleAnnotation": true}');
selectAnnotations();
runPlugin('qupath.imagej.detect.nuclei.WatershedCellDetection', '{"detectionImageBrightfield": "Hematoxylin OD", "requestedPixelSizeMicrons": 0.5, "backgroundRadiusMicrons": 8.0, "medianRadiusMicrons": 0.0, "sigmaMicrons": 1.5, "minAreaMicrons": 10.0, "maxAreaMicrons": 400.0, "threshold": 0.1, "maxBackground": 2.0, "watershedPostProcess": true, "excludeDAB": false, "cellExpansionMicrons": 5.0, "includeNuclei": true, "smoothBoundaries": true, "makeMeasurements": true}');

//Save the results to a predetermined folder
saveAnnotationMeasurements('D:\\Results\\Data',)
saveDetectionMeasurements('D:\\Results\\Data',)

Anyone using this would want to edit almost all of the image data and cell detection to tailor it to their own project, but those parts can be fairly easily swapped out.

On the Python side, in it's simplest form, my script is just:

import subprocess
import os

#Work from QuPath installation directory
os.chdir(r"C:\\Program Files\\QuPath\\app")

script = r"D:\\Script\\NoGUITest.groovy"
imageDirectory = r"D:\\Images to process\\"

fileList = os.listdir(imageDirectory)
for file in fileList:
 imageFile = imageDirectory + file
 subprocess.call(["java", "-jar", "QuPathApp.jar", '-image', imageFile, "-script", script])

The working directory is set to the QuPath app folder so that everything works, and the images are pulled from their expected folder, and the results placed in the data folder.

[micros...@gmail.com]

Building on this, I altered the Python script to take the images once processed and move them to a new folder:

import subprocess
import os


#Work from QuPath installation directory
os.chdir(r"C:\\Program Files\\QuPath\\app")

imageDirectory = r'D:\\Images to process\\'

script = r"D:\\Script\\NoGUITest.groovy"

fileList = os.listdir(imageDirectory)

for file in fileList:
 imageFile = imageDirectory+

file
 movedImage = 'D:\\Results\\Processed Images\\'+file
 subprocess.call(["java", "-jar", "QuPathApp.jar", '-image', imageFile, "-script", script])
 os.rename(imageFile, movedImage)

[micros...@gmail.com]

And a slight addition lets you run the script indefinitely, checking the input folder every X minutes for files, and then running the given script on them.  If you have more complex needs, you could look at different folders for different scripts, or alter one of the loops to look for different file names.

import subprocess
import os
import time

#Work from QuPath installation directory
os.chdir(r"C:\\Program Files\\QuPath\\app")
imageDirectory = r'D:\\Images to process\\'
script = r"D:\\Script\\NoGUITest.groovy"

minutesBetweenChecks = 30
//a slight delay at start
time.sleep(30)
while 1:

 fileList = os.listdir(imageDirectory)
 for file in fileList:
 imageFile = imageDirectory+file
 movedImage = 'D:\\Results\\Processed Images\\'+file
 subprocess.call(["java", "-jar", "QuPathApp.jar", '-image', imageFile, "-script", script])
 os.rename(imageFile, movedImage)

 time.sleep(minutesBetweenChecks*60)

[Murali S]

Hi Micro,

Is it possible to use xmlParser and JsonSlurper in the script which is used by command line? if so How?

because I am getting error while using xml parser and json slurper.

Thanks & Regards,

Murali S.

[micros...@gmail.com]

Hi! I saw your post in the other thread and I am afraid that this is my first attempt at doing anything with QuPath by command line. I do not know anything yet about the xml parser or JSON slurper, sorry! I appreciate the time you and Pete have put in so far though, as it made it much easier to get started.

[Pete]

These dependencies aren't available to QuPath.  Either add the relevant jar files to QuPath's extensions folder (or otherwise find a way to put them on the classpath when launching QuPath), or else find a way to avoid the additional external dependencies.  For example, Gson should be available and might work as an alternative to JsonSlurper.

[Murali S]

Hi Pete,

I added both groovy-json-2.1.7.jar and groovy-xml-2.1.1.jar files to the jars directory. Also I updated the classpath in the QuPath.cfg file. So the same script working well when I am using the QuPath app as GUI. But its not working when I am trying to use it through command line.

Thanks & Regards,

Murali S.

[Pete]

The relevant code for the command line is at https://groups.google.com/d/msg/qupath-users/HpZDvzm9jcs/Xq0-hT1_CAAJ

The script editor code is considerably more complex, and makes more effort to help set up the context in which the script is run: https://github.com/qupath/qupath/blob/v0.1.2/qupath-gui-fx/src/main/java/qupath/lib/scripting/DefaultScriptEditor.java

I don't know exactly where the problem arises in your case, but I'd start by investigating with simple command line scripts that print the classpath etc. to see if that helps track down the problem.  You might also look to set the classpath explicitly from the command line.

[Murali S]

Even I tried with following command still I am getting error.

java -classpath jars/*.jar -jar QuPathApp.jar -image <imgpath> -script <script path>

[micros...@gmail.com]

Well, I am not familiar enough with the XML or JSON based scripts to really test it myself, but if you feel like posting the entire script that works within QuPath, I can take a look at trying to make it work through command line on my computer.  I am probably about to get a little bit busy and won't be able to both mock up a test script and figure things out!  I assume it is based off of the script in your other thread?

[Murali S]

Hi Micro,

The JSON based code is given below. For test.txt PFA.

import qupath.lib.scripting.QP;

import groovy.util.*;
import groovy.json.JsonSlurper;

def textFile = "test.txt"

def inputFile= new File(textFile)

def InputJSON = new JsonSlurper().parseText(inputFile.text)

InputJSON.each{ println it }

Thanks & Regards,

Murali S.

[micros...@gmail.com]

Well, I think I got about as far as you did with that.  I did notice that placing the jar file (http://www.java2s.com/Code/Jar/g/Downloadgroovyjson243jar.htm) in either the base folder, app folder, or jars folder did nothing on its own, the script still failed after starting QuPath.  Once I dragged the JAR into QuPath to install it, the script worked within QuPath, but still does not work from Python.

Error:Script1.groovy: 3: unable to resolve class groovy.json.JsonSlurper

Not sure if it helps, but the JSON object InputJSON has :

INFO: class groovy.json.internal.LazyMap

Also, when reading up on classpath, I found the following:

It's also worth noting that when you use the  java -jar command line option to run your Java program as an executable JAR, then the CLASSPATH environment variable will be ignored, and also the -cp and -classpath switches will be ignored. In this case, you can set your Java classpath in the META-INF/MANIFEST.MF file by using the Class-Path attribute. In short Class-path attribute in manifest file overrides classpath specified by -cp, -classpath or CLASSPATH environment variable.

Found at: http://javarevisited.blogspot.com/2011/01/how-classpath-work-in-java.html

I haven't yet looked around for what the manifest.mf is though, and it's getting late for me, so maybe another day!

[micros...@gmail.com]

Turns out I couldn't leave well enough alone, and the manifest is in the original files at: Qupath-master\deploy\natives\META-INF\manifest.mf if you have cloned the GitHub files.  I am not sure we can generally make use of it though.

[Pete]

Launching it this way works on my Mac:

java -cp "./QuPathApp.jar:/path/to/groovy-json.jar" qupath.QuPath -script /path/to/script.groovy

If you are using Windows then you may need to replace the colon for the -cp argument with a semicolon.

[micros...@gmail.com]

Yes, that did it, with the semicolon!  I take it the qupath.Qupath is running the program because it now has access to the "mainclass" (as defined in the cfg file) in QuPathApp.jar through the classpath?  So much to learn...

[Murali S]

Yes Thank you very much Pete!. It worked for me too.

I am trying to get the available path classes using getQuPath().getAvailablePathClasses(); in the script, but it raised the following error.

javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingMethodException: No signature of method: org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.getQuPath() is applicable for argument types: () values: []

Possible solutions: getAt(java.lang.String), getClass()

        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:155)

        at qupath.QuPath.main(Unknown Source)

If I tried with QPEx.getQuPath().getAvailablePathClasses(); then I am getting following error.

javax.script.ScriptException: javax.script.ScriptException: java.lang.NullPointerException: Cannot invoke method getAvailablePathClasses() on null object

        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:155)

        at qupath.QuPath.main(Unknown Source)

Caused by: javax.script.ScriptException: java.lang.NullPointerException: Cannot invoke method getAvailablePathClasses() on null object

Thanks & Regards,

Murali S.

[Pete]

Launching QuPath this way, I don't think the .cfg file will have any impact.  It only matters when launching the 'normal' way.  The classpath includes the Jar containing the main class in the command line I gave above, and the name of the main class is also included as an additional argument.  The rest of the required jars are either specified from the command line or within the manifest file of another jar.

The GUI-related methods are not available when running a script like this, since there is no GUI.  To ensure this, instead of relying on QPEx.java you should use QP.java.

It isn't possible to request the available PathClasses shown within the GUI because the GUI hasn't been started, but I'm not sure why they'd be needed anyway.

[Murali S]

Ok. Is there any way to get already available classes and add new path class to the PathClassFactory while running the groovy script through command line?

[Pete]

They don't need to be available already.  Just request the PathClass you want from the factory, and it will be created if there isn't already one in stock.

The getAvailablePathClasses() method is only for finding out which PathClasses are presented to the user in the GUI, and has no relevance if the GUI isn't there.

[Murali S]

Okay Thank you very much Pete.