Thanks!
I'm not sure how to imagine the appearance of your cytology images (are they single 2D planes, or z-stack?), but it may well be the case that none of the existing (mostly histology-focussed) tools in QuPath are particularly good for cell detection here, no matter how the parameters are adjusted.
In that case, I wonder if there are any benefits in a very modest first detection step - maybe using ‘Fast cell counts’ or any of the superpixel methods. This is likely to give many false positives, but you could then use ‘Add intensity features’ to compute local textures, and apply one of the standard classifiers to filter out many of the uninteresting areas afterwards - applying further processing only to what remains, if necessary. 'Fast cell counts' would only give single points, but you can always create a rectangle annotation around each point in a script if needed.
This may or may not be very useful.
This allows you to extract part of the image (either at full or reduced resolution), send it to ImageJ, and explore any supported image processing operations there for segmenting the cells. If you can successfully generate ROIs for each cell in ImageJ, these can be added to an overlay and then sent back to QuPath - optionally along with some basic measurements.
If you find a suitable approach within ImageJ, this can then be automated either using an ImageJ macro or (more flexibly) by writing a Groovy script to run from within QuPath that makes direct use of ImageJ. For an example of this, see the 'Estimate_background_values.groovy' file among the 'Batch processing scripts' with the
bioRxiv preprint.
If you aren’t familiar with ImageJ, there are lots of resources online and a really good ImageJ forum. I also wrote a handbook
here.
Alternatively, you could tackle the problem in Java or Groovy using any additional Java libraries you require. OpenCV is already available. For other libraries, if you add the corresponding Jar files to the extensions directory of QuPath (e.g. by dragging the files onto the main QuPath window), then they should be available on the Java classpath from any Groovy scripts.
Finally, if you have a whole slide image then you might want to simply extract some manageable image tiles (e.g. using ImageJ) and then try processing with any other software or programming language that you prefer… such as CellProfiler, MATLAB, R, Python or C++. If this approach works well enough, then we could consider later how to connect the pieces together and make the processing more streamlined.