Large TIFF raw data to TIFF (pyramid+tiled) for QuPath

[Ajay Zalavadia]

Once in a while I do have to deal with a large raw TIF file (i.e. 65K X 50K pixel 8GB). Since these files are not in a tiled TIFF format, it becomes very difficult to do anything with these files. I was wondering if you guys have any suggestion, I was wondering if I could convert these files to Pyramid+Tiled TIFF using tools like Vips image processing library it may be possible to work with them in QuPath.

Is there a better alternative ! 

[micros...@gmail.com]

I vaguely recalled an example of something like this on the GitHub site:  https://github.com/qupath/qupath/issues/221

Maybe that will help, maybe it won't :)  Good luck.

[Pete]

The upcoming Bio-Formats 6.0.0 will also be able to write image pyramids: http://blog.openmicroscopy.org/file-formats/community/2018/11/29/ometiffpyramid/

One of the biggest advantages of this is the potential to incorporate lots of useful metadata.

[Ajay Zalavadia]

That's interesting.

[Ajay Zalavadia]

I was trying to use bfconvert command from bftools, I will post if I can successfully at least plug in magnification and pixel size.

Has anyone tried Bio-format for pyramid conversion already ?

[Pete]

You'll need to be using the v6.0.0 milestone (I think the blog links to m3 although I believe there's also an m4 by now...).  My latest (not using bfconvert) effort to incorporate the writing into QuPath is here: https://github.com/petebankhead/qupath/blob/pete-java-11/qupath-extension-bioformats/src/main/java/qupath/lib/images/writers/ome/OMEPyramidWriter.java#L79

I've used it to successfully write several image pyramids, but that code is subject to change (it's also a bit slow for large pyramids).

You can try searching for PyramidOMETiffWriter and https://github.com/openmicroscopy/bioformats/ to see some other code showing it in action.  This may also be useful: 

https://github.com/openmicroscopy/bioformats/blob/47eb93c49e4b283e8907961c9f2a644c689d329a/components/bio-formats-tools/src/loci/formats/tools/ImageConverter.java

[Ajay Zalavadia]

Glad to see the Pyramid Writer being incorporated in QuPath. It is going to be very useful. Thank you for the awesome work.

[Ajay Zalavadia]

Update,

I was able to use VIPS 8.6.3 executable to convert TIFF file (64k x 48k pixel) to Tiled TIFF (in 67 sec) which loaded fine with QuPath (without resolution information), I used your tip from github issue #223 and added the resolution via script. I think spatial calibration values added via script are only temporary, is there a way to make those value permanent (I am using IHC branch of QuPath 1.3) ?.

 

I have zero experience working with Java, I am learning as I go forward. Over the weekend I was able to compile Bioformat V6.0.0-SNAPSHOT (.m2) and use bfconvert to achieve the same. It produced a tiled TIFF, which was again compatible with QuPath (without MPP and magnification).

While both methods did produce tiled TIFF files compatible with QuPath (~almost compatible), it would be nice to have the ability to add resolution and magnification info.

How do I use your version of pyramid writer (OMEPyramidWriter.java#L79) ?

[Pete]

Good this much is working!

I've lost track of where I've written about it, but it's not possible to store the metadata permanently yet.  It needs to be possible one day, but it's tricky to know where... it's on my list of Important Things To Do, but wrapped up with a revision of how projects work and data is saved, which rapidly turns into a big job.  Planning to try to get a few days to look into it over the next month, but not yet sure what will emerge at the end of those days.

[nheathp...@gmail.com]

I'm bumping this to report and discuss means of getting raw, unpyrmamidal multi-channel WSI images into QuPath by means of conversion. In my work, I am often doing registration of WSIs (I'm happy to share a bit of this process) and getting out the registered images as large .tif files. Similar to Ajay, I've used VIPS and it is very effective at generating pyramidal images of 24-bit RGB images which OpenSlide will support in QuPath. However, when generating 16-bit multi-channel data via VIPS, QuPath will attempt to read the full plane of the highest magnification using BioFormats. With BioFormats support for pyramidal generation now available, I used ```bfconvert``` to change these .tifs into ome.tiffs which work and load appropriate metadata into QuPath. HOWEVER, the conversion is punishingly long. A 16-bit ```29600x22242x3``` image with 4 sub-resolutions took 3.2 hours and this is not a lot channels and comparatively, not a lot of pixels...

Any thoughts out there on best conversion practices for multi-channel images?

[Ajay Zalavadia]

I am also waiting for the solution of the same issue, I also have 7 channel 16-bit TIFF data but no way of loading it to QuPath. I am hoping Pete's   OMEPyramidWriter when finished will have a solution to this issue. Pete has made a few commits towards that direction, but it is work in progress.

Keep this thread up to date if you come across any other solution.

[nheathp...@gmail.com]

I should note that I did succeed in converting a 16-bit multi-channel fluorescence image into pyramidal ome.tiff and read it using into QuPath 0.1.2 using the bioformats extension just with the caveat that it's very slow.

bfconvert /data/mytif.tif /data/mytif.ome.tif -bigtiff -pyramid-scale 2 -pyramid-resolutions 5 -tilex 256 -tiley 256 -compression LZW

It's important to check and make sure your multiple layers are recognized as channels by bioformats (you can just the bio-formats showinf command line tool on your image) and not time points. If they are time points, you can open the images in imagej, stack them if they aren't already so then go to Color -> Make Composite, then save the composite .tif. Bio-formats will recognize the image as multi-channel.  There may be a better way of editing the metadata for this.

Heath

[Ajay Zalavadia]

Thank you for that info, I will give it a another try. I gave up last time because of the slow process.

[Ajay Zalavadia]

Unlike yours, I have individual Tiffs for each channel. Do you have a way to use bfconvert on seperate files and compile ome.tif ?

[Pete]

I'd be interested in understanding how to write pyramids faster as well... perhaps best to start a discussion at https://forum.image.sc/tags/bio-formats

You can also tag QuPath there: https://forum.image.sc/tags/qupath