slow opening of tif files using scifio-based ImgOpener

[Tobias Pietzsch]

Hi,

recently I ran across a problem with the scifio-based ImgOpener of ImgLib2.
I wanted to load stacks from series of tif files which were all in the same directory (about 100.000 of them).
With ImgOpener, this was imbearably slow, so I decided to load the files using ImageJ1 and wrapping the resulting ImagePluses into imglib2 containers.

Today, I created a benchmark, that illustrates the problem.
You can find it here: https://github.com/tpietzsch/TifBenchmark
It is derived from Johannes' https://github.com/imagej/minimal-ij1-plugin, so you can simply import the maven project in Eclipse and try it out easily.

The results are disillusioning.
With just a single tif, scifio opens the file 10x slower than ImageJ1. Bad, but maybe bearable.
With 100.000 tif files in the same directory, scifio is 400x slower! That is 0.1s for loading 50 tifs from ImageJ vs 55s using scifio!!!

Can anyone explain this behavior?
This needs to be fixed if we expect anyone to seriously use the ImgOpener…

best regards,
Tobias

[Melissa Linkert]

Hi Tobias,

Bio-Formats/SCIFIO is known to be slower than ImageJ when opening plain
TIFF files. See:

http://trac.openmicroscopy.org.uk/ome/ticket/9228

That ticket (and several other performance improvements in Bio-Formats) are at
the top of my to-do list for the next few weeks.

Regards,
-Melissa

> --
> --
> Please avoid top-posting, and please make sure to reply-to-all!
>
> Mailing list web interface: http://groups.google.com/group/fiji-devel
>
> ---
> You received this message because you are subscribed to the Google Groups "Fiji-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fiji-devel+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Tobias Pietzsch]

Hi Melissa,

That is awesome! Thanks a lot for tackling these problems.

Out of curiosity, could you briefly explain (if there is an easy explanation) why the 100.000-files-issue happens?
Is bioformats analyzing the directory the TIFF file lives in? Is there a way to turn this off?

best regards,
Tobias

[Albert Cardona]

2013/2/6 Tobias Pietzsch <piet...@mpi-cbg.de>

Hi Melissa,

That is awesome! Thanks a lot for tackling these problems.

Out of curiosity, could you briefly explain (if there is an easy explanation) why the 100.000-files-issue happens?
Is bioformats analyzing the directory the TIFF file lives in? Is there a way to turn this off?

Yes, Bio-formats analyzes the folder. Some formats are multi-file.

Albert

--
http://albert.rierol.net
http://www.ini.uzh.ch/~acardona/

[Lee Kamentsky]

On 2/6/2013 12:13 PM, Albert Cardona wrote:

2013/2/6 Tobias Pietzsch <piet...@mpi-cbg.de>

Hi Melissa,

That is awesome! Thanks a lot for tackling these problems.

Out of curiosity, could you briefly explain (if there is an easy explanation) why the 100.000-files-issue happens?
Is bioformats analyzing the directory the TIFF file lives in? Is there a way to turn this off?

Yes, Bio-formats analyzes the folder. Some formats are multi-file.

Sorry, chiming in late here. A few years back, we had a case where CellProfiler was using BioFormats to open files in a similarly-sized directory and the load generated by walking the files in the directory (duplicated because multiple blades were doing the same thing) brought the file server down. Melissa was helpful - she added directory cacheing to loci.common.Location and has made sure that all formats honor loci.formats.IFormatReader's GroupFiles setting (setGroupFiles / isGroupFiles). We always run with setGroupFiles(false) when accessing images in CellProfiler in order to reduce load.

Albert

--
http://albert.rierol.net
http://www.ini.uzh.ch/~acardona/ --

[Curtis Rueden]

Hi all,

> Is there a way to turn this off?

Yes, you can call imageOpener.setAllowOpenFiles(false) which will prevent Bio-Formats from doing any directory listings or opening up any files for the purposes of determining the file format. However, if you set this flag, some file formats will be improperly identified.

Out of curiosity: how much does the benchmark improve if that flag is set inside ImgOpener's createReader method? Another thing you could test, to determine the performance impact, is to change "new ImageReader" to "new TiffReader", which explicitly tells ImgOpener to assume everything is a TIFF.

If such changes vastly improve the ImgOpener performance, we could consider adding another ImgOpener signature that lets you explicitly specify the file format.

> Bio-formats analyzes the folder. Some formats are multi-file.

Indeed. And some files are even considered a different file format depending on which other files are present in the same or nearby directories.

Regards,

Curtis

P.S. to Melissa: Johannes has some ideas for improving the TIFF reader performance too, which he plans to explore over the next few days. We will keep you posted on any progress.

--

[Tobias Pietzsch]

Hi Curtis,

Ok, I tried the things you suggested as well as the setGroupFiles( false ) suggested by Lee.
Here are the results:

baseline (unmodified ImgOpener):
loading 50 tif images using ImageJ, 10000 other tif files in same directory
median: 141 ms
loading 50 tif images using ImgOpener, 10000 other tif files in same directory
median: 6953 ms

with setAllowOpenFiles( false ):
loading 50 tif images using ImgOpener, 10000 other tif files in same directory
median: 5972 ms

with setGroupFiles( false ):
loading 50 tif images using ImgOpener, 10000 other tif files in same directory
median: 2331 ms

with both, setAllowOpenFiles( false ) and setGroupFiles( false ):
loading 50 tif images using ImgOpener, 10000 other tif files in same directory
median: 1070 ms

with TiffReader:
loading 50 tif images using ImgOpener, 10000 other tif files in same directory
median: 5061 ms

with TiffReader and setGroupFiles( false ):
loading 50 tif images using ImgOpener, 10000 other tif files in same directory
median: 568 ms




For comparison, the case where there are no other files in the directory
baseline (unmodified ImgOpener):
loading 50 tif images using ImageJ, 0 other tif files in same directory
median: 140 ms
loading 50 tif images using ImgOpener, 0 other tif files in same directory
median: 1804 ms


with TiffReader and setGroupFiles( false ):
loading 50 tif images using ImgOpener, 0 other tif files in same directory
median: 552 ms

That's still 4x slower than ImageJ1 but already getting a lot closer. Nice!

best regards,
Tobias

[Johannes Schindelin]

Hi Tobias,

Thanks for the detailed comparison. I will add code to perform these
timing tests more automatically and try to get at the root of the problem
as I indicated in a private conversation previously (using OProfile).

Ciao,
Dscho

[Lee Kamentsky]

On 2/7/2013 10:21 AM, Tobias Pietzsch wrote:
> Hi Curtis,
>
> Ok, I tried the things you suggested as well as the setGroupFiles( false ) suggested by Lee.
> Here are the results:
>

> ...lots of good stuff...

> That's still 4x slower than ImageJ1 but already getting a lot closer. Nice!

Thanks Tobias. From the CellProfiler perspective, I suspect the ImageJ
1.0 opener (and similar - libtiff for instance) is not the same beast
as BioFormats. There's some sacrifice of raw speed in BioFormats for
consistency of interface and metadata handling and, for CellProfiler,
we're willing to accept ~ 1 sec for loading images considering our
typical processing budget of about a minute per imaging site and given
the wide variety of images used by our community. Having said that,
streamlining is appreciated when possible.

[Curtis Rueden]

Hi Lee & everyone,

> I suspect the ImageJ 1.0 opener  (and similar - libtiff for instance)

> is not the same beast as BioFormats.

Indeed, ImageJ 1.x does not properly support the baseline TIFF specification. In other words, there are some valid TIFFs that ImageJ 1.x's built-in support cannot handle, and you have to use Bio-Formats for those. Specifically, ImageJ treats TIFFs as streams and assumes that headers and pixels are in order within the file, whereas the TIFF specification allows arbitrary offsets to headers (i.e., IFDs), individual header entries, image planes, and even specific strips of specific planes. Supporting all of that correctly in the general case has performance implications. That said, it should be possible, in common cases, to detect that a TIFF file is "friendly" such that we can take such shortcuts. For example, if we detect that the TIFF was written by whitelisted software such as ImageJ or Bio-Formats itself, we can make certain assumptions for speed, and if those assumptions are not borne out in practice then fall back to the more general behavior (for an even greater performance penalty of course, but in practice it wouldn't happen often if ever).

-Curtis

To unsubscribe from this group and stop receiving emails from it, send an email to fiji-devel+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
   

--
--
Please avoid top-posting, and please make sure to reply-to-all!

Mailing list web interface: http://groups.google.com/group/fiji-devel

--- You received this message because you are subscribed to the Google Groups "Fiji-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fiji-devel+unsubscribe@googlegroups.com.

[Johannes Schindelin]

Hi Tobias,

On Thu, 7 Feb 2013, Tobias Pietzsch wrote:

> Ok, I tried the things you suggested as well as the setGroupFiles( false
> ) suggested by Lee.

I finally have some promising results to show. After writing a
light-weight profile based on the ideas of OProfile, but backed by
Javassist (hence it neither requires operating system support nor a
special Java agent or other start-up parameters to the Java virtual
machine), I could find a couple of bottlenecks, even if it seems that the
most important bottleneck I found was obtaining thread-specific CPU tick
counts (i.e. the profiling code itself that I subsequently replaced by
global ticks, not as precise, but way lower impact).

I performed all timings as best of 5 runs (which is a more meaningful
value than the Median), 50 tif images, 10,000 other tif files in same
directory. My current state can be found in the 'master' branch of
git://github.com/dscho/TifBenchmark.

The most surprising finding was that copying from planes into the
ArrayContainer contributed substantially to the execution time. Using a
PlanarContainer speeded that up.

Another speed-up (and nice simplification of the code) was to use
ByteBuffer instead of hand-coded byte array to float array. After
switching off profiling, it turned out not to be that much of an
improvement, though it does shave off about 200 milliseconds; given that
the reference we try to reach is only ~120ms it might be an important
improvement, still.

You can find this change (not yet cleaned up) in the 'performance' branch
of git://github.com/imagej/bioformats. I will work more on this branch.

Together with setGroupFiles(false), I achieved a dramatic speed-up. The
time is now dominated by createReader which initializes the Reader for
every slice, which in turn creates a new OMEXMLMetadata (taking a whopping
17 milliseconds every single time).

This change is in the 'performance' branch of
git://github.com/imagej/imglib. I will work more also on this branch.

ArrayContainer (i.e. Tobias' original benchmark): best of 5: 4,922 ms
ByteBuffer, ArrayContainer: best of 5: 4,838 ms
PlanarContainer: best of 5: 4,736 ms
ByteBuffer, PlanarContainer: best of 5: 4,540 ms
PlanarContainer, setGroupFiles(true): best of 5: 1,832 ms
ByteBuffer, PlanarContainer, setGroupFiles(true): best of 5: 1,629 ms
ImageJ 1.x: best of 5: 113 ms

Down from a 44x slower code to 14x slower. Still nothing to laugh about,
but an improvement.

Ciao,
Johannes

P.S.: just to illustrate the impact on execution time by the profiling
code itself: the ImageJ 1.x test runs 2ms longer (115ms instead of 113ms),
the best ImgOpener runs 3,523ms instead of 1,629ms, more than double as
long! Even when instrumenting the code but switching off the profiling
(i.e. every method still has to test one public static boolean and then a
local variable just before returning), the time was still 1,839ms, i.e.
200ms slower. If you switch on thread-specific tick counting via the
ThreadMXBean, ImageJ 1.x goes up to 568ms, and the best ImgOpener
benchmark goes up to more than 100 seconds(!).

[Johannes Schindelin]

Hi,

On Sun, 3 Mar 2013, Johannes Schindelin wrote:

> ByteBuffer, PlanarContainer, setGroupFiles(true): best of 5: 1,629 ms
> ImageJ 1.x: best of 5: 113 ms

More progress.

Switching on file mapping brought it down to 1,115ms:

https://github.com/dscho/TifBenchmark/commit/1ac041e8f4ce5d935151fcfb70023f67f2a086df

A minor improvement to 1,097ms was achieved by switching the ImageReader
from using the Location delegator to using the real thing instead:

https://github.com/imagej/bioformats/commit/2bbf001061d13882234d07cdbd8cdcbfb9778675

Switching the ZeissTIFFReader (which tries very hard to claim that it is
responsible to open the files) from using a delegator to the real
CaseInsensitiveLocation brought it down to 957ms:

https://github.com/imagej/bioformats/commit/ce94a9e8647a2f786bba3045b005ec0973df5e83

A real game changer was to disallow the ZeissTIFFReader to blow away the
cache of the CaseInsensitiveLocation class *every* *single* *time*,
bringing the time down to 317ms:

https://github.com/imagej/bioformats/commit/33447f50a51356cdc0609164002f0860c724d5e9

My hunch is that there is serious room for improvement in the
CaseInsensitiveLocation class. And probably even in the ZeissTIFFReader:
there must be a nicer way to detect whether we are dealing with a Zeiss
.tiff than to look at all files in the same directory, searching for files
with a similar name -- and maybe even differ in capitalization, thanks
Zeiss :-(.

I could even imagine that with .tiff being such a common case with such a
lot of subtypes that it will be *necessary* to add an own class of plugins
for TIFF subtypes, so that the base TIFF reader can be chosen quickly as
the one to use, without having to run through the complete list of readers
when determining what is the correct reader. The subtype plugins would
then have to implement a very fast test *after* we already know that it is
a TIFF. Some of them could then even state that scanning through all the
IFDs is not necessary (e.g. when it is detected that ImageJ 1.x wrote the
file.).

And finally, the profiling pointed out that I was right when I had a bad
feeling about the #isThisType() method and wanted something less
free-form, such as a combination of file extensions and patterns to match
against the first kilobyte (much like the HandleExtraFileTypes class does
it). Worse: the signature of isThisType() does not even allow sharing of a
RandomAccessInputStream! And sure enough, a dominant part of the
performance statistics are made up by opening gazillions of
RandomAccessInputStreams, all for the exact same file, all reading pretty
much the exact same bytes. Setting setAllowOpen(false) -- whose name I
mistook as first to imply that we refuse to open files at all, hence
preventing us from opening images altogether -- brings the time down to
186ms:

https://github.com/dscho/TifBenchmark/commit/d10db689c36b668bb190b041f0582add33ef1990

Summary so far: the target is to come close to 115ms [*1*]. I started out
with code that took a whopping 4,922ms. With a couple of tricks and
shortcuts, I got it down to 186ms, not quite the 43x improvement I hoped
for, but a 26x improvement still [*2*].

Ciao,
Dscho

P.S.: One thing I almost forgot to mention: initializing the logger takes
quite some time. It is a one-time cost, sure, but it is quite noticable
when the first slice is opened (hence I excluded the first run from the
performance profiling for now). We might want to do something about that,
e.g. constructing a super-light-weight logger a la scijava-common's
StderrLogService by default.

Footnote *1*: ImageJ 1.x' new ImagePlus(path) wrapped as a float Img). We
will probably not be able to reach that goal because the ImgOpener is
intended to handle many more file formats than just TIFF, and it should
also open images which are valid as far as the TIFF specification is
concerned but which ImageJ 1.x completely fails on.

Footnote *2*: The changes to SCIFIO, ImgOpener and to TifBenchmark are not
ready yet for prime time, unfortunately. For example, I doubt that it will
be a good idea to force people to use PlanarContainers all the time. We
need to accelerate ImgOpener with ArrayContainers because they will be the
common case (much better performance than PlanarContainers, but too
limited for truly large images). We need to teach SCIFIO to determine in a
more intelligent way when file grouping does not make sense at all, and to
switch it off in that case. The CaseInsensitiveLocation needs a *lot* of
improvement, I haven't even touched it yet. We also need to teach SCIFIO
to stop being stubborn and re-use the RandomAccessInputStream when it
comes to the readers to determine whether they want to handle the image.
Maybe that is not enough, even, and we have to re-design the complete
strategy how readers are picked. One thing is utterly clear, though: in
its current form, the ImgOpener is not fit for daily use.

[Lee Kamentsky]

On 3/3/2013 2:10 AM, Johannes Schindelin wrote:
> Hi,
>
> On Sun, 3 Mar 2013, Johannes Schindelin wrote:

Thanks for more excellent research, Dscho.

> I could even imagine that with .tiff being such a common case with such a
> lot of subtypes that it will be *necessary* to add an own class of plugins
> for TIFF subtypes, so that the base TIFF reader can be chosen quickly as
> the one to use, without having to run through the complete list of readers
> when determining what is the correct reader. The subtype plugins would
> then have to implement a very fast test *after* we already know that it is
> a TIFF. Some of them could then even state that scanning through all the
> IFDs is not necessary (e.g. when it is detected that ImageJ 1.x wrote the
> file.).
>
> And finally, the profiling pointed out that I was right when I had a bad
> feeling about the #isThisType() method and wanted something less
> free-form, such as a combination of file extensions and patterns to match
> against the first kilobyte (much like the HandleExtraFileTypes class does
> it). Worse: the signature of isThisType() does not even allow sharing of a
> RandomAccessInputStream!

I have to admit - I have at times mucked with the ordering so that the
base TIFF reader comes up first and fast when that's all I need. You can
either set up the order you want using a ClassList in the ImageReader
constructor or you can loop through your own list. I do the latter and I
use ImageReader.isThisType(RandomAccessStream). isThisType(byte []
block) might be good too - I've never tried it.

Here's some of the Javascript I use in CellProfiler (ok how many
languages can you use in one program?). The idea is to accept a reader
if it is absolutely sure it can handle the format (and the base TIFF
reader wins for all TIF on the first pass, I think), only test readers
which think they can handle the format on the second pass and then leave
it wide open on the third pass. At this point in the program, I'm making
minimal use of the metadata, so the base TIFF reader is good enough. The
stream in the code below is loci.common.RandomAccessInputStream.
var classes = class_list.getClasses();
var rdr = null;
var lc_filename = java.lang.String(filename.toLowerCase());
for (pass=0; pass < 3; pass++) {
for (class_idx in classes) {
var maybe_rdr = classes[class_idx].newInstance();
if (pass == 0) {
if (maybe_rdr.isThisType(filename, false)) {
rdr = maybe_rdr;
break;
}
continue;
} else if (pass == 1) {
var suffixes = maybe_rdr.getSuffixes();
var suffix_found = false;
for (suffix_idx in suffixes) {
var suffix =
java.lang.String(suffixes[suffix_idx]);
suffix = suffix.toLowerCase();
if (lc_filename.endsWith(suffix)) {
suffix_found = true;
break;
}
}
if (! suffix_found) continue;
}
if (maybe_rdr.isThisType(stream)) {
rdr = maybe_rdr;
break;
}
}
if (rdr) break;
}
rdr;
Maybe something like this could migrate its way into Bio-formats to be
used when the situation called for it?

--Lee

[Johannes Schindelin]

Hi Lee,

As you point out below, it would be nicer for SCIFIO to by cleverer by
default instead of forcing the developer to know a ton about SCIFIO
internals and about the particular images at hand.

> Here's some of the Javascript I use in CellProfiler (ok how many
> languages can you use in one program?

Yeah, since it depends already on NumPy, you have to add C to the fray,
too... and every additional language halves the number of your potential
contributors.

>). The idea is to accept a reader if it is absolutely sure it can handle
>the format (and the base TIFF reader wins for all TIF on the first pass,
>I think), only test readers which think they can handle the format on the
>second pass and then leave it wide open on the third pass. At this point
>in the program, I'm making minimal use of the metadata, so the base TIFF
>reader is good enough. The stream in the code below is
>loci.common.RandomAccessInputStream.

You mean ome.scifio.io.RandomAccessInputStream.

I see a big problem in having to jump through hoops there. What we need to
do is to teach SCIFIO to be way faster when determining which is the
appropriate Reader. I will have to talk to Curtis about that. And I will
have to let this wait for at least a week -- I have an important other
project to take care of.

Ciao,
Dscho

[Stephan Saalfeld]

Hi scifio-ers,

following on this topic, I just noticed that opening zip-compressed
files is mindblowingly slow with ImgOpener. I ran a test on this file
compressed and uncompressed:

https://github.com/imagej/imglib/blob/copy-realtransformaccess/examples/l1-cns.tif.zip

There is a low number of other files in the same path. Results reading
from an SSD:

Opening TIF with ImageJ ...
took 263ms
Opening ZIP with ImageJ ...
took 648ms
Opening TIF with ImgOpener ...
took 823ms
Opening ZIP with ImgOpener ...
took 46340ms

This is roughly 50x slower than without zip compression. Do you have
any ideas how to compensate for that?

Thanks in advance,
Stephan

[Curtis Rueden]

Hi Stephan,

> opening zip-compressed files is mindblowingly slow with ImgOpener.

Thanks for the report. This bug has been fixed on the master branch of scifio.git:
    https://github.com/scifio/scifio/commit/fed74262eb05c5dbacfb4081c32d502ced23db65

We are in the process of transitioning ImgOpener over to SCIFIO, so once that is done, ZIP performance will be vastly better.

Regards,
Curtis

P.S. to Melissa: The fix for ZipHandle in bioformats.git on develop is analogous.

[Stephan Saalfeld]

Thanks a lot for fixing this Curtis!

Best regards,
Stephan