Image processing library
Tim Holy
Is the examples/image.jl file intended to become the basis of a standard image
processing library? If so, as a long-time Matlab user who is annoyed at
limitations in their Image Processing Toolbox, I'd like to raise a few issues
here "at the beginning":
1. For people who do biomedical imaging, it's very limiting to have the
default assumption be that images are 2-dimensional. Many of us collect 3d
images over time, often in multiple colors for each volume ("5d images" = 3d
space, time, channel/color).
2. I worry that Matlab's way of handling RGB/multichannel images is often
wrong: in terms of memory layout, frequently I want the channel/color
coordinate first, then the spatial coordinates. For example, when you have to
do things like take dot products at each spatial location, one gets more
cache-friendly behavior by having the channel coordinate be in sequential
memory addresses. Also, consider taking a z-slice (see * below) from a 3d RGB
image: with color last, you get an object of size (x,y,1,3). This is cleaned
up considerably if you canonically use a color-first representation (c,x,y,z,t)
(c = channel, t = time), because then a z-slice is just (3,x,y), in agreement
with your canonical 2d RGB representation. It's also worth pointing out that
for some of our current/upcoming analyses, I rearrange my memory layout as
(c,t,x,y,z), because the behavior over time at the current pixel is the core
ingredient in various calculations, which I want to be cache-friendly.
Ideally, I imagine that some way of "tagging" the meaning of individual
coordinates could be useful, so that functions operating on images could know
what they're handling.
3. If one is considering developing a generic container "class" (sorry, I'm
still learning Julia and its terminology) for images, there are a number of
pieces of extra information that can also be extremely useful to carry around.
For example, the physical-units spacing among different coordinates can vary:
you might have the separation between pixels in x- and y- be 0.12 microns, but
have the z- spacing be 2 microns. Indeed, it's not always guaranteed that the
coordinates are orthogonal. (Image disk-formats like NRRD
http://teem.sourceforge.net/nrrd/ provide fields that allow one to gracefully
handle such situations.) The algorithms themselves often benefit from being
aware of image geometry.
4. For people with large data sets (in my lab, a typical experiment is a few
hundred gigabytes, and soon we'll be at many terabytes), a good framework for
"virtual" images, in which images are looked up from disk on-demand, is
essential. However, this is often needed only at "higher" levels, e.g.,
functions that process an entire experimental run. The core algorithms
(filtering, registration, etc) often can work on memory-resident images.
5. Following up on #4, in real-world imaging devices you often have things
like "known bad pixels." In a memory-resident image, you can often represent
these as NaNs, and craft your processing routines to treat these
appropriately. (For example, consider the bad things that happen when an IIR
filter operates on an image containing one NaN value.) However, for large
virtual images you might want to simply provide information like "each z-slice
has bad pixels at the following locations", as the alternative is to load the
entire experiment into memory at once and mark all bad pixels with NaNs at the
outset.
I recognize that one risk is making life harder for people who don't have to
deal with such complexities, and obviously that would not be a good outcome.
However, I fear that if these things aren't at least considered early, it
could cause a big mess later. If necessary, I could easily see a "simple
images" library and a "complex images" library, but obviously it would be good
to make these as cross-compatible as possible. I'm happy to contribute to
implementation. But first I'd like to hear your thoughts on the issues.
* z-slicing is the most typical slicing case, because of the way 3d images are
acquired and various physical limitations of optics/fMRI/whatever---your
resolution is frequently anisotropic, and the low resolution spatial axis is
typically third in terms of acquisition/memory layout.
Best,
--Tim
Stefan Kroboth
I'm the one who implemented a lot of these routines in image.jl and I
was hoping for someone to come around and help me define some basic
principles the library should be built on. For now, I assume images are
one or three channel with the values ranging from 0 to 1. This makes
dealing with images a lot easier, but I'm certain that this will not
meet all needs. I think Integer values up to 16bits (for medical
imaging) should be supported as well.
Currently, I'm orienting myself at Matlab's Image Processing Toolbox,
although I think it would be much better to make it similar to OpenCV.
The capabilities of Matlab's Image Processing Toolbox are very limited
compared to OpenCV.
I'm doing Biomedical Imaging myself, but most of my work is in the area
of MRI reconstruction, so I'm not familiar with your problems. And since
my CV knowledge is in its early stages, I might not be able to recognize
all usecases and problems that might arise.
I can't really comment on your suggestions but I assume you know what
you are talking about as you seem to have done a lot of work in this
area. What I don't understand is why having the channel as first index
should be more memory efficient. I assumed that img[:,:,1] would be one
continuous part in the memory, which should be pretty fast to access.
But I wouldn't be surprised if I'm wrong.
I also share your concern that this shouldn't be trimmed too
specifically to biomedical imaging as this could lead to newbies being
intimitated by the complexity. But I'm sure that a proper design can
take care of this.
Further it would be nice to have DICOM support, though I also don't have
any experience with DICOM data yet.
I've been thinking about creating a Wiki page dedicated to Computer
Vision, but until now I haven't gotten around to it. Maybe this would be
a great way to start organizing the development of the library.
I'm also unsure if my work will be part of an official CV library, but
I sure hope so and I'm willing to put a lot more work into it.
pavanky has started a project which integrates CUBLAS into Julia. I
think it would be great to use the power of GPUs too in the CV library
if the CUBLAS wrapper gets merged into Julia. I'll see if I can
contribute to pavenkys code.
Especially for the MRI reconstruction stuff I'm doing I need A LOT more
power than CPUs can provide. Developing fast MR imaging methods where
the reconstruction takes hours or even days doesn't make much sense.
Also, I'd like to get the other people working at the Institute where
I'm doing my Masters thesis to use Julia for their work. Good GPU
support could be THE feature to get them convinced ;)
I really hope that this will be as awesome as the rest of Julia is.
Therefore I would be pleased if you (and others) would join in to make
this a great library.
-Stefan
--
So long, and thanks for all the fish!
Stefan Kroboth
Student of Biomedical Imaging and Computer Vision
Graz University of Technology
Tim Holy
On Monday, March 12, 2012 05:54:22 am Stefan Kroboth wrote:
> I'm the one who implemented a lot of these routines in image.jl
Many thanks! It's an impressive beginning.
> Currently, I'm orienting myself at Matlab's Image Processing Toolbox,
> although I think it would be much better to make it similar to OpenCV.
> The capabilities of Matlab's Image Processing Toolbox are very limited
> compared to OpenCV.
I don't know OpenCV, sounds worth a look. Agreed that Matlab's are limited (I
find myself writing a lot of MEX functions).
> What I don't understand is why having the channel as first index
> should be more memory efficient. I assumed that img[:,:,1] would be one
> continuous part in the memory, which should be pretty fast to access.
> But I wouldn't be surprised if I'm wrong.
If you want to pull out all spatial pixels in a given channel, then indeed
you're absolutely right. And I should also say that, of the points I raised,
this is the one that is least likely to be important, and also the one for
which I have the greatest chance of being wrong. Still, let's see if I can
defend my intuition here.
For my own needs (and I can only speak for myself here), pulling out an image
corresponding to one of the channels is a relatively rare situation. The
situations in which I worry about performance most are operations similar to
"compute the correlation with all neighboring pixels, where neighbors are
defined as a 3x3 block," i.e., for a given RGB value p1 = [r1, g1, b1], compute
a 3x3 scalar-valued matrix pixelcorr, where pixelcorr[i,j] is the dot product
of p1 with im2(:,i+offseti,j+offsetj). I'd guess that one might be more likely
to be able to use registers and/or SSE instructions with
loop over pixels
compute dot product
end
than with
loop over channels
loop over pixels
accumulate contribution to dot product
end
end
But I could be wrong; I know next to nothing about compilers/SSE/how the
underlying hardware actually works. But certainly, version 1 is cache-friendly
only if the 3 color channels are all in the same block of memory. On the other
hand, now that I think about it (and google around), maybe version 2 would be
more likely to use SSE (the ADDSS instruction)? If so, my point #2 is perhaps
invalid. Comments desired.
> Further it would be nice to have DICOM support, though I also don't have
> any experience with DICOM data yet.
We don't use it in my lab, but its use seems widespread.
> I've been thinking about creating a Wiki page dedicated to Computer
> Vision, but until now I haven't gotten around to it. Maybe this would be
> a great way to start organizing the development of the library.
Sounds like a great idea! I'll be happy to help.
> I'm also unsure if my work will be part of an official CV library, but
> I sure hope so and I'm willing to put a lot more work into it.
>
> pavanky has started a project which integrates CUBLAS into Julia. I
> think it would be great to use the power of GPUs too in the CV library
> if the CUBLAS wrapper gets merged into Julia. I'll see if I can
> contribute to pavenkys code.
We've just begun playing with this ourselves, and it's looking very promising.
> I really hope that this will be as awesome as the rest of Julia is.
> Therefore I would be pleased if you (and others) would join in to make
> this a great library.
Sounds like fun! I'll keep working on teaching myself more Julia.
Best,
--Tim
Albert Cardona
I would like to draw your attention to the ImgLib2, an image
processing library born from years of dealing with biomedical images
with ImageJ (an NIH, public domain image processing application). NIH
now is sponsoring the recreation of ImageJ into what is called the
ImageJ 2.0 project (http://developer.imagej.net/).
A major aim is to write image processing algorithms that are type-,
dimension- and storage-independent. For example, the same code should
be able to process 8-bit, 16-bit, 32-bit, etc., with any number of
dimensions (2d,3d,4d,....) and with images stored as cubes in disk, or
planes in RAM, or accessed via URLs, etc. All of this greatly
facilitates handling biological research imagery, which comes in all
flavors but generally requires the same kind of processing algorithms.
Fantastically the developers managed to do so without sacrificing more
than a tiny percent of performance over hand-written, type-dependent
implementations, all thanks to the JVM JIT. See these metrics:
http://developer.imagej.net/imglib-benchmarks
(notice the pulldown menu with various testing options in realistic
conditions, such as "Expensive operations on various image
resolutions". Multiple data container variants are tested).
The design documentation with class and type diagrams are at the
imglib2/core/doc folder:
The git repository:
http://fiji.sc/cgi-bin/gitweb.cgi?p=imglib.git
A trivial example of code that uses the library, demonstrating how the
handling of image types has been removed away from the developer of
the algorithm:
http://fiji.sc/wiki/index.php/ImgLib1_(deprecated)#Example
Notice that ImgLib1 is now deprecated; ImgLib2 is the current version,
which is being used as the foundation for the ImageJ 2.0 project. Both
libraries are quite similar, with ImgLib2 being less verbose,
performing better, and being a superset of ImgLib1.
Before creating an image processing library from scratch, I suggest
studying the many choices and pains that mature projects like ITK (the
insight toolkit: http://www.itk.org/) and VIGRA (a C++ templating
library for computer vision, with a focus on extraordinary
performance) have gone through. Please do not reinvent a design
without being aware of why ImgLib2, ITK and VIGRA were designed the
way they have been. Same with the Matlab image library, whose design
though is dated.
Best,
Albert
Tim Holy
Thanks for drawing my attention to ImgLib2. I've been so Matlab-centric that I
wasn't aware of this at all, it looks like it could be a useful example.
On Monday, March 12, 2012 08:13:30 am Albert Cardona wrote:
> A major aim is to write image processing algorithms that are type-,
> dimension- and storage-independent. For example, the same code should
> be able to process 8-bit, 16-bit, 32-bit, etc., with any number of
> dimensions (2d,3d,4d,....) and with images stored as cubes in disk, or
> planes in RAM, or accessed via URLs, etc.
Interesting, particularly about the cubes. ImageVis3d bases its entire
visualization pipeline on them (they call them "bricks").
http://cibcwiki.sci.utah.edu/cibc/wiki/index.php/CIBC:ImageVis3D
For those who aren't in the "connectome" community, these bricks are really
important when you need to create locally-operating algorithms that work on
image volumes far larger than you can plausibly store in memory at one time.
> Before creating an image processing library from scratch, I suggest
> studying the many choices and pains that mature projects like ITK (the
> insight toolkit: http://www.itk.org/)
I briefly tried ITK a few years ago, and at least at the time I found it
somewhat heavy---even though I like C++ in general, the layers were getting in
the way of what I wanted to do, and performance was slow even compared to
Matlab. I don't doubt that it's much better now, though.
> and VIGRA (a C++ templating
> library for computer vision, with a focus on extraordinary
> performance) have gone through.
Ooh, this is a nice project. I've reinvented versions of some of those
iterator functionalities myself; wish I'd discovered VIGRA in my searches.
Perhaps at the time it was not multidimensionally-capable, so maybe I crossed
it off my list (too hastily).
What are your thoughts on, say, the virtues of building this up in Julia vs.
just sticking with existing toolkits in other languages? Certainly it will be
some work to implement, and I'm sensitive to that for my own lab. But usually
it's design that takes the most time, and as you say we can lean on the good
work of others.
Best,
--Tim
Albert Cardona
I wish I had the time to teach myself Julia down to all its details,
and to do so by re-implementing ImgLib2 (a library that I know well,
and to which I contributed some code) in Julia. Only then one could
compare performance. But I had a child 2 months ago and currently my
out-of-work time is tied up. Perhaps this Fall 2012.
As you reiterated it's design that takes the most time. ImgLib2's
design has been iterated to death, to squeeze performance out of a
JIT-based language (the JVM) without sacrificing flexibility.
VIGRA is wonderful, but as I understood from the mailing list, the
fact that it is a template library doesn't make it suitable to
wrapping trivially for use in Julia. About ITK, I agree with you on
the heaviness and suboptimal performance.
At this time, the best option seems like implementing a subset of
ImgLib2 in Julia and run performance benchmarks, and reevaluate the
choice.
Albert Cardona
http://fiji.sc/cgi-bin/gitweb.cgi?p=imglib.git;a=blob;f=imglib2/core/doc/imglib2.dia;hb=HEAD
(the doc/ folder contains other, older versions; we'll clean it up soon.)
Stefan Kroboth
thank you for your input. I haven't taken a close look at imglib2, but
skimmed through it a bit. It is probably the best to start by looking at
how others solved the problems.
If it leads to just writing wrappers for imglib2, I'll probably loose
interest, though ;)
Have you tried OpenCV? If yes, how do you think OpenCV relates to
imglib2? I mean, is imglib2 far superior to OpenCV?
Best,
Stefan
Stefan Kroboth
> For my own needs (and I can only speak for myself here), pulling out an image
> corresponding to one of the channels is a relatively rare situation. The
> situations in which I worry about performance most are operations similar to
> "compute the correlation with all neighboring pixels, where neighbors are
> defined as a 3x3 block," i.e., for a given RGB value p1 = [r1, g1, b1], compute
> a 3x3 scalar-valued matrix pixelcorr, where pixelcorr[i,j] is the dot product
> of p1 with im2(:,i+offseti,j+offsetj). I'd guess that one might be more likely
> to be able to use registers and/or SSE instructions with
> loop over pixels
> compute dot product
> end
> than with
> loop over channels
> loop over pixels
> accumulate contribution to dot product
> end
> end
> But I could be wrong; I know next to nothing about compilers/SSE/how the
> underlying hardware actually works. But certainly, version 1 is cache-friendly
> only if the 3 color channels are all in the same block of memory. On the other
> hand, now that I think about it (and google around), maybe version 2 would be
> more likely to use SSE (the ADDSS instruction)? If so, my point #2 is perhaps
> invalid. Comments desired.
I think I understand what you mean. I don't know anything about
compilers myself, thats why I can't comment on which version is more
efficient for your case. But I think this is more a question of which
usecase is more common: having all RGB values lined up in memory or
accessing each channel separately. For instance, my ROF-denoising as
well as imfilter work on each channel separately. If this makes a major
difference in terms of speed and memory efficiency, we could define two
image types with different indexing.
> > I've been thinking about creating a Wiki page dedicated to Computer
> > Vision, but until now I haven't gotten around to it. Maybe this would be
> > a great way to start organizing the development of the library.
>
> Sounds like a great idea! I'll be happy to help.
If I find some time tonight, I'll create a page on GitHub.
> Sounds like fun! I'll keep working on teaching myself more Julia.
Great! :)
Best,
Stefan
Albert Cardona
ha escrit:
> Hi Albert,
>
> thank you for your input. I haven't taken a close look at imglib2, but
> skimmed through it a bit. It is probably the best to start by looking at
> how others solved the problems.
> If it leads to just writing wrappers for imglib2, I'll probably loose
> interest, though ;)
>
> Have you tried OpenCV? If yes, how do you think OpenCV relates to
> imglib2? I mean, is imglib2 far superior to OpenCV?
OpenCV has a more dated design and is nor particularly robust (it's
easy to shoot oneself in the foot), but it's library is impressive.
There are many algorithms (e.g. random forests) out there implemented
for OpenCV that make it very valuable.
ImgLib2 is new, it's design sprung from the needs of the bioimaging
research (massive files, multidimensional, numerous image types that
are sometimes odd like 1-bit images, 6-bit, 12-bit, etc.), and while
it already has a sizable number of algorithms implemented in its
library (integral images, FFTs, efficient gaussians), many are still
missing.
It is the design of ImgLib2 that I find most interesting for the stage
that Julia's image processing library is at. Keep in mind that ImgLib2
is a java library, and wrappers then won't really be an option. You
could wrap opencv from julia if that fits, but I would see this as a
separate package (as in, OpenCV will exist no matter what and a
wrapper would be nice).
