Image data types
Curtis Rueden
One of ImageJDev's goals is to expand the data model (see http://imagejdev.org/proposal, Aim IC). Regarding core image types, we have pledged support for int8, uint32, and float64 images.
It would be easy enough to leave it at that, but we have an opportunity to remove the burden on users of the API to write case logic for each data type they wish to support. Stephan Preibisch and Stephan Saalfeld have proposed a "generic image processing library" (I believe Fiji's imagecontainer1 branch, mentioned below, was an older version of it) using Java generics that allows client code to write type-agnostic image processing code, with only a modest decrease in performance. (Stephan and Stephan: Is this code accessible from Fiji's Git repository? I couldn't find it.) In cases where performance is paramount, case logic can still be used. This scheme would also make it easier to add new core data types in the future (e.g., complex numbers), such that most existing image processing algorithms would immediately support them.
Gábor Bakos and Johan Henriksson also mentioned Scala and its "@specialized" tag, though we have not had time to research that option yet. However, I am concerned about introducing any dependence on Scala (even though it is Java-compatible) unless the benefits are very substantial, and cross-platform support remains intact. Johan also mentioned Javassist, though upon cursory inspection I am not sure how we could use that tool to address the image data types issue.
-Curtis
[was: API vs. Scripting?]
On Wed, Oct 7, 2009 at 10:58 AM, Wilhelm Burger <wil...@ieee.org> wrote:
3) I am reading about support for more (unsigned) primitive data
types, use of generics etc. I consider myself a Java person, but this
is where Java is particularly bad at, not to talk about some notorious
performance issues. How important is Java in this context?
I have seen some concerns in this list about the performance of usage of generics with numerical computations. I think this (@specialized annotation in Scala 2.8.x (to be released this year)), can be used to avoid performance problems, and keep writing generic algorithms. (blog, Numeric trait, discussion). (I am not sure, maybe Johan Hendriksson has been already solved this problem using pure Java. In that case I would be interested how, and is it available somewhere?)
(Regarding abstract classes <-> interfaces - In Scala you can use traits for interfaces and those can hold implementations too.)
I have also looked at the Fiji's imagecontainer1 branch. I think it is more modular, flexible than ImageJ, maybe the way to the future. (Although using public int in the Type class is not so good solution in my humble opinion.)
[was: Outsider's opinion]
On Fri, Oct 23, 2009 at 4:38 PM, Johan Henriksson <he.j...@gmail.com> wrote:
indeed the solution in java is to code @specialized, most likely with
javassist.
http://www.csg.is.titech.ac.jp/~chiba/javassist/
it provides run-time specialization
+ no need to recompile to support new datatypes
+ smaller binaries
- slower startup
Johan Henriksson
Hi everyone,
One of ImageJDev's goals is to expand the data model (see http://imagejdev.org/proposal, Aim IC). Regarding core image types, we have pledged support for int8, uint32, and float64 images.
I don't think this is enough. I think for most image operations we will be using float64 and (u/s)int32 but in reality, the others are quite useful too. in particular, float32 consumes only half the memory with a precision that is good enough for most of us. if you guys ever get to use openCL then you will run into other interesting formats e.g. float16.
unsigned integers are hellish to support in java because they are not built-in. for endrov I opted to remove them in fact, with the exception of uint8 (performance, shouldn't be used for any calculations really). an interesting issue for plugin writers is how to handle temporary overflows e.g.
(a+b)/2 can temporarily overflow but the result will never. hence the plugin must be clever enough to not just use the generics-imposed format for the intermediate but, say, a full int. even more fun is when the plugin takes two images and they are of different types; this breaks simple minded generics approaches.
Gábor Bakos and Johan Henriksson also mentioned Scala and its "@specialized" tag, though we have not had time to research that option yet. However, I am concerned about introducing any dependence on Scala (even though it is Java-compatible) unless the benefits are very substantial, and cross-platform support remains intact. Johan also mentioned Javassist, though upon cursory inspection I am not sure how we could use that tool to address the image data types issue.
I would prefer to avoid scala if possible, there are plenty reasons for doing so (tool support, avoid non-mainstream, avoid mixing languages etc). javassist allows you to take code (=the modified plugin template) and have it compiled on-the-fly. I find it easiest (but yet to confirm) to start from actual source than class binaries because the JVM treats scalars and classes rather different at byte code level. the original code should be possible to compile before javassist takes it apart, otherwise Eclipse et al won't work very well (e.g. refactoring). this all goes down to the nit-bits of the implementation so I will have to work on this. I will try (as planned) to do this in endrov and let you know how it works.
/Johan
-----------------------------------------------------------
Johan Henriksson
PhD student, Karolinska Institutet
http://mahogny.areta.org http://www.endrov.net
Gábor Bakos
On Thu, Dec 24, 2009 at 1:23 AM, Curtis Rueden <ctrued...@gmail.com> wrote:Gábor Bakos and Johan Henriksson also mentioned Scala and its "@specialized" tag, though we have not had time to research that option yet. However, I am concerned about introducing any dependence on Scala (even though it is Java-compatible) unless the benefits are very substantial, and cross-platform support remains intact. Johan also mentioned Javassist, though upon cursory inspection I am not sure how we could use that tool to address the image data types issue.
I would prefer to avoid scala if possible, there are plenty reasons for doing so (tool support, avoid non-mainstream, avoid mixing languages etc). javassist allows you to take code (=the modified plugin template) and have it compiled on-the-fly. I find it easiest (but yet to confirm) to start from actual source than class binaries because the JVM treats scalars and classes rather different at byte code level. the original code should be possible to compile before javassist takes it apart, otherwise Eclipse et al won't work very well (e.g. refactoring). this all goes down to the nit-bits of the implementation so I will have to work on this. I will try (as planned) to do this in endrov and let you know how it works.
--g
--
/Johan
-----------------------------------------------------------
Johan Henriksson
PhD student, Karolinska Institutet
http://mahogny.areta.org http://www.endrov.net
--
You received this message because you are subscribed to the Google Groups "ImageJX" group.
To post to this group, send email to ima...@googlegroups.com.
To unsubscribe from this group, send email to imagejx+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/imagejx?hl=en.
--
Skill is successfully walking a tightrope over Niagara Falls. Intelligence is not trying. ~~~...
Johan Henriksson
2009/12/24 Johan Henriksson <mah...@areta.org>On Thu, Dec 24, 2009 at 1:23 AM, Curtis Rueden <ctrued...@gmail.com> wrote:Gábor Bakos and Johan Henriksson also mentioned Scala and its "@specialized" tag, though we have not had time to research that option yet. However, I am concerned about introducing any dependence on Scala (even though it is Java-compatible) unless the benefits are very substantial, and cross-platform support remains intact. Johan also mentioned Javassist, though upon cursory inspection I am not sure how we could use that tool to address the image data types issue.
I would prefer to avoid scala if possible, there are plenty reasons for doing so (tool support, avoid non-mainstream, avoid mixing languages etc). javassist allows you to take code (=the modified plugin template) and have it compiled on-the-fly. I find it easiest (but yet to confirm) to start from actual source than class binaries because the JVM treats scalars and classes rather different at byte code level. the original code should be possible to compile before javassist takes it apart, otherwise Eclipse et al won't work very well (e.g. refactoring). this all goes down to the nit-bits of the implementation so I will have to work on this. I will try (as planned) to do this in endrov and let you know how it works.
Will this approach work for the android port too? (I have not seen mentioning the DalvikVM on the Javassist site, but the plans suggest that it should be supported.) For me this is not a priority, but for those who are targeting that platform might be. (Actually I do not know whether this will work in Scala either.)
there are the two levels of code manipulation; source code level and bytecode level. bytecode is the older one (in industry practice), typically you use something like
http://jakarta.apache.org/bcel/
android uses another JVM with another bytecode (register based, supposedly fasted, unofficially also a step away from Sun). while it can take OpenJDK bytecode and convert it (compatibility), I don't know if it can do so at runtime; I need to read up here. but if it cannot then a lot of bytecode generating programs will have to be ported. in case of source code generation instead, only the underlying library (eg javassist) need be ported but the program not. I don't like bytecode approaches since it locks us into one JVM and I don't think anyone likes the stack based machine. people have tossed money on it so it runs *decently* but hopefully there will be a next generation, like dalvikVM.
I let the scala users handle that part of the question.
I think porting to android is fairly low priority though. how many here are doing image processing on their phone? it'd be useful maybe as a library in part of a production system (eg online face recognition and tracking) but few of us do this sort of thing, and then we have to look into the entire license mess as well.
/Johan
--g--
/Johan
-----------------------------------------------------------
Johan Henriksson
PhD student, Karolinska Institutet
http://mahogny.areta.org http://www.endrov.net--
You received this message because you are subscribed to the Google Groups "ImageJX" group.
To post to this group, send email to ima...@googlegroups.com.
To unsubscribe from this group, send email to imagejx+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/imagejx?hl=en.
--
Skill is successfully walking a tightrope over Niagara Falls. Intelligence is not trying. ~~~...--
You received this message because you are subscribed to the Google Groups "ImageJX" group.
To post to this group, send email to ima...@googlegroups.com.
To unsubscribe from this group, send email to imagejx+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/imagejx?hl=en.
Dimiter Prodanov
If there is a company willing to step in and do it they are welcome.
But the main stream users use desktop, laptop and may be netbook
systems.
best regards,
Dimiter
Wilhelm Burger
On Dec 24, 1:23 am, Curtis Rueden <ctrueden.w...@gmail.com> wrote:
> Hi everyone,
>
> One of ImageJDev's goals is to expand the data model (seehttp://imagejdev.org/proposal, Aim IC). Regarding core image types, we have
> pledged support for int8, uint32, and float64 images.
I agree that it would be helpful to have some additional data types
but at the same time we should be modest and consider the
combinatorial number of required conversion methods. Since they are
hard to work with in Java, I would generally avoid UNsigned numeric
types, unless there is a *very* good reason to use them. Currently I
*only* see a good reason for unsigned 8-bit integers ("byte"), because
one wants to use the full 8-bit dynamic range and not loose one bit
for the sign.
In all other cases I would argue to work with SIGNED data to avoid
multiple conversions and make efficient use of Java's arithmetic
infrastructure. In particular, if you need 16 bit unsigned integers
you could use 32 bits signed data without loosing anything. Personally
I use the 32 bit float format universally for almost all numeric
processing, since it has enough precision, it can hold (intermediate)
negative results, calculations are efficient and it is easy to store
(eg. with TIFF).
So my personal "basic data formats wish list" would look like this:
* 8-bit unsigned integer ("byte")
* 16-bit signed integer ("short")
* 32-bit signed integer (int) - NEW
* 32-bit floating-point (float) - hopefully not to be just called
"32bit" any longer!
* 64-bit floating-point (double) - NEW
In addition, I would also wish for a true "binary" (bitmap) fomat, eg.
to represent binary masks, but this seems to be impossible to
implement in Java.
As far as RGB images are concerned, I would strongly argue NOT to use
the packed integer format any longer. While this is reasonable for
storage, it is computationally very inefficient. For practically any
calculation on color images, RGB integers must be decomposed and re-
assembled again. For example, filtering is usually done separately for
each color plane, and the current IJ implementation for color filters
first converts the image to three FloatProcessors, performes the
filter operation on each of them, and then converts back again to the
packed representation. This can be avoided by using a 3-plane data
model from the beginning that is open to use any primitive data type
for the individual color planes. In fact, I would treat RGB images
only as a special case of image stack.
--Wilhelm
Wilhelm Burger
On Dec 24, 11:51 am, Johan Henriksson <maho...@areta.org> wrote:
> unsigned integers are hellish to support in java because they are not
> built-in. for endrov I opted to remove them in fact, with the exception of
> uint8 (performance, shouldn't be used for any calculations really). an
> interesting issue for plugin writers is how to handle temporary overflows
> e.g.
> (a+b)/2 can temporarily overflow but the result will never. hence the plugin
> must be clever enough to not just use the generics-imposed format for the
> intermediate but, say, a full int. even more fun is when the plugin takes
> two images and they are of different types; this breaks simple minded
> generics approaches.
Arithmetic overflow is really an interesting issue that is too often
ignored although it causes many hard-to-find (and possibly yet
undetected) errors. As far as I know, there is currently no way in
Java (and C?) to check for arithmetic overflows - although all ALUs do
raise a corresponding flag in such a case. I remember one could easily
handle overflows with PL/I or ADA back in the old days, while Java
designers seem to have forgotten about this important feature. It
would be nice if Java could be asked to throw an exception in such a
case. I think this is one lesson (of many) learned by the C# people.
--Wilhelm
Johan Henriksson
In addition, I would also wish for a true "binary" (bitmap) fomat, eg.
to represent binary masks, but this seems to be impossible to
implement in Java.
not less possible than any other language but they are messy to implement. it all just boils down to plenty of bit operators. thanks, I had forgotten about these. I've considered them for a while for the space efficiency, and for extremely fast set operations. I was on a talk recently about imaris and yep, some users had memory issues when they had too many masks. but I'm not sure they are that useful, in many cases I actually prefer to have 0=no object, n=id of some object, so I actually use up all those numbers. masks mainly pop up as intermediates of logic operations and binary morphology. can we think of user cases when it makes sense to have a ridiculous number of binary masks to make it worthwhile?
>Arithmetic overflow is really an interesting issue that is too often
>ignored although it causes many hard-to-find (and possibly yet
>undetected) errors. As far as I know, there is currently no way in
>Java (and C?) to check for arithmetic overflows - although all ALUs do
As far as RGB images are concerned, I would strongly argue NOT to use
I've gotten rid of RGB-images entirely from endrov. I don't think they fit in biological imaging (but I can have infinite channels, wells, recordings etc instead)
IJX might need it for backwards compatibility however.
Wilhelm Burger
> in biological imaging (but I can have infinite channels, wells, recordings
> etc instead)
That might be true (don't know). But color images are prevalent in
many other areas and I was actually hoping that IJX could make ImageJ
attractive much beyond biological applications. So I think RGB images
should be considered carefully not only for compatibility reasons.
--Wilhelm
Gabriel Landini
> > I've gotten rid of RGB-images entirely from endrov. I don't think they
> > fit in biological imaging (but I can have infinite channels, wells,
> > recordings etc instead)
I am puzzled by this, can you expand what do you mean "[not] fit in biological
imaging"?
> That might be true (don't know). But color images are prevalent in
> many other areas and I was actually hoping that IJX could make ImageJ
> attractive much beyond biological applications. So I think RGB images
> should be considered carefully not only for compatibility reasons.
I agree. Also, now that we are at it, I would like to suggest improved support
for cinvertion to HSB, Lab and some other colour space formats.
Currently I do not think that one can convert back and forward between RGB and
HSB without some noticeable rounding errors. This becomes apparent for example
when rotating the hue of an RGB image repeatedly.
Regards
Gabriel
Wilhelm Burger
On Dec 25, 12:35 pm, Gabriel Landini <G.Land...@bham.ac.uk> wrote:
> I agree. Also, now that we are at it, I would like to suggest improved support
> for cinvertion to HSB, Lab and some other colour space formats.
> Currently I do not think that one can convert back and forward between RGB and
> HSB without some noticeable rounding errors. This becomes apparent for example
> when rotating the hue of an RGB image repeatedly.
Yes, this is a mess which partially must be blamed on the (bad)
implementation of color conversion in AWT (eg., applying forward/
inverse transforms will generally not produce the original color
values). There is obviously a lack of precision if 8-bit unsigned
integers are used, which also cannot represent out-of-gamut values
that may be produced as intermediate results. I therefore wrote my own
implementation using float components (stacks) to represent colors and
allow precise conversions. For those interested I'll be happy to share
this.
(Not) considering the non-linearities associated with most color
images is another sad issue.
--Wilhelm
Johan Henriksson
On Friday 25 December 2009, Wilhelm Burger wrote:
> > I've gotten rid of RGB-images entirely from endrov. I don't think they
> > fit in biological imaging (but I can have infinite channels, wells,
> > recordings etc instead)
I am puzzled by this, can you expand what do you mean "[not] fit in biological
imaging"?
the imaging I've mostly been doing involve non-RGB methods;
* multiple fluorophores, need not be 3 colors (and they are not rgb)
* the new spectral unmixers, up to 32 channels
* mainly DIC, which shouldn't even be considered a color
since an RGB in the end can just be decomposed into 3 channels (just values), it seems moot to me to keep RGB as an algorithm concept. I decided to leave it as a concept in the users head; it's still possible to display multiple channels superimposed (the RGB interpretation) or feeding all channels simultaneously to a filter (eg spectral unmixing) but this is dealt with by the user.
this solved the design issue I had with how to both support both superposing channels (as in DIC+gfp) and superposing RGB, and how to work on individual channels; one concept was simply superfluous.
/Johan
Johannes Schindelin
On Wed, 23 Dec 2009, Curtis Rueden wrote:
> It would be easy enough to leave it at that, but we have an opportunity
> to remove the burden on users of the API to write case logic for each
> data type they wish to support. Stephan Preibisch and Stephan Saalfeld
> have proposed a "generic image processing library" (I believe Fiji's
> imagecontainer1 branch, mentioned below, was an older version of it)
> using Java generics that allows client code to write type-agnostic image
> processing code, with only a modest decrease in performance. (Stephan
> and Stephan: Is this code accessible from Fiji's Git repository? I
> couldn't find it.)
Please bear with us, we are unfortunately very busy with other things
(among other things, we also are offline a lot these days due to partying
Christmas).
So for the moment, I will have to restrict myself with a general
description of this work:
- while we use generics in that library, at no point do we use basic types
in the specialization (for those who know the term "boxing": this is
what makes that specialization undesirable).
- due to a very clever encapsulation scheme devised by Stephan Saalfeld
and Stephan Preibisch, the library allows data type, dimension and
storage layout independent implementations of algorithms, while
maintaining performance close to hand-crafted specific code.
- by the same token, abstraction between the different components implies
that it is very easy to add new data types, such as complex numbers
(float16 comes to mind).
I am sorry, but at this point, I cannot do more than describe the
capabilities in these general terms, and say that the library is in heavy
use (proving above claims in practice). Proper documentation, example
code and more is in preparation, but will take some time.
In particular, I want to stress the fact that this library does not
restrict the developer in such a way that basically everything would be
converted to double -- or int, as has been claimed falsely (which would
most often only waste time and memory).
It is also important to point out that the library does not limit any
developer to a scripting language such as Scala, or to a set of platforms
that exclude some popular ones such as Android.
Ciao,
Johannes