ImageJ class hierarchy suggestions
Curtis Rueden
Next week the ImageJDev team will be drafting an updated ImageJ API. I am writing because many of you have experience writing large applications that plug into or extensively use ImageJ (Bio7, TrakEM2, Fiji, Micro-Manager, Endrov, etc.), and it is likely that we have each run across different issues and limitations.
Thus, we would be very interested in any specific feedback you have regarding ImageJ's API design (e.g., class/package structure) that you would like to see improved or changed. We have now documented most of what we plan to do on our Trac (http://imagejdev.org/trac/imagej/roadmap) but so far it is very general and does not discuss many specifics.
Most valuable would be thoughts on specific class relationships and design decisions that caused you trouble when attempting to leverage ImageJ.
My example: in developing the Bio-Formats plugins, we had a hard time supporting the additional file formats within ImageJ in a completely integrated way (i.e., so that File>Open would support the existing formats seamlessly). We needed to add code to HandleExtraFileTypes—which arguably causes a licensing conflict as it creates an indirect dependency on Bio-Formats—but there are many edge cases where a file may not be handled well with File>Open. E.g., ImageJ cannot handle all the TIFF variants that Bio-Formats can. Thus, we want to create a PlugInIO interface that allows I/O plugins more declarative power about exactly what they do.
Please let us know if you have any suggestions of your own!
Thanks,
Curtis
Jean-Yves
On Feb 25, 2010, at 11:12 PM, Curtis Rueden wrote:
Most valuable would be thoughts on specific class relationships and
design decisions that caused you trouble when attempting to leverage
ImageJ.
Hi all.
I would have a suggestion on GUI, and more specifically on ROIs.
I hope to merge with Michael ideas he sent on the ImageJX list.
It is rather lengthy email, sorry.
-----------------------------
I recently tried to code weird shapes as ROIs in ImageJ. They were the
results of a segmentation with constrained shapes. Because I wanted to
have something nice for the user, The ROIs had to be mouse-interactive
(resizable, moveable etc..).
I had a difficult time.
Johannes proposed on the Fiji-devel list an abstract class whose goal
was to facilitate this interaction. (If you are curious, my
implementation of the Arrow plugin uses it.)
http://pacific.mpi-cbg.de/cgi-bin/gitweb.cgi?p=fiji.git;a=blob;f=src-plugins/Arrow_/fiji/util/AbstractTool.java;hb=HEAD
But we still gave to comply to ImageJ ij.gui.Roi master class, which
is a concrete class in charge of drawing rectangle ROIs. Inside this
class, there is everything: the logic to draw it, to interact with the
user, with the image container, and the image data. Any homemade ROI
must inherit from this class, there is no interface to implement.
I tried to get my brain around ImageJ mechanism for ROI, and drafted a
diagram for it. You can find it here: (please be patient with it, it
is just a draft I did for myself):
http://creately.com/app?diagID=g45j59hu1
What I would like to propose here is to go for an interface hierarchy
for ROIs, that is well decoupled, and that would allow the flexible
design of new ROIs. This is just an early suggestion, and you will
find no usable code in this post. On top of that, I could not have
unambiguous and clear schema of what to choose, so I am sure you will
find something to say.
-------------------------
We use ROIs for many purposes, for instance:
- user interaction:
- draw a rectangle to crop an image
- measure intensity with a complex area
- add non-destructive annotations
- etc...
- as input for plugins
- as output for plugins, for instance a result of segmentation
From this you can see that they need to
- know how to draw themselves as an overlay
- comply to some interface to be an input of some plugins
- know how to interact with mouse clicks and drag
-------------------------
Let us take an example of a cube ROI that we want to draw over a 3D
image stack. We suppose we all agree on how to describe images in
ImageJ; for instance to use imglib.
We want ImageJ 2.0 to be able to deal with different way to represent
the image data on screen. For instance, we can imagine that there will
be 2 "displayers":
- one that display 3D data the same way ImageJ does now (with a
slider at bottom that allow to scroll through slices),
- a second one that generate a 3D rendering (a la Benjamin Schmidt's
3D viewer)
All displayers will implement a Displayer.java interface that signals
them as taking in charge the rendering on the screen.
Now, a Displayer should be able to display many things, not only
images. We can even consider that there is no reason to privilege the
image. So the Displayer must be able to deal with an array of stuff to
draw.
At the other side of the hierarchy lies our 3D cube. The cube itself
can be implemented within a Cube.java class specialized in geometric
shapes. It would have as fields a side length and the x,y,z
coordinates of its botton-north-east corner, and that is enough to
characterize it entirely. As a side subject, we can imagine it belongs
to a shape hierarchy that contains spheres, points, etc, ...
But there is not point for the Cube class to have methods for drawing
itself. It does not care to be drawn in 2D or 3D, it just wants to
deal with geometric data, that can be used by other classes. We must
avoid to couple it to a Displayer. So we have to insert another class
here, let us call it Drawer.java that will wrap the Cube object, and
will be fed to the displayer.
Because the way the cube will be drawn is not the same in 2D than in
3D, the Drawers will be implemented for each mother Displayer. When
ask to redraw, the Displayer will loop over all Drawers it has been
given, and call their draw() method (more detail lower).
The same goes for the image data: is should be wrapped in a Drawer
that knows how to render it in the 2D case, and another one for the 3D
case.
-------------------------
If we try to make this example concrete, it will look like this.
Displayer_2D.java: This one is only able to display 3D data, nothing
more
public class Displayer_2D implements Displayer {
protected ArrayList<Drawer2D> drawers;
protected int current_slice;
...
public void repaint() {
Graphics g = getGraphics();
for (Drawer_2D dw : drawers) {
dw.draw(g, current_slice);
}
}
...
}
Drawer2D.java
public interface Drawer2D {
public void draw(Graphics g, int current_slice);
}
DrawerImage2D.java
public class DrawerImage2D implements Drawer2D {
...
/** Constructor that takes an imaglib image. */
public DrawerImage2D(Image<T> _image) {
this.image = _image;
}
public void draw(Graphics g, current_slice) {
// Now this is PSEUDOCODE
slice = image.getSlice(current_slice);
drawImage(Graphics g, slice);
}
...
}
Cube.java:
public class Cube extends GeometricShape { // GeometricShape could be
a mother class for others shapes
public double a; // cube side
public double xc, yc, zc; // cube corner location
...
}
DrawerCube2D.java. However, we can imagine a clever class that would
be general for all GeometricShapes, and use a common method to draw.
Here we keep it cumbersome but simple
public class DrawerCube2D implements Drawer2D {
...
public DrawerCube2D(Cube _cube) {
this.cube = _cube;
}
public void draw(Graphics g, int current_slice) {
double current_z = getCurrentZ(current_slice); // do something to
get physical coords
if (current_z > cube.zc + cube.a || current_z < cube.zc) {
return;
}
drawRectangle(g, cube.xc, cube. yc, cube. a, cube. a);
}
}
-----------------------------------------------------------------
Now we have a nice decoupled hierarchy, with classes that are in
charge of drawing, others in charge of holding physical information,
others in charge of rendering.
But this asses only the drawing part of the problem. It is still to
design a few things:
- What do we pass to our plugins? It makes sense to pass only the
class that has the physical information. In the above example, a
plugin only cares for the Image<T> and the Cube, not their Drawer.
However, what common interface should they implement to be able to be
passed to a plugin?
- On top of that, all shapes are not the same: for instance, there is
lines and cubes. A plugin that crop an image can accept a cube, but
not a line. So there might be another hierarchy to specify here
(Maskable? Samplable? ... )
- How do we deal with mouse clicks? I feel that should be the work of
Drawers. A Displayer that receives user interaction stuff, e.g. mouse
clicks, should be able to decide to what Drawer forward it (if we
click near the cube ROI, we want to modify it), and Drawers should
have a logic to modify the object they wrap from these clicks. Also,
we have to modify behavior according to what is selected as tool in
the toolbar.
But what is cool with this hierarchy is that now having 3D of even 4D
or whatever ROIs is not extremely simple. We just have to make sure
the draw method receive the correct localization info (what slice do
you want me to render) and that's all.
Also, we have decoupled the ROI from the displayer: we can have ROI
objects that can be drawn either in a 2D context or in a 3D context
simply by changing drawer.
-----------------------------------------------------------------
What do you think?
Best
jy
--
Jean-Yves Tinevez, PhD
postdoc in PFID - Imagopole
Institut Pasteur
25-28 rue du Docteur Roux
75015 Paris, France
tel: +33 1 40 61 31 77
Wilhelm Burger
this sounds great and we have seen some good ideas on this list
earlier. I guess a key question is how far are you willing to go in
terms of (keeping/breaking) backward compatibility?
-- Wilhelm
Curtis Rueden
this sounds great and we have seen some good ideas on this list
earlier. I guess a key question is how far are you willing to go in
terms of (keeping/breaking) backward compatibility?
After our meeting, we will post a rough draft of the updated API including proposed class delegation. Our goal is to post & announce this before 3/14. And some time after 3/27 there will likely be a significant update to the proposal based on feedback from the Fiji developers.
-Curtis