ImageJ class hierarchy suggestions

[Curtis Rueden]

Hi everyone,

Next week the ImageJDev team will be drafting an updated ImageJ API. I am writing because many of you have experience writing large applications that plug into or extensively use ImageJ (Bio7, TrakEM2, Fiji, Micro-Manager, Endrov, etc.), and it is likely that we have each run across different issues and limitations.

Thus, we would be very interested in any specific feedback you have regarding ImageJ's API design (e.g., class/package structure) that you would like to see improved or changed. We have now documented most of what we plan to do on our Trac (http://imagejdev.org/trac/imagej/roadmap) but so far it is very general and does not discuss many specifics.

Most valuable would be thoughts on specific class relationships and design decisions that caused you trouble when attempting to leverage ImageJ.

My example: in developing the Bio-Formats plugins, we had a hard time supporting the additional file formats within ImageJ in a completely integrated way (i.e., so that File>Open would support the existing formats seamlessly). We needed to add code to HandleExtraFileTypes—which arguably causes a licensing conflict as it creates an indirect dependency on Bio-Formats—but there are many edge cases where a file may not be handled well with File>Open. E.g., ImageJ cannot handle all the TIFF variants that Bio-Formats can. Thus, we want to create a PlugInIO interface that allows I/O plugins more declarative power about exactly what they do.

Please let us know if you have any suggestions of your own!

Thanks,
Curtis

[Jean-Yves Tinevez]

On Feb 25, 2010, at 11:12 PM, Curtis Rueden wrote:

Most valuable would be thoughts on specific class relationships and design decisions that caused you trouble when attempting to leverage ImageJ.

Hi all. 

I would have a suggestion on GUI, and more specifically on ROIs. 

I hope to merge with Michael ideas he sent on the ImageJX list.

It is rather lengthy email, sorry.

-----------------------------

I recently tried to code weird shapes as ROIs in ImageJ. They were the results of a segmentation with constrained shapes. Because I wanted to have something nice for the user, The ROIs had to be mouse-interactive (resizable, moveable etc..).

I had a difficult time.

Johannes proposed on the Fiji-devel list an abstract class whose goal was to facilitate this interaction. (If you are curious, my implementation of the Arrow plugin uses it.) 

http://pacific.mpi-cbg.de/cgi-bin/gitweb.cgi?p=fiji.git;a=blob;f=src-plugins/Arrow_/fiji/util/AbstractTool.java;hb=HEAD

But we still gave to comply to ImageJ ij.gui.Roi master class, which is a concrete class in charge of drawing rectangle ROIs. Inside this class, there is everything: the logic to draw it, to interact with the user, with the image container, and the image data. Any homemade ROI must inherit from this class, there is no interface to implement. 

I tried to get my brain around ImageJ mechanism for ROI, and drafted a diagram for it. You can find it here: (please be patient with it, it is just a draft I did for myself):

http://creately.com/app?diagID=g45j59hu1

What I would like to propose here is to go for an interface hierarchy for ROIs, that is well decoupled, and that would allow the flexible design of new ROIs. This is just an early suggestion, and you will find no usable code in this post. On top of that, I could not have unambiguous and clear schema of what to choose, so I am sure you will find something to say. 

-------------------------

We use ROIs for many purposes, for instance:

- user interaction:

- draw a rectangle to crop an image

- measure intensity with a complex area

- add non-destructive annotations

- etc...

- as input for plugins

- as output for plugins, for instance a result of segmentation

From this you can see that they need to 

- know how to draw themselves as an overlay 

- comply to some interface to be an input of some plugins

- know how to interact with mouse clicks and drag

-------------------------

Let us take an example of a cube ROI that we want to draw over a 3D image stack. We suppose we all agree on how to describe images in ImageJ; for instance to use imglib. 

We want  ImageJ 2.0 to be able to deal with different way to represent the image data on screen. For instance, we can imagine that there will be 2 "displayers":

- one that display 3D data the same way ImageJ does now (with a slider at bottom that allow to scroll through slices), 

- a second one that generate a 3D rendering (a la Benjamin Schmidt's 3D viewer)

All displayers will implement a Displayer.java interface that signals them as taking in charge the rendering on the screen. 

Now, a Displayer should be able to display many things, not only images. We can even consider that there is no reason to privilege the image. So the Displayer must be able to deal with an array of stuff to draw.

At the other side of the hierarchy lies our 3D cube. The cube itself can be implemented within a Cube.java class specialized in geometric shapes. It would have as fields a side length and the x,y,z coordinates of its botton-north-east corner, and that is enough to characterize it entirely. As a side subject, we can imagine it belongs to a shape hierarchy that contains spheres, points, etc, ...

But there is not point for the Cube class to have methods for drawing itself. It does not care to be drawn in 2D or 3D, it just wants to deal with geometric data, that can be used by other classes. We must avoid to couple it to a Displayer. So we have to insert another class here, let us call it Drawer.java that will wrap the Cube object, and will be fed to the displayer. 

Because the way the cube will be drawn is not the same in 2D than in 3D, the Drawers will be implemented for each mother Displayer. When ask to redraw, the Displayer will loop over all Drawers it has been given, and call their draw() method (more detail lower). 

The same goes for the image data: is should be wrapped in a Drawer that knows how to render it in the 2D case, and another one for the 3D case.

-------------------------

If we try to make this example concrete, it will look like this.

Displayer_2D.java: This one is only able to display 3D data, nothing more

public class Displayer_2D implements Displayer {

protected ArrayList<Drawer2D> drawers;

protected int current_slice;

...

public void repaint() {

Graphics g = getGraphics();

for (Drawer_2D dw : drawers) {

dw.draw(g, current_slice);

}

}

...

}

Drawer2D.java

public interface Drawer2D {

public void draw(Graphics g, int current_slice);

}

DrawerImage2D.java

public class DrawerImage2D implements Drawer2D {

...

/** Constructor that takes an imaglib image. */

public DrawerImage2D(Image<T> _image) {

this.image = _image;

}

public void draw(Graphics g, current_slice) {

// Now this is PSEUDOCODE

slice = image.getSlice(current_slice);

drawImage(Graphics g, slice);

}

...

}

Cube.java:

public class Cube extends GeometricShape { // GeometricShape could be a mother class for others shapes

public double a; // cube side

public double xc, yc, zc; // cube corner location

...

}

DrawerCube2D.java. However, we can imagine a clever class that would be general for all GeometricShapes, and use a common method to draw. Here we keep it cumbersome but simple

public class DrawerCube2D implements Drawer2D {

... 

public DrawerCube2D(Cube _cube) {

this.cube = _cube;

}

public void draw(Graphics g, int current_slice) {

double current_z = getCurrentZ(current_slice); // do something to get physical coords

if (current_z > cube.zc + cube.a || current_z < cube.zc) {

return;

}

drawRectangle(g, cube.xc, cube. yc, cube. a, cube. a);

}

}

-----------------------------------------------------------------

Now we have a nice decoupled hierarchy, with classes that are in charge of drawing, others in charge of holding physical information, others in charge of rendering.

But this asses only the drawing part of the problem. It is still to design a few things:

- What do we pass to our plugins? It makes sense to pass only the class that has the physical information. In the above example, a plugin only cares for the Image<T> and the Cube, not their Drawer. However, what common interface should they implement to be able to be passed to a plugin? 

- On top of that, all shapes are not the same: for instance, there is lines and cubes. A plugin that crop an image can accept a cube, but not a line. So there might be another hierarchy to specify here (Maskable? Samplable? ... )

- How do we deal with mouse clicks? I feel that should be the work of Drawers. A Displayer that receives user interaction stuff, e.g. mouse clicks, should be able to decide to what Drawer forward it (if we click near the cube ROI, we want to modify it), and Drawers should have a logic to modify the object they wrap from these clicks. Also, we have to modify behavior according to what is selected as tool in the toolbar.

But what is cool with this hierarchy is that now having 3D of even 4D or whatever ROIs is not extremely simple. We just have to make sure the draw method receive the correct localization info (what slice do you want me to render) and that's all.

Also, we have decoupled the ROI from the displayer: we can have ROI objects that can be drawn either in a 2D context or in a 3D context simply by changing drawer. 

-----------------------------------------------------------------

What do you think?

Best

jy

--

Jean-Yves Tinevez, PhD

postdoc in PFID - Imagopole

Institut Pasteur

25-28 rue du Docteur Roux

75015 Paris, France

tel: +33 1 40 61 31 77

[Nico Stuurman]

Hi Curtis and others,

Thus, we would be very interested in any specific feedback you have regarding ImageJ's API design (e.g., class/package structure) that you would like to see improved or changed. We have now documented most of what we plan to do on our Trac (http://imagejdev.org/trac/imagej/roadmap) but so far it is very general and does not discuss many specifics.

Most valuable would be thoughts on specific class relationships and design decisions that caused you trouble when attempting to leverage ImageJ.

Sorry for the delay.  I am not a software architect and can not really give comments on design decisions, but will attempt to convey some of the problems our project (Micro-Manager) run into.

1. The Brightness/Contrast tool.  Display of the histogram cannot be reliably set to the dynamic range of the camera (i.e., it always automatically goes back to the range of the minimum and maximum pixel value in the image, which can be extremely deceptive). No gamma correction.  No method to update histogram when the image changes.  No log display of the histogram.  We ended up writing our own, but things are still clunky because acquired images (shown in a modified Image5D viewer) can only be controlled by the ImageJ B&C tool.

2. Lack of plugin API.  We have been bitten a number of times by internal changes in ImageJ breaking our code.  Wayne is very responsive, but this still causes confusion.

3. Lack of standard for Multi-Dimensional viewer.  We ended up using Image5D viewer, Hyperstacks came later.  My impression is that the UI of Image5D is easier for users than the UI of Hyperstacks.  In any case, we will be helped by a standard viewer for multi-dimensional images that integrates nicely with other ImageJ tools (like 3D viewers), and that is extensible (we do need to add a number of buttons that interface with image acquisition).  

4. MDI versus SDI. Not sure if this was on your list already (all of you have certainly debated this in the past!), but it seems that many people prefer the MDI model.  On the Mac, it is pretty weird that a single application has different menus depending on which window you select (in our case, ImageJ windows versus Micro-manager window).  

These are more, but I think these ones are the most pressing.

Let me know if you need more info.

Best.

Nico