Image quirks

[Albert Cardona]

We know that ImageJ is great. That said, and in the spirit of a
defining what ijx is to become, here is a list of small and big quirks
that must be addressed:

* The command recorder:
Recording commands is a great idea. Currently, the output is one of
the these two:
- directly as macro language, which is has great utility on the user end
- as events from the ij.CommandListener, which is currently very
limited: only the command, not the arguments neither the image over
which the command has been applied.

Generalizing the command recorder could lead to automatic code
generation in any scripting language of choice, including java.

* The min and max: a call to setMinAndMax on a GRAY8 or COLOR_RGB
image has the effect of expanding the desired range to the 8-bit range
of the grey or each color channel. Not so for GRAY16 or GRAY32. The
result is that a number of successive calls to setMinAndMax with the
same min, max values has different effects on each image, depending on
type.

* The severe lack of thread safety. I have patched some of that over
the last year with the addition of Thread-local ij.Macro.getOptions()
and in Thread-local ij.WindowManager.getCurrentImage(). Yet, there are
numerous other places affected: for example, numerous ij.plugin.* java
files use static fields to store the parameters as edited from a
GenericDialog, be it from a user-driven dialog or from a macro-driven
dialog. The result is a collision: different threads calling the same
class with different macro arguments for the dialogs result in
unexpected parameters being used in computations.
This design error should not be repeated, in view of the increasingly
multi-core of CPUs.

* The lack of multidimensional data representation. The current adhoc,
limited solutions, namly hyperstacks and CompositeImage, are, I'm
sorry to say, horrendous. They fix the immediate problem, but they
introduce numerous others, particularly: inconsistency in dimension
handling, and extra complexity in core classes like ImagePlus.
To this effect, I would create a proper data structure that can handle
any number of dimensions, i.e. a so-called tensor representation.

Which leads to:
** The difficulty in managing stacks programmatically. The
ImageStack fails me every time I need to manipulate its pixels. Even
simple tasks like duplicating a stack are not trivial. The
ij.plugin.CanvasResizer has somewhat helped in at least resizing the
2D dimensions of a stack, but there is no way to resize it in 3D for
example. Proper tensor implementation ought to enable resizing in any
dimensions, and provide trivial copy mechanisms of any subpart.

** The limitations in ImageProcessor: it's only 2D, and completely
unware of the third and other dimensions. Numerous plugins, like
current TransformJ package, perform operations purely on 2D, even if
they can handle stacks.

** The lack of image source abstraction. Does it matter if an image
lives in a database, in an url, in a text file, in a series of paged
blocks (2D like mipmaps, or 3D in an octree of diminishing cube edge
length), in a remote server, or as an actual file? It should not
matter at all. The current ij.io.FileInfo manages the file and the url
cases, but the model is not extensible.

Just to say that we within the Fiji project have started a general
implementation for tensors, with pixel iterators optimized for each
data source type (2D sections like ImageProcessor, or paged cubes,
etc; even handling complex numbers for 3D FFT; mostly the work of
Stephan Preibisch).

* The difficulty in calling plugins from other plugins. The current
model uses ij.IJ.run("Name", "arguments"); or as of recently and
because of improved Thread isolation capabilities,
ij.IJ.run(imageplus, "Name", "arguments");
Still, there is no generic way to express the need to suppress calls
to ImagePlus.show() in plugins and instead gather the otherwise shown
images for the caller plugin to use.
What I think we are seeing here is the use of a java plugin as a
script, if we define script as "a small program to do one relatively
simple task using mostly high-level commands and not intended to be
reused by other plugins." For this purpose, the current javascript
support is great. But actual java plugins ought to be more expressive
and controlable. I would like to see more expressive PlugIn-like
interfaces.

* The current practice of not packaging plugins; i.e. a "Class_Name"
instead of a "org.my.Class_Name". Which leads to very undesirable name
collisions and on occasions incomprehensible errors when compiling
other plugins. The plugin-from-plugin call problem is aggravated by
this problem as well.

* The presence of public fields, like the ImagePlus.changes. These are
the least friendly at all for a program to manage or listen to. Such
practice should be banned outright; sharing state publically in a
non-final way is the antithesis of concurrent programming (aka Thread
safety).

* The lack of an event system. The current ImageListener interface is
a start, but very insufficient. Ideally I want to listen to anything
that occurs to an image. I suspect the duality between ImagePlus and
ImageProcessor, which sometimes suplant each other in current ImageJ,
is part of the problem.

* The inability to insert plugins as commands anywhere in the menus.
The current ij.Menus is a mess that I gave up on fixing. Johannes
Schindelin gave it a long shot and came up with a cleanup, which
enabled adding commands just about anywhere in the menus. Such
cleaned-up class is part of the ImageJA/fiji branch that we use as the
source for the core ij.jar of Fiji.

* The lack of a plugin manager, resulting in fragmented documentation,
difficulty in finding plugins, imposibility of finding plugin source
code on occasions. The lack of a plugin manager is very related to the
lack of a centralized set of tested, safety-verified plugins. We are
aiming at solving this problem within Fiji, with institutional support
from the Max Plank in Dresden, Germany.

* Numerous inconsistencies in the API. Simple, small gotchas like the
setMinAndMax that I explained above are abundant. For example:
** The ROI exists in the ImagePlus and in the ImageProcessor, but
could be programmatically two different ROI if one is to set it
directly to the ImageProcessor. This is part of the duality between
ImageProcessor and ImagePlus, which must be resolved one what or
another.
** Calling ImageProcessor.fill() has different effects depending
upon the type of ROI present in the ImageProcessor. For a correct
fill, one must call ImageProcessor.fill(mask), where mask is the
ImageProcessor mask of the ROI when the ROI is not a ij.gui.Roi (a
Rectangle).

* The GUI classes. As others and myself pointed out, the GUI classes
are a mixture of GUI, control and data representation. This leads to:
** Inability to run ImageJ headless, without a virtual screen
buffer, even if no GUI classes are instantiated at all (but are
present within the import list of some class that is instantiated).
** The sometimes not small problem of avoiding null pointers in
ImageCanvas when not contained in an ImageWindow.
** Lack of flexibility with a GUI toolkit. I don't think Swing is
great (I use it for TrakEM2, but it's a headache), but java3d and jogl
are nice alternatives. The hoops and workarounds to make java3d work
with ImageJ are enormous, and result in, for example, Bene Schmid's 3D
Viewer basically reinventing its own data structures and delegating
some calls to an ImageCanvas and ImageWindow for ROI and ImageJ
interaction.

I am sure I'm missing a few of my informal list of grievances with
current ImageJ. As I said, ImageJ works and its great, but all the
above are limiting us in what we can do in our application fields.

I would love to see what other aspects of ImageJ you consider in need
of attention, or expansions on the above.

Albert

[wil...@ieee.org]

Albert,

this is an impressive and well-motivated list, I share much of this,
thank you for this input. I think it is important in this early stage
to lay down all ideas that are out there and work out a common goal,
before talking about any specific details. On the way we will
hopefully also find out how the project can be positioned relative to
the existing ImageJ.

My personal goal is to create a substrate for building complex, multi-
layered and reliable computer vision systems. Java is a good platform
for this for many reasons and ImageJ is great for experimenting and
algorithm development. It is not well suited, however, for deployment
or smooth integration into larger systems. Consequently, my priorities
are somewhat different, more bottom-up and less GUI-oriented. Here is
the top of my little list:

1. Data structures: What is an image? The distinction between
ImageProcessor and ImagePlus is confusing and not necessary
(ironically, ImagePlus does not hold the image data while
ImageProcessor is a data container but not a processor). One simple
image class should be sufficient. This has to be cleaned up.

2. Establish a standard model for processing images. Allow operations
to be chained easily without manually duplicating images (this affects
return values, expression handling etc.).

3. Add platform-specific "performance packs" in a transparent fashion
to compensate for Java's performance limitations.

4. Separate data representation and processing from viewing. Create
base libraries that can be used in other environments (head-less),
minimize interdependencies between classes.

5. Maximize compile-time safety. Avoid dangerous typecasts and
eliminate the need for run-time type checking wherever possible. Use
enum types instead of integer constants and interfaces to instead of
bit-vector flags.

6. Improve and extend the plugin mechanism. Do not invoke or pass
arguments to plugins (and other methods) unless the type specification
allows it. Standardize the use of plugins (e.g., are dialogs allowed
inside the run method?). Extend plugins to operate on multiple images
and on data structures other than images (e.g. sets of primitives
etc.).

7. Improve orthogonality. Provide clear semantics of different
elements and recommend a *single* way to implement any given task.

8. Additional implementation details: Minimize the public API. Apply
good programming practices throughout (use generics where appropriate,
limit the use and functionality of constructors - use factories
instead, thread safety, ...).

I got started on a small demo implementation that (as of now)
peacefully coexists with unmodified ImageJ. It shares the standard
ImageJ GUI but works with its own data structures. I redesigned the
plugin protocol and wrote an extended plugin executer (which is a
standard IJ plugin itself) using the new standard java compiler. I can
share this any time but it was only meant to get a feel how much would
be involved in overhauling this.

As a last note, I have also considered a conversion to C# and .NET,
which I personally think (but don't kill me now!) is definitely
superior as a whole to the Java-environment for this purpose. To be
frank: If the ImageJ community is not interested in an approved API,
necessary changes are unacceptable and/or full backward-compatibility
turns out to be impossible, I would probably prefer to invest my time
into porting ImageJ to C#/.NET. But at this moment I still feel for
Java ;-)

Wilhelm

[Volker Baecker]

Hello,
I agree with most of your points.
For me the very first step seems to be to create a java5 clean version
of ImageJ. That is a version, in which all or at least most of the
warnings are gone. That would be a more or less mechanical task.
Use generics, remove type casts and simplify loops.
The next step seems to be to remodel ImageJ according to mvc. Never a
piece of useful code must directly talk to a view! That will be much
harder to do. When it is done, ImageJ will be usable as a library and
the switching of the gui framework would be possible.
Has someone already done work along these lines?
Once we are there we can start the interesting part, i.e. your point 1
or the multidimensional data representation proposed by Albert.
Another thing that I would like to see is the creation of a test suite
and the usage of unit tests. I think that would especially be valuable
for a more community based development process.

What do you think?
Volker

I would then try to separate the

wil...@ieee.org a écrit :

[wil...@ieee.org]

Hello group,

I made a quick sketch of how I think IJX could perhaps be coarsely
structured:
http://groups.google.com/group/imagejx/web/ijx-architecture-01.png
There is also the PowerPoint file if someone wants to play with it.

"Control" in the top layer represents the C-component of a MVC
structure. Other things should be self-explanatory. Let me know what
you think. Anything important missing?

Wilhelm

[Curtis Rueden]

Hi Wilhelm,

Looks good! However, there are some fairly disparate things in "core."
For example, while I would agree that statistical functions, image
operators and utility functions should definitely be part of the main
distribution, they are less "core" than the bottom three items, in
that they can be implemented using the plugin infrastructure. And
doing so is a great way to thoroughly exercise said infrastructure
while providing a wealth of examples to new programmers. So maybe a
fourth category called "core plugins" or some such?

-Curtis

[joris meys]

Hi all,

while I can understand the idea of a set of "core plugins", I'm afraid this will quickly overload the plugin menu. I learned a lot about ImageJ by looking at the source code of the core program actually, more than looking at plugins.

I would be in favor of implementing these methods and operators in the core distribution. As the ImageJX is conceptualized, that sounds more logic too : these methods and operators are essentially tools to be used in plugins, and providing them as plugins just makes things more complicated than necessary if you have to implement them in another custom plugin. As far as I understood, the whole idea is to make ImageJX more than a program, i.e. a "language for image analysis".

Kind regards
Joris

[Bill Mohler]

On the question of plugins vs core code:

I am very new to effort at modifying the java code of either core or
plugin classes (I'm more of a user wanting to be able to develop via
better understanding the structure of ImageJ).

With the current structure, how much of a performance loss is incurred
if an algorithm is run as a plugin rather than as part of the core ij
package?
Is there a real difference? Or is it just a matter of "structure and style"?

Bill

[wil...@ieee.org]

Hello Curtis,

thanks for the feedback. I understand the "core" of IJX as a carefully
defined, well-tested collection of base classes that cover most
standard image processing data structures and functionalities. In
contrast to plugins these classes are completely known at compile
time.

In this context, one should really reconsider the plugin concept in
general. I agree with Albert as he states:

> What I think we are seeing here is the use of a java plugin as a
> script, if we define script as "a small program to do one relatively
> simple task using mostly high-level commands and not intended to be
> reused by other plugins." For this purpose, the current javascript
> support is great. But actual java plugins ought to be more expressive
> and controlable. I would like to see more expressive PlugIn-like
> interfaces.

My growing opinion is that Java plugins, as they are defined now, are
probably not needed at all. They are just classes that are loaded
dynamically with a rather loose (and unsafe) coupling to the main
runtime. Scripts can do the same quicker and with lower demands on
programming expertise. (Btw. there is NO performance difference
between a method defined in a plugin compared to a regular Java
class.)

To keep the environment configurable, I would instead - or in addition
- suggest to allow for "extensions" (name to discuss), i.e., Java
classes with a strictly defined interface that are loaded at startup
or per request.

-- Wilhelm

> > Wilhelm- Zitierten Text ausblenden -
>
> - Zitierten Text anzeigen -

[Grant]

I think the distinction between a "script", a "plug-in", an
"extension", and a "module" is interesting. It relates, perhaps, to
the distinction between an API and an SPI.

"Service Provider Interface (SPI): a software mechanism to support
replaceable components. It is the implementer-side equivalent of an
API; a set of hooks that can or must be overridden. (In the Java
Runtime Environment, SPIs are used in JDBC, JCE, JNDI, JAXP, JBI, Java
Sound and Java Image I/O.)"

Clients use the API while extensions/plugins use the SPI.

More on API/SPI:

The Capability Pattern - Future-Proof Your APIs (http://
weblogs.java.net/blog/timboudreau/archive/2008/08/the_capability.html)

NetBeans API http://openide.netbeans.org/tutorial/api-design.html#design.apiandspi

Creating Extensible Applications With the Java Platform (http://
java.sun.com/developer/technicalArticles/javase/extensible/)

-- Grant

[wil...@ieee.org]

Very good point - and interesting article! I think this is really a
central issue. Must read ...

-- Wilhelm

[wil...@ieee.org]

In the same context of building modular Java systems - has anyone in
the group looked at the "Open Services Gateway
Initiative" (www.osgi.org) yet?

--Wilhelm

[Grant]

OSGi is part of a large, but very important, can of worms. These
worms include plugin frameworks, extension mechanisms, modules,
service loaders, dependency injection... I'll try to put together some
of the material I have gathered and post it, but I'm interested to
hear other thoughts on these issues.
- Grant

[Adrian Daerr]

On 16 mar, 21:27, Wilhelm wrote:
> Very good point - and interesting article! I think this is really a
> central issue. Must read ...

I agree. Just to throw in another thought, how usefull do you think
Java's ImageProducer/ImageConsumer interfaces are as an example of SPI/
API in this context ?
In the effort to categorize scripts/plug-ins/modules/extensions/foo,
the most easily identifiable building block of image processing are
processing units which read and output zero or more images (at least
basic "scripts" and "plug-ins"). Could they be served by a
ImageProducer/ImageConsumer-inspired interface ? Allowing the plugin
to declare need for input or output plumbing according to the job
("creators" (e.g. image acquisition, readers), "filters" (one or
several images in, one ore more out), "consumers" (statistics,
writers,...)).
In a GUI context, a basic plugin could expect 'standard in' to be the
current image, and 'standard out' another image, without having to
worry about the plumbing. Yet other plug-ins can re-use the code by
redirecting the output as needed.

In brief, is the UNIX "everything is a file descriptor" concept,
reborn as Producer/Consumer interfaces, powerfull enough (and easy-to-
use enough) for at least a certain category of code ?

--
Adrian

[Dimiter]

On Mar 24, 7:26 pm, Adrian Daerr <adrian.da...@gmx.de> wrote:
> On 16 mar, 21:27, Wilhelm wrote:
>
> > Very good point - and interesting article! I think this is really a
> > central issue. Must read ...
>
> I agree. Just to throw in another thought, how usefull do you think
> Java's ImageProducer/ImageConsumer interfaces are as an example of SPI/
> API in this context ?

This is more or less abandoned in Java2D.

> In the effort to categorize scripts/plug-ins/modules/extensions/foo,
> the most easily identifiable building block of image processing are
> processing units which read and output zero or more images (at least
> basic "scripts" and "plug-ins"). Could they be served by a
> ImageProducer/ImageConsumer-inspired interface ? Allowing the plugin
> to declare need for input or output plumbing according to the job
> ("creators" (e.g. image acquisition, readers), "filters" (one or
> several images in, one ore more out), "consumers" (statistics,
> writers,...)).
> In a GUI context, a basic plugin could expect 'standard in' to be the
> current image, and 'standard out' another image, without having to
> worry about the plumbing. Yet other plug-ins can re-use the code by
> redirecting the output as needed.

I would say that you forget the circular mode :). When you actually
change the current image.

Then there is also the measurement/statistical mode where from an
image you get some numbers. E.g.
a ResultTable.
I in this line, I think that the ResultTable should be per image/
processor and not a global entity for the entire IJ.