Work on the ROI model

[Roger Leigh]

Hello,

Regarding the recent discussion about the ROI model, I would like to
share what I've been doing over the last few weeks, and discuss in more
detail about what everyone would like out of the model so that we can
cover all the different use cases well.

The model is hosted here:
https://github.com/scijava/roi-model
(git://github.com/scijava/roi-model.git)
with the generated documentation here:
http://www.scijava.org/roi-model/

To work on the model, clone it and run "make" (needs sphinx, python,
graphviz, texlive). The documentation is in restructured text (sphinx),
with the model being defined by tables in the spec/ directory. Most of
the documentation is generated directly from the spec; there's a
top-level python script, "introspect", to read, validate and generate
the specification.

The main pages to look at for the moment are the "Shapes" and "Shape
representations" pages, which are generated, and the "Fundamental data
types" and "Geometric shape primitives" pages, which provide some of the
background for them. There is a rather large number of shape and
representation types. Don't panic--we don't have to support all these!
They are just included for relative completeness--I'd like to define a
subset of these needed by all implementations which we can use to start
with.

The main concept with the current design is that a "shape" is the
realisation of some geometric (or other in nD) form, and this can be
used to visualise, measure, or convert to a mask for iteration, image
processing etc. i.e. it is a concrete entity you can use to do stuff.
The "representation" is a means for specifying a shape, i.e. its
underlying geometry; one shape can have several representations, since
there may be several alternative ways to specify a shape. The question
here is: which form (or forms) should be the definitive form(s) we use
for storage/exchange, and should we allow more than one form per shape?
Note that these can also be used as a basis for versioning the shape
behaviour if we need to revise the specification.

The shapes and representations are used to define a simple serialisation
format which is tightly specified. This could take several concrete
forms for exchange: packed binary, plain text, or XML for the
appropriate implementation schema (e.g. the OME ROI model). But we are
not tied to this design at all--please do suggest alternatives.


Some questions for discussion:
• is the above sensible, or are there any fundamental problems that I've
not considered?
• what are the basic shapes we want to support initially?
• for each of those shapes, what should the representation(s) used to
specify those shapes be?
• should we support separate 1D, 2D and 3D geometric primitives, or
should all these primitives be 3D, with 1D and 2D representations for
convenience? (This has the scope to greatly simplify the implementation.)


Once we have some of that a bit better defined, I would like to start
work on a reference implementation (generated from the spec) that we can
start to use for testing and integrating.


Regards,
Roger

--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
College of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364

The University of Dundee is a registered Scottish Charity, No: SC015096

[Johannes Schindelin]

Hi Roger,

On Wed, 24 Oct 2012, Roger Leigh wrote:

> Regarding the recent discussion about the ROI model, I would like to
> share what I've been doing over the last few weeks, and discuss in more
> detail about what everyone would like out of the model so that we can
> cover all the different use cases well.
>
> The model is hosted here:
> https://github.com/scijava/roi-model
> (git://github.com/scijava/roi-model.git)
> with the generated documentation here:
> http://www.scijava.org/roi-model/

Thanks.

> To work on the model, clone it and run "make" (needs sphinx, python,
> graphviz, texlive). The documentation is in restructured text (sphinx),
> with the model being defined by tables in the spec/ directory. Most of
> the documentation is generated directly from the spec; there's a
> top-level python script, "introspect", to read, validate and generate
> the specification.

I put a minimal README into the root (which you merged while I was writing
this mail) with the essential information for people who might otherwise
be puzzled what to do with roi-model.git.

A general comment on the specification: it seems to be very focused on
serialization. What I will work on in Dundee is more focused on the
image processing side, though.

IMHO we need to focus more on the applications than on implementation
details or optimization.

Applications I can think of right now:

- serialization
- visualization (AKA rendering)
- processing (AKA iterating, both *inside* and *outside*)
- set operations (i.e. merging, intersecting, set difference)
- interaction (i.e. letting the users define ROIs)

I am sure that I forgot some.

The main concern I have for this meeting is that we might end up with
something that is limiting us too much in the future.

For example, we need to think far enough ahead to allow for
backwards-compatible and future-proof extensions (as Tobias suggested, we
need to define a way how to ignore unimplemented features safely).

Another limitation we should avoid at all cost is the limitation to a
certain number of dimensions (such as an inherent 2D model that somehow
gets extruded to nD; my experience with the Virtual Insect Protocol shows
that this is a terrible way to think about regions of interest).

And I would also like to caution against making the specification so
centric on serialization.

Ciao,
Johannes

[Josh Moore]

All,

Sorry for the late posting, but a few points summarizing most if not all the chats we had in Dundee this past week while I was visiting (sorry that I couldn't stay for this week!).

On Nov 9, 2012, at 6:11 PM, Johannes Schindelin wrote:

> Hi Roger,
>
> On Wed, 24 Oct 2012, Roger Leigh wrote:
>
>> Regarding the recent discussion about the ROI model, I would like to
>> share what I've been doing over the last few weeks, and discuss in more
>> detail about what everyone would like out of the model so that we can
>> cover all the different use cases well.
>>
>> The model is hosted here:
>> https://github.com/scijava/roi-model
>> (git://github.com/scijava/roi-model.git)
>> with the generated documentation here:
>> http://www.scijava.org/roi-model/
>
> Thanks.
>
>> To work on the model, clone it and run "make" (needs sphinx, python,
>> graphviz, texlive). The documentation is in restructured text (sphinx),
>> with the model being defined by tables in the spec/ directory. Most of
>> the documentation is generated directly from the spec; there's a
>> top-level python script, "introspect", to read, validate and generate
>> the specification.
>
> I put a minimal README into the root (which you merged while I was writing
> this mail) with the essential information for people who might otherwise
> be puzzled what to do with roi-model.git.
>
> A general comment on the specification: it seems to be very focused on
> serialization.

Though at first glance the tight, binary serialization that Roger has proposed might be quite useful, I agree it's not a priority. In fact, we'll likely have multiple serializations, since we'll need something (unfortunately) that'll work well in XML, etc. etc. Nevertheless, I think more of the specification is about getting the model API straight. And this week is about hammering on it, modifying, repeating, etc.

> What I will work on in Dundee is more focused on the
> image processing side, though.
>
> IMHO we need to focus more on the applications than on implementation
> details or optimization.
>
> Applications I can think of right now:
>
> - serialization
> - visualization (AKA rendering)
> - processing (AKA iterating, both *inside* and *outside*)
> - set operations (i.e. merging, intersecting, set difference)
> - interaction (i.e. letting the users define ROIs)
>
> I am sure that I forgot some.
>
> The main concern I have for this meeting is that we might end up with
> something that is limiting us too much in the future.
>
> For example, we need to think far enough ahead to allow for
> backwards-compatible and future-proof extensions (as Tobias suggested, we
> need to define a way how to ignore unimplemented features safely).

+1, though it's of course hard to know what one is forgetting. In terms of dealing with the unimplemented or undesirable features, I'd vote for allowing one to choose between either the implementations (say via a factory interface) or the error-handling itself (say via a strategy interface) to allow for minimally "safe" and "unsafe" versions. (More to that below)

> Another limitation we should avoid at all cost is the limitation to a
> certain number of dimensions (such as an inherent 2D model that somehow
> gets extruded to nD; my experience with the Virtual Insect Protocol shows
> that this is a terrible way to think about regions of interest).

The proposal as it stands certainly has limitations. Some we discussed this week, which Roger is either working on now, or we'll be mentioning once everyone is in Dundee. But even with those changes, one of the initial points of discussion will be the utility and mathematics behind higher-dimensional shapes (i.e. beyond 3D). Very roughly, the updated proposal will be that 3D shapes be combined with 3D transforms to produce what we workingly called "Shape trees". There is one of these per ROI along with an embedding in other higher dimensions.

We discussed 2 of the possibly several down-sides from others' perspectives. The first is that one can't use an arbitrary MxN transformation across all the dimensions in the shape tree. We couldn't figure out the mathematics behind why that would be useful (or legal). Perhaps someone could talk us through that. The second is that this form of "N-3" restriction doesn't work with the ImgLib API.

If we can agree that this second reason is actually the bigger issue, then we'd propose:

* Make the ROI package something external to ImgLib, e.g. package "scijava.roi"

* Make some or all scijava.roi classes implement the proper ImgLib interfaces

* Add Safe and Unsafe strategies as above so that if ImgLib tries to perform a higher-dimensional transformation that's not supported, the implementation can either throw (Safe) or blindly convert the ROI+shapes to a NDim matrix (Unsafe).

For a division like that to work, it'll be important to nail down the minimal number of ImgLib interfaces that this package (whatever the name) MUST implement.

There may already be too many ROI elements in ImgLib for this to be feasible without burdensome refactoring; if so, back to the drawing board. But in an effort to prevent everything from falling into ImgLib over time, we'd suggest at least discussing a place for it in the Dresden-DAG, similar to the RnR package.

Cheers,
~Josh

P.S. Another, secondary question might be: how will axis ordering/naming be specified at the ROI level (say for transferring a ROI between 2 images)? What's the naming scheme used? But that seems like getting into details at this point.

[Johannes Schindelin]

Hi Josh,

On Sun, 11 Nov 2012, Josh Moore wrote:

> On Nov 9, 2012, at 6:11 PM, Johannes Schindelin wrote:

Please note that GoogleGroups has no options to recommend reply-to-all.
Yet, on scijava, only reply-to-all makes sense. You *will* lose people
otherwise.

So, unless you want to lose conversation partners, reply-to-all every
single time you respond to a mail to scijava.

Ciao,
Johannes

[Johannes Schindelin]

Hi Josh,

I hope we did not lose anybody in the conversation, but it is really too
much to ask from me to re-instate the Cc:list.

On Sun, 11 Nov 2012, Josh Moore wrote:

> On Nov 9, 2012, at 6:11 PM, Johannes Schindelin wrote:
>
> > On Wed, 24 Oct 2012, Roger Leigh wrote:
> >
> > A general comment on the specification: it seems to be very focused on
> > serialization.
>
> Though at first glance the tight, binary serialization that Roger has
> proposed might be quite useful, I agree it's not a priority. In fact,
> we'll likely have multiple serializations, since we'll need something
> (unfortunately) that'll work well in XML, etc. etc. Nevertheless, I
> think more of the specification is about getting the model API straight.
> And this week is about hammering on it, modifying, repeating, etc.

No. The focus needs to be on the use cases of ROIs. Otherwise we do not
even *know* what to serialize.

Just think about the following use case: a user chooses Li auto-threshold
to select voxels.

What to serialize? Li? The value Li calculated? The BinaryType mask
representing the selected voxels? A geometric approximation of the
hyper-volume?

This is not a theoretical question. It has very real consequences on what
we can, and what we cannot do, with the ROI model.

In my opinion, we have to concentrate on *what* people need to do with
regions of interest. For example, in the above use case, scientists
frequently need to be able to store the fact that it was a Li
auto-threshold. But to apply the same ROI to another channel, the binary
mask must be transported.

However, my primary focus these two days lies purely in implementing ROIs
on top of ImgLib2. Everything else would be premature, or too limited in
applicability.

> > Applications I can think of right now:
> >
> > - serialization
> > - visualization (AKA rendering)
> > - processing (AKA iterating, both *inside* and *outside*)
> > - set operations (i.e. merging, intersecting, set difference)
> > - interaction (i.e. letting the users define ROIs)
> >
> > I am sure that I forgot some.

No comments here???

> > The main concern I have for this meeting is that we might end up with
> > something that is limiting us too much in the future.
> >
> > For example, we need to think far enough ahead to allow for
> > backwards-compatible and future-proof extensions (as Tobias suggested,
> > we need to define a way how to ignore unimplemented features safely).
>
> +1, though it's of course hard to know what one is forgetting.

The point, of course, is to allow ourselves to forget something. We need
to use duck-typing on the implementation side, and versioning on the
specification side (although I sincerely believe that we have to get the
implementation side right first before we can dare to think about the
specification).

> In terms of dealing with the unimplemented or undesirable features, I'd
> vote for allowing one to choose between either the implementations (say
> via a factory interface) or the error-handling itself (say via a
> strategy interface) to allow for minimally "safe" and "unsafe" versions.

No. The only way that proved practical was to use an interface-driven
design, not a factory-driven one. I.e. "what can it do?" instead of "give
me something, but I don't know what, exactly"

> > Another limitation we should avoid at all cost is the limitation to a
> > certain number of dimensions (such as an inherent 2D model that
> > somehow gets extruded to nD; my experience with the Virtual Insect
> > Protocol shows that this is a terrible way to think about regions of
> > interest).
>
> The proposal as it stands certainly has limitations. Some we discussed
> this week, which Roger is either working on now, or we'll be mentioning
> once everyone is in Dundee. But even with those changes, one of the
> initial points of discussion will be the utility and mathematics behind
> higher-dimensional shapes (i.e. beyond 3D). Very roughly, the updated
> proposal will be that 3D shapes be combined with 3D transforms to
> produce what we workingly called "Shape trees". There is one of these
> per ROI along with an embedding in other higher dimensions.

Again, this is too focused on low dimensions. 2D, 3D are not what we need
to think. If we limit ourselves that way, we can stop thinking about a
general ROI specification that will serve us well for decades to come
right now.

Instead, we need to focus on defining what a ROI can do. Completely
independent of the number of dimensions of the space it lives in.

> We discussed 2 of the possibly several down-sides from others'
> perspectives. The first is that one can't use an arbitrary MxN
> transformation across all the dimensions in the shape tree. We couldn't
> figure out the mathematics behind why that would be useful (or legal).
> Perhaps someone could talk us through that. The second is that this form
> of "N-3" restriction doesn't work with the ImgLib API.

In real life, transformations on ROIs are rarely needed, unless
accompanied by a transformation on the original image's space. In which
case it is a space transformation applied to the ROI.

Thinking about the problem that way lets us liberate ourselves from the
very limited fixed-dimensional view of ROIs. We can say that a certain
aspect some ROIs accomodate is to provide a
RealRandomAccessibleInterval<ProbabilityType> instead. Much more useful
when you have to work *with* the ROIs as opposed to just trying to save
and load them.

> If we can agree that this second reason is actually the bigger issue,
> then we'd propose:
>
> * Make the ROI package something external to ImgLib, e.g. package
> "scijava.roi"

There is little reason to make the ROI package independent of ImgLib.
ImgLib is a versatile library to work with n-dimensional data, and ROIs
are very much subsets of those data. Trying to overgeneralize will only
cost us unnecessary gray hair.

> * Add Safe and Unsafe strategies as above so that if ImgLib tries to
> perform a higher-dimensional transformation that's not supported, the
> implementation can either throw (Safe) or blindly convert the
> ROI+shapes to a NDim matrix (Unsafe).

Again, transformations are not something inherently specific to ROIs. So
they should not be part of the specification of ROIs. Instead, ROIs that
can be transformed (my Li example from above cannot, unless realized into
a specific RandomAccessible<BitType>) should implement an according
interface.

> For a division like that to work, it'll be important to nail down the
> minimal number of ImgLib interfaces that this package (whatever the
> name) MUST implement.

Now we're talking.

> There may already be too many ROI elements in ImgLib for this to be
> feasible without burdensome refactoring;

We are already sure that there needs to be refactoring. This is what this
hackathon is about.

> if so, back to the drawing board.

Or alternatively, hacking. That is what a hackathon is good for.

Remember, a hackathon is not about nailing down a design document. It is
about hacking on code. If that code serves as a proof-of-concept, or as a
demonstration why a particular approach does not work, does not matter.

A hackathon is about code.

> But in an effort to prevent everything from falling into ImgLib over
> time, we'd suggest at least discussing a place for it in the
> Dresden-DAG, similar to the RnR package.

Since ImgLib2 is already shared between so many SciJava projects, it does
not make sense to bend over to define things outside it.

Instead, we should implement things on top of ImgLib2. Even better: there
is already a ton of code out there, on top of ImgLib2 (for those
unfamiliar with ImgLib2, I highly recommend to familiarize yourself with
it, you will have to do that in the future anyway).

> P.S. Another, secondary question might be: how will axis ordering/naming
> be specified at the ROI level (say for transferring a ROI between 2
> images)? What's the naming scheme used? But that seems like getting into
> details at this point.

Indeed.

So far, the assumption of ImgLib2's core is that if you work on two images
simultaneously, the axis ordering/naming is the same. And if it is not, it
is very easy to use Views to make it so. Therefore I deem this an
unconcern for the moment.

> > And I would also like to caution against making the specification so
> > centric on serialization.

I really worry about this. If we focus too much on specification, we will
end up -- again -- with no tangible code.

Ciao,
Johannes

[Stephan Saalfeld]

Hi,

sorry for not contributing much to the conversation, I have a pressing
deadline on Thursday which keeps me really busy. So here are just three
loose comments that I've discussed with Tobias last week and that come
up here again:

[...]

> Instead, we need to focus on defining what a ROI can do. Completely
> independent of the number of dimensions of the space it lives in.
>

Good---Tobias has already commented on this, however, I think that
interfaces for a specific dimensionality are a fine idea and can be
completely orthogonal with other aspects. Many concepts exist only for
a specific dimensionality. E.g. 2D polygons are 2D.

> > We discussed 2 of the possibly several down-sides from others'
> > perspectives. The first is that one can't use an arbitrary MxN
> > transformation across all the dimensions in the shape tree. We couldn't
> > figure out the mathematics behind why that would be useful (or legal).
> > Perhaps someone could talk us through that. The second is that this form
> > of "N-3" restriction doesn't work with the ImgLib API.
>

ImgLib2 has an interface for transformations. In analogy to above,
creating an orthogonal interface hierarchy that limits the
dimensionality is, I think, a good idea. I propose that this would be
the same hierarchy and that they be extensions of the n-dimensional
EuclidenSpace interface (EuclideanSpace2D extends EuclideanSpace).
Transformations being interfaces and streamable (XML-able) enables to
save compact representations of specific transformation classes beyond
or below affine matrices. We have this concept implemented in my mpicbg
library and the TrakEM2 extensions (they extend the mpicbg interfaces
with XML export and import). Find a textual description of the
interfaces and XML attached. Check code examples and dive in further
(do F4 in Eclipse):

https://github.com/axtimwalde/mpicbg/blob/master/mpicbg/src/main/java/mpicbg/models/CoordinateTransform.java

https://github.com/axtimwalde/mpicbg/blob/master/mpicbg/src/main/java/mpicbg/models/CoordinateTransformList.java

and the corresponding XML-ers for TrakEM2:

http://fiji.sc/cgi-bin/gitweb.cgi?p=trakem2.git;a=blob;f=mpicbg/trakem2/transform/CoordinateTransform.java

http://fiji.sc/cgi-bin/gitweb.cgi?p=trakem2.git;a=blob;f=mpicbg/trakem2/transform/CoordinateTransformList.java

concrete for a 2d rigid transformation

http://fiji.sc/cgi-bin/gitweb.cgi?p=trakem2.git;a=blob;f=mpicbg/trakem2/transform/RigidModel2D.java

Our plan was always to have a cleaned up architecture of the mpicbg
transformations with the same principle goals migrated into ImgLib2.
This sounds like the right moment to getting this nailed.

> In real life, transformations on ROIs are rarely needed, unless
> accompanied by a transformation on the original image's space. In which
> case it is a space transformation applied to the ROI.
>

I have a different opinion about that. Many geometric primitives such
as hypercubes, spheroids or hyperspheroids can be expressed by a base
shape (e.g. a [hyper-]sphere of radius 1 or a [hyper-]cube between
(0,0,...) and (1,1,...) ) and an accompanying transformation. If some
of you remembers the fabulous RayTracer POVRay, its scene description
language does exactly that (not completely clean though ;)) with great
and intuitive representation of the weirdest geometric primitive
combinations. Read:

http://www.povray.org/documentation/view/3.6.1/273/

if interested. Doing so would save us from a lot of overcomplicated
constructs (such as a rotated rectangle which is just the 2d unit box,
some scale in x and y, a rotation and a translation) and keep us much
more flexible for extensions (e.g. bend this box or transform it
perspectively).

I am looking forward to see what has happened after Thursday.

I really regret not to be with you...

Cheers,
Stephan

[Josh Moore]

On Nov 12, 2012, at 12:38 AM, Johannes Schindelin wrote:

> Hi Josh,
>
> I hope we did not lose anybody in the conversation, but it is really too
> much to ask from me to re-instate the Cc:list.

The only other person on the To: list of your email was Roger, who I know to be on scijava. No one was lost by my reply.

> On Sun, 11 Nov 2012, Josh Moore wrote:
>
>> On Nov 9, 2012, at 6:11 PM, Johannes Schindelin wrote:
>>
>>> On Wed, 24 Oct 2012, Roger Leigh wrote:
>>>
>>> A general comment on the specification: it seems to be very focused on
>>> serialization.
>>
>> Though at first glance the tight, binary serialization that Roger has
>> proposed might be quite useful, I agree it's not a priority. In fact,
>> we'll likely have multiple serializations, since we'll need something
>> (unfortunately) that'll work well in XML, etc. etc. Nevertheless, I
>> think more of the specification is about getting the model API straight.
>> And this week is about hammering on it, modifying, repeating, etc.
>
> No. The focus needs to be on the use cases of ROIs. Otherwise we do not
> even *know* what to serialize.

I'd agree. For me, knowing what the model is is about knowing what code, i.e. what use cases, it will support.

> Just think about the following use case: a user chooses Li auto-threshold
> to select voxels.
>
> What to serialize? Li? The value Li calculated? The BinaryType mask
> representing the selected voxels? A geometric approximation of the
> hyper-volume?
>
> This is not a theoretical question. It has very real consequences on what
> we can, and what we cannot do, with the ROI model.
>
> In my opinion, we have to concentrate on *what* people need to do with
> regions of interest. For example, in the above use case, scientists
> frequently need to be able to store the fact that it was a Li
> auto-threshold. But to apply the same ROI to another channel, the binary
> mask must be transported.

This is fair, and certainly something that hasn't yet been taken into account. Would that kind of information be present at the ImgLib layer, or only closer to the user & the algorithms, say at the OMERO or ImageJ2 layers?

> However, my primary focus these two days lies purely in implementing ROIs
> on top of ImgLib2. Everything else would be premature, or too limited in
> applicability.

That's of course vital. And I wish I had been there to finally learn more about the ImgLib API. Nevertheless, as you said yourself, it's important to also think of all the use cases, which is why the other folks may well likely be looking at the problem from other angles.

>>> Applications I can think of right now:
>>>
>>> - serialization
>>> - visualization (AKA rendering)
>>> - processing (AKA iterating, both *inside* and *outside*)
>>> - set operations (i.e. merging, intersecting, set difference)
>>> - interaction (i.e. letting the users define ROIs)
>>>
>>> I am sure that I forgot some.
>
> No comments here???

I don't think anything I have is not particularly covered by your list. I can say that my key point is to make sure that the model can losslessly represent anything we will find in a pre-existing file or be told by a user via any arbitrary UI for the foreseeable future. That includes painstakingly defining what inside, outside, and set operations exactly mean such that we won't break that with a future release. But those are all subtopics of what you listed.

>>> The main concern I have for this meeting is that we might end up with
>>> something that is limiting us too much in the future.
>>>
>>> For example, we need to think far enough ahead to allow for
>>> backwards-compatible and future-proof extensions (as Tobias suggested,
>>> we need to define a way how to ignore unimplemented features safely).
>>
>> +1, though it's of course hard to know what one is forgetting.
>
> The point, of course, is to allow ourselves to forget something. We need
> to use duck-typing on the implementation side, and versioning on the
> specification side (although I sincerely believe that we have to get the
> implementation side right first before we can dare to think about the
> specification).

Fair enough. Looking forward to seeing the prototype then.

>> In terms of dealing with the unimplemented or undesirable features, I'd
>> vote for allowing one to choose between either the implementations (say
>> via a factory interface) or the error-handling itself (say via a
>> strategy interface) to allow for minimally "safe" and "unsafe" versions.
>
> No. The only way that proved practical was to use an interface-driven
> design, not a factory-driven one. I.e. "what can it do?" instead of "give
> me something, but I don't know what, exactly"

I'm all for an interface-driven design, but I don't think you will be to get around a factory or minimally a context (amounts to the same thing) in which one can configure choices between implementations, or alternatively configuration options for a single implementation.

>>> Another limitation we should avoid at all cost is the limitation to a
>>> certain number of dimensions (such as an inherent 2D model that
>>> somehow gets extruded to nD; my experience with the Virtual Insect
>>> Protocol shows that this is a terrible way to think about regions of
>>> interest).
>>
>> The proposal as it stands certainly has limitations. Some we discussed
>> this week, which Roger is either working on now, or we'll be mentioning
>> once everyone is in Dundee. But even with those changes, one of the
>> initial points of discussion will be the utility and mathematics behind
>> higher-dimensional shapes (i.e. beyond 3D). Very roughly, the updated
>> proposal will be that 3D shapes be combined with 3D transforms to
>> produce what we workingly called "Shape trees". There is one of these
>> per ROI along with an embedding in other higher dimensions.
>
> Again, this is too focused on low dimensions. 2D, 3D are not what we need
> to think. If we limit ourselves that way, we can stop thinking about a
> general ROI specification that will serve us well for decades to come
> right now.
>
> Instead, we need to focus on defining what a ROI can do. Completely
> independent of the number of dimensions of the space it lives in.

On the ROI level, I think we all agree. But on the shape level? Perhaps we table that until after the presentation.

>> We discussed 2 of the possibly several down-sides from others'
>> perspectives. The first is that one can't use an arbitrary MxN
>> transformation across all the dimensions in the shape tree. We couldn't
>> figure out the mathematics behind why that would be useful (or legal).
>> Perhaps someone could talk us through that. The second is that this form
>> of "N-3" restriction doesn't work with the ImgLib API.
>
> In real life, transformations on ROIs are rarely needed, unless
> accompanied by a transformation on the original image's space. In which
> case it is a space transformation applied to the ROI.
>
> Thinking about the problem that way lets us liberate ourselves from the
> very limited fixed-dimensional view of ROIs. We can say that a certain
> aspect some ROIs accomodate is to provide a
> RealRandomAccessibleInterval<ProbabilityType> instead. Much more useful
> when you have to work *with* the ROIs as opposed to just trying to save
> and load them.

Agreed. I certainly couldn't have listed the proper interface as here, but I agree that whatever it is you all say these things should implement, should be something generic. But as you said, it's also about making the work with the implementation straight-forward, e.g. making illogical things hard if not impossible, etc. Which is why I think having separate layers is beneficial: one in which we have API purity so that people working *with* the ROIs don't have to bend over backwards, and one in which we have model security so that things can explode and/or silently transform depending on the situation.

>> If we can agree that this second reason is actually the bigger issue,
>> then we'd propose:
>>
>> * Make the ROI package something external to ImgLib, e.g. package
>> "scijava.roi"
>
> There is little reason to make the ROI package independent of ImgLib.
> ImgLib is a versatile library to work with n-dimensional data, and ROIs
> are very much subsets of those data. Trying to overgeneralize will only
> cost us unnecessary gray hair.

I'm certainly fine for it to be "imglib.roi" but that wasn't really my place to suggest. But I would strongly argue that we don't want the ROI implementations (interfaces yes, but not the implementations) seeping throughout ImgLib core.

>> * Add Safe and Unsafe strategies as above so that if ImgLib tries to
>> perform a higher-dimensional transformation that's not supported, the
>> implementation can either throw (Safe) or blindly convert the
>> ROI+shapes to a NDim matrix (Unsafe).
>
> Again, transformations are not something inherently specific to ROIs. So
> they should not be part of the specification of ROIs. Instead, ROIs that
> can be transformed (my Li example from above cannot, unless realized into
> a specific RandomAccessible<BitType>) should implement an according
> interface.

Hmmm....our use of words might be off. Probably need to talk it through before debating.

>> For a division like that to work, it'll be important to nail down the
>> minimal number of ImgLib interfaces that this package (whatever the
>> name) MUST implement.
>
> Now we're talking.

Of course we are. ;)

>> There may already be too many ROI elements in ImgLib for this to be
>> feasible without burdensome refactoring;
>
> We are already sure that there needs to be refactoring. This is what this
> hackathon is about.
>
>> if so, back to the drawing board.
>
> Or alternatively, hacking. That is what a hackathon is good for.
>
> Remember, a hackathon is not about nailing down a design document. It is
> about hacking on code. If that code serves as a proof-of-concept, or as a
> demonstration why a particular approach does not work, does not matter.
>
> A hackathon is about code.

I think a hackathon is about what each of us needs it to be about, Johannes.

>> But in an effort to prevent everything from falling into ImgLib over
>> time, we'd suggest at least discussing a place for it in the
>> Dresden-DAG, similar to the RnR package.
>
> Since ImgLib2 is already shared between so many SciJava projects, it does
> not make sense to bend over to define things outside it.

As above: but it should be a proper subproject of ImgLib (like OPS??) such that it's later in the DAG.

> Instead, we should implement things on top of ImgLib2. Even better: there
> is already a ton of code out there, on top of ImgLib2 (for those
> unfamiliar with ImgLib2, I highly recommend to familiarize yourself with
> it, you will have to do that in the future anyway).

Agreed.

>> P.S. Another, secondary question might be: how will axis ordering/naming
>> be specified at the ROI level (say for transferring a ROI between 2
>> images)? What's the naming scheme used? But that seems like getting into
>> details at this point.
>
> Indeed.
>
> So far, the assumption of ImgLib2's core is that if you work on two images
> simultaneously, the axis ordering/naming is the same. And if it is not, it
> is very easy to use Views to make it so. Therefore I deem this an
> unconcern for the moment.
>
>>> And I would also like to caution against making the specification so
>>> centric on serialization.
>
> I really worry about this. If we focus too much on specification, we will
> end up -- again -- with no tangible code.
>
> Ciao,
> Johannes

~J.

[Tobias Pietzsch]

Hi all,

regarding the inside/outside-imglib question I would think that parts
of the implementation are definitely beyond imglib2.
I think there will be a set of interfaces in the core (if we even need
new interfaces, we'll see...) and a sub-project (similar to "ops" or
"algorithms")
that has some very basic implementations. Rendering and serialisation
(probably) and manipulation (definitely) are out of the scope.
Meta-information such as how the ROI was generated (the example
Johannes mentioned) is also out of scope.

In principle, the core should even be able to work with different
implementations of Rois (I know that's opposite to what this hackathon
is about, I'm just saying, in principle...).
I want to avoid to shove a full-blown
serialize/render/manipulate/...-implementation down everyones throat
who maybe just wants to use a bitmask for some toy project.
I think Josh is right, there should be a separate project.
And of course I also think Johannes is right, that we should nail down
the appropriate interfaces to imglib.

Please note, that I'm not suggesting to create a new project right
away (these two days will be over before we know it...). It may well
start as a imglib branch or whatever your choice is.

best regards,
Tobias

[Johannes Schindelin]

Hi Josh,

On Mon, 12 Nov 2012, Josh Moore wrote:

> On Nov 12, 2012, at 12:38 AM, Johannes Schindelin wrote:
>
> > In my opinion, we have to concentrate on *what* people need to do with
> > regions of interest. For example, in the above use case, scientists
> > frequently need to be able to store the fact that it was a Li
> > auto-threshold. But to apply the same ROI to another channel, the
> > binary mask must be transported.
>
> This is fair, and certainly something that hasn't yet been taken into
> account. Would that kind of information be present at the ImgLib layer,
> or only closer to the user & the algorithms, say at the OMERO or ImageJ2
> layers?

In our discussion today, Roger suggested that this could be a custom
property. So my current thinking is this: sampled (discrete) ROIs would
store bitmasks, with the parameters to recreate them (if possible) stored
as custom properties. Likewise, resolution-independent ROIs would be
stored using polygons, Bezier curves or meshes, with the parameters for
the actual ROI type (think Ricard's Snakes) stored in custom properties.

> > However, my primary focus these two days lies purely in implementing
> > ROIs on top of ImgLib2. Everything else would be premature, or too
> > limited in applicability.
>
> That's of course vital. And I wish I had been there to finally learn
> more about the ImgLib API. Nevertheless, as you said yourself, it's
> important to also think of all the use cases, which is why the other
> folks may well likely be looking at the problem from other angles.

Correct. It already proves very diverse and interesting.

> >>> Applications I can think of right now:
> >>>
> >>> - serialization
> >>> - visualization (AKA rendering)
> >>> - processing (AKA iterating, both *inside* and *outside*)
> >>> - set operations (i.e. merging, intersecting, set difference)
> >>> - interaction (i.e. letting the users define ROIs)
> >>>
> >>> I am sure that I forgot some.
> >
> > No comments here???

Things to add from the discussions:

- rasterizable
- transformable
- iterable
- boundary-iterable

Maybe we also need a concept of "is expensive to compute". For example
(did not come up in the discussions, but just occured to me), if
you want to dilate a ROI 5 times, it makes no sense to perform on-the-fly
operations because they get increasingly more expensive. You definitively
want to cache results there.

> >>> The main concern I have for this meeting is that we might end up
> >>> with something that is limiting us too much in the future.
> >>>
> >>> For example, we need to think far enough ahead to allow for
> >>> backwards-compatible and future-proof extensions (as Tobias
> >>> suggested, we need to define a way how to ignore unimplemented
> >>> features safely).
> >>
> >> +1, though it's of course hard to know what one is forgetting.
> >
> > The point, of course, is to allow ourselves to forget something. We
> > need to use duck-typing on the implementation side, and versioning on
> > the specification side (although I sincerely believe that we have to
> > get the implementation side right first before we can dare to think
> > about the specification).
>
> Fair enough. Looking forward to seeing the prototype then.

Me, too.

> >> In terms of dealing with the unimplemented or undesirable features, I'd
> >> vote for allowing one to choose between either the implementations (say
> >> via a factory interface) or the error-handling itself (say via a
> >> strategy interface) to allow for minimally "safe" and "unsafe" versions.
> >
> > No. The only way that proved practical was to use an interface-driven
> > design, not a factory-driven one. I.e. "what can it do?" instead of "give
> > me something, but I don't know what, exactly"
>
> I'm all for an interface-driven design, but I don't think you will be to
> get around a factory or minimally a context (amounts to the same thing)
> in which one can configure choices between implementations, or
> alternatively configuration options for a single implementation.

Beatiful thing is: we can always add that when we come across the need.
Tobias' preferred way is to add static methods, which are kind of
factories. (The difference is that you cannot pass such factories around.)

> > Instead, we need to focus on defining what a ROI can do. Completely
> > independent of the number of dimensions of the space it lives in.
>
> On the ROI level, I think we all agree. But on the shape level? Perhaps
> we table that until after the presentation.

Careful with the term "to table it"... means opposite things in British vs
American English ;-)

> I'm certainly fine for it to be "imglib.roi" but that wasn't really my
> place to suggest. But I would strongly argue that we don't want the ROI
> implementations (interfaces yes, but not the implementations) seeping
> throughout ImgLib core.

This is the strong opinion of Tobias and Christian as well. I'd still like
to keep the package name imglib2.roi to make certain people understand the
mind-set of it.

Ciao,
Dscho

[Josh Moore]

On Nov 12, 2012, at 2:43 PM, Johannes Schindelin wrote:

> Hi Josh,
>
> On Mon, 12 Nov 2012, Josh Moore wrote:
>
>> On Nov 12, 2012, at 12:38 AM, Johannes Schindelin wrote:
>>
>>> In my opinion, we have to concentrate on *what* people need to do with
>>> regions of interest. For example, in the above use case, scientists
>>> frequently need to be able to store the fact that it was a Li
>>> auto-threshold. But to apply the same ROI to another channel, the
>>> binary mask must be transported.
>>
>> This is fair, and certainly something that hasn't yet been taken into
>> account. Would that kind of information be present at the ImgLib layer,
>> or only closer to the user & the algorithms, say at the OMERO or ImageJ2
>> layers?
>
> In our discussion today, Roger suggested that this could be a custom
> property. So my current thinking is this: sampled (discrete) ROIs would
> store bitmasks, with the parameters to recreate them (if possible) stored
> as custom properties. Likewise, resolution-independent ROIs would be
> stored using polygons, Bezier curves or meshes, with the parameters for
> the actual ROI type (think Ricard's Snakes) stored in custom properties.

My strongest desire would be to make the custom properties as generally consumable as possible.

Guess I was thinking we'd primarily complain (i.e. raise an exception) if slow things were done, unless it was done explicitly. But I could imagine an interface to inspect "is expensive" though don't know how to do that well a priori.

Yeah, I'm anti-static methods due to extensibility limitations.

>>> Instead, we need to focus on defining what a ROI can do. Completely
>>> independent of the number of dimensions of the space it lives in.
>>
>> On the ROI level, I think we all agree. But on the shape level? Perhaps
>> we table that until after the presentation.
>
> Careful with the term "to table it"... means opposite things in British vs
> American English ;-)

Interesting. Will have to be more careful.

>> I'm certainly fine for it to be "imglib.roi" but that wasn't really my
>> place to suggest. But I would strongly argue that we don't want the ROI
>> implementations (interfaces yes, but not the implementations) seeping
>> throughout ImgLib core.
>
> This is the strong opinion of Tobias and Christian as well. I'd still like
> to keep the package name imglib2.roi to make certain people understand the
> mind-set of it.

Perhaps start there, and see where we end up? If the package grows sufficiently, then perhaps we'll want to rename.

~J.

> Ciao,
> Dscho