thoughts about editing control points

[kfj]

Control point editing/visualisation is a core issue of the panorama
creation mechanism used in hugin. It would deserve being focused on
more. I'd like to advise caution and extensive discussion of the
matter, and I've some ideas to share. When I first tried to get my
head around it, the conceptual difficulties were great - particularly
those arising from confusion about which features would be part of the
input space and which part of the output space. I fear that if some
quick fix is applied to add features in either the CP editor tab or
the preview, the situation might not improve.

This post started out in another thread:
http://groups.google.com/group/hugin-ptx/browse_thread/thread/dbb85dad10715586/06bc804cf2d3a234#06bc804cf2d3a234
But I felt the topic deserved a new thread, so here it is.

The CP editor tab shows images in their original projection. Many
users will not be aware that there is choice involved here - they have
rectilinear images, lines look like lines, why can't they just draw
lines, either on one image or on several, why not show lines in the
editor? Features on overlapping images look the same, why can't they
just overlay them somehow? Even seemingly rectilinear images, though,
aren't perfectly free from distortion (a, b, c) and with a more wide-
angled lens, lines towards the image's edges may well become visibly
bent. Other users use different lenses, be they fisheye or
stereographic, to maybe make 360 degree panos. These users would be
well aware of the bent straight lines - after all it's extremely
obvious in their case. They suffer from the opposite problem: Often
parts of the images they try to match are so distorted or tilted they
can't figure out what is where and offering them the images as
captured by the sensor makes editing CPs at times excruciatingly
difficult.

Displaying the native images, next to each other, in separate images,
makes sense when there is no initial notion of the images' connection.
This would be the case in a setup without an automatic CPG, or in a
situation where automatic CP generation fails utterly. I would say
that these situations are now rather the exception than the rule:
automatic CPG are quite reliable, and hugin now has a distributable
CPG, so there is at least one CPG available. To find oneself in a
situation where CP editing has to start from scratch is now mainly a
thing of the past; today it's more a matter of finetuning the CPG's
result.

The preview on the other hand shows what the output would look like,
given the current position and warping of the input images, and
projected according to the chosen output projection. Again, different
users will deal with different types of images, where for example,
all, or only some straight lines will show up straight, and there are
uses where large distortions are deliberate - noone would want to use
those projections to edit control points with: The preview should be
that - a preview of the final image. This is why I have at times
wished for a preview-like display to edit CPs in rather than CP
editing in the preview.

Considering these two intentions: working on the input images to find
and/or manipulate control points, and previewing the output, there is
a distance to travel, and the question is, how this distance can be
made navigable for everyone's benefit.

Having considered the matter thoroughly, I feel that displaying the
input images in the CP editor has relatively little merit. Direct
visualisation of the input images would be useful to get a quick idea
of what they look like - like you would navigate by looking at
thumbnails when looking at a folder full of images. Such a facility
would logically belong into the images tab - most file managers, image
databases etc. will give you at least an option to see thumbnails of
the objects they present to you, so as to allow you to do what you
want in a GUI: point and click. It should be no different in a GUI for
panorama creation, and hugin's way of showing a single thumbnail of
the currently selected image is nice, but there should be an option to
show thumbnails of all images at the same time, and I feel the UI
would benefit from that.

When it comes to editing control points, the usefulness of displaying
images in their native projection becomes more limited. I also feel
that having to work on two separate images by default is at times
awkward, particularly with the current UI, which doesn't seem very
intuitive to me - it feels more like a relic from another time. My
idea would be to use only one standard projection for the purpose
which lends itself particularly well to the task and can be
efficiently calculated, and only one view. As far as standard
panoramic work is concerned, I feel that equirectangular projection is
best suited. Sine CP editing is mainly close-up work, the distortions
would be small, and the most-commonly used line of reference - the
horizon - comes out as a line in an equirect if all is well. The
projection could be done just like in the preview window. The images
could be shown in the position they are currently thought to be in,
there would be a mechanism to focus on individual images, pairs,
groups of images and the lot to set points on one, two or several
synchronously. I imagine controlling visibility and focus with an
interface like a mixing desk, where you have solo buttons, mute
buttons etc. - an extension of the switch on/off buttons in the
preview window. CP setting would be in this display, with an interface
which should be well thought-out and discussed and quite definitely
new and not just the existing preview with a few features added. The
idea is to always use this particular projection for CP editing and
choose it for that purpose, not giving any choice in the matter - with
the notable exception of mosaic mode, which would require a
rectilinear display. The CP editor should allow zooming with good
resolution and offer several overlay modes, like the 'normal' and
'difference' modes in the preview.

The preview, finally, is well as it is. There are even two of it
already. These two may not be used so much anymore if the CP editor
can be used as an equirect and rectilinear preview as well, but they
wouldn't do any harm either - and to display the whole rich set of
output projections and faithfully show a preview of the final product,
their existence is well justified. Some editing capability which can
be conceptualized more easily as belonging to the output space could
remain in them.

Maybe a 'new CP editor' could start out alongside the current one,
just as the openGL preview has come up alongside it's elder brother,
as an experimental feature, to gain experience and feedback. Take it
as my item on the wishlist.

with regards
Kay

[Yuval Levy]

Hey Kay,

I finally have time to catch up with these and I agree with most of what you
write about editing control points. But..

But the option need to be available. Let's relegate the CP tab to a context
menu option on the connection between two images, as displayed in the layout
mode.

> The preview on the other hand shows what the output would look like,
> given the current position and warping of the input images, and
> projected according to the chosen output projection. Again, different
> users will deal with different types of images, where for example,
> all, or only some straight lines will show up straight, and there are
> uses where large distortions are deliberate - noone would want to use
> those projections to edit control points with: The preview should be
> that - a preview of the final image.

and a navigation tool to zoom in on those areas of the images that still need
manual work of some sort.

agree

> which lends itself particularly well to the task and can be
> efficiently calculated, and only one view. As far as standard
> panoramic work is concerned, I feel that equirectangular projection is
> best suited.

disagree. Try placing a CP on the nadir.

> Sine CP editing is mainly close-up work, the distortions
> would be small

you assume rotating the sphere so that the point of interest is around the
horizon, where the distortion is indeed minimal; but this is also true of
every other perspective. If we go this way of zooming into close-up work,
then I would rather have that projection rectilinear (think: straight lines)
or stereographic (think: reducing distortion).

> , and the most-commonly used line of reference - the
> horizon - comes out as a line in an equirect if all is well.

wrong. the horizon is not the most commonly used line of reference. On many
pictures it does not appear (interiors!). I don't think we can make any
assumption about most commonly used lines of references. Verticals are for
interiors. Horizons for flat outdoors. And sometimes there is neither nor.
Rectlinear projection served both of them equally well (and can be used also
for horizontal lines parallel to the Horizon).

Agree with the rest.

Yuv

[kfj]

On 25 Dez., 17:35, Yuval Levy <goo...@levy.ch> wrote:
> Hey Kay,
>
> I finally have time to catch up with these and I agree with most of what you
> write about editing control points.  But..

Ha! I thought noone would ever take notice of my outpourings...

> But the option need to be available.  Let's relegate the CP tab to a context
> menu option on the connection between two images, as displayed in the layout
> mode.

sounds reasonable.

> > ...

> > which lends itself particularly well to the task and can be
> > efficiently calculated, and only one view. As far as standard
> > panoramic work is concerned, I feel that equirectangular projection is
> > best suited.
>
> disagree.  Try placing a CP on the nadir.

I haven't made myself clear here. I thought to use a view where the
projection axis remains in the center of the display. And indeed as
you point out

> you assume rotating the sphere so that the point of interest is around the
> horizon, where the distortion is indeed minimal;  but this is also true of
> every other perspective.  If we go this way of zooming into close-up work,
> then I would rather have that projection rectilinear (think: straight lines)
> or stereographic (think: reducing distortion).

The diference, particularly if closed up, wouldn't be too great.
Probably the rectilinear projection would be most familiar, and look
most like what most users input. I was just thinking of making it
computationally efficient and thought equirectangular would be the
most efficient one. Shouldn't make that much difference, though?

>
> > , and the most-commonly used line of reference - the
> > horizon - comes out as a line in an equirect if all is well.
>
> wrong.  the horizon is not the most commonly used line of reference.  On many
> pictures it does not appear (interiors!).  I don't think we can make any
> assumption about most commonly used lines of references.  Verticals are for
> interiors.  Horizons for flat outdoors.  And sometimes there is neither nor.  
> Rectlinear projection served both of them equally well (and can be used also
> for horizontal lines parallel to the Horizon).

You are right. I must have failed to see this because I mainly do
landscapes.

Kay

[Tom Sharpless]

Thanks, Kay,

I too think it is time for us to take a hard look at the whole problem
of aligning source images.

Editing control points is one important aspect of that problem.
Finding CPs in the first place is a more fundamental one. Your
suggestions about focusing on local areas where images actually
overlap is, I think, the key to both.

It is amazing to me that Hugin (and the other PT based stitchers)
still treat the alignment problem as if the source images were taken
in random directions, when in fact all serious panographers take a lot
of trouble to orient their views systematically. Simply taking
advantage of what the photographer already knows about the alignment
would eliminate a great deal of useless computation, and make it
feasible to stitch large arrays automatically (inability to do that is
one of Hugin's major weaknesses). Furthermore, it would let Hugin
show us detail views of the overlapping areas right from the outset;
then after some manual adjustments the CP finding problem should be
almost trivial. Or CPs might even be bypassed altogether in favor of
a direct optimization based on correlating patches within the
overlaps.

I strongly support the suggestion that any such detail views should be
re-projected to rectilinear (or stereographic, for wide zoomed-out
views). I'd suggest two ways to make those views even more useful: 1)
the center of the rectilinear (or stereographic) projection should be
at the middle of the region of interest (i.e. the 'overlap patch' we
are examining); 2) any lens corrections we already know about should
be applied. BTW if this were done, the 2 views of the overlap could
then be submitted to an auto CP finder with very high probability of
getting good CPs.

I concur with Yuv about the horizon not being generally useful.
Vertical lines are much more so, and should perhaps be given special
status. But in fact the most important thing is to make it as easy as
possible to designate the same straight line in both views, regardless
of orientation. If backed up with some automatic edge-finding, that
could make strong curved edges just as useful for alignment as
straight lines.

The general principle here is to let the user specify rough local
alignments, and have the SW refine them into a precise global
alignment. And to take advantage of pre-calibrated lenses and
shooting patterns.

Regards, Tom

On Dec 25, 11:35 am, Yuval Levy <goo...@levy.ch> wrote:
> Hey Kay,
>
> I finally have time to catch up with these and I agree with most of what you
> write about editing control points.  But..
>
> On December 16, 2010 09:22:29 am kfj wrote:
>
>
>
> > Control point editing/visualisation is a core issue of the panorama
> > creation mechanism used in hugin. It would deserve being focused on
> > more. I'd like to advise caution and extensive discussion of the
> > matter, and I've some ideas to share. When I first tried to get my
> > head around it, the conceptual difficulties were great - particularly
> > those arising from confusion about which features would be part of the
> > input space and which part of the output space. I fear that if some
> > quick fix is applied to add features in either the CP editor tab or
> > the preview, the situation might not improve.
>
> > This post started out in another thread:

> >http://groups.google.com/group/hugin-ptx/browse_thread/thread/dbb85da...

>  signature.asc
> < 1KViewDownload

[kfj]

On 25 Dez., 19:15, Tom Sharpless <tksharpl...@gmail.com> wrote:

> I too think it is time for us to take a hard look at the whole problem
> of aligning source images.
>
> Editing control points is one important aspect of that problem.
> Finding CPs in the first place is a more fundamental one.  Your
> suggestions about focusing on local areas where images actually
> overlap is, I think, the key to both.

Maybe even thinking in terms of control points is a limiting bias.
What is really needed is making parts of warped images overlap. One
has to consider where the whole methodology of the current CPGs like
autopano-sift-c and panomatic actually comes from: If I'm not
mistaken, SIFT and SURF originate from pattern recognition in AI. They
were made to find common points of reference in images which were
decidedly not taken from the same spot with good knowledge of the
camera position (or a reasonable guess) - more for autonomous robots
trying to navigate a real environment. This is why they were designed
to allow for invariance to affine transformations - which is precisely
the parameters you want to be immune against when moving the camera
(hoping parallax won't ruin it for you). Using them for CP generation
is more of a spinoff:

- lens distortions locally look very similar to affine transformations
- many users shoot handheld, introducing corresponding errors some of
which SIFT and SURF cope well with
- different lenses in a project make use of the scale invariance

so, SURF and SIFT work for us, but, especially in a very properly set
up take with pano head and well-calibrated lens, they may be overkill.
On the other hand, there are ceratin drawbacks to SURF and SIFT when
using them in a photographic context, especially with fisheye lenses.
(I'm a bit out of my depth here, so please correct me if I'm
mistaken). The problem is that any convolution on a pixel matrix
yields results that are determined by the geometry of the pixel
matrix. The assumption here is of course that the pixel matrix is a
set of equidistant sample points, which, in the case of a fisheye
image, and towards it's edges, is true for the sensor, but not the
corresponding points in the captured scene. To make convolution
results comparable, you either have to do mathematic trickery (like
making your detector invariant to affine transforms), or you have to
reproject the image locally so the convolution kernel will see the
same image geometry. If you know all relevant parameters and have just
one shared point of reference, you can reproject two images to each
have their projection axis going to the shared reference point. If you
match these reprojected images, you should be able to get away with
less involved mathematics since you don't have to be transform- or
scale-invariant, just rotation-invariant. I have the feeling that a
very fast and efficient semi-manual detection could be set up for well-
parametrized situations where the user would only have to supply one
CP, and the software could take it from there.
I have actually experimented a bit here, reprojecting images from my
Walimex 8mm stereographic fisheye by shifting the projection axis so
that the axes go through corresponding image points and running the
matching process on these reprojected images. My results are
inconclusive - The quality of the matching even without reprojection
is extremely good with autopano-sift-c in full scale, so I din't look
much further in that direction, being happy with what is there
already. At lower resolutions and with other CPGs, I felt the CP
detection was better (particularly, more evenly distributed) after
such a reprojection. Maybe further investigation and play in that
direction would be rewarding.

Another drawback of relying on control points is their one-dimensional
nature. Of course, in case of the match between two SIFT/SURF feature
points, the environment of the points is present in the feature
vectors, but everything else in the further distance may be totally
unaligned. So to have a good result, more CPs are generated. But these
won't be evenly distributed, and if the projection and camera position
is known, an analysis of the assumed overlapping region should be
possible which makes a clearer statement of the quality of the overlap
than a few feature point correspondences.

> It is amazing to me that Hugin (and the other PT based stitchers)
> still treat the alignment problem as if the source images were taken
> in random directions, when in fact all serious panographers take a lot
> of trouble to orient their views systematically.

do keep in mind though that a portion of the users (I'm not sure about
how big a portion...) are either

- casual
- non-technical
- inexperienced
- have accidentally got it wrong
- or unable to be extremely precise due to circumstances

I, for example, often take handheld shots because I'm out in the
wilderness trecking and can't drag a tripod around with me because I'd
rather have something to eat in my pack.

These users musn't be left behind just because there is also a fair
portion of 'serious panographers'. In fact, the tool should (and,
currently, does) adapt to different qualities of input, but if it
deals with the serious panographer's input, it could maybe be put into
a mode with fewer degrees of freedom to exploit the comfortable data
situation.

> ...

> The general principle here is to let the user specify rough local
> alignments, and have the SW refine them into a precise global
> alignment.  And to take advantage of pre-calibrated lenses and
> shooting patterns.

Yes. I agree. And have proper pattern matching as a fallback option if
there is a lack of known constraints.

Kay

[Yuval Levy]

Hey Kay, Tom, and all the others with sensible detailed vision but lack of
implementation time / skill.

On December 25, 2010 12:33:22 pm kfj wrote:
> Ha! I thought noone would ever take notice of my outpourings...

don't worry, I read selectively based on the subject line and the person
posting; and since your posting tend to be interesting, I tend to read them.

The problem with such outpourings is that there is very little that I (or you)
can do about them in the immediate here and now. Those are thoughts that
deserve longer consideration, discussion, and eventually planning and
implementation.

Launchpad has a feature for that, called Blueprints.

Go ahead and flesh your ideas as Blueprints at [0]. This will give them a
better chance to survive the unstructured chatter of a mailing list with short
memory. Eventually I hope that we will use the Blueprints feature to flesh
out future Google Summer of Code or similar projects, when a student has about
three months time to work full time on making a Blueprint reality.

Yuv

[0] <https://blueprints.launchpad.net/hugin>

[kfj]

On 26 Dez., 20:06, Yuval Levy <goo...@levy.ch> wrote:

> Launchpad has a feature for that, called Blueprints.
>
> Go ahead and flesh your ideas as Blueprints at [0].

Thanks for the hint. The more I see of Canonical's services the better
I like them. I'll investigate.

Kay