Region of Interest (ROI) for Control Point Finding/Image Alignment?

[Daniel Reetz]

Hi All,

AFAIK, Hugin always searches the entire area of each image for control
points. I use Hugin for all kinds of image registration [1], and often
I would get much better results if I could specify a "region of
interest" where the CP generator could search.

Is this possible with the present version of Hugin? Is there a trick
to getting the Align Images step to search in a limited area in each
image?

Of course I can set CPs manually, but it's a lot of work.

Thanks,
Daniel Reetz

[1]I really love using/abusing Hugin as an image registration tool. In
particular, I like to take in sequences of images, register them all
to one plane in the image, and then average the result for "synthetic
focus":

Poor example:
http://danreetz.com/for_hugin/NissanSynAp.jpg

Better example with custom software my friend Matti wrote (Warning, large!):
http://www.futurepicture.org/images/sinclair_1_full.jpg

[Jeffrey Martin]

I was thinking also...

for panos, the centers of images usually do not overlap.

isn't searching the WHOLE of images a big waste?

again for 4 image panos this is not a big deal. but for 20 or 200 images, it is very significant.

just a thought :)

[kfj]

On 13 Feb., 11:13, Jeffrey Martin <360cit...@gmail.com> wrote:
> I was thinking also...
>
> for panos, the centers of images usually do not overlap.
> isn't searching the WHOLE of images a big waste?

It seems wasteful, but you also have to consider the time you have to
invest to pre-position the images roughly so the ROI can be detected
as roughly that area where the pre-aligned images overlap - and the
time it takes to extract/mask the ROI. Scanning the whole image
assures that the images can be positioned and oriented in any way and
all overlaps will be found.

There is another aspect which becomes more important with fisheyes and
stereographic images: if you feed the CPG with a partial image, it
can't decipher the warp to adapt it's feature point detector: the
image geometry is quite different from center to margin. So you can't
just use partial images, but you have to use the whole images and mask
them appropriately. Masking means introduction of additional data into
the process, either in the shape of a separate mask or as an alpha
channel. Using an alpha channel might be a reasonalbly inexpensive and
transparent (hah) way of doing the needful, but I'm not sure if the
current CPGs honour alpha channels - I rather doubt it.

Finally, if you use a reasonable overlap of 30% and calculate how much
of your images is left over in areas which are not overlapping, you
may find out that these areas are quite small after all, since the
overlap is on all margins. Even with 25% overlap, much less than half
of the image is outside overlaps.

If you take all of this into account, I think the gain is not worth
the effort for everyday work. On the other hand, there may be special
situations where the savings would be significant. To cater for these,
the mechanism of limiting the scan for feature points to a ROI should
be available as an optional feature. In fact, this sounds like an
ideal scenario for a plugin. I'd expect stuff like this to be among
the first things to be implemented as a plugin as soon as the plugin
facility becomes maintream (currently verification of the implemented
mechanism on Mac OS is pending, but I hope it won't be too much
longer).

I am currently toying with this mechanism for use in another demo
plugin, but I want to throw in rewarping of the parts of the images
that correspond to the ROIs to a common projection to make them
geometrically as similar as possible, thus improving CPG performance
especially with fisheye images and avoiding warp-related CPG problems
- some sort of high-end matching which would produce very good
quality, well-distributed CPs, particularly for applications like lens
calibration. Also, it would be nice to get quick access to the warped
partial images for visual inspection.

To put a final tag on it - nice to have, but not crucial.

Kay

[Yuval Levy]

On February 12, 2011 10:38:15 pm Daniel Reetz wrote:
> AFAIK, Hugin always searches the entire area of each image for control
> points. I use Hugin for all kinds of image registration [1], and often
> I would get much better results if I could specify a "region of
> interest" where the CP generator could search.

Or specify a "region of interest" and discard all CPs generated outside it.

> Is this possible with the present version of Hugin? Is there a trick
> to getting the Align Images step to search in a limited area in each
> image?

Not AFAIK, but there are some elements that can be "re-purposed" for your
application.

If I understand your application correctly, you would like to have CPs
generated on the object to synthetically focus on. You will need to mask that
object in one way or another? Hugin has a mask editor. The one function that
would need to be implemented to achieve your expeted result is pruning CPs
that are outside (or inside) a selected mask. Not excessively difficult to
implement.

There are other pruning mechanism within Hugin as well, but they may not be
helpful in your case.

Celeste prunes control points from the sky. Initially we wanted to exclude
the sky from the CP generation, but it turned out to be computationally more
efficient to do it the other way around.

"clean control points" (a button on the Images tab) prunes statistical
outliers. Must press the button a few times. I found it to be excellent at
removing CPs from moving subjects; and very good at removing CPs from
foreground subjects (those pesky CPs that are more parallax-affected than the
far away CPs).

> Of course I can set CPs manually, but it's a lot of work.

You don't want that. That said, adding CP's manually in Hugin is a very fast
process. Click left, click right, right-click. You don't have to position
your mouse perfectly on the spot on the left and right image. On right-
clicking Hugin fine-tunes the location of the CP, so you just need to target
the approximate area (+/- 10pixels).

Yuv

[john doe]

OpenCV can do this...

[Carl von Einem]

Yuval Levy schrieb am 13.02.11 18:21:

> On February 12, 2011 10:38:15 pm Daniel Reetz wrote:
>> AFAIK, Hugin always searches the entire area of each image for control
>> points. I use Hugin for all kinds of image registration [1], and often
>> I would get much better results if I could specify a "region of
>> interest" where the CP generator could search.
>
> Or specify a "region of interest" and discard all CPs generated outside it.
>
>> Is this possible with the present version of Hugin? Is there a trick
>> to getting the Align Images step to search in a limited area in each
>> image?
>

> If I understand your application correctly, you would like to have CPs
> generated on the object to synthetically focus on. You will need to mask that
> object in one way or another? Hugin has a mask editor. The one function that
> would need to be implemented to achieve your expeted result is pruning CPs
> that are outside (or inside) a selected mask. Not excessively difficult to
> implement.

Isn't this already achieved with "Remove control points in masks" from
the 'Edit' menu?

Carl

[Yuval Levy]

On February 13, 2011 01:03:02 pm Carl von Einem wrote:
> Isn't this already achieved with "Remove control points in masks" from
> the 'Edit' menu?

Duh! Of course. Shows how much I have been using the masking function... not.

Thanks for completing the info.
Yuv

[Bob Bright]

On 11-02-13 09:21 AM, Yuval Levy wrote:
> On February 12, 2011 10:38:15 pm Daniel Reetz wrote:
>> AFAIK, Hugin always searches the entire area of each image for control
>> points. I use Hugin for all kinds of image registration [1], and often
>> I would get much better results if I could specify a "region of
>> interest" where the CP generator could search.
> Or specify a "region of interest" and discard all CPs generated outside it.
>
>
>> Is this possible with the present version of Hugin? Is there a trick
>> to getting the Align Images step to search in a limited area in each
>> image?
> Not AFAIK, but there are some elements that can be "re-purposed" for your
> application.
>
> If I understand your application correctly, you would like to have CPs
> generated on the object to synthetically focus on. You will need to mask that
> object in one way or another? Hugin has a mask editor. The one function that
> would need to be implemented to achieve your expeted result is pruning CPs
> that are outside (or inside) a selected mask. Not excessively difficult to
> implement.

I believe cpfind is already exclude mask aware. Just set an exclude
mask on any parts of an image where you don't want control points, run
cpfind, and you'll get control points only in regions that haven't been
excluded.

Here's an example:

http://VictoriaVR.ca/tmp/cpfind-without-mask.jpg
http://VictoriaVR.ca/tmp/exclude-mask.jpg
http://VictoriaVR.ca/tmp/cpfind-with-mask.jpg

Cheers,
BBB

--
Bob Bright
Vancouver Island Digital Imaging
(250) 857-9887
BBBr...@VictoriaVR.ca
http://VictoriaVR.ca

[Daniel Reetz]

Thanks, everyone, for the help and suggestions. I'm going to go
further with using the built-in mask facilities and see what I can
come up with.

Regards,
Daniel Reetz

[Jeffrey Martin]

Hmm yes. Honoring a mask/alpha channel seems a very reasonable way to make this work.

[Pablo d'Angelo]

Am 13.02.2011 16:25, schrieb kfj:
>
>
> On 13 Feb., 11:13, Jeffrey Martin<360cit...@gmail.com> wrote:
>> I was thinking also...
>>
>> for panos, the centers of images usually do not overlap.
>> isn't searching the WHOLE of images a big waste?
>

> There is another aspect which becomes more important with fisheyes and
> stereographic images: if you feed the CPG with a partial image, it
> can't decipher the warp to adapt it's feature point detector: the
> image geometry is quite different from center to margin. So you can't
> just use partial images, but you have to use the whole images and mask
> them appropriately. Masking means introduction of additional data into
> the process, either in the shape of a separate mask or as an alpha
> channel. Using an alpha channel might be a reasonalbly inexpensive and
> transparent (hah) way of doing the needful, but I'm not sure if the
> current CPGs honour alpha channels - I rather doubt it.

cpfind does honor alpha channels and masks defined in the pto file.

It removes any matches found in the alpha channels (this is needed for
remapped images, otherwise one could get "false" matches at the
corners/edges of a remapped image.)

It will still analyse all parts of the image, though, so there is not
much computational benefit. I think the typical savings are not really
worth the coding effort.

> I am currently toying with this mechanism for use in another demo
> plugin, but I want to throw in rewarping of the parts of the images
> that correspond to the ROIs to a common projection to make them
> geometrically as similar as possible, thus improving CPG performance
> especially with fisheye images and avoiding warp-related CPG problems
> - some sort of high-end matching which would produce very good
> quality, well-distributed CPs, particularly for applications like lens
> calibration.

Yes, something like that would be very nice. Or one could try Least
squares matching. (estimates local affine transformation as part of the
matching process, is typically accurate to ~ 0.1 pixel). However, it
needs good initialisations (1-2 pixels error, good estimate for the
initial affine transform).

ciao
Pablo

[kfj]

I'm getting ahead slowly with the code for my plugin, because the
precise masking out of the non-overlapping parts isn't trivial. Sadly
the code for masks in Mask.cpp and Mask.h is totally undocumented. My
initial attempt was this:

If I have an overlap of two images, I sample one image's boundaries,
transform that to the other image's coordinates and use it as a mask
for the other image. This would be fine if I could use what's outside
of the image as a mask as exclude mask, but the obvious flag in
Masks.cpp ( Mask.setInverted() ) doesn't seem to do what I want, the
Mask is forever just the inside of the Polygon.

So I'm now writing python code for the masks, which is cleaner anyway
than my quick-and-dirty approach (it's even fun) - but it' sad that
the Mask code isn't a bit more user-friendly.

Once I get it to run I'll post again.

Kay