Improving the Optimizer
Tom Sharpless
I'm planning some changes to the libpano13 optimizer, used by hugin,
aimed at making it easier to calibrate fisheye and ultrawide lenses,
and possibly improving overall results on that kind of image. I
invite your comments and suggestions. Here are my current ideas:
Motivation
The current PT optimizer measures aligment errors of
ordinary control points in an equirectangular projection,
but uses the user-selected pano projection for evaluating
errors of horizontal, vertical and straight-line CPs.
This needlessly limits the usefulness of the "line" CP
types to cases where the line is indeed straight in the
final panorama -- in practice mainly rectilinear panos.
In fact it should be possible to put straight-line CPs
on any line that is straight in the real world, and have
the optimzer take its true straightness into account, no
matter how it curves in the input or output images.
Straight line constraints are especially valuable for
estimating the lens correction parameters. A generalized
straight-line CP capability would make it easier to apply
the straight line calibration method to fisheye lenses.
The existing code for straight line control points also
does not really measure the error correctly; improving
that should make the fitted lens parameters more reliable.
It is important for optimizer stability that all errors
be measured in the same units, using identical projection
parameters. The current PT optimzer code has a lot of
complexities which make it hard to verify that. So another
goal is to use a single projection function for evaluating
all CP errors.
Method
Straight lines in the real world correspond to great circles
on the panosphere, or equivalently to planes through the
center of that sphere. These can be fitted by projecting
the control points onto the sphere in Cartesian coordinates,
and using the singular value decomposition to find the direction
of minimum variance, which is the normal to the least-squares
plane, equivalently the axis of the least-squares great circle.
The distance of each point from this plane is the appropriate
error for the optimzer.
The projection has to take into account all of the mapping
parameters, including the position of the optical axis, and
the alignment of the images (in case CPs for one line have been
placed on more than one image). So the only reasonable way
is to rely on the PT mapping code, as the present evaluation
routines do. The needed Cartesian spherical coordinates are
in fact computed in an intermediate stage of the PT mapping,
it is just a matter of capturing them.
The current practice of measuring alignment errors in the
equirectangular pano space may or may not be a good idea. In
theory it is wrong because that space is not isometric and
greatly magnifies errors at large vertical angles. But it does
work, and has the advantage of automatically splitting the
error into two orthogonal components -- something the optimizer
now relies on to greatly speed final convergence.
In theory it would be best to evaluate all errors as arc lengths
on the panosphere, or as the corresponding distances in 3D
Cartesian coordinates (which are easier to compute, and to split
into orthogonal components). It is quite possible that this would
improve optimization results with wide angle and fisheye images;
so I think it deserves to be tried.
It looks feasible to implement all of this by rewriting a single
function in adjust.c (the one that computes and posts error estimates
for all the control points) and adding some new subroutines for its
private use.
Cheers, Tom
Klaus
Adressing the way errors are computed, you have a valid point when you
suggest to computer the relative angle in solid angle space.
However, your suggestion to allow great circles only as straight
lines, I fear it is taking away functionality. I would like to see
that further discussed.
And the choice of projection influencing how horizontal and vertical
alignment points affect the alignment: I have done several panoramas
where I optimised alignment using one projection, and after alignment
switched to another output projection.
> In theory it would be best to evaluate all errors
> as arc lengths on the panosphere
I do second it.
Cheers
Klaus
Bruno Postle
>
>However, your suggestion to allow great circles only as straight
>lines, I fear it is taking away functionality. I would like to see
>that further discussed.
My understanding is that Tom's 'great circles' proposal only
concerns 'straight-line' control points and not horizontal/vertical
control points:
http://wiki.panotools.org/Horizontal_control_points
http://wiki.panotools.org/Vertical_control_points
http://wiki.panotools.org/Straight_line_control_points
These 'straight lines' currently are only much use in 'rectilinear'
output. I think that Tom is proposing that the 'rectilinear' case
will remain more-or-less the same, but straight lines will become
similarly useful with any output projection.
--
Bruno
Tom Sharpless
On Jul 23, 5:02 pm, Bruno Postle <br...@postle.net> wrote:
> On Wed 23-Jul-2008 at 13:48 -0700, Klaus wrote:
>
>
>
> >However, your suggestion to allow great circles only as straight
> >lines, I fear it is taking away functionality. I would like to see
> >that further discussed.
straight line is a great circle on the panosphere, so the proposed
change would only increase, not decrease, the utility of straight line
CPs. If a straight line has CPs in two or more images, then it will
also be an alignment constraint, much like h/v CPs but more general.
> My understanding is that Tom's 'great circles' proposal only
> concerns 'straight-line' control points and not horizontal/vertical
> control points:
solve specific alignment problems, and I would not want to break
that. But I guess far fewer use straight line controls.
>
> http://wiki.panotools.org/Horizontal_control_pointshttp://wiki.panotools.org/Vertical_control_pointshttp://wiki.panotools.org/Straight_line_control_points
> These 'straight lines' currently are only much use in 'rectilinear'
> output. I think that Tom is proposing that the 'rectilinear' case
> will remain more-or-less the same, but straight lines will become
> similarly useful with any output projection.
>
> --
> Bruno
Klaus
On 24 Jul, 00:13, Tom Sharpless <TKSharpl...@gmail.com> wrote:
> Hi Klaus & Bruno
>
> On Jul 23, 5:02 pm, Bruno Postle <br...@postle.net> wrote:
> > My understanding is that Tom's 'great circles' proposal only
> > concerns 'straight-line' control points and not horizontal/vertical
> > control points:
>
> Yes that is right. I think many people know how to use h/v CPs to
> solve specific alignment problems, and I would not want to break
> that. But I guess far fewer use straight line controls.
> > output. I think that Tom is proposing that the 'rectilinear' case
> > will remain more-or-less the same, but straight lines will become
> > similarly useful with any output projection.
world, which is different from a great circle. I have to admit I did
not use straight lines there yet. So far I have used straight lines
only to align cables, structures that are essentially translational
invariant in one direction. Choosing straight lines to represent great
circleson the panosphere is a different paradigma to h/v CPs where the
choice of projection has an influence on how they act.
But maybe the usefulness of great circles compensates for that
difference in behaviour.
Cheers
Klaus
Yuval Levy
> I think many people know how to use h/v CPs to
> solve specific alignment problems, and I would not want to break
> that.
actually true for v CPs only. h CPs are confusing to newbies who often
think of them as "parallel to the horizon" rather than "on the
horizon/equator of the sphere".
I agree with your analysis and look forward for the improved optimizer.
One thing I kindly as you to consider (and of which I have not thought
through how much complexity it would introduce): does it make sense to
separate input space lines from output space lines?
Yuv
Tom Sharpless
You have identified what will probably be the worst source of
confusion about using general straight line controls: the fact that
such lines don't necessarily look straight in either the input or
output images, which are both projections of the real world. A line
that is straight in the real world will be straight in an image only
if the image is an ideal rectilinear projection, or in certain special
positions for other projections.
What I'm talking about is using real world straightness as an
optimization constraint. In the ideal spherical projection that is
the "geometry reference space" for PanoTools, real world straight
lines correspond one-to-one with great circle arcs. That gives the
optimizer a "handle" on truly straight lines, regardless of the lens
and pano projections, which is just what is needed to optimize the
parameters of those projections.
I'm not suggesting that anybody needs to try to visualize those arcs.
To use a straight line control all you need to do is identify the
image of a straight line in your picture. Most people should find
that rather easy, even in fisheye images, because straight lines are
so ubiquitous and so important to us.
Regards. Tom
Klaus
I think it is confusing when straight lines are not a generalisation
of the h/v CPs. As it stands, they relate to the chosen output
projection, chosen at the optimisation time, which of course can be
changed before stitching.
<mode=Paxman>
What is the point in altering the software code, if you can get the
great circle functionality by simply switching to rectilinear output
at optimisation time?
</mode=Paxman>
One reason might be, that you want to use that funcionality with fov
>180 degrees. Does that happen often enough?
Cheers
Klaus
Tom Sharpless
On Jul 24, 11:42 am, Klaus <k...@ph.ed.ac.uk> wrote:
> Hi Tom,
>
> I think it is confusing when straight lines are not a generalisation
> of the h/v CPs. As it stands, they relate to the chosen output
> projection, chosen at the optimisation time, which of course can be
> changed before stitching.
straightness in the output projection. That is what I think needs to
be changed, for line controls anyway.
A "true straight line" optimizaton would make it possible to use line
controls as you suggest -- as a generalization of h/v controls. If
defined with CPs in two or more images, a true line control will force
alignment along that line, not just in some arbitrary "horizontal" or
"vertical" direction -- and regardless of whether the line is straight
in either the input or the pano: the only requirement is that it is
straight in reality.
> <mode=Paxman>
> What is the point in altering the software code, if you can get the
> great circle functionality by simply switching to rectilinear output
> at optimisation time?
> </mode=Paxman>
>
> One reason might be, that you want to use that funcionality with fov
>
> >180 degrees. Does that happen often enough?
like 125 degrees. Someone making spherical panos, as I mainly do,
could often make good use of straight lines that span a bigger angle.
But aside from that there is good reason to change the existing code:
it isn't correct, even for the limited applications it presently
supports. It assumes that every line contains exactly 4 control
points, in two pairs, even though that is not enforced by the GUIs --
and should not be, as the usefulness of a line control increases with
the number of control points on it. This results in some valid error
estimates being lost, and possibly some bogus ones being reported, and
must be considered a serious bug. Secondly, the reported errors are
only a rough approximation to the error that the LM algorithm is
designed to minimize, so the resulting fits will not be as good as
they might be.
>
> Cheers
> Klaus
Regards, Tom
Jim Watters
For straight lines to work it is necessary to have more than a pair of
points.
For h & v lines use one pair of points.
I am ok with h&v lines not being tied to the output projection. But I
still want to be able to do perspective correction by setting h & v
control points.
Evaluating all the points that make a straight line and giving an error
to each of the points should improve the accuracy of straight lines.
Having just looked at the current algorithm for straight lines I can see
room for improvement.
The current algorithm for straight lines takes a give pair of line
points and finds the first pair of line points that is not itself that
has the same id.
It uses these 4 points to find the furthest two points to make a line
and calculates the error of the other two points from this line.
The biggest problem with this is the first pair of points is always used.
Another area of improvement would to updating the fine-tune control
points to shift straight line points perpendicular to this line.
Something similar can be done with h & v points.
--
Jim Watters
jwatters @ photocreations . ca
http://photocreations.ca
Klaus
While one is tempted to see the straight lines as a generalisation of
the h/v CPs, I have to admit they are not.
h/v CPs (mostly) affect the general alignment of an image cluster on
the panosphere and less the relative image alignment. Their main
purpose is *orientation*.
The straight lines are quite close to standard CPs, at least in the
case of two points in image A and two points in image B. When these
images are located on a panosphere, each pair of points on one image
corresponds to a virtual CP defined by the normalised vector product
of the points' position vectors. Superpose the virtual CPs, and you
have *alignment*.
At least the happens if straight lines are defined as great circles,
as you propose. So you have a point in (re)defining straight lines
that way.
Just a little note: this virtual CP has a twofold ambiguity, as it
could be at either of two diametrically opposed positions on the
panosphere.
Cheers
Klaus
Tom Sharpless
On Jul 24, 2:37 pm, Jim Watters <jwatt...@photocreations.ca> wrote:
> I don't think we can change how h and v control points are added.
>
> For straight lines to work it is necessary to have more than a pair of
> points.
> For h & v lines use one pair of points.
>
> I am ok with h&v lines not being tied to the output projection. But I
> still want to be able to do perspective correction by setting h & v
> control points.
>
controls. H/v would remain tied to the output projection.
I know that many people find the h and v functionality very useful as
it is; nor can I foresee enhancement that would not be better served
by the true straight line control -- which should work like a
generalized h/v also.
> Evaluating all the points that make a straight line and giving an error
> to each of the points should improve the accuracy of straight lines.
>
> Having just looked at the current algorithm for straight lines I can see
> room for improvement.
> The current algorithm for straight lines takes a give pair of line
> points and finds the first pair of line points that is not itself that
> has the same id.
> It uses these 4 points to find the furthest two points to make a line
> and calculates the error of the other two points from this line.
> The biggest problem with this is the first pair of points is always used
spent time placing lots of control points on a line, to find that most
of them were being ignored. And this estimate of the error is not a
correct basis for least-squares minimization, either.
> Another area of improvement would to updating the fine-tune control
> points to shift straight line points perpendicular to this line.
> Something similar can be done with h & v points.
Regards, Tom
Tom Sharpless
Are you starting to agree with me? :)
In fact the whole line is a "virtual CP", but it will will (directly)
affect only those images in which its "real" CPS are located. There
is no ambiguity, the optimizer sees the errors in those points, not in
some virtual ones.
Cheers, Tom
Klaus
On 24 Jul, 21:59, Tom Sharpless <TKSharpl...@gmail.com> wrote:
> Hi Klaus,
>
> Are you starting to agree with me? :)
Seriously, with changes, one wins something and one looses something.
So I wanted to point out what one would loose implementing your
proposal, so it is not overlooked. It is not a veto, it is to prepare
the ground for an informed decision. No point for me in reiterating
your valid "pro" points.
Before we turn too overenthusiastic: is there a limit on the number of
straight lines?
Cheers
Klaus
Erik Krause
> You have identified what will probably be the worst source of
> confusion about using general straight line controls: the fact that
> such lines don't necessarily look straight in either the input or
> output images, which are both projections of the real world. A line
> that is straight in the real world will be straight in an image only
> if the image is an ideal rectilinear projection, or in certain special
> positions for other projections.
I don't think this is a show stopper. Confusion is far larger as it
is now: That a line has to be straight in output projection.
Apparently it is not easy to learn and understand that fact but once
you done so, you should understand the new concept even better. For
any new user of panotools it will be more intuitive and far more
versatile to have the new model. All those telephone wires and other
featureless line structures would be far easier to stitch...
best regards
Erik Krause
http://www.erik-krause.de
Klaus
...but that is a minor quibble only. If one places the linear CP close
together, it still very much helps with the alignment. Well, if you
look locally, there is not much difference between the old and the new
model, but conceptually it is easier to explain.
Cheers
Klaus
Tom Sharpless
On Jul 25, 4:42 am, Klaus <k...@ph.ed.ac.uk> wrote:
> Seriously, with changes, one wins something and one looses something.
> So I wanted to point out what one would loose implementing your
> proposal, so it is not overlooked. It is not a veto, it is to prepare
> the ground for an informed decision. No point for me in reiterating
> your valid "pro" points.
>
there is always a downside to changing anything that has been in use
for a while. However so far I have not seen any strong opposition to
the proposed change, and several people that I recognize as experts
have said it would be useful. So I am proceeding with it.
> Before we turn too overenthusiastic: is there a limit on the number of
> straight lines?
>
GUIs or their parsers impose any limits.
J. Schneider
> Having just looked at the current algorithm for straight lines I can see
> room for improvement.
> The current algorithm for straight lines takes a give pair of line
> points and finds the first pair of line points that is not itself that
> has the same id.
> It uses these 4 points to find the furthest two points to make a line
> and calculates the error of the other two points from this line.
> The biggest problem with this is the first pair of points is always used.
As I can't read the code myself, I try to understand from your
description what would be the best method to place line cps to have them
properly interpreted with the algorithm as it is.
Is it: 1. place a pair (1a+1b) at the ends of the distorted line to be
straightened and 2. place another pair a bit further inside (2a+2b)
( ----1a---------2a------------2b--------1b--)
or: place consecutive pairs starting from one end
(--1a ---------1b-----------2a-------2b--) ?
Are the 4 points of two pairs interpreted pairwise? I understand: no.
Does the problem occur only when there are more than 4 points?
I just started to use this feature and I am going to use it for the
seemingly pretty easy method of lens calibration described here some
days ago. So I would appreciate some help on how to make the most of it.
regards
Joachim
Tom Sharpless
Joachim,
I believe the first scheme (working inward from the ends) is
correct. That is what the Wiki tutorial on the straight line method
of lens calibrations specifies. And I have gotten very good
corrections for a fisheye lens using that placement, on an image of
several straight black threads taped to a white wall. However it
takes several optimizations; if you try to optimize all lens
parameters at once (including the center shifts d,e) the result is
usually far from correct. I have used the following sequence: first
optimize a,b,c; next d,e; finally all 5 together. I don't think it
makes sense to include the fov, v, when optimizing on a single image.
I save the optimized lens parameters and when aligning a panorama,
load them back and do not optimize a,b,c again. I do allow center
shift adjustment as on my camera there is a possibility of significant
change each time a lens is mounted. And I optimize fov, too, as that
seems the only way to get a clean 360 degree seam -- you have to be
watchful about that as changing the cropping circle may change the fov
parameter.
All,
I have begun testing "spherical" straight line optimization, and must
admit that it has problems. Although it agrees that the lens
parameters a,b,c,d,e found by the above procedure are in fact an
optimum, I have been unable to reach that optimum by any simple series
of optimizations -- the lens parameters are much more likely to "blow
up" with spherical errors compared to the rectilinear ones. This
"blowing up" is already a well known problem with PanoTools; and it
seems that using a more homogeneous error space only makes it worse.
It remains to be seen whether the spherical method helps or hurts the
use of straight line controls for image alignment.
If anyone would like to help with this testing, I have a package to
build the alternative libpano, using cmake scripts, that should work
on any system that can build hugin. I have linked hugin's
AutoOptimizer tool with this library for my first tests, but intend to
link a full version of hugin with it soon, so that I can use its
convenient GUI features, especially the optimizer tab. Let me know if
you would like to try it.
And if you have any ideas on how the lens parameter optimization might
be constrained to make it more likely to find the real optimum.
Cheers, Tom
Daniel M German
Tom> Hi Jim
Tom> On Jul 24, 2:37 pm, Jim Watters <jwatt...@photocreations.ca> wrote:
>> I don't think we can change how h and v control points are added.
>>
>> For straight lines to work it is necessary to have more than a pair of
>> points.
>> For h & v lines use one pair of points.
>>
>> I am ok with h&v lines not being tied to the output projection. But I
>> still want to be able to do perspective correction by setting h & v
>> control points.
>>
Tom> I don't propose to change how h and v controls work, just line
Tom> controls. H/v would remain tied to the output projection.
Tom> I know that many people find the h and v functionality very useful as
Tom> it is; nor can I foresee enhancement that would not be better served
Tom> by the true straight line control -- which should work like a
Tom> generalized h/v also.
Go for it Tom. Add it as an optional feature (i.e. create another
parameter that specifies if your new method is to be used or not, by
default it will work as usual, when set it will use your
method). Once it is in panotools people can start using if they want
to. No point on stopping a feature when it can be an extension to the
current ones (not a replacement).
--dmg
--
--
Daniel M. German
http://turingmachine.org/
http://silvernegative.com/
dmg (at) uvic (dot) ca
replace (at) with @ and (dot) with .
Tom Sharpless
I am indeed implementing everything as purely optional, with the
default being "do it the old way".
I could modify the hugin GUI to switch the new features on and off
from he preferences tab; however GUI changes in the trunk are
forbidden, pending release of version 0.7, due to a freeze on string
translations. Also since the optimizer is called from several hugin
modules in several executables, it might take me a while to find the
right place(s) in the hugin code to set the working control flags.
And that would not help people who use libpano via other apps.
For now what I plan to do is make available
-- a source package to build a hugin-specific subset libpano13
containing the true straight lines feature. This should work on any
system that can build hugin, as it uses a hugin-compatible cmake
script.
-- an "experimental use only" version of hugin for Windows that I have
linked with that library, with true straight lines enabled.
Anyone who knows how to build hugin should be able make that variant
on other platforms, and of course fiddle with the optimizer code.
Somebody who knows how to build libpano the regular way could easily
merge my changes into a "full" version that would support the PT
tools, PTAssembler, etc, as they are confined to 3 source files and
make no changes to the published API. But I don't think that should
happen unless and until the feature has been proved useful.
Regards, Tom
Daniel German
your implementation
working. It could be enabled using other methods, such as using an
environment variable,
or a small configuration file.
I don't think you need to define a new type of control points, do you?
At first release it as a patch, so some interested people can try it.
And don't worry about the freeze for Hugin. There is always a new
version next. But first
we need to see it working.
--dmg
Yuval Levy
Rather than a behavior switch in the GUI and the tools using the project
file, how about extending the specs of the project file itself at the
parser?
I am looking at the specs, shared by PTOptimizer and nona/hugin
The affected lines would be the 'c' lines, for the control points,
starting around line 232.
unfortunately it is not possible to extend the optional switch "t"
without changing its parameters type from the current unsigned integer,
so to avoid breaking existing parsers we'll have to leave it alone and
add a new optional switch, let's call it "s".
0 - default: old behavior
1 - CP in "real space", i.e. what you intend to implement
this will leave place for other types of CP such as output space CPs
Yuv
Daniel German
>
> Hi Tom,
>
> Rather than a behavior switch in the GUI and the tools using the project
> file, how about extending the specs of the project file itself at the
> parser?
>
> I am looking at the specs, shared by PTOptimizer and nona/hugin
>
> <http://panotools.svn.sourceforge.net/viewvc/panotools/trunk/libpano/doc/Optimize.txt?revision=777&view=markup>
>
> The affected lines would be the 'c' lines, for the control points,
> starting around line 232.
>
this is a very good idea. Hugin (if I remember correctly) lets you
edit the optimization script before it is send to the optimizer.
--dmg
Tom Sharpless
I thought Yuv's suggestion was brilliant, and then I thought "Drat, I
really want to set the control points in hugin, which won't know about
my hew switch". Now that Dan has mentioned it, I see for the first
time the little checkbox for "edit script before optimizing".
I shall put a new "m" line option in my modified libpano, so the
script can invoke the new way of handling straight line control
points. There will also be global control vars for an app to use.
Cheers, Tom
On Jul 28, 7:14 pm, "Daniel German" <d...@uvic.ca> wrote:
> On Mon, Jul 28, 2008 at 1:27 PM, Yuval Levy <goo...@levy.ch> wrote:
>
> > Hi Tom,
>
> > Rather than a behavior switch in the GUI and the tools using the project
> > file, how about extending the specs of the project file itself at the
> > parser?
>
> > I am looking at the specs, shared by PTOptimizer and nona/hugin
>