idea: limit the amplitude of variation of a free value

[Stéphane Péneau]

Hi,


When stitching thousands of pano with my script, sometimes I have some
broken pano. The reasons for these failures vary, but It usually ends up
with totally wrong values for lens FOV or yaw/pitch/roll.

For example, if I indicate that the FOV of a lens is 125°, and Hugin
finds that the best solution is 50°, it can't be correct.
If I indicate a yaw of 90°, a final result of -30° is necessarily incorrect.

I can parse the final pto file to detect if the pano is wrong, but I was
wondering if Hugin could not integrate a function to limit the amplitude
of variations of a parameter which would reduce the number of failed pano.

Just an idea

Best,

Stéphane

[johnfi...@gmail.com]

I think that problem is important and ought to get some attention and can be solved.

But I think your suggested solution would not be effective.  Either the user would need to know enough in setting the limits that they might as well just set the value and not optimize the variable, or automatically set limits would do more harm than good in other cases.

Based entirely on use, without yet even looking at the relevant part of the code, I'm guessing both sources of a control point are projected into their theoretical positions and compared there.  That approach would be inherently unstable and lead to finding wrong solutions that have lower "error" than the right solution.  That fits the observed behavior, thus my guess.

Projecting one point to its theoretical position and then back to the other image (and computing the error there) would be inherently much more stable.  But given varying zoom level across the input, that could inappropriately weight control points.  Effective zoom level could vary within an image due to the original lens distortion as well as varying original zoom in the images.  But my guess is that the weight problem could be solved more easily than other approaches to fixing the original problem.

I might be 100% wrong about all of this.  I hope to find time to look at that part of the code and really understand why optimize is so likely to find wrong answers.  But in working on many other optimize problems in many other domains, my experience tells me that stricter limits are likely to be a bad cover up for a problem that needs to be fixed elsewhere.

[Stéphane Péneau]

Hi !

Thank you for your taking some time on this.

I'll try to find some examples during the next week, and put a link to
download the individual images.

[Jeff Welty]

This may or may not be helpful -- but in the SkyFill program I found a version of levenberg marquardt that will fit with constraints on the parameters.  I *think* it is derived from the source code hugin has (lmdir.c), so functionally might be quite close to what it in hugin.  You can find it at:

http://cow.physics.wisc.edu/~craigm/idl/cmpfit.html

It's set up so any parameters can have an upper or lower constraint (or both, or neither).   So the user could request the FOV (if it is a parameter in the fit) have a final fit value between the upper and lower constraint.  In the little bit of testing I have done it seems to help avoid falling into a local minimum for the optimization.

Jeff

[Florian Königstein]

In my version "fastptoptimizer" of libpano I have the options to define a min and a max limit for each parameter, i.e. the optimizer won't change the parameter beyond the limits. The min / max options define "hard" limits for the parameter.

You can alternatively define "soft" limits by setting a weight for some or all parameters. The sum of squares that the optimizer tries to minimize is increased by the weighted squared deviation of the actual parameter from its initial value (before optimization), i.e. the deviation is squared and then multiplied by the weight for the parameter. So a deviation of the parameter from its initial value is punished by an increase of the sum of squares. The value 1 / sqrt(weight) (the inverse of the square root of the weight) can be interpreted as a standard deviation (or "soft" limit) for the initial parameter value.

However, I have not yet implemented the "hard" and "soft" limits in my Hugin branch (Hugin++) - at the moment they are only accessible via the command line optimizer in "fastptoptimizer" (PToptimizer.exe). PToptimizer.exe expects the filename of a script file. In this file the "hard" limits can be defined in the 'v' lines as in the following example:

v r0 min-10 max10

v p0 min-20 max-10

v y0 min30 max40

v r1 min-20 max20

...

In the following example I define weights for parameters.

v r0 w4

v p0 w1

v y0 w0.25

v r1 w0.01

E.g. in the line

v y0 w0.25

the weight is 0.25, i.e. the "standard deviation" (soft limit) is 2.

If you have problems optimizing large panoramas, you may also like my "weights for control points" feature that is accessible via Hugin++.

[Florian Königstein]

I just tested my min / max feature on PToptimizer.exe  and it seems that there is some bug. It doesn't work as expected.

But it did work in earlier versions of "fastptoptimizer". I think that I'll be able to fix it soon.

[johnfi...@gmail.com]

Other than field of view, what parameters are able to produce wrong results that have lower total error than any correct result?

If I understand correctly, that is the major reason for wanting limits.  But in case I've misunderstood:

In many other optimization problems hard limits are used to speed up optimization and/or keep it out of non convergence areas (where derivative based optimization sees a wrong direction and/or the error function might be NaN) even though those bad areas hold no false solutions.  Is that also a significant issue in hugin?

I still think a more stable objective function is a better idea than limits for solving the basic problem.  But for discussing the value limits might have, I'd like to understand what aspect (if any) is general across many parameters, vs. what is specific to field of view.

[Florian Königstein]

I have sometimes also the problem getting totally wrong results for yaw, pitch and roll. Yes, limits can help from converging to a local minimum with wrong parameters. E.g. you may know that the rotation of all images is nearly zero or nearly 90 degrees. Then you can force the optimizer to keep rotation e.g. in -10 ... 10 degrees or 80 .. 100 degrees.

Since most cameras yield a good estimation of the FOV, I generally don't optimize the FOV.

Optimizing FOV is generally a difficult task. If you must for some reason optimize the FOV, try setting the initial FOV to a value that is higher than the expected value.

Assume the FOVs that the optimizer actually assumes are relatively small. If the optimizer makes the FOVs smaller so that it goes to zero, the projection of the images onto the unit sphere get smaller - nearly linear/proportional with the FOV. And the error also gets smaller with the FOV. So the optimizer would estimate FOVs of zero and would set the yaw, pitch and roll of all images equal. All control points would be projected onto the same point on the unit sphere.

Of course this in not what we want. To prevent the optimizer from estimating a FOV of zero, the sum of squares of the errors is artificially divided by the square of a factor being proportional to the FOVs. To be accurate, this factor is equal to the ratio of the actual AVERAGE FOV of the images to the initial AVERAGE FOV of the image, so the error doesn't tend to zero if the FOV goes to zero (and all yaw, pitch, roll tend to be equal).

In my version of libpano is probably not a bug in the code. But when I compile it now, there seems to be some incompatibilites with SuiteSparse or with libraries that SuiteSparse uses.

[Florian Königstein]

I forgot to say: The optimizer, only divides the sum of squares by the square of the ratio of the actual AVERAGE FOV of the images to the initial AVERAGE FOV of the images if this ratio is smaller than one, i.e. if the  actual AVERAGE FOV is smaller than the initial AVERAGE FOV. Otherwise, the error would get smaller if the FOV increases and the optimizer would tend to estimnate big FOVs.

[Florian Königstein]

There was not a bug in my code, but there were some problems with linking to incompatible libraries.

Further more, the PTOptimizer.exe that was as binary in my Hugin++ bundle was compiled with a false option: Instead of printing messages to the console

it tried unsuccesfully to open a window.

Now I compiled it correctly and uploaded the Windows installer to:

https://sourceforge.net/projects/huginplusplus/files/development/Hugin++Setup_2021-1.1-Windows-64bit.exe

The feature with min / max or weights ("soft limits") for parameters should work with the command line program PTOptimizer.exe.