How line CPs really work (to level a panorama)

[ChameleonScales]

Please correct me where I'm wrong or confirm if I'm right.

To my experience and reasoning, horizontal or vertical line CPs never affect the relative positions of photos when optimizing. Only the orientation of the panoramic sphere as a whole.

Also, it seems the distance displayed in the CP list dialog (shortcut F3) is only the vertical component of the distance between the control points. In other words, if you imagine a horizontal plane that intersects the first point, the vertical segment from the second point to that plane gives you the distance, expressed in pixels at the output resolution.

As to how these line CPs are used by the optimizer to level a panorama, here's how I think it works (I will use horizontal lines in this example but this applies to vertical lines as well):

Let's consider the following horizontal line added in the control points tab (imagine the horizon is actually on this red line):

I first wondered whether the geometric optimization considers the geodesic line or the straight line in 3D space between the 2 points, but according to my tests, the latter is true.

A geodesic line on a sphere always curves across a unique plane. Meaning if the optimizer used geodesics, it would always orient the panorama the same way no matter its pitch/roll orientation before optimization, which I didn't find to be the case.

With a straight line however, there is an infinite number of possible pitch/roll orientations that keep that line horizontal.

Notice how in this gif, the straight line remains horizontal during the entire animation loop while the geodesic line is on a horizontal plane in only 2 frames of the animation (each time it crosses the straight line):

According to my tests, the optimization uses the straight line and choses the minimum pitch and roll transformation values make it horizontal, meaning it can give any of the results shown in the above gif depending on the orientation of the panorama before optimization.

If you add a second line however, no matter how you draw it relative to the first one, there is always a unique and conflict-free solution to level both lines using pitch and roll, as shown in this gif :

This works because geometrically speaking, any 2 straight lines in 3D space always exist on 2 parallel planes no matter their orientation.

Therefore, optimizing with 1 or 2 line CPs should always result in their distance dropping to zero, which is what I find in my tests.

Only when I add a 3rd line does the distance no longer drop to zero, which makes sense, since adding a third line when there was already a unique solution will always result in a conflict:

So starting at 3 line CPs, the optimizer has to find the orientation that is closest to all lines being horizontal, based on an average calculation, hence the distance remaining positive.

Conclusion : Using exactly 2 lines to level your panorama will produce the most predictable and precise result, since it has a unique solution without using any statistic calculation.

Did I misinterpret anything?

[T. Modes]

chamele...@protonmail.com schrieb am Mittwoch, 22. Mai 2024 um 18:01:15 UTC+2:

Also, it seems the distance displayed in the CP list dialog (shortcut F3) is only the vertical component of the distance between the control points.

The distance is calculated in the output projection, not in the 3D space. So the selected projection affects the line control points error.

For horizontal line points it is the vertical deviation, but for vertical line control points it is the horizontal deviation in the output canvas.

 

Conclusion : Using exactly 2 lines to level your panorama will produce the most predictable and precise result, since it has a unique solution without using any statistic calculation.

Did I misinterpret anything?

Keep in mind, that the error distance is calculated in the output projection.

In an equirectangular projection only the horizon is a horizontal line. All other "horizontal" lines are bend in an equirectangluar projection. But in this projection all vertical lines remain vertical.

So using horizontal lines only (in equirectangluar projection) has do be done very carefully. Better use vertical lines, these remain straight in this projection.

Horizontal line control points are best use in rectilinear output projection to correct the perspective - in the general 360 deg case vertical line should be preferred (for levelling).

[Bruno Postle]

On Wed, 22 May 2024, 17:01 'ChameleonScales' wrote:

Please correct me where I'm wrong or confirm if I'm right.

To my experience and reasoning, horizontal or vertical line CPs never affect the relative positions of photos when optimizing. Only the orientation of the panoramic sphere as a whole.

You can use line control points to align images with each other. This can be useful when the only features are linear and have no identifiable 'points'.

-- 

Bruno

[ChameleonScales]

The distance is calculated in the output projection, not in the 3D space. So the selected projection affects the line control points error.

I was wondering if that was the case. Thanks for the correction.

In an equirectangular projection only the horizon is a horizontal line. All other "horizontal" lines are bend in an equirectangluar projection

Just to clarify: I assume you meant that any straight horizontal line in 3D space will bend once projected to equirectangular unless that line sits on the same plane as the equator of the panosphere. I don't think you meant "horizontal lines" as in "longitude lines", since those remain horizontal in equirectangular projection. You can sit your camera tripod on the center of a round table and use that table as a horizontal line (which wouldn't be on the horizon) for your equirectangular panorama, something you can't do with a sidewalk for example. All that I was aware of.

I should note that the above 3D illustrations use random CP positions to demonstrate how I think the optimizer algorithm works. In practice, I use horizontal lines on the sea horizon.

However, CP positions are very precise float values, so the exact same principles as illustrated apply, just with less extreme transformations, meaning that even in my practical use case, adding only one line CP or more than 2 will often produce bad results, which is why I'm trying to get a better mathematical understanding of how the optimizer works.

You can use line control points to align images with each other. This can be useful when the only features are linear and have no identifiable 'points'.

Indeed. However, as soon as I have at least 2 regular CPs, the line CP no longer affects the relative position of photos. Which partly makes sense because you can pivot an image around a single point, not 2. I say partly because it's not necessary to do so. You could keep the current relative positions and reorient the panorma to level a horizontal line. In any case, my assumption holds true so long as you have more than 1 regular CP per couple of photos.

So I take it my theory and 3D illustrations as to how the optimizer straightens the panorama and why it's best to have exactly 2 line CPs were also correct? I mean, even if nothing is calculated in 3D space, the resulting behavior seems to be the exact same.

[ChameleonScales]

Sorry, accidental mail delivery before review. And I made a mistake: I said "longitude" but meant latitude.

[T. Modes]

chamele...@protonmail.com schrieb am Mittwoch, 22. Mai 2024 um 22:16:59 UTC+2:

Just to clarify: I assume you meant that any straight horizontal line in 3D space will bend once projected to equirectangular unless that line sits on the same plane as the equator of the panosphere. I don't think you meant "horizontal lines" as in "longitude lines", since those remain horizontal in equirectangular projection.

"Longitudinal" and "latitudinal" lines are mathematical constructs. I haven't see real one in images.

 

So I take it my theory and 3D illustrations as to how the optimizer straightens the panorama and why it's best to have exactly 2 line CPs were also correct?

No, again. You can't trust that you can level a pano with 2 horizontal line control points.

Hugin is using horizontal line cp and not a "horizon line".

Take the follow example:

 

The panorama is level all both horizontal line cp have an error of 0.00.

And now the same pano with a different pitch. The same 2 horizontal line cp have now also an error of 0.00

The optimizer tries to minimize all cp error. But when different results give the same error the result is undefined and depends on the actual starting values.

[ChameleonScales]

You did find a specific case I didn't think about. However, not only does that not disprove the fact that 3D straight lines behave the same (in fact it proves it even further), but that same problem applies to as many line CPs as you want. Here with 4 line CPs:

See, the case you show in your example is one where both lines are parallel in 3D space. In this situation, there is indeed a continuum of solutions to keep them horizontal, as shown here :

And obviously you can do that with as many lines as you want:

However, to make any 2 segments on the surface of a sphere parallel, you need to also meet a symmetry condition : both lines midpoints have to be on the same central plane (in Hugin that means the CP points you placed have to be spaced evenly around the same axis):

So I'll slightly rectify my stand : 2 line CPs is still the most predictable, but like with any other number of lines, beware of not making them parallel in 3D space. The best is to make them cover different parts of the panorama horizontally (either with partial overlap like in the 2nd animated gif of my first message, or no overlap at all). If one is entirely contained in the other, you risk losing precision.

[David W. Jones]

Hmm. Are you using horizontal lines correctly?

This reminds me of some of the problems I used to have making horizontal panoramas where the sea was the horizon. I initially tried a single horizontal line between the first and the last image, but that didn't really work. Then I tried horizontal lines between other pairs of image, with no better luck.

Putting a single horizontal line on each individual image (not spanning multiple images) worked fine. I've always thought that Hugin basically things there's just one "horizon" and the horizontal line just tells Hugin which way to rotate and vertically position an image along that "horizon".

I also found I get the best results if I add the horizontal lines first, then run the regular control point search. Maybe the regular CP search benefits from having the horizontal lines?

Not that I know any theory or algorithm behind how Hugin levels panoramas. Just my empirical experience.

--
A list of frequently asked questions is available at: http://wiki.panotools.org/Hugin_FAQ
---
You received this message because you are subscribed to the Google Groups "hugin and other free panoramic software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hugin-ptx+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hugin-ptx/V0gSn1x2VDPdTEpxeL04J_TOofHlzk1rXo7ZsC7DHY8575xanv608wvjinPK33lYfpRgNSfWqLTlJoD-vUaZmJhVgdOaQHmoXTpO5wGEv68%3D%40protonmail.com.

-- 
David W. Jones
gnome...@gmail.com
wandering the landscape of god
http://dancingtreefrog.com
My password is the last 8 digits of π.

[T. Modes]

chamele...@protonmail.com schrieb am Samstag, 25. Mai 2024 um 03:36:28 UTC+2:

So I'll slightly rectify my stand : 2 line CPs is still the most predictable, but like with any other number of lines, beware of not making them parallel in 3D space.

When you are using horizontal lines on the horizon this is always the case - and should avoided.

All other horizontal lines in 3 D space are bend lines in the images - so it is impossible to set them in the photos. So this remains a mathematical problem, but nothing meaningful for stitching with real photos.

[ChameleonScales]

When you are using horizontal lines on the horizon this is always the case - and should avoided.

Sorry but that's incorrect in several ways (or maybe phrased poorly).

• First, you seem to imply that all lines that follow the horizon are parallel. That is not the case in 3D space unless they meet that specific condition I explained in my previous message. Here's a loop showing the only 2 moments when 2 lines that follow the horizon can be parallel (when the "=" sign blinks):
  

• Second, none of the 4 lines I showed in my hugin example are placed on the horizon (or on the same plane for that matter) and as you can see it still produces bad results (because they are still parallel in 3D). I attached that same hugin project to this message so you can see it for youself and reproduce what I showed in my example if you want.

• Third, line CPs on the horizon should not be avoided. If they are placed such that they are not parallel in 3D space (and it's easy to ensure, even if you only look in 2D), Hugin will correctly level the panorama.

The problem of using line CPs correctly has nothing to do with them being on the horizon or not and has everything to do with them not being parallel in 3D space. You can say what you want about how the algorithm is coded but I bet you can't find an example that contradicts this logic.

At this point I'm convinced that's how it works and it has already produced good results in my work but I think it's important that we agree on it and not potentially mislead other users who may also need high precision and predictable results. If we can agree on the best way to use this feature, maybe we could also improve documentation on it (I'd gladly provide illustrations if you want me to).

All other horizontal lines in 3D space are bend lines in the images - so it is impossible to set them in the photos

Yes they are bent but you can still use them if you're standing on the center of a circular feature (of course it will likely not be as clean as the sea horizon).

P.S.:

"Longitudinal" and "latitudinal" lines are mathematical constructs. I haven't see real one in images.

Me neither 😄. But those constructs can be used on any sphere including the panoramic sphere. Again, if you stand on the center of a circular feature, its boundary defines a latitude line on your panosphere. We're past that part but I forgot to reply earlier.

[ChameleonScales]

And just to put this to rest, here's a practical test with a spherical photo I captured on top of a round orientation table (I again attached the project files to this message in case you want to see it for yourself)

The original image was straight:

but I purposely rotated it, re-exported it and used the rotated result in Hugin to make it the starting point for this test:

First, I add 2 horizonal lines : one on the edge of the table and the other on a centered circle that was drawn on the table (just to show that it doesn't even matter if the lines aren't on the same latitude).

At first I make them parallel in 3D space (they don't appear parallel in projected space but as shown on the panosphere, they are in 3D) :

As expected, the optimizer fails to straighten the panorama despite dropping CP distance to zero:

Then, I make sure the 2 lines are no longer parallel in 3D:

As predicted, the optimizer now succeeds:

And the lines follow the circles in the Control Points editor:

Still not fully convinced? Let's do it with 4 horizontal lines. I'll even add one on the horizon for good measure.

Now, since I took this photo in the mountains, I colorized the top half of the original straight panorama in order to visualize the true horizon (higher res in attached zip file):

 

First with the 4 lines being parallel in 3D space. Again, not visible in 2D. Also, the panosphere in Hugin displays the geodesic lines in blue, so I overlayed le straight 3D lines in red to better show that they are indeed parallel:

As expected, the optimizer fails straightening the panorama despite the CP distances lowering close to zero:

but as soon as I move a single line along its designated circle so that it's no longer parallel with all the others:

wouldn't you know it, now it succeeds:

Now, as you noted, in Hugin this applies when the output projection is set to equirectangular (and also cylindrical). In theory it could be made to work with any other projection (e.g. if it calculated in 3D space). It could especially be made to work in Mercator projection since that one preserves the horizontality of latitude lines just like equirectangular. So depending on your use case, you may want to temporarily switch to equirectangular before preforming geometric optimization.

Conclusion:

In equirectangular or cylindrical output projection, 2 horizontal lines are enough when they are used on either 1 or 2 reliable features that follow a latitude line on the sphere of view. If you use them on an approximate feature (e.g if the sea is windy and rough with waves and the horizon isn't clean, or if your camera was loosely centered on a circular feature), then additional points may help average that roughness and get closer to the real horizon. Regardless, if all your lines get too close to being parallel in 3D space, you lose precision. However, that's very easy to prevent.

[ChameleonScales]

For whatever reason, 2 images failed to upload in my last post, which is quite annoying as it breaks the explanation I took hours to write.

At least their names are shown in place, so you can follow the explanation by replacing "Screenshot_2024-05-26_20-37-45-2.jpg" by:

and "4 parallel lines.jpg" by:

Sorry for the inconvenience.