Opinions on warping to fine tune the results of control point optimization?

[johnfi...@gmail.com]

The most important use case for this idea would also depend on support for low priority control points, which IIUC is in a fork of Hugin that I haven't had time to look at yet.

Assume that control points are very accurately placed, but still don't optimize very well.  So the remapped images don't align very well.  For high contrast areas, anything worse than a single pixel of misalignment is a very poor alignment.  The usual Hugin answers to that seem to be based on avoiding blending high contrast areas.  Sometimes that is a good solution.  Sometimes it isn't.

One cause of major misalignment is the combination of translation (moving the point of view between images, rather than using a tripod with a perfectly adjusted nodal slide) with subjects of the photo being at significantly varying distance.

The translation optimization necessarily depends on all connections between two images (including indirect connections through other images) being at the same distance from the camera.  For multiple subject distances, there is no correct remapping for translation.

Assume either user action (or maybe some automation I haven't though of) lowers the priority of all control points that connect two images at other than the most important subject distance.  Then the alignment for that distance is great, but other distances are a problem.

If high contrast important features at other than the preferred distance are continuous beyond the width of an image, there is no decent solution:  warping would distort the shape, while any other approach would blur the seams.

But more often, the high contrast important features outside the preferred distance are isolated.  If those features are tagged with low priority control points, and the less important areas between them don't contain many control points, then the necessary shape distortion of fine tuning by warp would land in the less important areas.

Some reshaping is fundamentally necessary for that combination of viewpoint shift plus multi-distance.  Blurring across a seam can concentrate the reshaping where it matters least.  But in many cases I think warping would do a better job with less user effort.

In case it wasn't obvious, my idea is to compute all the remapping implied by the optimization results (with prioritized control points) then apply additional remapping by warping to move the control point whatever extra is needed to get from close to perfect.  Then compute the resulting pixels based on the complete movement and blending or original pixels.

I probably won't take on a chunk of work this big.  But I'm thinking about it (for after the easier enhancements I want to make to Hugin).  But I am considering and would appreciate feedback as if I would do it.

Without having tried it, my intuition as a user of Hugin is that it would be enough better that I'm surprised it wasn't designed this way.  Feel free to tell me what I might be misunderstanding.

[Bruno Postle]

See the ptomorph proof of concept from ten(!) years ago here:
https://groups.google.com/g/hugin-ptx/c/UripOuuYXCQ?pli=1

This works incredibly well, with no need for low-priority control
points, but I never pursued it, and it needs some thought regarding
getting it to work with more than two photos.

--
Bruno

[johnfi...@gmail.com]

On Monday, January 31, 2022 at 5:06:20 PM UTC-5 bruno...@gmail.com wrote:

See the ptomorph proof of concept from ten(!) years ago here:
https://groups.google.com/g/hugin-ptx/c/UripOuuYXCQ?pli=1

This works incredibly well, with no need for low-priority control
points, but I never pursued it, and it needs some thought regarding
getting it to work with more than two photos.

I guess I need to learn more about the pano scripting etc.  At this point, a few gaps in my knowledge stop me from even following the discussion there.

Looking at the full stitched version  vs. the full morphed version, the stitching error in the full morphed version (where the silver pole crosses the boundary between the top and second to top window of the glass windowed cabinet) is surprising for the described method, though I expect a few control points could fix it.

The overall shape differences are more concerning.  That specific scene is one in which broad shape differences are hard to visually parse.  So I can't validate my guess that in other examples the shape change of that method would be too big.

Both the big shape change and the issues of chaining the methodt across more photos are reasons that I was thinking of the morph as a fine tune on top of existing methods, rather than as a replacement.

For many examples, just the morph would be a lot better than the existing methods, so it is sad that you didn't have time to pursue that as another choice within the Hugin GUI.

From that discussion:

> Panotools also uses a not-very-nice distortion where it splits the image up into triangles and performs an affine transformation on each triangle separately

That is certainly what I had in mind (possible just because I don't know better ways).  I expect that using this as a fine tune, rather than as the primary method, would eliminate the problems with that method.

Anyway, thanks for providing a lot more information on this topic than I expected to get (even though I have significant learning to do before I can try using any of that).

[Bruno Postle]

On Mon, 31 Jan 2022 at 22:42, johnfi...@gmail.com <johnfi...@gmail.com> wrote:

On Monday, January 31, 2022 at 5:06:20 PM UTC-5 bruno...@gmail.com wrote:

See the ptomorph proof of concept from ten(!) years ago here:
https://groups.google.com/g/hugin-ptx/c/UripOuuYXCQ?pli=1

This works incredibly well, with no need for low-priority control
points, but I never pursued it, and it needs some thought regarding
getting it to work with more than two photos.

 

Looking at the full stitched version  vs. the full morphed version, the stitching error in the full morphed version (where the silver pole crosses the boundary between the top and second to top window of the glass windowed cabinet) is surprising for the described method, though I expect a few control points could fix it.

Probably I hadn't noticed this error and didn't put in any points to fix it.

 

The overall shape differences are more concerning.  That specific scene is one in which broad shape differences are hard to visually parse.  So I can't validate my guess that in other examples the shape change of that method would be too big.

This is a more standard scene I did at the same time, though not correcting parallax, the machine moved between shots: https://www.flickr.com/photos/brunopostle/7103560665/

 

Both the big shape change and the issues of chaining the methodt across more photos are reasons that I was thinking of the morph as a fine tune on top of existing methods, rather than as a replacement.

The remapping and stitching is performed as usual by the Hugin toolchain, ptomorph just manipulates the input images a bit to make them stitch better.

The question was always: should this morphing be added as an overall polynomial distortion in the panotools lens model so that it became another part of the optimisation step, or is it just a stitch-time fix (as in the ptomorph examples). The problem for me is that doing it in the lens model is more elegant, and the problem of three or more images doesn't exist because you wouldn't be forcing an exact alignment; but on the other hand I was getting *really good* results using the ptomorph approach with the 'Shepards' stretch to fit distortion between two images.

--

Bruno

[johnfi...@gmail.com]

On Tuesday, February 1, 2022 at 6:56:17 AM UTC-5 bruno...@gmail.com wrote:

The remapping and stitching is performed as usual by the Hugin toolchain, ptomorph just manipulates the input images a bit to make them stitch better.

By "input" do you mean original?  I can't imagine how that data flow would work.  Maybe I'm also confused by the terminology, but I think I understand what the "remapping" and "stitching" steps are.  In that context, I saw "fine tuning" as an adjustment logically after remapping and before stitching.  I would not want that actually after remapping, because I think the interpolation of moving pixels non-integer distance is best done all at once.  So I would want the morphing to occur during the remapping (digging into that code is the biggest obstacle to my actually doing this whole thing).

The question was always: should this morphing be added as an overall polynomial distortion in the panotools lens model so that it became another part of the optimisation step,

That is another thing for which I can't even imagine the workflow (or the point) so I must be misunderstanding what you mean.   I am assuming a morphing that goes all the way to perfect alignment of every control point, so it would overwhelm the point of optimization if it were in optimization.  So I would think it must be after optimization.

I am only guessing at the data flow inside remapping, but based on that guess my idea is:

First do optimization additionally computing something that I think isn't currently output but easily could be:  the "correct" yaw and pitch of each control point.  A control point exists in two images (could be more than two if other ideas I have were mixed in).  The "correct" yaw and pitch of a control point would be necessarily the same across all images that contain that control point.  For each image the control point is in, you would also compute the "incorrect" yaw/pitch of that control point by applying the optimization results for that image to that control point.

In remapping, I assume you work on an integer row/column position of the output and need to compute the corresponding (non integer) row/column position in the input.  I assume you have the parameters to convert the output row/column to yaw/pitch, then you would identify the triangle of control points (in correct yaw/pitch coordinates of control points).  Then you need to adjust the point you are working on by a small amount based on where it is in the triangle and the difference between the "correct" and "incorrect" positions of each corner of the triangle.  Finally, you apply the optimization results of the input image to convert that yaw/pitch back to row/column.

When the triangles are very large in yaw/pitch, because either masking or other factors left no details needing to be fixed by extra control points, it is important that the adjustment by the error of the triangle is tiny compared to the adjustment by image's optimization results.  So shape within that triangle is preserved as well as possible by the chosen projection method.

[Bruno Postle]

On Tue, 1 Feb 2022 at 12:53, johnfine2017 wrote:

On Tuesday, February 1, 2022 at 6:56:17 AM UTC-5 bruno...@gmail.com wrote:

The remapping and stitching is performed as usual by the Hugin toolchain, ptomorph just manipulates the input images a bit to make them stitch better.

By "input" do you mean original?  I can't imagine how that data flow would work.  Maybe I'm also confused by the terminology, but I think I understand what the "remapping" and "stitching" steps are.  In that context, I saw "fine tuning" as an adjustment logically after remapping and before stitching.  I would not want that actually after remapping, because I think the interpolation of moving pixels non-integer distance is best done all at once.  So I would want the morphing to occur during the remapping (digging into that code is the biggest obstacle to my actually doing this whole thing).

ptomorph was a proof of concept, so it mapped the coordinates of both sides of each control-point to the output panorama, mapped the half-way point back to each source photo, and used this as the distortion destination. The source photos were distorted into temporary files and a new PTO project created that used the temporary images as input.

As I remember, the conclusions were that:

1. Squashing triangles as implemented in panotools looks terrible, it doesn't work, the 'Shepards' stretching/morphing as implemented in ImageMagick is much better. They both force an exact alignment, the pixel ends up exactly where you tell it to go, but Shepards does it right.

2. High-order polynomials work ok for correcting parallax. Though with a polynomial you get a smooth transition, so the pixels don't end up exactly where you tell them to go. The problem with a high-order polynomial distortion is you need a huge number of control points over the whole image or optimisation goes a bit mad.

3. Low-order polynomials are more stable, but less useful for correcting parallax errors which are often only visible in small-scale features.

4. The Shepards distortion is not an easy fit with the panotools lens model, so it would need to be implemented as a pre-processor (like ptomorph) or as a post-processor, either way it could be rolled-into nona.

5. Shepards distortion (and squashing triangles) won't work so well when three or more images overlap. Either: you accept that there is no perfect solution, you implement multiple-image control points (ew), or you only allow control points that are very close to the seam boundary (forcing a two image solution everywhere).

--

Bruno

[Robert Krawitz]

On 1/31/22 11:41, johnfi...@gmail.com wrote:
> The most important use case for this idea would also depend on support for low priority control
> points, which IIUC is in a fork of Hugin that I haven't had time to look at yet.
>
> Assume that control points are very accurately placed, but still don't optimize very well.  So the
> remapped images don't align very well.  For high contrast areas, anything worse than a single pixel
> of misalignment is a very poor alignment.  The usual Hugin answers to that seem to be based on
> avoiding blending high contrast areas.  Sometimes that is a good solution.  Sometimes it isn't.
>
> One cause of major misalignment is the combination of translation (moving the point of view between
> images, rather than using a tripod with a perfectly adjusted nodal slide) with subjects of the photo
> being at significantly varying distance.

That happens to me a lot because of my shooting style. Any time I see an interesting scene I'll
shoot a pano sequence, and it's almost always hand held, much less a pano head (which I don't even
have). And to make matters more fun, there's often both near-field and distant objects.

> The translation optimization necessarily depends on all connections between two images (including
> indirect connections through other images) being at the same distance from the camera.  For multiple
> subject distances, there is no correct remapping for translation.

My way of dealing with that is to shoot lots of extra shots, throw out all of the near field control
points, and then use masks to ensure to the degree possible that only one frame will contain the
offending (close) object. When I can't do that, I either have to accept the problem, crop the
panorama down, or not use it. It's a somewhat tedious process, because I have to build it many
times, but usually I manage to get something clean enough. I have on occasion resorted to manually
warping the result in GIMP, which is really tedious.

> Assume either user action (or maybe some automation I haven't though of) lowers the priority of all
> control points that connect two images at other than the most important subject distance.  Then the
> alignment for that distance is great, but other distances are a problem.
>
> If high contrast important features at other than the preferred distance are continuous beyond the
> width of an image, there is no decent solution:  warping would distort the shape, while any other
> approach would blur the seams.

I'd have to see examples of what the distortion would look like.

> Some reshaping is fundamentally necessary for that combination of viewpoint shift plus
> multi-distance.  Blurring across a seam can concentrate the reshaping where it matters least.  But
> in many cases I think warping would do a better job with less user effort.

If I understand what you're saying, yes.