Problem with GL preview

[Robert Krawitz]

I've observed a problem with the GL preview that makes it very hard to
use for many of my panoramas. Particularly when I use a wide angle
lens, I find that after I do enough optimization steps the preview
gets badly garbled such that it's essentially unusable. If I use a
normal or short telephoto lens, this usually doesn't happen.

The sequence of steps I usually use is:

1) Create control points

2) Perform an initial optimization step, then do position/view/barrel
distortion, then optimize everything except translation (lens
distortion is significant with my 8-16).

3) Use the GL viewer to center and straighten the pano.

4) Calculate optimal size and FoV in the stitch panel (I hate that
3000 pixel size that all panos start with; it's one extra step I
have to do).

5) Crop out foreground and sky CP's (Celeste isn't perfect, and I
usually don't use a tripod much less a pano head) and re-optimize.

6) Remove all control points except between adjacent images (I shoot
panos with a lot of overlap, and I've found that control points
between non-adjacent images lead to larger errors in the output).

7) Crop out any foreground points that made it through the original
crop, and other control points with a lot of error (carefully -- I
want to make sure that the points with large error are actual bad
points; if there are a lot of foreground points, sometimes the good
points are the ones with large residual error). I run multiple
passes, gradually bringing down the acceptable error. At this
stage I usually try to get it down to about 3 units.

8) Fine tune all points, remove all points with correlation
coefficient less than 0.8, reoptimize, remove the points that the
fine tune botched.

9) Continue with iterative optimization until I get worst case error
under 1 unit.

10) Do photometric optimization.

Anyway, at some point during all of this the preview gets garbled. I
found a particular case with a particular pano where the GL preview is
good, but the next optimization step results in a garbled preview.

The original images in question are all in
http://rlk.smugmug.com/Photography/Hugin-demo/n-NbPQS/ . The "good"
preview is
http://rlk.smugmug.com/Photography/Hugin-demo/n-NbPQS/i-tmdKB7J; the
next optimization step results in
http://rlk.smugmug.com/Photography/Hugin-demo/n-NbPQS/i-vSbVSWP .
This is independent of projection (at least equirectangular and
Mercator).

The .pto file is attached.

In case it matters, I'm using the current repository (although this
problem has been around for at least a few years, I think) with a
Radeon HD5870 using the radeon driver with kernel 3.11.10 and the
following Mesa bits:

DirectFB-Mesa-1.6.3-4.1.3.x86_64
Mesa-10.2.4-381.1.x86_64
Mesa-32bit-10.2.4-381.1.x86_64
Mesa-demo-8.2.0-37.2.x86_64
Mesa-demo-x-8.2.0-37.2.x86_64
Mesa-devel-10.2.4-381.1.x86_64
Mesa-libEGL-devel-10.2.4-381.1.x86_64
Mesa-libEGL1-10.2.4-381.1.x86_64
Mesa-libEGL1-32bit-10.2.4-381.1.x86_64
Mesa-libGL-devel-10.2.4-381.1.x86_64
Mesa-libGL1-10.2.4-381.1.x86_64
Mesa-libGL1-32bit-10.2.4-381.1.x86_64
Mesa-libGLESv1_CM-devel-10.2.4-381.1.x86_64
Mesa-libGLESv1_CM1-10.2.4-381.1.x86_64
Mesa-libGLESv2-2-10.2.4-381.1.x86_64
Mesa-libGLESv2-devel-10.2.4-381.1.x86_64
Mesa-libglapi-devel-10.2.4-381.1.x86_64
Mesa-libglapi0-10.2.4-381.1.x86_64
Mesa-libglapi0-32bit-10.2.4-381.1.x86_64
libOSMesa-devel-10.2.4-381.1.x86_64
libOSMesa9-10.2.4-381.1.x86_64
libOSMesa9-32bit-10.2.4-381.1.x86_64

--
Robert Krawitz <r...@alum.mit.edu>

MIT VI-3 1987 - Congrats MIT Engineers 6 straight men's hoops tourney
Tall Clubs International -- http://www.tall.org/ or 1-888-IM-TALL-2
Member of the League for Programming Freedom -- http://ProgFree.org
Project lead for Gutenprint -- http://gimp-print.sourceforge.net

"Linux doesn't dictate how I work, I dictate how Linux works."
--Eric Crampton

[Terry Duell]

Hello Robert,

On Tue, 05 Aug 2014 11:53:35 +1000, Robert Krawitz <r...@alum.mit.edu>
wrote:

> I've observed a problem with the GL preview that makes it very hard to
> use for many of my panoramas. Particularly when I use a wide angle
> lens, I find that after I do enough optimization steps the preview
> gets badly garbled such that it's essentially unusable. If I use a
> normal or short telephoto lens, this usually doesn't happen.
>
> The sequence of steps I usually use is:
>

[snip]

> 6) Remove all control points except between adjacent images (I shoot
> panos with a lot of overlap, and I've found that control points
> between non-adjacent images lead to larger errors in the output).
>

I see the problem here, I'm running Hugin-2014.1.0 (current development
source).
When I first open your .pto all is OK, but it quickly goes haywire when
any refinement is done.
I note your comment (6), above, but wonder if you still have the problem
is you try to stitch with less overlap, say leaving out every second image.
I can't try that here as I only have dummy images and there aren't the
control points between the non adjacent images.

Cheers,
--
Regards,
Terry Duell

[Robert Krawitz]

On Tue, 05 Aug 2014 12:22:26 +1000, Terry Duell wrote:
> Hello Robert,
>
> On Tue, 05 Aug 2014 11:53:35 +1000, Robert Krawitz <r...@alum.mit.edu>
> wrote:
>
>> I've observed a problem with the GL preview that makes it very hard to
>> use for many of my panoramas. Particularly when I use a wide angle
>> lens, I find that after I do enough optimization steps the preview
>> gets badly garbled such that it's essentially unusable. If I use a
>> normal or short telephoto lens, this usually doesn't happen.
>>
>> The sequence of steps I usually use is:
>>
>
> [snip]
>
>> 6) Remove all control points except between adjacent images (I shoot
>> panos with a lot of overlap, and I've found that control points
>> between non-adjacent images lead to larger errors in the output).
>>
>
> I see the problem here, I'm running Hugin-2014.1.0 (current development
> source).
> When I first open your .pto all is OK, but it quickly goes haywire when
> any refinement is done.

Correct, but at this point I've already performed quite a bit of
refinement without hitting the problem.

> I note your comment (6), above, but wonder if you still have the problem
> is you try to stitch with less overlap, say leaving out every second image.
> I can't try that here as I only have dummy images and there aren't the
> control points between the non adjacent images.

Even if I remove all but one image from the viewer, I have the same
problem.

[Terry Duell]

Hello Robert,

On Tue, 05 Aug 2014 12:25:57 +1000, Robert Krawitz <r...@alum.mit.edu>
wrote:

> On Tue, 05 Aug 2014 12:22:26 +1000, Terry Duell wrote:

>> I see the problem here, I'm running Hugin-2014.1.0 (current development
>> source).
>> When I first open your .pto all is OK, but it quickly goes haywire when
>> any refinement is done.
>
> Correct, but at this point I've already performed quite a bit of
> refinement without hitting the problem.
>
>> I note your comment (6), above, but wonder if you still have the problem
>> is you try to stitch with less overlap, say leaving out every second
>> image.
>> I can't try that here as I only have dummy images and there aren't the
>> control points between the non adjacent images.
>
> Even if I remove all but one image from the viewer, I have the same
> problem.

OK. Can you provide a .pto from a very early stage, i.e. prior to doing
any refinement, say just after generating control points for all images
and no optimisation steps performed?

[Robert Krawitz]

Here it is. I presume that control point generation is deterministic.

[Brandon]

I get this problem from time to time. When it is messed up, if I go through the preview one image at a time I generally find that it is caused by just a few pictures and not all of them are a mess.

Near as I can tell it is caused when optimizing some of the lens parameters(I have not narrowed it down to a single one yet, currently I am thinking that any of the lens settings can mess it up). My work around has been to either ingorne it as it does not show up in the final pano or use the other preview. If I really want to use the GL preview I have found that reseting the lens parameters and optimizing again will solve the problem almost every time.

For debugging I find this in both the 2012 and 2013 versions. As far as lens on the camera I have found it with every camera that I have used, a point and shoot as well as a 30mm and 12 mm(after adjusting for sensor size) lens.

I am pretty sure the problem is that the GL preview has a problem with the lens optimizations from time to time. At one point I went through and messed up the settings one at a time to see what each does and it is easy to take a normal good looking pano and make it look like modern art with the GL preview. To do that take any good pano, go to the lens tab and change any of the lens settings by 2 or more whole numbers and it will be way out of whack, some times it takes a lot less.

Near as I can tell the other preview does not do it.

[Robert Krawitz]

On Tue, 5 Aug 2014 08:16:06 -0700 (PDT), Brandon wrote:
> I get this problem from time to time. When it is messed up, if I go through the preview one image at a time I generally find that it is caused by just a few pictures and not all of them are a mess.

In this case, all (or perhaps all but one) are a mess.

> Near as I can tell it is caused when optimizing some of the lens parameters(I have not narrowed it down to a single one yet, currently I am thinking that any of the lens settings can mess it up). My work around has been to either ingorne it as it does not show up in the final pano or use the other preview. If I really want to use the GL preview I have found that reseting the lens parameters and optimizing again will solve the problem almost every time.

I've already optimized the same set of parameters. Resetting the lens
parameters temporarily cleared it; when I reoptimized, it came right
back.

> For debugging I find this in both the 2012 and 2013 versions. As far as lens on the camera I have found it with every camera that I have used, a point and shoot as well as a 30mm and 12 mm(after adjusting for sensor size) lens.

Both of those are wide angle. I've done some panoramas with longer
lenses than that, and generally haven't see the problem.

> I am pretty sure the problem is that the GL preview has a problem with the lens optimizations from time to time. At one point I went through and messed up the settings one at a time to see what each does and it is easy to take a normal good looking pano and make it look like modern art with the GL preview. To do that take any good pano, go to the lens tab and change any of the lens settings by 2 or more whole numbers and it will be way out of whack, some times it takes a lot less.

Unfortunately, the "regular" preview lacks a lot of features that the
GL preview offers. The ones that matter most to me are on the fly
cropping, positioning, and exposure adjustment. Positioning and
exposure adjustment I can work around, because there's usually enough
usable imagge. Cropping is another matter.

> Near as I can tell the other preview does not do it.

It doesn't.

[Terry Duell]

Hello Robert,

On Tue, 05 Aug 2014 22:07:49 +1000, Robert Krawitz <r...@alum.mit.edu>
wrote:

> On Tue, 05 Aug 2014 12:34:31 +1000, Terry Duell wrote:

[snip]

>> OK. Can you provide a .pto from a very early stage, i.e. prior to doing
>> any refinement, say just after generating control points for all images
>> and no optimisation steps performed?
>
> Here it is. I presume that control point generation is deterministic.

Using this .pto (almanor-16-bad-1.pto), if I deselect images
1,3,5,7,9,11,12 I can then optimise everything without all hell breaking
loose. I optimised in steps, positions, positions and view, positions and
barrel, positions view and barrel, then everything. Are you able to
reproduce this behaviour?
Whilst this might not be an acceptable workaround for you, the mean error
is 3.1 pix, max 12.4, it is still considered a good fit.
It doesn't solve the basic problem, but maybe adds a bit more info to help
track down the cause.

[Robert Krawitz]

On Wed, 06 Aug 2014 08:24:25 +1000, Terry Duell wrote:
> Hello Robert,
>
> On Tue, 05 Aug 2014 22:07:49 +1000, Robert Krawitz <r...@alum.mit.edu>
> wrote:
>
>> On Tue, 05 Aug 2014 12:34:31 +1000, Terry Duell wrote:
>
> [snip]
>
>>> OK. Can you provide a .pto from a very early stage, i.e. prior to doing
>>> any refinement, say just after generating control points for all images
>>> and no optimisation steps performed?
>>
>> Here it is. I presume that control point generation is deterministic.
>
> Using this .pto (almanor-16-bad-1.pto), if I deselect images
> 1,3,5,7,9,11,12 I can then optimise everything without all hell breaking
> loose. I optimised in steps, positions, positions and view, positions and
> barrel, positions view and barrel, then everything. Are you able to
> reproduce this behaviour?

It still eventually happens, after enough iterations of removing bad
points, reoptimizing everything but translation, and fine tuning all
points. In this case, when I got the max error under 2 (average .6,
max 1.7), the problem showed up again. This is about when I'd
typically expect it to show. I've attached it at the point it shows up.

> Whilst this might not be an acceptable workaround for you, the mean error
> is 3.1 pix, max 12.4, it is still considered a good fit.
> It doesn't solve the basic problem, but maybe adds a bit more info to help
> track down the cause.

It would be helpful if there were a way to trace every operation I
perform. My description isn't very specific, but it's very hard to be
precise when there are thousands of control points and all that.

I don't consider max 12.4 to be a good fit at all. At that level,
there's usually some visible discontinuities in the background, where
they're most objectionable (IMO). Maybe I'm too finicky, but I like
to get it under 1.

[Terry Duell]

On Wed, 06 Aug 2014 08:49:42 +1000, Robert Krawitz <r...@alum.mit.edu>
wrote:

> On Wed, 06 Aug 2014 08:24:25 +1000, Terry Duell wrote:

[snip]

>> Using this .pto (almanor-16-bad-1.pto), if I deselect images
>> 1,3,5,7,9,11,12 I can then optimise everything without all hell breaking
>> loose. I optimised in steps, positions, positions and view, positions
>> and
>> barrel, positions view and barrel, then everything. Are you able to
>> reproduce this behaviour?
>
> It still eventually happens, after enough iterations of removing bad
> points, reoptimizing everything but translation, and fine tuning all
> points. In this case, when I got the max error under 2 (average .6,
> max 1.7), the problem showed up again. This is about when I'd
> typically expect it to show. I've attached it at the point it shows up.
>

Ok. I didn't remove any bad points or do any fine tuning, simply optimised
the project 'as was' in steps.

>> Whilst this might not be an acceptable workaround for you, the mean
>> error
>> is 3.1 pix, max 12.4, it is still considered a good fit.
>> It doesn't solve the basic problem, but maybe adds a bit more info to
>> help
>> track down the cause.
>
> It would be helpful if there were a way to trace every operation I
> perform. My description isn't very specific, but it's very hard to be
> precise when there are thousands of control points and all that.
>

That is the 'difficult part', something that needs to be done in a
debugging mode where one can watch variables. Unfortunately I don't know
enough to do that. Hopefully someone who does can use your project file to
track down the problem.

[Brandon]

Tonight this bug is really bothering me. I have hit my undo button often enough that I have noticed the change that is causing the problem tonight. On the stitching tab if I have the canvas set to the optimal size (23514) the bug is not there, but if I set it smaller than 11750 (about half) the problem starts happening. Conversely if I set the output any bigger than optimal size it shows up. The further I get from that general range the worse the blurring gets.

[Robert Krawitz]

If this is the problem I was reporting (with corruption in the GL
preview, http://rlk.smugmug.com/Photography/Hugin-demo/n-NbPQS/), then
in my case it isn't affected by canvas size (I always set the canvas
size to optimal right away).