Determining lens distortion parameters from multiple panoramas

[Jeff Welty]

I have a new camera (Canon G1X mark iii).

The default lens distortion at the widest angle (15mm) in lensfun definitely does not match my particular copy of this camera, so I can't export output Darktable with lens correction.

I have a dozen or so mini-panoramas with 2 to 5 images each, of which I can get very good distortion fits with hugin.  Same projection is used for all panoramas.

One approach is take the average value of each coefficient from each of the separate panoramas, and update the lensfun data with the average value.  I did a quick check and this appears mathematically consistent.

Another approach, which I haven't seen in the FAQ, would be to combine the panoramas algorithmically -- changing the control point numbers and image numbers so they all remain unique.   The panorama view would make no sense, but I don't think hugin would encounter any problems with that.  

If I did this, I believe hugin could perform a custom optimization and output a good estimate for distortion coefs based on all the panorama data I have for a given focal length.   Do you think this will work?

---

On a side note -- I have yet to use a tripod, I am pretty good at rotating the camera around the exit pupil.  But a little translation can make a big difference.  I hacked in the constrained Levenberg Marquardt method so the optimization doesn't get lost looking for large translation parameters.  This seems to be working very well.  Hard coded to not let Tx,Ty or Tz change more than +/- .05 from the starting value found when the optimization begins.

[Bruno Postle]

Lensfun and panorama tools have a slightly different lens model, so it
isn't always possible to transfer parameters (I'm not sure if there is
a converter, it wouldn't be difficult to write one if it doesn't
exist).

The idea of optimising a merged project to get more accurate lens
parameters should work. There is a tool called ptomerge in the
Panorama::Script perl module for merging projects, I'm not sure if it
merges lenses, but if it doesn't you can reassign the lens in Hugin.

I like the idea of a 'handheld' mode for limiting translation
parameters, though since these parameters are unitless I'm not sure
how to pick the thresholds reliably.

--
Bruno

On Thu, 28 Apr 2022 at 02:34, Jeff Welty wrote:
>
> I have a new camera (Canon G1X mark iii).
>
> The default lens distortion at the widest angle (15mm) in lensfun definitely does not match my particular copy of this camera, so I can't export output Darktable with lens correction.
>
> I have a dozen or so mini-panoramas with 2 to 5 images each, of which I can get very good distortion fits with hugin. Same projection is used for all panoramas.
>
> One approach is take the average value of each coefficient from each of the separate panoramas, and update the lensfun data with the average value. I did a quick check and this appears mathematically consistent.
>
> Another approach, which I haven't seen in the FAQ, would be to combine the panoramas algorithmically -- changing the control point numbers and image numbers so they all remain unique. The panorama view would make no sense, but I don't think hugin would encounter any problems with that.
>
> If I did this, I believe hugin could perform a custom optimization and output a good estimate for distortion coefs based on all the panorama data I have for a given focal length. Do you think this will work?
>

> On a side note -- I have yet to use a tripod, I am pretty good at rotating the camera around the exit pupil. But a little translation can make a big difference. I hacked in the constrained Levenberg Marquardt method so the optimization doesn't get lost looking for large translation parameters. This seems to be working very well. Hard coded to not let Tx,Ty or Tz change more than +/- .05 from the starting value found when the optimization begins.

--
Bruno

[Jeff Welty]

On Thursday, April 28, 2022 at 2:33:53 AM UTC-7 bruno...@gmail.com wrote:

Lensfun and panorama tools have a slightly different lens model, so it
isn't always possible to transfer parameters (I'm not sure if there is
a converter, it wouldn't be difficult to write one if it doesn't
exist).

I thought the "ptlens" distortion model in lensfun was identical to hugin's model.  The lensfun tutorials show the a,b,c parameters from hugin's optimization being directly used in lensfun data.  The equations documented sure looks the same, but you have far, far more experience at it so I will look harder at it.  FWIW here's the web page I referenced documenting the lensfun "ptlens" model:

https://lensfun.github.io/calibration-tutorial/lens-distortion.html

The idea of optimising a merged project to get more accurate lens
parameters should work. There is a tool called ptomerge in the
Panorama::Script perl module for merging projects, I'm not sure if it
merges lenses, but if it doesn't you can reassign the lens in Hugin.

Oooh, thanks for that.  I'm perl fluent so that's a nice place to start.

After a little more research, I read where image stabilization could be a cause for the "d,e" parameters to legitimately be different for each image.   So I may need to link only the a,b,c parameters and allow d&e to be optimized for each indvidual image.

I like the idea of a 'handheld' mode for limiting translation
parameters, though since these parameters are unitless I'm not sure
how to pick the thresholds reliably.

The best understanding I have is that the translation parameter units are relative to the panosphere, where the panosphere radius is 1.   I got that understanding from this page:

http://hugin.sourceforge.net/docs/manual/Stitching_a_photo-mosaic.html

I think if a "handheld" mode was to be incorporated into hugin, translation change limits would be set in the preferences.   I have it hard-coded with a maximum change of 0.05. So if the true TrX is 0.1, and the optimization ran "perfectly" it would take two runs of the optimizer for Trx to get to 0.1   I also contrained the Tpy and Tpp for translation mapping plane to not change by more than 0.9, but I haven't optimized those so far.

If you are interested in this, I'll bundle up the changes I have in a zip or tar file.  As I recall 4 or 5 source code files were affected.  I did not attempt to get the LM starting step sizes for the parameters correctly set, those are hardcoded, and the same for every parameter, it's still a FIXME issue.

Jeff

[T. Modes]

Hi Jeff,

eljef...@gmail.com schrieb am Donnerstag, 28. April 2022 um 03:34:47 UTC+2:

One approach is take the average value of each coefficient from each of the separate panoramas, and update the lensfun data with the average value.  I did a quick check and this appears mathematically consistent.

This is what Hugins lens database does (integrated into the GUI and hugin_lensdb command line tool): It stores the parameters for each stitch and can reuse the average for new panoramas when requested.

Another approach, which I haven't seen in the FAQ, would be to combine the panoramas algorithmically -- changing the control point numbers and image numbers so they all remain unique.   The panorama view would make no sense, but I don't think hugin would encounter any problems with that.  

Beside ptomerge in Panorama::Script perl module there is also pto_merge from Hugin which allows the merge several panoramas.

 

I think if a "handheld" mode was to be incorporated into hugin, translation change limits would be set in the preferences.   I have it hard-coded with a maximum change of 0.05. So if the true TrX is 0.1, and the optimization ran "perfectly" it would take two runs of the optimizer for Trx to get to 0.1   I also contrained the Tpy and Tpp for translation mapping plane to not change by more than 0.9, but I haven't optimized those so far.

The last one is too restrictive: both parameters are angles - Tpy 0 - 360 and Tpp -90 - 90. So a limit of 0.9 is significant too low for the general case.

 

If you are interested in this, I'll bundle up the changes I have in a zip or tar file.  As I recall 4 or 5 source code files were affected.  I did not attempt to get the LM starting step sizes for the parameters correctly set, those are hardcoded, and the same for every parameter, it's still a FIXME issue.

This would be nice.

For providing the changes a diff file (or patch) would be preferred. But if you bundle all modified source files it would also be a good starting point.

Thomas

[Bruno Postle]

On Thu, 28 Apr 2022 at 15:57, Jeff Welty wrote:

> On Thursday, April 28, 2022 at 2:33:53 AM UTC-7 Bruno Postle wrote:
>>
>> Lensfun and panorama tools have a slightly different lens model, so it
>> isn't always possible to transfer parameters (I'm not sure if there is
>> a converter, it wouldn't be difficult to write one if it doesn't
>> exist).
>>
> I thought the "ptlens" distortion model in lensfun was identical to hugin's model. The lensfun tutorials show the a,b,c parameters from hugin's optimization being directly used in lensfun data. The equations documented sure looks the same, but you have far, far more experience at it so I will look harder at it. FWIW here's the web page I referenced documenting the lensfun "ptlens" model:

I may be wrong, it has been a while. The panotools/Hugin lens model
does have a couple of flaws: the a & c parameters have an odd number
power, whereas usually only even number powers are used for lens
models (which is why we generally recommend starting with just the b
parameter); the angle of view parameter is locked to the width of the
image, but the lens correction parameters are scaled to the narrowest
dimension of the image (which is the height for landscape shots and
the width for portrait shots), the result is that you can't reuse the
same parameters for landscape/portrait images. These have never been
fixed because it would break existing PTO projects and they don't
cause any problems in practice.

>> The idea of optimising a merged project to get more accurate lens
>> parameters should work. There is a tool called ptomerge in the
>> Panorama::Script perl module for merging projects, I'm not sure if it
>> merges lenses, but if it doesn't you can reassign the lens in Hugin.
>>
> Oooh, thanks for that. I'm perl fluent so that's a nice place to start.
> After a little more research, I read where image stabilization could be a cause for the "d,e" parameters to legitimately be different for each image. So I may need to link only the a,b,c parameters and allow d&e to be optimized for each indvidual image.

To the optimiser, the d & e parameters resemble the TrX and TrY
parameters at the small scale, so sometimes you can get strange
results mixing them.

--
Bruno

[Gunter Königsmann]

My first smartphone (a htc wildfire) had a slightly different scale at the top left edge of the image than on the bottom right one. ...and lens parameters determined with landscape photos didn't work with ones that were made in the other orientation.
Perhaps this discussion has now explained a reason why that might have been....

[Jeff Welty]

Hi Thomas,

On Thursday, April 28, 2022 at 8:40:05 AM UTC-7 T. Modes wrote:

One approach is take the average value of each coefficient from each of the separate panoramas, and update the lensfun data with the average value.  I did a quick check and this appears mathematically consistent. 

This is what Hugins lens database does (integrated into the GUI and hugin_lensdb command line tool): It stores the parameters for each stitch and can reuse the average for new panoramas when requested.

 

Oooohh.  Very nice.   I was looking at that database the other day, and wondered how all those distortion coefs got in there.    I'm going to have a hard look at that.  Thanks!!

 

If you are interested in this, I'll bundle up the changes I have in a zip or tar file.  As I recall 4 or 5 source code files were affected.  I did not attempt to get the LM starting step sizes for the parameters correctly set, those are hardcoded, and the same for every parameter, it's still a FIXME issue.

This would be nice.

For providing the changes a diff file (or patch) would be preferred. But if you bundle all modified source files it would also be a good starting point.

 Here you go.   The mpfit.*  go in the main directory of libpano.   adjust.c is where constraints are added.   There is a default mode as I described earlier, and, for testing, if a pto_constraints.txt file exists, it will use constraints found in there.  I had no intent at all that is is how constraints should be found and applied, it was just a simple way I could test the concept.   Any parameter passed to the optimizer can have an upper, lower (or both) limit.

The use of the LM_WITH_CONSTRAINTS #define in optimize.c is perhaps a little strange for a real implementation.  When I'm testing ideas I use this approach so the original code is not modified and I can refer to it while editing a source file.

As you can probably tell, the patch is for modifications made to libpano13-2.9.21

[Jeff Welty]

On Thursday, April 28, 2022 at 8:49:58 AM UTC-7 bruno...@gmail.com wrote:

I may be wrong, it has been a while. The panotools/Hugin lens model
does have a couple of flaws: the a & c parameters have an odd number
power, whereas usually only even number powers are used for lens
models (which is why we generally recommend starting with just the b
parameter); the angle of view parameter is locked to the width of the
image, but the lens correction parameters are scaled to the narrowest
dimension of the image (which is the height for landscape shots and
the width for portrait shots), the result is that you can't reuse the
same parameters for landscape/portrait images. These have never been
fixed because it would break existing PTO projects and they don't
cause any problems in practice.

If the documentation is correct, the hugin eqn is:

                 Rsrc = (a * Rdest^3 + b * Rdest^2 + c * Rdest + d) * Rdest

So when you multiply the elements inside the parenthesis by outside the parenthesis, "Rdest", the powers all increase by one, and will match the lensfun eqn.

But as you point out, if the scaling of the maximum dimension for scaling is not the same, then the coefs are not compatible.   Thanks for sharing your insight.   I suspect TCA coefs are also incompatible for the same reason.

Like you say, a little algebra would convert from one system to another.

To the optimiser, the d & e parameters resemble the TrX and TrY
parameters at the small scale, so sometimes you can get strange
results mixing them.

I hadn't even considered that, but I see the problem.  Probably best not to optimize d&e if TrX and TrY are being optimized.

Thanks Bruno!

Jeff

 

 

[Klaus Foehl]

On 28.04.22 17:49, Bruno Postle wrote:
> On Thu, 28 Apr 2022 at 15:57, Jeff Welty wrote:
>> On Thursday, April 28, 2022 at 2:33:53 AM UTC-7 Bruno Postle wrote:
>>> Lensfun and panorama tools have a slightly different lens model, so it
>>> isn't always possible to transfer parameters (I'm not sure if there is
>>> a converter, it wouldn't be difficult to write one if it doesn't
>>> exist).
>>>
>> I thought the "ptlens" distortion model in lensfun was identical to hugin's model. The lensfun tutorials show the a,b,c parameters from hugin's optimization being directly used in lensfun data. The equations documented sure looks the same, but you have far, far more experience at it so I will look harder at it. FWIW here's the web page I referenced documenting the lensfun "ptlens" model:
> I may be wrong, it has been a while.

No, you are not.

> The panotools/Hugin lens model
> does have a couple of flaws: the a & c parameters have an odd number
> power, whereas usually only even number powers are used for lens
> models (which is why we generally recommend starting with just the b
> parameter);

Hence my suggestion to include more parameters from the Brown-Conrady
model into hugin. Please. Keep the flawed parameters a and c for
backwards compatibility.

On why these odd power parameters are mathematically flawed, I've
written on that in postings long ago. I can revisit the topic if desired.

[Jeff Welty]

I took a look at the lensfun code, and it sure looks like it has a consistent formula with what is described in hugin, and what you (Bruno) said hugin does for lens distortion in terms of where radius=1.

To confirm this in practice, I used a panorama where I have an excellent fit of lens distortion (using only a,b,c parameters).   I let the TrX, TrY and TrZ also be optimized per image.  The difference mode in the panorama viewer shows almost entirely black.  The tifs were created only using TCA correction in darktable.

Then I exited darktable, updated the lens distortion coefs in the appropriate lensfun *.xml file, restarted darktable (so it would re-read the *.xml file), and recreated the tifs, but this time enabling distortion correction as well as TCA.

I made a copy of the *.pto file, and loaded the newly created tifs in hugin, and set a,b,c to 0.0

Looking at the panorama in difference mode, it looks identical to the one where I let hugin do the distortion correction.

I'm convinced, at least for distortion correction, you can indeed use the same a,b,c parameters for hugin and lensfun (where you specify the distortion model to be "ptlens"

I would assume that vignetting and TCA are also consistent between hugin and lensfun.

On Thursday, April 28, 2022 at 8:49:58 AM UTC-7 bruno...@gmail.com wrote: