Help stitching from camera rig

[Poul Jensen]

Hello,

I've built a camera rig for 4 full frame cameras with Sigma 15mm diagonal fisheyes - see photo included in zip-file linked below. One camera is vertical, the 3 others at 120 degree intervals horizontally and 70 degrees from vertical. This provides sufficient overlap for stitching, capturing everything down to about 20 degrees below the horizon. It obviously suffers from significant parallax, but when put in an open space it should be possible to get a good stitch, using control points on far subjects only.

The test shot I have been able to make for now (see images and PTGui project in zip-file) suffers from several issues, but nevertheless I was surprised to find that:
1) PTGui could generate no control points at all
2) Even with manual control points (only on far subjects) linking all images, the stitch is in PTGui's own words "very bad".

I am hoping someone with more experience with PTGui will care to take a look at the project file and perhaps identify how to get a decent stitch - or, if it's plain impossible, explain why PTGui is unable to make a good stitch. I tried setting yaw/pitch/roll for each image manually, but it seems lens distortion correction is not applied correctly without running the optimizer. Perhaps because the distortion coefficients a, b and c under lens correction default to zero? I do have an accurate lens mapping function (polar angle theta -> normalized radius, 5th order polynomial), but not sure how to derive the distortion coefficients (tried with AI, but the derived values completely messed up stitching).

Here's a link to the project file and (flawed) test images:
PTGui-4-camera-rig.zip

Thank you,
Poul Jensen

[PTGui Support]

Hi Poul,

I got a much better result by enabling 'individual lens parameters' for
all lenses, and optimizing the lens parameters. In your original project
the lens parameters were not optimized.

I also deleted 3 control points which had a very large distance,
probably they were misplaced.

Project file is attached.

You can even improve it further by enabling viewpoint optimization for
all images except one, this compensates for some of the parallax.

Kind regards,

Joost Nieuwenhuijse
www.ptgui.com

> PTGui-4-camera-rig.zip <https://ktisisimiut-my.sharepoint.com/:u:/g/
> personal/pofl_kti_gl/IQBfIrGf9WP3QIiy5x9pNTHZAVQDnscWU325b-Lygrr2hvY?
> e=tdE7em>
>
> Thank you,
> Poul Jensen
>
> --
> You received this message because you are subscribed to the Google
> Groups "PTGui Support" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to ptgui+un...@googlegroups.com
> <mailto:ptgui+un...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/ptgui/
> fccaeeda-5c34-41fe-9e39-a536b0680a73n%40googlegroups.com <https://
> groups.google.com/d/msgid/ptgui/fccaeeda-5c34-41fe-9e39-
> a536b0680a73n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Poul Flyvholm Jensen]

Hi again,

Joost, thanks a lot for taking a look. Your edited project file made me
realize how to optimize lens parameters in PTGui. However, given that I
already have accurate lens mapping functions for each lens (polar angle
theta -> normalized radius r, derived from star positions in images) I
would like to calculate the lens distortion coefficients a, b and c used
by PTGui (https://wiki.panotools.org/Lens_correction_model) so I can set
these manually. My approach has been to do a least squares fit to match
the PTGui model to the known source projection r_src, using an
equidistant projection as the target (r_dest = k*theta) - see attached
Python script.

The wiki page suggests that a, b and c should be close to zero (which is
also what the optimizer in PTGui finds). I'm getting approximate values
of a = -0.37, b = 1.3 and c = -1.45, so it seems I must have done
something wrong. Any idea what the problem is?

Thank you,
Poul Jensen

[PTGui Support]

Hi Poul,

I can't follow exactly what the python script is attempting to do, but I
see several problems: PTGui normalizes such that the smaller sides of
the image have coordinates -1 and 1. Your script normalizes to the
diagonal radius. I also see a sqrt(13) which would mess up the
normalization (perhaps in an attempt to convert from 2:3 to the
diagonal?). And the focal length / field of view is missing from the
equation. Looks like the LLM doesn't really know what it's doing.

I would just use the lens parameters as optimized by PTGui.

Kind regards,

Joost Nieuwenhuijse
www.ptgui.com

[Poul Flyvholm Jensen]

Hi Joost,

Thanks for looking at the code. To clarify:

The field of view is given by theta_max = 1.52, the maximum polar angle
(radians) captured on the sensor (corner). And yes, the sqrt(13)/2 is
the factor needed to convert from diagonal = 2 (my lens mapping
function) to short side = 2 (PTGui model) on a 3:2 sensor (you can do a
quick Pythagoras to confirm).

While a good stitch is most important, I'd also like to maintain as
"realistic" of a view as possible with stars at their correct relative
positions. Given that PTGui produces large errors on precisely placed
control points at infinity (stars) I suspect the lens parameters as
optimized by PTGui are compromised to help solve the problem with
parallax error. Perhaps it is impossible to get a good stitch using the
accurate lens distortion parameters, but I would at least like to try.

To further try to explain what the code does, I create an array of polar
angles (theta) corresponding to the diagonal field of view. I use my
lens mapping function to convert the polar angles to actual positions on
the sensor, r_src, but apply the normalization factor of sqrt(13)/2 so r
= 1 is at top of sensor instead of corner. The target projection is
equidistant, so r_dest = k*theta where k = (sqrt(13)/2)/theta_max is
again normalized so r = 1 is at top of sensor.

Finally I do a least squares fit to fit the PTGui model with the target
projection r_dest to the actual source projection r_src.

While true that the LLM doesn't know what it's doing, in this case
whatever the error is, it is mine. And I can't spot it. :-/

Best regards,
Poul

[PTGui Support]

Hi Poul,

Sorry, I see! sqrt(13)/2 makes sense, I didn't notice the /2 at first.

What values are you using for the focal length and fisheye factor in
PTGui? Have you seen 3.28?
https://ptgui.com/faq#3_28

For equidistant, set the fisheye factor to 0. Then R=f*theta

For full frame sensor, R would be 21.63mm. So f would be 14.23mm

Or a different value if the camera is not full frame.

Kind regards,

Joost Nieuwenhuijse
www.ptgui.com

[Poul Flyvholm Jensen]

Hi Joost,

I probably misunderstood - I thought the fisheye factor model (f/k) and the PanoTools lens correction model (a/b/c) were separate, given that you have to choose between fisheye factor and a/b/c when optimizing. Let me see if I understand now:
- The fisheye factor model (FAQ 3.28) gives a first approximation to the lens profile
- The PT lens correction model serves as fine tuning, taking the fisheye factor model as input (r_dest), outputting a slightly adjusted profile (r_src) that can match the actual lens profile quite accurately. In other words, the constants f/k/a/b/c are all applied together in PTGui to produce an accurate match.

Based on the above understanding I calculated optimal values of f/k/a/b/c that produce very close matches to the accurate lens mapping functions I derived for each of my lenses using stars in photos. However, it doesn't work as expected in PTGui. In the zip-file linked below you will find a project file (and images) where I haven't optimized anything, just entered all the known values for my lenses and the yaw/pitch/roll for my fixed camera rig under Image Parameters. That produces very wrong results with strange artifacts - not the minor errors I was expecting (other than foreground parallax). So clearly I am still not understanding how PTGui works. Can you explain what's going on?

Link to zip-file

Thank you,
Poul Jensen

[PTGui Support]

Hi Poul,

Actually you're right, you would be able to separate the lens projection
from the distortion correction. It's possible to set the fisheye factor
to 0 and PTGui will use a simple R=f*theta mapping. But you could also
incorporate the fisheye factor as you did now, it gives an extra degree
of freedom for fitting the parameters.

I'm sorry, I don't know what's going wrong. I do see that image 5 is
significantly different from the others. Perhaps plot your lens
correction parameters against PTGui's f/k/a/b/c curve and see if they
match? There must be a difference.

Kind regards,

Joost Nieuwenhuijse
www.ptgui.com

> Link to zip-file <https://ktisisimiut-my.sharepoint.com/:u:/g/personal/
> pofl_kti_gl/IQCHUL7Ts_bmQK6Imkp9I3LDAV_bHku2i9w1n9_SBncpjLY?e=2sekXt>

>
> Thank you,
> Poul Jensen
>
> --
> You received this message because you are subscribed to the Google
> Groups "PTGui Support" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to ptgui+un...@googlegroups.com
> <mailto:ptgui+un...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/

> ptgui/7f1db2aa-729c-4cdf-b24c-f54ecd39e4d8%40gmail.com <https://
> groups.google.com/d/msgid/ptgui/7f1db2aa-729c-4cdf-b24c-
> f54ecd39e4d8%40gmail.com?utm_medium=email&utm_source=footer>.

[Poul Flyvholm Jensen]

Hi Joost,

I got it figured out. There was a wrong parameter value for image 5,
causing the weird artifacts. But more importantly, there was indeed an
LLM that did not quite know what it was doing. It suggested pitch angles
of +/- 45 degrees for my cube setup, which sounded plausible to me. But
it turns out that the correct pitch angles are instead +/-
arccos(sqrt(2/3)) = 35.26 degrees. After correcting this I got a stitch
that was reasonably close to good (attached). It does need a bit of
optimization, and the tricky part is to keep the horizon level, which
makes it difficult (if not impossible) to let pitch, roll and viewpoint
be variables in the optimization. If there are any tricks in this regard
I am all ears.

The next issue I am facing is regarding batch processing. I will start a
separate thread for that so it will be easier for others to find when
searching.

Thank you,
Poul Jensen

[John Houghton]

Poul,  I got a good result by treating the sky and land areas as separate groups, and applying viewpoint correction to the land images.  Project file attached.

John

[Poul Flyvholm Jensen]

Hi John,

Great input. In this case I also managed to get a good stitch without
viewpoint correction, but it does take some tinkering, and I can imagine
applying VP correction to selected images only will prove useful in
future projects.

Thank you,
Poul Jensen