Documentation of Fields in PTO i-lines

[kfj]

Dear group!

I'm working on a script which modifies PTO scripts. For PTO format, I'm using this reference:

http://hugin.sourceforge.net/docs/manual/PTOptimizer.html

I noticed that there are quite a few fields which are not covered in this documentation. In a recent PTO script generated by hugin, the i-lines end with these fields:

TrX0 TrY0 TrZ0 Tpy0 Tpp0 j0

I assume that TrX, TrY and TrZ are translation parameters. Tpy, Tpp and j mean nothing to me. Is there maybe more documentation for PTO format somewhere? If not, how about we create it?

Kay

[Bruno Postle]

There is more documentation in nona.txt in the Hugin sources https://sourceforge.net/p/hugin/hugin/ci/default/tree/doc/nona.txt

--
Bruno

[kfj]

Thanks, Bruno, that explains TrX0 TrY0 TrZ0 Tpy0 Tpp0 j0, and it has much more information than the docu I was using!

I have another issue with the i-lines. In PTOptimizer.html, there is the option to use cropped source images with the 'C' field:

# C100,600,100,800 Crop(l,r,t,b), Only pixels inside the rectangle will be used for conversion.

#                                  Cropped image size is used for all image parameters

#                                  (e.g. field-of-view) refer to the cropped part of the image.

I tried to use a PTO file with such C fields with hugin, but the cropping information was silently changed to a selection field ('S' tag)

Is this field type not used by the hugin processing chain?

Kay

[Bruno Postle]

PTOptimizer.txt is for PTOptimizer, which is a standalone tool that is independent of Hugin. The nona.txt file describes the behaviour of the Hugin stitcher.

Panotools supports both 'C' and 'S' image parameters, they do similar things, describing the rectangle of interest in the photo, but are not interchangeable.

'C' parameters discard all data outside the rectangle, and then all image parameters (angle of view, lens distortion) relate to this new image boundary. This is a model that makes a lot of sense if all your photos are digitised with a flat-bed scanner - the cropped-out areas are not part of the photo.

'S' parameters set the area outside the rectangle to 'transparent', but the image parameters still relate to the original image frame. This model makes more sense with digital cameras where all the pixels have come from the sensor, just that some of them are not wanted (in Hugin there is a lot of overlap with the masking functionality, which came much later).

It is not straightforward to convert from a project that uses one method to the other method, so it makes sense for a GUI tool to support one only (since you don't need both). This is the original schism between Ptgui and Hugin - Ptgui uses 'C' and Hugin uses 'S'.

So if you are importing a Ptgui project into Hugin, you need to re-optimise lens parameters because the C parameters have been silently converted to S.

--
Bruno

On 12 September 2021 07:46:03 BST, 'kfj' via hugin and other free panoramic software wrote:
>Thanks, Bruno, that explains TrX0 TrY0 TrZ0 Tpy0 Tpp0 j0, and it has much
>more information than the docu I was using!
>
>I have another issue with the i-lines. In PTOptimizer.html

><http://hugin.sourceforge.net/docs/manual/PTOptimizer.html>, there is the

[Kay F. Jahnke]

Am 12.09.21 um 11:47 schrieb Bruno Postle:

> PTOptimizer.txt is for PTOptimizer, which is a standalone tool that is independent of Hugin. The nona.txt file describes the behaviour of the Hugin stitcher.
>
> Panotools supports both 'C' and 'S' image parameters, they do similar things, describing the rectangle of interest in the photo, but are not interchangeable.
>
> 'C' parameters discard all data outside the rectangle, and then all image parameters (angle of view, lens distortion) relate to this new image boundary. This is a model that makes a lot of sense if all your photos are digitised with a flat-bed scanner - the cropped-out areas are not part of the photo.
>
> 'S' parameters set the area outside the rectangle to 'transparent', but the image parameters still relate to the original image frame. This model makes more sense with digital cameras where all the pixels have come from the sensor, just that some of them are not wanted (in Hugin there is a lot of overlap with the masking functionality, which came much later).
>
> It is not straightforward to convert from a project that uses one method to the other method, so it makes sense for a GUI tool to support one only (since you don't need both). This is the original schism between Ptgui and Hugin - Ptgui uses 'C' and Hugin uses 'S'.
>
> So if you are importing a Ptgui project into Hugin, you need to re-optimise lens parameters because the C parameters have been silently converted to S.

Thank you, Bruno, for the explanation. I stumbled upon this because I
was working on images done with a Xiaomi Mi Sphere 360 - these images
have two circular fisheye images next to each other, and I wanted to
mask them so that I could use the image twice, once for the left half of
the sphere and once for the right half, with, otherwise, identical lens
parameters. I hoped to use 'C' which would have made the job simple.

With 'S' I could create correct masks, but I had to introduce differing
horizontal shift (d parameter) for the two half-images. So I needed two
separate lenses. I also had to assign a hfov of over 360 degrees (for
the whole image, the (square) half-images have about 193 degrees fov).
All of this made the optimization a bit tricky...

I succeeded in the end, but I was wondering what the 'C' was all about.
Now it's clear. I was also a bit miffed that hugin would silently
consume my PTO with a C field to a PTO with an S field, when the two
have quite different meaning.

Kay

[T. Modes]

kfj schrieb am Sonntag, 12. September 2021 um 13:04:25 UTC+2:

 Now it's clear. I was also a bit miffed that hugin would silently
consume my PTO with a C field to a PTO with an S field, when the two
have quite different meaning.

And  I could be a bit miffed that you ignored the dual lens assistant which does all the steps needed for such images.

(Maybe it needs some fine-tune for a specific model, but in principle it does all steps needed.)

Thomas

[Kay F. Jahnke]

Am 19.09.21 um 19:52 schrieb T. Modes:

> kfj schrieb am Sonntag, 12. September 2021 um 13:04:25 UTC+2:
>
>  Now it's clear. I was also a bit miffed that hugin would silently
> consume my PTO with a C field to a PTO with an S field, when the two
> have quite different meaning.
>
> And  I could be a bit miffed that you ignored the dual lens assistant
> which does all the steps needed for such images.

Okay, what's the 'dual lens assistant' and where can I find it?

> (Maybe it needs some fine-tune for a specific model, but in principle it
> does all steps needed.)

My criticism was that the C field was silently changed into an S field,
which has different meaning. Maybe emitting a diagnostic would be
appropriate?

Kay

[T. Modes]

kfj schrieb am Montag, 20. September 2021 um 10:19:39 UTC+2:

Okay, what's the 'dual lens assistant' and where can I find it?

Start with new project and add single image. Then run user defined assistant:

Edit>User defined assistant>Assistant for dual lens images

[Kay F. Jahnke]

Am 20.09.21 um 20:12 schrieb T. Modes:

Okay, thanks, I found it.

The result is disappointing:

- The cropping area of the two circular fisheye images is not placed
correctly, parts of the image which show the inside of the lens are
inside the cropping area, some content is cut off so much that the
output has black areas. The cropping area it's too small and off-center.

- the assistant produces two separate hfov values for the two images,
but both cameras in the device have identical hfov, so the second i-line
should have v=0. The only thing that differs are d and e, and y, p, r
for the second image

- CPs are few and far between, half of them high in the sky and in the
lawn near the nadir. Adding additional CPs with the CP tool in the
overlap and +/- 30 degrees off the horizon helps.

This is the assistant-generated PTO (p-line and i-lines only:

# hugin project file
#hugin_ptoversion 2
p f2 w6496 h3248 v360 k0 E14.2769 R0 n"TIFF_m c:LZW r:CROP"
m i0

# image lines
#-hugin autoCenterCrop=1 cropFactor=1
i w6912 h3456 f2 v402.651775543405 Ra0 Rb0 Rc0 Rd0 Re0
Eev14.2769324951933 Er1 Eb1 r0 p0 y0 TrX0 TrY0 TrZ0 Tpy0 Tpp0 j0 a0 b0
c0 d-1615.03702730985 e3.9466693357031 g0 t0 Va1 Vb0 Vc0 Vd0 Vx0 Vy0
S113,3569,108,3356 Vm5 n"image.jpg"
#-hugin autoCenterCrop=1 cropFactor=1
i w6912 h3456 f2 v383.086221346944 Ra=0 Rb=0 Rc=0 Rd=0 Re=0
Eev14.2769324951933 Er1 Eb1 r0.144827568515012 p-0.223075234348641
y166.756224026178 TrX0 TrY0 TrZ0 Tpy0 Tpp0 j0 a=0 b=0 c=0
d1605.72455893899 e16.1069820972669 g=0 t=0 Va=0 Vb=0 Vc=0 Vd=0 Vx=0
Vy=0 S3334,6790,120,3368 Vm5 n"image.jpg"

For comparison, here's a manually reworked PTO file for the same image
which produces a near-perfect stitch:

# hugin project file
#hugin_ptoversion 2
p f2 w6496 h3248 v360 k0 E14.2769 R0 n"TIFF_m c:LZW r:CROP"
m i0

# image lines
#-hugin cropFactor=1
i w6912 h3456 f2 v392.926531308803 Ra0 Rb0 Rc0 Rd0 Re0
Eev14.2769324951933 Er1 Eb1 r0 p0 y0 TrX0 TrY0 TrZ0 Tpy0 Tpp0 j0 a0 b0
c0 d-1597.21304025071 e2.31010096005214 g0 t0 Va1 Vb0 Vc0 Vd0 Vx0 Vy0
S4,3460,97,3345 Vm5 n"image.jpg"
#-hugin cropFactor=1
i w6912 h3456 f2 v=0 Ra0 Rb0 Rc0 Rd0 Re0 Eev14.2769324951933 Er1 Eb1
r0.123583351129881 p0.0411329889565548 y165.690011847671 TrX0 TrY0 TrZ0
Tpy0 Tpp0 j0 a0 b0 c0 d1602.86788254359 e14.1561252521546 g0 t0 Va1 Vb0
Vc0 Vd0 Vx0 Vy0 S3457,6913,103,3351 Vm5 n"image.jpg"

The assistant-generated PTO was made with hugin Pre-Release
2020.1.0.a7e93b18e270, so maybe the assistant has improved with newer
versions.

Kay

[T. Modes]

kfj schrieb am Dienstag, 21. September 2021 um 11:28:33 UTC+2:

- The cropping area of the two circular fisheye images is not placed
correctly, parts of the image which show the inside of the lens are
inside the cropping area, some content is cut off so much that the
output has black areas. The cropping area it's too small and off-center.

This assistant may need some fine-tuning for a specific model, as already written. The dual lens assistant was meant as a starting point, not as final product.

 

- the assistant produces two separate hfov values for the two images,
but both cameras in the device have identical hfov, so the second i-line
should have v=0. The only thing that differs are d and e, and y, p, r
for the second image

I doubt that the two lenses have really identical hfov - at least at a pixel scale. So the idea was to take the deviation between the two lenses into account.

Some models will have more deviation, other are similar. YMMV.

[Kay F. Jahnke]

Am 21.09.21 um 18:23 schrieb T. Modes:

> kfj schrieb am Dienstag, 21. September 2021 um 11:28:33 UTC+2:
>
> - The cropping area of the two circular fisheye images is not placed
> correctly, parts of the image which show the inside of the lens are
> inside the cropping area, some content is cut off so much that the
> output has black areas. The cropping area it's too small and
> off-center.
>
>
> This assistant may need some fine-tuning for a specific model, as
> already written. The dual lens assistant was meant as a starting point,
> not as final product.

I hope my observations will help improve it.

>
> - the assistant produces two separate hfov values for the two images,
> but both cameras in the device have identical hfov, so the second
> i-line
> should have v=0. The only thing that differs are d and e, and y, p, r
> for the second image
>
> I doubt that the two lenses have really identical hfov - at least at a
> pixel scale. So the idea was to take the deviation between the two
> lenses into account.

Producing hfov values which are 20 degrees apart is definitely wrong. I
suspect that the assistant relies on too few control points. If you want
to stick with separate v values for the two i-lines, my suggestion would
be to try something like this:

start out with the same v for both i-lines (so v=0 in the second one)
and optimize for y,p,r,d,e in the second image. This gives you a good
base to start from to detect pixel-level differences in v for the two
cameras, because with the initial result you can generate more and
better CPs. Once you have these, you can decouple v and see if your
optimization gets any better. I tried that, and with 27 CPs and v
decoupled the difference the optimizer produced was a mere .003 degrees.

Typically, the overlap in this type of device is small and in an area
where the lens does not 'behave well', so hoping to get the lens
characteristic right by optimizing based on (too few) CPs in this
overlap region is likely to produce suboptimal results, especially if
the initial guess which cpfind needs to find good CPs is not very
accurate. I am convinced that assuming equal FOV is a better initial
guess, at least as a starting point.

> Some models will have more deviation, other are similar.

I'd be curious to see measurements from others working with these
devices. I'd assume that the differences you get from device to device
result from slight differences in how the individual cameras are mounted
in the device, and are reflected in the outcome of the optimization for
y,p,r,d and e, while the hfov should come out very similar, and widely
different values are most likely wrong.

I hope to get more sample images in the near future, also from a
different device, and I've also asked for samples with the camera yawed
by 90 degrees which would help greatly in working out the fov issue.
I'll post again when I have the data.

Kay