Help stitching a large (and messy) aerial mosaic

[Derek Dohler]

Hello,

I am attempting to stitch a large and fairly messy set of aerial photos, and not having much luck. That's not a huge surprise because the photos were hand-shot out of a moving aircraft, so they were certainly not taken with this application in mind. However, because the area being photographed is coastal barrier islands, which are fairly flat, I was hoping I might nonetheless be able to get something that appears reasonable (even if it is not the same as a georeferenced map image).

I have been trying to follow the tutorials on stitching mosaics and flat scanned images (https://hugin.sourceforge.io/tutorials/Mosaic-mode/en.shtml , https://hugin.sourceforge.io/tutorials/scans/en.shtml ).

However, whenever I get beyond 4-6 images, the optimization generally seems to break down and I get ridiculous results or the panorama fails to stitch at all. I am scripting this due to the large number of images, and my flow is as follows:

1. pto_gen -o {pto_file} {images}

2. cpfind --celeste --multirow --cache --fullscale -o {pto_file} {pto_file}

3. pto_var --opt y,p,r,TrX,TrY,TrZ --anchor={middle_image} -o {pto_file} {pto_file}

4. autooptimiser -n -o {pto_file} {pto_file}

5. pano_modify --projection=0 --fov=AUTO --canvas=AUTO --crop=AUTO --ldr-file=JPG --ldr-compression=99 -o {pto_file} {pto_file}

6. nona -m TIFF_m -o {prefix} {pto_file}

7. enblend --compression=99 -o {output_name} {prefix}*.tif

The images are public, so in case I can convince anyone more knowledgeable than me to give this a try, I'm using the Raw images from here: https://experience.arcgis.com/experience/010503b4c64b4ff6a7f3570220a53647/page/Data-Explorer-

Specifically, I've been experimenting with "21May2018 Camera2-[6290-6325].jpg"

Thank you so much for any tips you can provide!

Derek

[wirz]

Hello Derek,

> I am attempting to stitch a large and fairly messy set of aerial photos,
> and not having much luck. That's not a huge surprise because the photos
> were hand-shot out of a moving aircraft, so they were certainly not taken
> with this application in mind. However, because the area being photographed
> is coastal barrier islands, which are fairly flat, I was hoping I might
> nonetheless be able to get something that appears reasonable (even if it is
> not the same as a georeferenced map image).

I agree that this should be possible. Manually removing control points
from moving waves and anything higher than sea level may help (but I
didn't find it necessary).

> However, whenever I get beyond 4-6 images, the optimization generally seems
> to break down and I get ridiculous results or the panorama fails to stitch
> at all. I am scripting this due to the large number of images, and my flow
> is as follows:
>
> 1. pto_gen -o {pto_file} {images}
> 2. cpfind --celeste --multirow --cache --fullscale -o {pto_file} {pto_file}
> 3. pto_var --opt y,p,r,TrX,TrY,TrZ --anchor={middle_image} -o {pto_file}
> {pto_file}
> 4. autooptimiser -n -o {pto_file} {pto_file}
> 5. pano_modify --projection=0 --fov=AUTO --canvas=AUTO --crop=AUTO
> --ldr-file=JPG --ldr-compression=99 -o {pto_file} {pto_file}
> 6. nona -m TIFF_m -o {prefix} {pto_file}
> 7. enblend --compression=99 -o {output_name} {prefix}*.tif
>
> The images are public, so in case I can convince anyone more knowledgeable
> than me to give this a try, I'm using the Raw images from
> here: https://experience.arcgis.com/experience/010503b4c64b4ff6a7f3570220a53647/page/Data-Explorer-
>
> Specifically, I've been experimenting with "21May2018
> Camera2-[6290-6325].jpg"

For easier download:
https://twi-aviandata.s3.amazonaws.com/HighResolutionImages/2018/21May2018%20Camera1-${i}.jpg

I agree that it is not a simple example. I'm getting this:
http://78.46.190.157:8080/birds.jpg, ie, good alignment but the mapping
is a little wonky. While I don't think my result is great I have some
thoughts about that:

-- the images overlap far too much. I have removed 2 out of every 3.
That could be done systematically, if they always overlap that much.
One could also truncate each image.

-- I assume the images have been shot out the side of the plane, ie, not
vertically down. Then it is not correct to set yaw and pitch of the
anchor to zero. (I'm getting a yaw of ~20degrees).

-- freely optimising y/p/r/Tx/Ty/Tz of many images hardly ever converges
into the intended minimum. The obvious strategies are to optimise a few
images, add a few more, optimise them together, etc. Here, I have
optimised first Tx/Ty for all, then y/p/r for all except the anchor,
then y/p for the anchor. Tz has only been included for some images --
optimising it for all yields better agreement but a worse mapping (and I
assume the plane has been at a consistent height). I don't have a good
idea how to deal with that -- the problem is obviously that the degrees
of freedom are not orthogonal.

-- having any right angles on the ground and using them for line control
points would help very much.

best regards, lukas wirz

[Derek Dohler]

Thank you Lukas! This is extremely helpful.

I agree that it is not a simple example. I'm getting this:
http://78.46.190.157:8080/birds.jpg, ie, good alignment but the mapping
is a little wonky. While I don't think my result is great I have some
thoughts about that:

This is better than anything I've managed so far, this looks really good! Thanks for trying!

 

-- the images overlap far too much. I have removed 2 out of every 3.
That could be done systematically, if they always overlap that much.
One could also truncate each image.

Okay, that's good to have confirmed. I suspected something like this might be happening, and at one point I tried stitching "adjacent" images (based on EXIF timestamp) into mini-panoramas before stitching the bigger one, which seemed to help somewhat. I can repurpose that logic to simply skip images taken close together instead.

 

-- I assume the images have been shot out the side of the plane, ie, not
vertically down. Then it is not correct to set yaw and pitch of the
anchor to zero. (I'm getting a yaw of ~20degrees).

Yes, this is correct. And thank you! I don't think I would have ever realized this about the anchor on my own.

 

-- freely optimising y/p/r/Tx/Ty/Tz of many images hardly ever converges
into the intended minimum. The obvious strategies are to optimise a few
images, add a few more, optimise them together, etc. Here, I have
optimised first Tx/Ty for all, then y/p/r for all except the anchor,
then y/p for the anchor. Tz has only been included for some images --
optimising it for all yields better agreement but a worse mapping (and I
assume the plane has been at a consistent height). I don't have a good
idea how to deal with that -- the problem is obviously that the degrees
of freedom are not orthogonal.

Okay, perfect, this sounds workable. I hadn't considered a multi-pass optimization like this, but it makes sense to do positions and y/p/r separately to prevent them from confounding each other during optimization. I think that the plane GPS track may be available somewhere in the data dump, so it might be possible to impute an elevation to the images based on timestamps. It probably wouldn't be precise enough to use directly, but perhaps it could be used as a starting point that is close to reality. But that sounds like an idea for v2. :)

 

-- having any right angles on the ground and using them for line control
points would help very much.

This is good to know. There aren't always going to be straight lines available but in some cases there may be due to debris. I will keep an eye out for opportunities to use this.

Thank you very much for your help!

Best,

Derek

 

best regards, lukas wirz

[Derek Dohler]

Update: this has worked very well. I have converted the multi-pass optimization you described into my scripted pipeline and it is working quite well. I have managed to successfully mosaic up to 65 images at a time (albeit skipping some of them to avoid excessive overlap). I am going to continue expanding to even larger batches.

Thanks again for your help!

Best,

Derek