Aligning a days worth of photos from a timelapse against an anchor

[spmp]

I have a camera taking images every two minutes of an outdoor scene. The mount moves enough that the images need aligning. I am in the process of creating scripts to align and remap a days worth of images at a time (once a day) against a chosen anchor image and am looking for advice on how best to proceed.

What I have done and am aiming to do:

Found the lens parameters. The FOV is 98deg, so required for CPFind as it remaps.

I have a script which finds all images at midday, then I have created control points pairwise on all pairs to find the most suitable image to use as an anchor. The image with the lowest contrast (i.e overcast) matched with all other days so far and with the lowest total distance (I wish you could export the control point matrix to CSV!)

I have found that as the scene has street lights in at morning and evening that I can create control points between sequential pairs for a single day reliably without disjoint groups. Finding control points between days is highly problematic due to varying light and weather.

My idea is to in a scriptwise fashion:

1. Create a PTO file for a days worth of images with includes just the images, lens parameters and exclusion mask (mask out trees, sky, etc) based on a PTO file saved from Hugin.
2. Find the image closest to midday and insert the anchor image into the PTO file after it, making it the anchor for position.
3. Run CPFind using 'linearmatch', 'fullscale', and caching. Your input on good parameters here is highly appreciated!
4. Optimise the positions - Which parameters should I include and not... At this stage thinking y,p,r,TrX,TrY,TrZ.
5. Caclulate the field of view, crop, and output size etc. I have no idea how to do this so all images will align perfectly from day to day. I am assuming that the use of an anchor image will tie them all together.
   
6. Remap all images except the anchor (or with the anchor it doesn't matter)

• Is this the best way to do this? What other options do I have if I want to do it day at a time?
• Is there a tool for manipulating PTO files, such as removing an image etc?
  
• Are there Python bindings for hugin-tools?
• As in the notes above, how can I ensure consistent remapping day to day?
• How can I 'straighten' the anchor image? If I just included this and straightened using the GL viewer or similar, what parameters in the PTO file reflect this please?

I think you get the idea 8) I am hoping that there is a tool that does this already, but if not and this is the way to go I am looking forward to the challenge of it 8)

Cheers

[Bruno Postle]

There is a tool in the Panotools::Script Perl module called `nona-deshake` that does most of this, see this thread: https://groups.google.com/g/hugin-ptx/c/yFE6VF-mtGk/m/CCErzyeGDwAJ

You basically create a Hugin project with a single photo, straighten, crop, change projection, whatever. Then this is used as an anchor to align all the other photos in the sequence.

I planned on extending it to support multiple photos in this anchor project (so you could represent all different lighting conditions) but this hasn't happened yet.

There has been at least one attempt to create a python equivalent of Panotools::Script, I don't have any links :(

--
Bruno

On 16 May 2021 04:53:37 BST, spmp wrote:
>
>I have a camera taking images every two minutes of an outdoor scene. The
>mount moves enough that the images need aligning. I am in the process of
>creating scripts to align and remap a days worth of images at a time (once
>a day) against a chosen anchor image and am looking for advice on how best
>to proceed.
>
>What I have done and am aiming to do:
>Found the lens parameters. The FOV is 98deg, so required for CPFind as it
>remaps.
>
>I have a script which finds all images at midday, then I have created
>control points pairwise on all pairs to find the most suitable image to use
>as an anchor. The image with the lowest contrast (i.e overcast) matched
>
>with all other days so far and with the lowest total distance (I wish you
>could export the control point matrix to CSV!)
>
>I have found that as the scene has street lights in at morning and evening
>that I can create control points between sequential pairs for a single day
>reliably without disjoint groups. Finding control points between days is
>highly problematic due to varying light and weather.
>
>My idea is to in a scriptwise fashion:
>

>1. Create a PTO file for a days worth of images with includes just the

>images, lens parameters and exclusion mask (mask out trees, sky, etc)
>based
> on a PTO file saved from Hugin.

>2. Find the image closest to midday and insert the anchor image into the

> PTO file after it, making it the anchor for position.

>3. Run CPFind using 'linearmatch', 'fullscale', and caching. Your input

>
> on good parameters here is highly appreciated!

>4. Optimise the positions - Which parameters should I include and not...

> At this stage thinking y,p,r,TrX,TrY,TrZ.

> 5. Caclulate the field of view, crop, and output size etc. I have no

>idea how to do this so all images will align perfectly from day to day. I
>am assuming that the use of an anchor image will tie them all together.

> 6. Remap all images except the anchor (or with the anchor it doesn't
> matter)
>
>
> - Is this the best way to do this? What other options do I have if I

> want to do it day at a time?

>- Is there a tool for manipulating PTO files, such as removing an image
>
> etc?
> - Are there Python bindings for hugin-tools?
> - As in the notes above, how can I ensure consistent remapping day to day?
>- How can I 'straighten' the anchor image? If I just included this and

[spmp]

Thanks @Bruno,

 It looks like your script aligns each image to the anchor rather than sequentially, so... damn, I may have to go with my clunky approach.

I guess most of the magic is done for you with 'align_image_stack'.

Perl scares me TBH, so not sure if I will be able to extend 'nona-deshake' for my use case 8(

So many of the original questions remain unfortunately 8(

Ideas?

[Bruno Postle]

The problem with matching hundreds of photos sequentially is that the view drifts over time, so if you want perfectly stable output you need to keep referring back to an anchor.

align_image_stack has the same interface as cpfind, but only works where the images are nearly aligned already, this is not a issue with webcam footage that is already closely aligned.

The main difference here is that cpfind characterises and compares features, which works best when the photos were all taken at the same time, since this characterisation is very sensitive to changes in lighting conditions.

align_image_stack just compares the pixel data for any feature that looks like a 'corner' and shuffles the alignment to find the least-worst match, this is much more robust when comparing photos taken over a period of weeks or between night and day.

Perl is pretty similar to python if you ignore all the ::->~= stuff.

--
Bruno

On 18 May 2021 04:48:09 BST, jasper mackenzie wrote:
>Thanks @Bruno,
> It looks like your script aligns each image to the anchor rather than
>sequentially, so... damn, I may have to go with my clunky approach.
>I guess most of the magic is done for you with 'align_image_stack'.
>Perl scares me TBH, so not sure if I will be able to extend 'nona-deshake'
>for my use case 8(
>
>So many of the original questions remain unfortunately 8(
>
>Ideas?
>

[spmp]

I have had some time to come back to this after a long while... Thanks for your tips so far.

 I have tried a few things, but the best bet is still doing something like the following for each days images (the idea is to script this stuff once I find a good solution):

1. Load a template pto which only contains the output projection settings.
   
2. Add days images to a hugin project, loading the lens data.
3. Add a predetermined anchor image and move it to be the position in the photo stack which is just after solar midday.
4. Change the geometric optimisation to "Positions and Translations - yprxyz" (#hugin_optimizerMasterSwitch 34).
5. Save the pto file.
6. Run cpfind over the pto file using linear find with a match length of 12 to a new pto file.
7. Remove the mask line from the pto file.
8. Run auotoptimier over the pto file above, generating another.
9. Run nona over the pto file keeping intermediate images.

Some days match perfectly, but others, mostly earlier in the year, are almost right, but are quite wrong at the edges as evident top left.

Can you please look at these images and see if you can determine what I could do to fix this. Are my lens parameters wrong?

I suspect the difference is the physical translation of the camera - still relatively small amount, circa 10mm.

Images are attached, basically equidistant temporally.

Cheers,

Jasper