Question regarding image_align

[Ariane]

Hi,

I was wondering, if the image_align tool is suitable for aligning non-mapprojected images as well and if that would even make sense? I was experimenting with two images with RPCs, but ran into the following problems:

1) When using the disparity produced from previously running parallel_stereo instead of IP matches to find the alignment transform, the points seem to be off. I am attaching two screenshots from the stereo_gui. The IP matches image_align generates from the disparity do not correspond to the images I provided as input to parallel_stereo and image_align, but rather to the rectified images *-L.tif and *-R.tif. Is this mismatch an issue?

2) As both reference and source image are not georeferenced, the aligned image won't have a reference either and loses the RPC metadata. Is there any way to transfer the alignment transform to the camera of the second image though? I was thinking about something like:
bundle_adjust $img2 -t rpc -o ba/run --apply-initial-transform-only --initial-transform align/run-transform.txt
and then re-creating the RPCs with cam2rpc, however, bundle_adjust requires a 4x4 matrix and image_align only provides 3x3.

Thanks in advance,

Ariane

[Oleg Alexandrov]

Dear Ariane,

Thank you for trying the tools. 

I was wondering, if the image_align tool is suitable for aligning non-mapprojected images as well and if that would even make sense?

Yes, the tool should work that way. It should be also easy to adapt our mosaicking tool to merge the aligned images with no mapprojection.

1) When using the disparity produced from previously running parallel_stereo instead of IP matches to find the alignment transform, the points seem to be off. I am attaching two screenshots from the stereo_gui. The IP matches image_align generates from the disparity do not correspond to the images I provided as input to parallel_stereo and image_align, but rather to the rectified images *-L.tif and *-R.tif. Is this mismatch an issue?

That indeed can be an issue, in general. But, in the example in the documentation, at https://stereopipeline.readthedocs.io/en/latest/tools/image_align.html,  it is shown that parallel_stereo is invoked with the option --correlator-mode, when the hope is that L.tif is same as left input image, and same with R.tif being the right input image, as alignment method is then set to none. If you tried it that way and it produced the above misalignment, that's a bug which I would need to look into.

 

2) As both reference and source image are not georeferenced, the aligned image won't have a reference either and loses the RPC metadata. Is there any way to transfer the alignment transform to the camera of the second image though? I was thinking about something like:
bundle_adjust $img2 -t rpc -o ba/run --apply-initial-transform-only --initial-transform align/run-transform.txt
and then re-creating the RPCs with cam2rpc, however, bundle_adjust requires a 4x4 matrix and image_align only provides 3x3.

 

The above won't work, because the --initial-transform is meant to use a 3D transform among objects in space in units of meters, while this one from image_align is a pixel transform in 2D. 

What can work is to just merge the obtained aligned second image with the first image it was aligned to, as they now agree pixel-wise wherever they overlap. Then, the obtained merged image will be in the camera coordinates of the first image, as long as the second aligned image is a "continuation of the first image" without changing the (0, 0) pixel, so the relationship between camera model and pixels does not change.

I am not sure what you would gain by doing this though. Note that the two input images that you have, which are raw, are acquired from different perspectives, to do stereo. So they cannot be merged by a simple 2D transform in pixel space. The only way to merge images from different perspectives is to mapproject them first, onto a terrain very similar to what the images look at, using the same ground resolution, then mosaic them with dem_mosaic.

Happy to hear your thoughts. This is a new tool, and its applications are still not clear.

Oleg

--
You received this message because you are subscribed to the Google Groups "Ames Stereo Pipeline Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/7375c41b-ac8b-4c45-a3fb-325cf3752339n%40googlegroups.com.

[Ariane]

Dear Oleg, 

I have yet another question regarding the image_align tool: When I use a disparity as input, I sometimes don't get matching interest points over the entire image overlap, even though the disparity raster does have valid pixels in the missing area. What is the reason for that? Are the points or the disparity filtered somehow, or is there a limit to the number of samples I can take? Is there a way to force the generation of interest point matches over the entire extent of the disparity?

I am attaching two examples of IP matches generated with image_align, where parts of the corresponding disparity (dy is shown) are missing, and one good example where points are distributed over the entire scene.

Happy to hear from you.

Ariane

[Oleg Alexandrov]

Normally the interest point matches should be everywhere where there is disparity. I just looked at the code, it simply iterates over the disparity with a given sample rate and creates an interest point for each read disparity pixel.

I also looked at a testcase I have and there are matches everywhere. I am not sure what is going on in your case. Note that there is inlier filtering using RANSAC, maybe it throws matches out. In my case, it did not, I got the following text when running image_align:

        Found 177371 / 177371 inliers using RANSAC.

If you see that a lot of the matches are removed, you can try relaxing the ransac parameters. so --inlier-threshold.

It looks btw that you plotted the matches in some external tool, likely with python. ASP's stereo_gui can show them too, overlaid onto the left and right image.

To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/dfd891f8-0d5b-4ed4-9e85-6d051d88e1d3n%40googlegroups.com.

[Oleg Alexandrov]

The disparity computed by stereo does not mind if the right image has some local distortions relative to the left image, in fact, that is almost always the case. However, when the disparity is used by image_align, this tool tries to find the best-fitting rigid or similarity transform, so areas where the right image is locally deformed relative to the left don't fit that kind of transform, so the disparity in that region is thrown out.

You can surely use a high outlier threshold to keep matches over the whole image, as long as you are fine with the computed alignment transform being some kind of compromise over the regions where the right and left image don't agree up to such an alignment transform very well.

On Fri, Aug 5, 2022 at 2:45 AM Ariane Müting <ariane....@gmail.com> wrote:

Dear Oleg,

You were right - it appears that RANSAC discarded some matches:

Found 5564 / 8309 inliers using RANSAC.

Setting the inlier threshold to 30 pixels worked to retrieve matches over the entire disparity.  Would it be an issue to set a really high threshold here, e.g. 1000 pixels or so, to make sure to always get matches over the entire disparity? Technically, the disparity should have already been filtered when running parallel_stereo, so there would be no need to remove outliers.

Best,

Ariane