idea/feature request: use ipfind/ipmatch georeferencing (aliging map projected images) and for improved pc_align instead of using hillshades

91 views
Skip to first unread message

Andrew Annex

unread,
Apr 29, 2020, 7:08:36 PM4/29/20
to Ames Stereo Pipeline Support
Hey all,

I was playing around with ipfind/ipmatch and I was wondering two things:

1. could it be used to align normal visible rasters, ie instead of using arcgis/qgis to georegister/co-register map projected images (I tried and although the matching worked great I didn’t see a way to apply to transformations).

2 as an option instead of using the hill shades to align in pc_align. I used ipfind/match on a hirise and ctx pair which worked great, but I couldn’t use the transformation because I guess there is no associated Z information, but maybe there could be an advanced mode that uses the pixel positions of the matched points to get Z values from corresponding DEMs. Often when attempting to use the hill shade align I will end up with low numbers of ip matches (less than 30), which is totally adequate, but once I was left with 4 matches that somehow caused a 180 degree rotation of the hirise DEM.

Alexandrov, Oleg (ARC-TI)[KBR Wyle Services, LLC]

unread,
Apr 29, 2020, 7:47:46 PM4/29/20
to Andrew Annex, Ames Stereo Pipeline Support
Andrew,

The pc_align tool with the --initial-transform-from-hillshading option already uses ipfind and ipmatch under the hood.  One can pass options to those two tools in pc_align. (Or maybe I misunderstand what you say.)

> 1. could it be used to align normal visible rasters, ie instead of using arcgis/qgis to georegister/co-register map projected images (I tried and although the matching worked great I didn’t see a way to apply to transformations).

Yeah, we have no option to apply a transform to a georeferenced image. A non-smart way of doing it is to use pc_align with two georeferenced images with an --initial-transform that you want to apply. Then those images will be interpreted as DEMs with the image value as a height. It will move that image as a cloud, and can save that aligned cloud to disk as a PC file. Then point2dem can be used to rasterize it to get back a (transformed) image. Ideally we should have the ability to apply a 2D transform to a 2D image to get back a 2D image using interpolation. Maybe one day. Should be easy to do in python with the relevant gis packages I guess, like gdal, or something.

Oleg

From: ames-stereo-pi...@googlegroups.com <ames-stereo-pi...@googlegroups.com> on behalf of Andrew Annex <andrew...@gmail.com>
Sent: Wednesday, April 29, 2020 4:08 PM
To: Ames Stereo Pipeline Support <ames-stereo-pi...@googlegroups.com>
Subject: [EXTERNAL] idea/feature request: use ipfind/ipmatch georeferencing (aliging map projected images) and for improved pc_align instead of using hillshades
 
--
You received this message because you are subscribed to the Google Groups "Ames Stereo Pipeline Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/92d2ff8a-15ee-46fa-bb8c-cc46740b0a5f%40googlegroups.com.

Shashank Bhushan

unread,
Apr 29, 2020, 8:17:09 PM4/29/20
to Andrew Annex, Ames Stereo Pipeline Support
Oleg did suggest a workaround to align mapprojected images using pc_align  transform: https://github.com/NeoGeographyToolkit/StereoPipeline/issues/265

Perhaps one can use the formula here: https://scikit-image.org/docs/dev/api/skimage.transform.html#skimage.transform.AffineTransform and here: https://stackabuse.com/affine-image-transformations-in-python-with-numpy-pillow-and-opencv/ to apply the transformation matrix output by pc_align to the image geo-transform using python.  
The transformation matrix can also be found in python once you have the coordinates of the match points in the 2 images using ipfind/ipmatch. See this simple implementation here: https://scikit-image.org/docs/dev/auto_examples/transform/plot_matching.html#sphx-glr-auto-examples-transform-plot-matching-py. Specifically these 2 blocks:
"""
# find correspondences using simple weighted sum of squared differences
src = []
dst = []
for coord in coords_orig_subpix:
    src.append(coord)
    dst.append(match_corner(coord))
src = np.array(src)
dst = np.array(dst)

"""
The source and dst array will contain the match point coordinates in the 2 images.
and 
"""# robustly estimate affine transform model with RANSAC
model_robust, inliers = ransac((src, dst), AffineTransform, min_samples=3,
                               residual_threshold=2, max_trials=100) 
outliers = inliers == False
"""
The model_robust variable will contain the transformation parameters.

I will try to check what I wrote here makes any sense by running it in sometime. I tried these with sample image data in skimage, and it worked well there, but have not tried on satellite images yet, so these are suggestions, not answers :) .

--
You received this message because you are subscribed to the Google Groups "Ames Stereo Pipeline Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/92d2ff8a-15ee-46fa-bb8c-cc46740b0a5f%40googlegroups.com.


--
Shashank Bhushan
Graduate Student/Research Assistant
Civil & Environmental Engineering
University of Washington, Seattle

Andrew Annex

unread,
Apr 29, 2020, 8:32:26 PM4/29/20
to Ames Stereo Pipeline Support
Hey oleg, let me clarify my ideas a bit, yes these were both inspired by using the --initial-transform-from-hillshading option.

For my first point, my idea would be to use the matched points on visible images to make an affine or some other transformation to align two (or more) map projected images. From the rest of your reply it sounds like I was clear enough.

For the second point, my idea was to use the visible images for the initial XY translation/rotation for DEM alignment, but to then derive the vertical (Z axis) offset from corresponding DEMs at the positions of those point matches in XY coordinates. Even with resolution differences, by matching sift features on the panchromatic images from say HiRISE and CTX, many SIFT feature matches can be found:




Without having read the source code for pc_align recently, I assume that a similar process must exist already for the hillshade alignment. Hillshade alignment works well when I get a decent number of point matches, but I sometimes run into situations with N<10 where things go wrong unless I change some of the ipfind options to boost the number of potential points to match. I would hope that matching on the imagery would be a bit more reliable than the hillshades.

I think that this would not be as straight forward if point clouds are being used however.


On Wednesday, April 29, 2020 at 11:47:46 PM UTC, Alexandrov, Oleg (ARC-TI)[KBR Wyle Services, LLC] wrote:
Andrew,

The pc_align tool with the --initial-transform-from-hillshading option already uses ipfind and ipmatch under the hood.  One can pass options to those two tools in pc_align. (Or maybe I misunderstand what you say.)

> 1. could it be used to align normal visible rasters, ie instead of using arcgis/qgis to georegister/co-register map projected images (I tried and although the matching worked great I didn’t see a way to apply to transformations).

Yeah, we have no option to apply a transform to a georeferenced image. A non-smart way of doing it is to use pc_align with two georeferenced images with an --initial-transform that you want to apply. Then those images will be interpreted as DEMs with the image value as a height. It will move that image as a cloud, and can save that aligned cloud to disk as a PC file. Then point2dem can be used to rasterize it to get back a (transformed) image. Ideally we should have the ability to apply a 2D transform to a 2D image to get back a 2D image using interpolation. Maybe one day. Should be easy to do in python with the relevant gis packages I guess, like gdal, or something.

Oleg


Sent: Wednesday, April 29, 2020 4:08 PM
To: Ames Stereo Pipeline Support <ames-stereo-pi...@googlegroups.com>
Subject: [EXTERNAL] idea/feature request: use ipfind/ipmatch georeferencing (aliging map projected images) and for improved pc_align instead of using hillshades
Hey all,

I was playing around with ipfind/ipmatch and I was wondering two things:

1. could it be used to align normal visible rasters, ie instead of using arcgis/qgis to georegister/co-register map projected images (I tried and although the matching worked great I didn’t see a way to apply to transformations).

2 as an option instead of using the hill shades to align in pc_align. I used ipfind/match on a hirise and ctx pair which worked great, but I couldn’t use the transformation because I guess there is no associated Z information, but maybe there could be an advanced mode that uses the pixel positions of the matched points to get Z values from corresponding DEMs. Often when attempting to use the hill shade align I will end up with low numbers of ip matches (less than 30), which is totally adequate, but once I was left with 4 matches that somehow caused a 180 degree rotation of the hirise DEM.

--
You received this message because you are subscribed to the Google Groups "Ames Stereo Pipeline Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline-support+unsub...@googlegroups.com.

Andrew Annex

unread,
Apr 29, 2020, 8:43:10 PM4/29/20
to Ames Stereo Pipeline Support
thanks Shashank, I hadn't seen that github issue before. It would be interesting to make a command line tool with this to see how robust it would be.


On Thursday, April 30, 2020 at 12:17:09 AM UTC, Shashank Bhushan wrote:
Oleg did suggest a workaround to align mapprojected images using pc_align  transform: https://github.com/NeoGeographyToolkit/StereoPipeline/issues/265

Perhaps one can use the formula here: https://scikit-image.org/docs/dev/api/skimage.transform.html#skimage.transform.AffineTransform and here: https://stackabuse.com/affine-image-transformations-in-python-with-numpy-pillow-and-opencv/ to apply the transformation matrix output by pc_align to the image geo-transform using python.  
The transformation matrix can also be found in python once you have the coordinates of the match points in the 2 images using ipfind/ipmatch. See this simple implementation here: https://scikit-image.org/docs/dev/auto_examples/transform/plot_matching.html#sphx-glr-auto-examples-transform-plot-matching-py. Specifically these 2 blocks:
"""
# find correspondences using simple weighted sum of squared differences
src = []
dst = []
for coord in coords_orig_subpix:
    src.append(coord)
    dst.append(match_corner(coord))
src = np.array(src)
dst = np.array(dst)

"""
The source and dst array will contain the match point coordinates in the 2 images.
and 
"""# robustly estimate affine transform model with RANSAC
model_robust, inliers = ransac((src, dst), AffineTransform, min_samples=3,
                               residual_threshold=2, max_trials=100) 
outliers = inliers == False
"""
The model_robust variable will contain the transformation parameters.

I will try to check what I wrote here makes any sense by running it in sometime. I tried these with sample image data in skimage, and it worked well there, but have not tried on satellite images yet, so these are suggestions, not answers :) .

On Wed, Apr 29, 2020 at 4:14 PM Andrew Annex <andre...@gmail.com> wrote:
Hey all,

I was playing around with ipfind/ipmatch and I was wondering two things:

1. could it be used to align normal visible rasters, ie instead of using arcgis/qgis to georegister/co-register map projected images (I tried and although the matching worked great I didn’t see a way to apply to transformations).

2 as an option instead of using the hill shades to align in pc_align. I used ipfind/match on a hirise and ctx pair which worked great, but I couldn’t use the transformation because I guess there is no associated Z information, but maybe there could be an advanced mode that uses the pixel positions of the matched points to get Z values from corresponding DEMs. Often when attempting to use the hill shade align I will end up with low numbers of ip matches (less than 30), which is totally adequate, but once I was left with 4 matches that somehow caused a 180 degree rotation of the hirise DEM.

--
You received this message because you are subscribed to the Google Groups "Ames Stereo Pipeline Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline-support+unsub...@googlegroups.com.

Alexandrov, Oleg (ARC-TI)[KBR Wyle Services, LLC]

unread,
Apr 29, 2020, 11:47:45 PM4/29/20
to Andrew Annex, Ames Stereo Pipeline Support
I see. What you propose would be easy enough. It would assume that images are overlayed accurately on top of the DEMs. Then matches would be found among the images, and those would be transferred on top of the DEMs. 

In fact, if each of your DEMs has precisely the same resolution as the image mapprojected on top of it, the current approach can be used with very small changes. First, ipfind and ipmatch is used to find the matches, among the images, then pc_align is invoked with the two DEMs and the value of the option --match-file being obtained from the images. Here it is implicitly assumed that the DEMs and images have one-to-one correspondence among the pixels. Then pc_align will extract the height info for each of those pixels and find the best-fitting 3D transform. 

In short, it is easy to write a little script that would do calls to ipfind, ipmatch, then pc_align with that match file option. 
From: ames-stereo-pi...@googlegroups.com <ames-stereo-pi...@googlegroups.com> on behalf of Andrew Annex <andrew...@gmail.com>
Sent: Wednesday, April 29, 2020 5:32 PM

To: Ames Stereo Pipeline Support <ames-stereo-pi...@googlegroups.com>
Subject: Re: [EXTERNAL] idea/feature request: use ipfind/ipmatch georeferencing (aliging map projected images) and for improved pc_align instead of using hillshades
 
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/148d4132-f7c7-42f9-8863-953c39b91de9%40googlegroups.com.

Andrew Annex

unread,
Apr 30, 2020, 11:40:02 AM4/30/20
to Ames Stereo Pipeline Support
I'll give that a try, but I think I tried that same process before but I got some error saying the match files had to be on the dems not the DRGs, but I'll double check later today to see what error it was.

On Thursday, April 30, 2020 at 3:47:45 AM UTC, Alexandrov, Oleg (ARC-TI)[KBR Wyle Services, LLC] wrote:
I see. What you propose would be easy enough. It would assume that images are overlayed accurately on top of the DEMs. Then matches would be found among the images, and those would be transferred on top of the DEMs. 

In fact, if each of your DEMs has precisely the same resolution as the image mapprojected on top of it, the current approach can be used with very small changes. First, ipfind and ipmatch is used to find the matches, among the images, then pc_align is invoked with the two DEMs and the value of the option --match-file being obtained from the images. Here it is implicitly assumed that the DEMs and images have one-to-one correspondence among the pixels. Then pc_align will extract the height info for each of those pixels and find the best-fitting 3D transform. 

In short, it is easy to write a little script that would do calls to ipfind, ipmatch, then pc_align with that match file option. 

Amaury Dehecq

unread,
May 1, 2020, 4:06:31 AM5/1/20
to Ames Stereo Pipeline Support
Hi Andrew,

Just a small addition to the other replies. My experience with --initial-transform-from-hillshading is that it can work really well with the right set of parameters. I typically increase the values of IPs to be detected first compared to the default value. The issue is that you can end up with a lot of outliers. Increasing the number of RANSAC iterations or changing the inlier-threshold will help getting rid of those if you have enough inliers. The unfiltered/filtered matches are saved in the pc_align folder (the filtered one have extension -clean.match) and can be visualized to check that you get good matches first. Hopefully this will help getting more matches even with more difficult terrain. I use for example the following parameters '--ip-per-image 5000000' and '--ransac-iterations 100000'
Then, if you final transformation is really bad, you can play with the '--alignment-method' and '--max-displacement' options. A method using similarity will have more degrees of freedom and is more likely to fail. So you can call pc_align a first time with a simpler transform to get DEMs that are closer, then call pc_align a second time with a similarity transform and apply a lower max-displacement threshold (Note that when using the hillshading option, the threshold is applied AFTER the transformation from the hillshades has been applied, so it can be much lower usually).

For Earth images, it's much easier to find matches between shaded relief that between images due to the changing surface conditions. But I can imagine that on Mars, surface conditions don't change that much and the relief can have very few features. As Oleg said, it won't be too difficult to runt ipfind and ipmatch on your images, calculate a transformation using your matches in an external script and use this transformation as input to pc_align --initial-transform.
Attached is a Python script to read/write ASP match files.

Amaury
read_match.py

Shashank Bhushan

unread,
Oct 21, 2020, 10:48:31 PM10/21/20
to Andrew Annex, Ames Stereo Pipeline Support
Andrew, here is a script shared by Amaury which can do what you want (write binary match files).
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Ames Stereo Pipeline Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Ames Stereo Pipeline Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/126d6e0e-8912-44f5-88cb-564f9361578e%40googlegroups.com.
read_match.py

Andrew Annex

unread,
Oct 21, 2020, 10:53:02 PM10/21/20
to Shashank Bhushan, Ames Stereo Pipeline Support
Cheers! Was this posted before on the google group? I think I had a vague memory of a similar discussion but I didn’t find this, possibly due to how late it is for me.

Thanks though that should help me make progress in the morning!

-Andrew Annex

On Oct 21, 2020, at 10:48 PM, Shashank Bhushan <sbag...@uw.edu> wrote:


<read_match.py>

andrew...@gmail.com

unread,
Oct 21, 2020, 10:55:57 PM10/21/20
to Ames Stereo Pipeline Support
and now I see it, heh okay I knew it sounded familiar... just missed it
Reply all
Reply to author
Forward
0 new messages