Umbra SAR - bundle adjustment fails to find interest points

[Ryan Rock]

I am attempting to replicate the Umbra SAR example in the ASP docs with some GEC images for a forest in Alabama. I have 16 GEC images and have reviewed the metadata to select pairs of images for testing based on the azimuth and incidence angles. Also, visually, the illumination in the images I selected looks similar enough. I am using a 1m USGS DEM for reference and have used dem_geoid to adjust it and have mapprojected the images.

So far, any pair I try will fail to find enough interest points under the default settings. I can force the bundle adjustment to work by running:

bundle_adjust -t rpc img1.tif img2.tif --remove-outliers-params "75.0 3.0 50 50" --mapprojected-data "img1_proj.tif img2_proj.tif dem-adj.tif" --ip-per-tile 1000 -o ba/run

then can run stereo using:

parallel_stereo -t rpc --bundle-adjust-prefix ba/run --stereo-algorithm asp_mgm --ip-per-tile 1500 --min-num-ip 15 img1_proj.tif img2_proj.tif stereo/run dem-adj.tif

But the resulting DEM after doing the remaining steps in the example (point2dem/pc_align/point2dem) is full of masked pixels. 

[Oleg Alexandrov]

SAR is very challenging for ASP. 

You can try inspecting the left and right mapprojected images when overlaid with georeference. They should hopefully be similar enough and in the same place. You can also look at the stereo convergence angle (there's a mention in the introduction). 

The "masked pixels" likely means that there's a lot of locations where correlation fails. 

You can also see what the stereo program prints about the search range, homography transform, and could even share that. 

There's also a file called GoodPixelMap.tif (in the stereo output directory) that shows the correlation mask before triangulation. Here trying to understand if the failure is during correlation, when this mask should be bad, or during triangulation.

If no luck, you can share privately a pair of images that you think are reasonable enough, with a good convergence angle, etc, but it still fails.

--
You received this message because you are subscribed to the Google Groups "Ames Stereo Pipeline Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/be4f6d31-8801-4ad4-9c2c-b823cb55594cn%40googlegroups.com.

[Ryan Rock]

When mapprojecting the images using ASP I get a warning that the images are already map projected (they are in WGS84) but it will complete the operation, resulting in images that have been resampled to 0.5m and are in the same projection as my reference DEM, which is a 1m image in WGS84 UTM 16N. 

When viewing these mapprojected images in a desktop GIS however, the resulting images are rotated 90 degrees with respect to the original images and do not align with the basemap/other data layers. Because of this, I tried to project the images in other software then use the projected images in the bundle adjustment command, replacing the projected images from ASP. This still fails during the "Filtering interest point matches using homography" stage, even though it initially find 667 matches. The RANSAC continually retires with fewer and fewer inliers until it fails to find matches. Command:

bundle_adjust -t rpc img4.tif img6.tif --ip-per-tile 500 --mapprojected-data "img4_arcProj.tif img6_arcProj.tif dem-adj-clip.tif" -o ba13/run

However if I run the bundle adjustment with the original images only using

bundle_adjust -t rpc img4.tif img6.tif -o ba/run,

it succeeds in finding 87 matches. Is this because it is skipping the homography filtering step and does this mean something is wrong with the DEM input? But because these images do not have the same resolution I cannot run parallel stereo on them.

I feel pretty confident that my images are similar enough and have sufficient (almost complete) overlap to be at least marginally successful so I will zip up my inputs and send them to you privately. I appreciate the help.

[Oleg Alexandrov]

I am on travel and can't say much now. Will look over within a couple of weeks. 

That you get wrong rotation after mapprojection suggests something is not right with cameras or your process. Do you have other data to try? 

As before, we only tested with GEC images.

You can ignore the warming about images already mapprojected. 

To view this discussion visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/091f46ee-ab25-4044-917a-200c94450ec3n%40googlegroups.com.

[David Shean]

Hi Ryan,

Sounds like your issue may be related to the fact that the GEC images contain both RPCs and an affine geotransform (which is what desktop GIS reads and uses for georeferencing by default).  You can potentially try stripping the geotransform from the header using GDAL to edit metadata.  

And to complicate things further, since the GEC/SIDD images are projected onto a ground plane, they truly are “already map projected”, just using a constant height above ellipsoid (approximation for the tangent plane), instead of a DEM for more traditional orthorectification.  This is why we added the ortho-heights option (which is also required to process L2A “ortho-ready” or “stereo-ready” optical images distributed by multiple vendors).  See end of this section: https://stereopipeline.readthedocs.io/en/latest/stereodefault.html#interest-point-determination.

I’ve encountered similar issues around failed matching and homography in the stereo_pprc steps with Umbra GEC images, even when using disparity seeding mode 2 (reference DEM to model disparities, not relying on feature matches).  I think there are some issues with the existing ASP logic when handling the somewhat unusual GEC metadata, both with and without the standard mapproject workflow, and how the default stereo_pprc matching and alignment (homography vs. none) is set.  We want to ensure that the user can skip initial matching and homography (used to set the search window) in these cases.  I’ve documented some of this from recent GEC stereo processing tests.  We don’t have much (really any) time available to work on this with current funding, but when Oleg is back, I’m hoping that we can quickly review and address these issues.  It will be good to have a few test cases.

-David

--
David Shean
Civil and Environmental Engineering
University of Washington
https://www.ce.washington.edu/facultyfinder/david-shean

201 More Hall, Box 352700
3760 E. Stevens Way NE
Seattle, WA 98195-2700
Office: (206) 543-3105, Wilcox Hall 265

To view this discussion visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/CADDkNmO8nCoosVkNTAX20-2-jFi-jx2wGS64NvvKyqCRSa46Pg%40mail.gmail.com.

[Oleg Alexandrov]

I took a look. The issue seems to be that the images have fine-level differences, likely due to the larger stereo convergence angle, at 25 degrees, and given how SAR is sensitive to that.

What worked was to resample the mapprojected images to a 2x coarser resolution by local averaging. I added an addendum to the doc at: https://stereopipeline.readthedocs.io/en/latest/examples/umbra_sar.html#handling-failure

I am not sure this is the best long-term solution, but it works for now.

Here is the resulting hillshaded DEM, The triangulation error (after using point2dem --errorimage) is kind of high, at 76 m, even after using bundle adjustment, which suggests modeling issues with the RPC camera. Maybe one can try without bundle adjustment and see if it is any different.

[Ryan Rock]

Thanks so much for the help. Resampling did indeed resolve the interest point matching issue I was having. I came across some other issues.
When running point2dem, the extent of the output DEM is expanded relative to the PC image. It seems like the output is being stretched across an area double the size of the input image. I was able to correct this by running point2dem with the --t_projwin set explicitly:

point2dem --t_projwin 575660.250, 3443789.750, 580954.750, 3449035.250 --errorimage --tr 2.0 stereo-50pct-noBA/run-PC.tif

Second, my output DEM is much more "swiss cheese-like" than the image you shared above and I am wondering why mine would have so many more holes than yours. Did you use the input dem file I provided or use one of your own? Are you using any of the dem fill parameters in the point2dem command? Here is my DEM output after correcting the extent issue.



Thanks again!

[Oleg Alexandrov]

Ryan,

When running point2dem, the extent of the output DEM is expanded relative to the PC image. It seems like the output is being stretched across an area double the size of the input image. I was able to correct this by running point2dem with the --t_projwin set explicitly:

This is likely a bug in our toolchain. I get same issue with the extent. But I do get a nice DEM, but the extent is larger. I have a screenshot at the bottom.

Maybe this boosting of resolution by resampling is a bad idea. Maybe one can mapproject at 1 meter / pixel to start with, rather than doing it at 0.5 m / pixel followed by resampling.I will look at this later.

 

Second, my output DEM is much more "swiss cheese-like" than the image you shared above and I am wondering why mine would have so many more holes than yours. Did you use the input dem file I provided or use one of your own? Are you using any of the dem fill parameters in the point2dem command? Here is my DEM output after correcting the extent issue.

I used the data as you gave them to me. I used the images with "2x coarser resolution by local averaging" for both bundle adjustment and stereo. Here's all my commands. You can let me know if there is no luck.

Precise commands:

mapproject --tr 0.5 dem-adj-clip.tif img4.tif img4.map.tif

mapproject --tr 0.5 dem-adj-clip.tif img6.tif img6.map.tif

# Reduce the resolution by averaging
gdal_translate -r average -outsize 50% 50% img4.map.tif img4.tr1.map.tif
gdal_translate -r average -outsize 50% 50% img6.map.tif img6.tr1.map.tif

# Bundle adjustment with lower-res mapprojected images
bundle_adjust -t rpc                                     \
  --remove-outliers-params "75.0 3.0 50 50"              \
  --ip-per-tile 1000                                     \
  img4.tif img6.tif                                      \
  --mapprojected-data                                    \
    "img4.tr1.map.tif img6.tr1.map.tif dem-adj-clip.tif" \
  -o ba_tr1/run

# Stereo
parallel_stereo -t rpc              \
  --bundle-adjust-prefix ba_tr1/run \
  --stereo-algorithm asp_mgm        \
  --allow-different-mapproject-gsd  \
  img4.tr1.map.tif img6.tr1.map.tif \
  stereo/run                        \
  dem-adj-clip.tif                  \
      

point2dem --errorimage stereo/run-PC.tif 

Screenshot showing that the DEM is good, but the extent is twice as big. Likely the image resampling did not handle that properly. To be looked into.