Stereo error

[davidef...@gmail.com]

Hi all,

I am trying tu run parallel_stereo on an Ikonos image pair.

Here is the command I am using:
parallel_stereo -t rpc --stereo-algorithm asp_mgm_final --corr-timeout 300 2009083010222400000011606113/2009083010222400000011606113/po_1398039_0000000/po_1398039_pan_0000000.tif 2009083010231260000011606114/2009083010231260000011606114/po_1398455_0000000/po_1398455_pan_0000000.tif try1/stereo

If I use the original block matching algorithm, everything works as expected. When using SGM/MGM, stereo performs correlation up to very last tile, and then throws this error:

Traceback (most recent call last):
  File "/gpfs/home/users/davide.fugazza/ELPRIM/StereoPipeline-3.0.0-2021-09-21-x86_64-Linux/libexec/parallel_stereo", line 870, in <module>
    spawn_to_nodes(step, settings, self_args)
  File "/gpfs/home/users/davide.fugazza/ELPRIM/StereoPipeline-3.0.0-2021-09-21-x86_64-Linux/libexec/parallel_stereo", line 463, in spawn_to_nodes
    generic_run(cmd, opt.verbose)
  File "/gpfs/home/projects/ELPRIM/StereoPipeline-3.0.0-2021-09-21-x86_64-Linux/libexec/stereo_utils.py", line 124, in generic_run
    raise Exception('Failed to run: ' + cmd_str)
Exception: Failed to run: parallel --will-cite --env ASP_DEPS_DIR --env PATH --env LD_LIBRARY_PATH --env PYTHONHOME -u -P 32 -a /gpfs/home/projects/ELPRIM/Forni_sat/Ikonos/tmp3f4sbpsz "/gpfs/home/users/davide.fugazza/ELPRIM/StereoPipeline-3.0.0-2021-09-21-x86_64-Linux/bin/python /gpfs/home/users/davide.fugazza/ELPRIM/StereoPipeline-3.0.0-2021-09-21-x86_64-Linux/libexec/parallel_stereo -t rpc --stereo-algorithm asp_sgm --corr-timeout 300 2009083010222400000011606113/2009083010222400000011606113/po_1398039_0000000/po_1398039_pan_0000000.tif 2009083010231260000011606114/2009083010231260000011606114/po_1398455_0000000/po_1398455_pan_0000000.tif try1/stereo --skip-low-res-disparity-comp --processes 32 --threads-multiprocess 8 --entry-point 1 --stop-point 2 --work-dir /gpfs/home/projects/ELPRIM/Forni_sat/Ikonos --isisroot /gpfs/home/users/davide.fugazza/ELPRIM/StereoPipeline-3.0.0-2021-09-21-x86_64-Linux --tile-id {}"

It seems as if it is trying to work on an empty tile.. Any idea what might be causing this? I have read/write permissions on the project folders and memory usage seems ok. Am I missing something?

Cheers,

Davide

[Alexandrov, Oleg (ARC-TI)[KBR Wyle Services, LLC]]

There is not enough information in the log you provided to figure out what is going on. 

Since you say it fails on the "last tile", you can peek in that tile and see its stereo_corr log and see what it says. Then you can try running that stereo_corr command for that particular tile by hand and examine if it complains about anything.

Or, you can simply look at your current failed output and see if there's any more info being dumped on screen. 

My suspicion may be that it could have ran out of memory with MGM on a given tile. In that case one may try to monitor memory usage with "top" and/or start parallel_stereo with fewer processes, and/or examine that search range is printed in the stereo_corr log files. But this is a guess.

You can also download the very latest build from https://github.com/NeoGeographyToolkit/StereoPipeline/releases, and try to add to your parallel_stereo command the option --alignment-method local_epipolar. In this mode stereo will perform only 1D search and will result in less memory usage.

Another thing is to simply use --stereo-algorithm mgm instead of --stereo-algorithm asp_mgm_final.

---

From: ames-stereo-pi...@googlegroups.com <ames-stereo-pi...@googlegroups.com> on behalf of davidef...@gmail.com <davidef...@gmail.com>
Sent: Saturday, October 2, 2021 4:57 AM
To: Ames Stereo Pipeline Support <ames-stereo-pi...@googlegroups.com>
Subject: [EXTERNAL] Stereo error

 

--
You received this message because you are subscribed to the Google Groups "Ames Stereo Pipeline Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/59ca82e5-d358-40a3-bcee-ece489a11f28n%40googlegroups.com.

[davidef...@gmail.com]

Hi Oleg,

thanks for the advice.

I have tried using local epipolar alignment, which did not crash but created a goodpixelmap with no valid pixels. I was using the daily build from 21/09, I will try to work with the latest one if there have been changes in the code.

I checked memory usage with top: overall memory consumption is fine, and it does not seem like any tile is exceeding its dedicated memory. The last tile is not problematic per se, I meant to say that instances of stereo-corr are spawned for all tiles. At the end, the running instances close one by one, then the error i described is thrown. 

Still, throughout the process there are tiles where correlation is not completed, and I get some messages like "stereo step1: correlation failed" and "Warning: Unable to compute valid search ranges for SGM input!." 

In fact, the image has some clouds and correlation is obviously a problem there (which is also why I set corr-timeout to a lower value than default), so maybe  that's what is somehow causing the error.

What would you suggest to deal with the clouds? Perhaps cropping the image (they are all in the lower part so I might be able to do that without sacrifing useful data)?

Cheers,

Davide

[Alexandrov, Oleg (ARC-TI)[KBR Wyle Services, LLC]]

Clouds are an issue. You can try running stereo_gui with the same options as you do for parallel_stereo, and then select an area without clouds and run parallel_stereo from the menu. That will display in the terminal the command you run, so at this stage one can cancel the gui and run it directly from the command line, if desired.

This way you can avoid the region with clouds. (You can also run a small clip from the GUI first as a sanity check). 

I put just recently an option to filter the low-resolution disparity D_sub for outliers. That may help with clouds as well. I don't recall now if it is checked in before or after the 9/21 build. It is good to examine how the low-resolution disparity gets computed, after preprocessing, by reading carefully what it prints in the menu, and look for a line which says: "Number (and fraction) of removed outliers by the triangulation error check".

You can also examine what search range is printed in the terminal, clouds may result in an outrageous range, perhaps. 

---

From: ames-stereo-pi...@googlegroups.com <ames-stereo-pi...@googlegroups.com> on behalf of davidef...@gmail.com <davidef...@gmail.com>

Sent: Monday, October 4, 2021 7:16 AM

To: Ames Stereo Pipeline Support <ames-stereo-pi...@googlegroups.com>

Subject: Re: [EXTERNAL] Stereo error

 

To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/b3ac6c7b-89be-4b5b-9795-09c234ea9428n%40googlegroups.com.

[Shashank Bhushan]

Hi folks,

I was having a similar problem when I was processing some cloudy WV tiles over Chamoli. The issue is that even if one  or two tile fails to run properly (due to very bad D_subs aka the low resolution disparity), the entire next step of the stereo process is abandoned. This was not looking right to me, and to make things work, I just uncommented the raise_Exception block in the def generic_run function to make it proceed with the next step and not complain about the lonely tile which is not working.

This function is in the stereo_utils.py program in the libexec directory.

```# A very simple wrapper around subprocess

def generic_run(cmd, verbose):
    cmd_str = asp_string_utils.argListToString(cmd)
    if verbose:
        print(cmd_str)

    try:
        code = subprocess.call(cmd)
    except OSError as e:
        raise Exception('%s: %s' % (cmd_str, e))
    #if code != 0:
            #raise Exception('Failed to run: ' + cmd_str)

```

I think Oleg's fix for D_sub outliers checking might have fixed it, so you may be able to complete the processing with the newer builds. 

Hacks like these just come out of desperation, so should not be used until a pressing situation due to some deadline or something is there :)

Cheers,

Shashank

To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/MN2PR09MB54498B552584C3A7E8A0754388AE9%40MN2PR09MB5449.namprd09.prod.outlook.com.

--

Shashank Bhushan

Graduate Student/Research Assistant

Civil & Environmental Engineering

University of Washington, Seattle

[Alexandrov, Oleg (ARC-TI)[KBR Wyle Services, LLC]]

For the recent "--alignment-method local_epipolar" all failures are caught without a given instance of stereo_corr crashing and bringing down the whole run, and the correlation result for that tile is set to an empty disparity. Without this alignment things can be trickier and I'd need a testcase to figure out why it crashes.

I am a bit surprised Davide says this alignment method does not work for that testcase with clouds. 

I would actually want that testcase and the command being used to test it myself, but I know that sharing data can be a pain, so I'd like to have it but it is a bit of an unreasonable request.

---

From: ames-stereo-pi...@googlegroups.com <ames-stereo-pi...@googlegroups.com> on behalf of Shashank Bhushan <sbag...@uw.edu>
Sent: Monday, October 4, 2021 6:21 PM
To: Alexandrov, Oleg (ARC-TI)[KBR Wyle Services, LLC] <oleg.al...@nasa.gov>
Cc: davidef...@gmail.com <davidef...@gmail.com>; Ames Stereo Pipeline Support <ames-stereo-pi...@googlegroups.com>

To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/CA%2BryyFQ3-RST7j%2BRE%2BvZdnM15zUJju%2BHU6aUwBbAS%3DC3qOpKhg%40mail.gmail.com.

[davidef...@gmail.com]

Hi Oleg,  Shashank

thanks for the info and tips. Shashank, luckily I have no strict deadlines for this as it is more of a side project, so I have some time to figure out what's going on :)

Oleg, I tried again stereo with local alignment and it worked, so I must have been doing something wrong before; maybe I was reusing an ill low-res disparity map from a previous run.

Still, I had no success with the default affine epipolar alignment. Indeed the low-res disparity map has some outliers due to clouds which might be the issue here. 

I don't mind sharing the data, if that can help solve the issue and maybe fix it for everyone else; I'll just give it a try with the latest build to see if the outlier removal filter in the low res disparity already fixes it

Cheers,

Davide

[Alexandrov, Oleg (ARC-TI)[KBR Wyle Services, LLC]]

This stereo dataset turned out to be quite tricky. It has both big mountains, a valley, and clouds. The disparity search range turned out to be quite big even when there were no clouds, so only the mountains, so it would run out of memory and crash. 

Another thing is that due to the terrain versatility the filtering of clouds in disparity was not very effective, since the disparity was big even without the clouds. Maybe the height of clouds is comparable to the height of the mountains (the clouds show up above a valley). 

I eventually ran the whole thing without crashing using 8 processes for each supercomputer node (parallel_stereo has an option for that), with each node having 64 GB of RAM. Some correlation subprocesses used up more than 4 GB of RAM. 

So there does not seem to be any giant bug, just a hard case.  I will suggest getting a first DEM by whatever means, making it smoother by running point2dem with a grid maybe 20x bigger than what is expected, fill holes, and mapproject both images on it and see if they align well, and then run stereo with mapprojected images while keeping an eye on what search range stereo_corr prints out (anything bigger than 300 pixels is likely big enough that memory usage will still be high). There's more info here: https://stereopipeline.readthedocs.io/en/latest/next_steps.html#mapproj-example

Also, mapprojection may help with a valley which was not resolved so well now, likely because the global image alignment was not so good there. (Such an alignment does best for gentler terrains.) 

This is a nice tough dataset, I appreciate sharing the data. I will use it for more tests when having time for ASP again in a few months.

Oleg

---

From: ames-stereo-pi...@googlegroups.com <ames-stereo-pi...@googlegroups.com> on behalf of davidef...@gmail.com <davidef...@gmail.com>

Sent: Wednesday, October 6, 2021 6:41 AM

To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/e2c27638-3af1-4a0d-80be-3edfbae6c621n%40googlegroups.com.

[davidef...@gmail.com]

Oleg,

thanks for the tips. I can run stereo with the classic ASP algorithm, so I will try and downsample the resulting DEM and use that as a basis for mapprojection. I will keep you updated.

No worries about the data, I'm glad if it can be helpful for future ASP improvements

Cheers,

Davide

[Oleg Alexandrov]

I made some improvements in software and documentation when it comes to handling clouds. I also added a section on that here: https://stereopipeline.readthedocs.io/en/latest/tutorial.html#dealing-with-clouds

For this testcase, however, the clouds is not as much of a problem as the fact that even after affine epipolar alignment the disparity search range is big. That can be determined by opening L.tif and R.tif in stereo_gui and clicking on the same feature in the left and right image and see the resulting pixel value and their difference, and compare with results from another feature (for example a mountain peak vs a valley).

The solution which worked for me was the usual approach suggested in this case, which is to mapproject the data. I expanded the doc for that, emphasizing that the left and right image must be mapprojected at the same resolution, and how to get an input DEM to project on. See here: https://stereopipeline.readthedocs.io/en/latest/next_steps.html#mapproj-example.

After following those steps with this testcase (with the latest software from the GitHub Release page), I got a disparity search range of  57 by  42 pixels, which is not bad. One can use the newer option --max-disp-spread if some clouds still make this range go over 150 pixels or so when it would be too big.

I recently ran ASP on quite nasty and large WorldView datasets as well, with plenty of mountains and clouds, including, but not only, on the Chamoli dataset mentioned by Shashank, and it does rather well with this approach.