parallel_stereo exhausts system resources without explicit flags and works very slow even on high-core machines

[forwork]

hi,

there's an issue when running parallel_stereo on a high-resource machine without explicitly specifying --processes, --threads-multiprocess, --threads-singleprocess, or memory limits, the command attempts to spawn too many processes. It rapidly hits the system's ulimit (e.g., fork: Resource temporarily unavailable) or exhausts resources and crashes, even if no other jobs are running on the server. This happens regardless of the machine, user, or ASP version being used.

The command that I use overall with plus and minus of flags:

parallel_stereo \
  --stereo-algorithm asp_mgm \
  --processes 1\
  --threads-multiprocess 100 \
  --corr-memory-limit-mb <~24GB> \
  --corr-tile-size 1024 \
  --clean-match-files-prefix ba/run \
  img1.cub img2.cub \
  ba/img1-adjusted-state.json img2-adjusted-state.json \

  stereo/run

and the hardware configurations are:

• Hardware: 128 CPU cores, 500GB RAM 

• ASP Version: 3.6

• The data is a long strip approx 72000 lines and 12000 samples.
  
  Also, I need guidance for generating good quality dem which should perfectly coincidences with the global grouthtruth. My dem gets slightly more rotated everytime, its not able to coincide perfectly. So, what are the recommended flag combinations in different commands like point2dem, pc_align,etc that I can use to fine tune the dem, specifically tuned to prevent this extreme rotation and achieve a perfectly aligned DEM using a reference dem.tif as refernce.
  

[Oleg Alexandrov]

there's an issue when running parallel_stereo on a high-resource machine without explicitly specifying --processes, --threads-multiprocess, --threads-singleprocess, or memory limits, the command attempts to spawn too many processes. It rapidly hits the system's ulimit (e.g., fork: Resource temporarily unavailable) or exhausts resources and crashes, even if no other jobs are running on the server. This happens regardless of the machine, user, or ASP version being used

 

The default behavior for SGM/MGM algorithms (like asp_mgm) is --threads-multiprocess 8 and --processes defaults to round(num_cpus / 8). You can try without setting these options at all, and monitor your system for number of processes and threads. In your case, with 128 cores you should see 16 stereo_corr processes and 8 threads. Or else we have a bug. 

Note that asp_mgm can be very memory-hungry. We suggest mapprojecting the images if you are not doing that already, and inspecting if they look similar enough and about in the same place after mapprojection. This will reduce the search range and improve the quality. 

In either case, monitor the memory usage too.

• Also, I need guidance for generating good quality dem which should perfectly coincidences with the global grouthtruth. My dem gets slightly more rotated everytime, its not able to coincide perfectly. So, what are the recommended flag combinations in different commands like point2dem, pc_align,etc that I can use to fine tune the dem, specifically tuned to prevent this extreme rotation and achieve a perfectly aligned DEM using a reference dem.tif as refernce.
  

Errors in alignment in the output DEM are almost surely from issues in cameras. We recommend the pc_align tool for fixing these. It is suggested to estimate the expected shift between these and run this tool with the --max-displacement option with this shift, per the doc. We also have the default point-to-plane algorithm, and you may try point-to-point. There is also the alignment from hillshading option, if your input DEM and output one are similar enough.

You can try also mapprojecting your input images and cameras without bundle adjustment at all. That will give you a hint if the rotation is introduced as part of bundle adjustment or was preexisting.