Error parallel_stereo with SGM on multiple nodes (SLURM)

19 views
Skip to first unread message

César Deschamps-Berger

unread,
Apr 4, 2024, 10:58:44 AMApr 4
to Ames Stereo Pipeline Support
Hi,

I am having problems when running parallel_stereo with SGM on multiple nodes (SLURM management).
I applied the advice of the doc (creating $nodesList, --nodes-list $nodesList).
This is a small test area of a pair of Pléiades images. The very same code runs smoothly on a single node.

The error:
Traceback (most recent call last):
  File "/home/ad/deschampsc/Apps/ASP/StereoPipeline-3.1.0-2022-05-18-x86_64-Linux/libexec/parallel_stereo", line 1027, in <module>
    contract_tiles = using_padded_tiles)
  File "/home/ad/deschampsc/Apps/ASP/StereoPipeline-3.1.0-2022-05-18-x86_64-Linux/libexec/parallel_stereo", line 270, in build_vrt
    raise Exception('No tiles were generated')
Exception: No tiles were generated

Find the code, stereo.default, logs and output folder attached.
In case anybody had a similar experience...
Thanks for any hints, best regards.

César Deschamps-Berger
stereo.default_SGM191015
log.err
log.out
SMASH_a_GrandesRousses_20240223_ASP_test_cml_5000_multinode.sh
ll_output.txt

Oleg Alexandrov

unread,
Apr 4, 2024, 12:04:09 PMApr 4
to César Deschamps-Berger, Ames Stereo Pipeline Support
I see in your log file the messages:

Warning: Permanently added 'trex031,10.120.117.31' (ECDSA) to the list of known hosts.
Warning: Permanently added 'trex034,10.120.117.34' (ECDSA) to the list of known hosts.
Connection closed by 10.120.117.29 port 22
Connection closed by 10.120.117.30 port 22

I am not sure but I think it is something about your own system. Are you able to run GNU Parallel alone? That is the program named 'parallel' that we ship. 

Here's an example for how to run that program, without ASP itself: https://sulis-hpc.github.io/advanced/ensemble/gnuparallel.html

You could have a shell script that only prints the name of the node, and pass it to that tool.

So, I think it is not ASP itself that is the problem but likely something in the communication. 

If that is not the problem, let us know. 




--
You received this message because you are subscribed to the Google Groups "Ames Stereo Pipeline Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ames-stereo-pipeline...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ames-stereo-pipeline-support/2b28412e-b508-4a41-8613-564f39c5242dn%40googlegroups.com.

César Deschamps-Berger

unread,
Apr 5, 2024, 4:54:27 AMApr 5
to Ames Stereo Pipeline Support
Thanks, I will look into this direction.

César
Reply all
Reply to author
Forward
0 new messages