Problem testing mpiexec (execution works in just one computer)

[Juan Pedro Carbonell]

Good morning everyone,
I have recently started working with FDS, and I am trying to perform parallel processing in a scenario. The first step I have done was to install FDS on three networked computers, executing the following command:

mpiexec -n 10 -hosts 2 tele-lima.upvnet.upv.es 1 tele_batman.alumno.upv.es 1 test_mpi

Being tele-lima.upvnet.upv.es and tele_batman.alumno.upv.es the hostnames of the computers. The result I get is the following:

Hello world: rank 0 of 10 running on
 Tele-Lima

 Hello world: rank 1 of 10 running on
 Tele-Lima

 Hello world: rank 2 of 10 running on
 Tele-Lima

 Hello world: rank 3 of 10 running on
 Tele-Lima

 Hello world: rank 4 of 10 running on
 Tele-Lima

 Hello world: rank 5 of 10 running on
 Tele-Lima

 Hello world: rank 6 of 10 running on
 Tele-Lima

 Hello world: rank 7 of 10 running on
 Tele-Lima

When the process terminates on Tele-Lima it freezes, not running on tele_batman. I have pinged all the PCs, and they have visibility between them, so I don't know what could be happening. Has anyone experienced something similar?

[Kevin]

Are you running under Linux, Windows, or MacOS?

[Juan Pedro Carbonell]

Both computers are running under Windows 10 Pro. "tele-lima.upvnet.upv.es" is the localhost from which the process is launched.

On the other hand, I just realized that in the second host, the computational demand is huge for a test. I don't know what is happening, but the processes are consuming 100% of the CPU:

This is the console output when I run the command line setting the verbose:

C:\Users\juacarri>mpiexec -v -hosts 2 tele-lima.upvnet.upv.es 1 tele_batman.alumno.upv.es 1 test_mpi
[mpiexec@tele-lima] Launch arguments: C:\Program Files\firemodels\FDS6\bin\\mpi\hydra_bstrap_proxy.exe --upstream-host tele-lima.upvnet.upv.es --upstream-port 61294 --pgid 0 --launcher service --launcher-number 0 --base-path C:\Program Files\firemodels\FDS6\bin\\mpi --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --service_port 0 --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 560 C:\Program Files\firemodels\FDS6\bin\\mpi\hydra_pmi_proxy.exe --usize -1 --auto-cleanup 1 --abort-signal 9
[proxy:0:1@tele_batman] Warning - oversubscription detected: 6 processes will be placed on 4 cores
[proxy:0:0@tele-lima] pmi cmd from fd 500: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@tele-lima] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@tele-lima] pmi cmd from fd 544: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@tele-lima] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@tele-lima] pmi cmd from fd 500: cmd=get_maxes
[proxy:0:0@tele-lima] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@tele-lima] pmi cmd from fd 544: cmd=get_maxes
[proxy:0:0@tele-lima] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@tele-lima] pmi cmd from fd 500: cmd=get_appnum
[proxy:0:0@tele-lima] PMI response: cmd=appnum appnum=0
[proxy:0:0@tele-lima] pmi cmd from fd 544: cmd=get_appnum
[proxy:0:0@tele-lima] PMI response: cmd=appnum appnum=0
[proxy:0:0@tele-lima] pmi cmd from fd 612: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@tele-lima] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@tele-lima] pmi cmd from fd 500: cmd=get_my_kvsname
[proxy:0:0@tele-lima] PMI response: cmd=my_kvsname kvsname=kvs_22356_0
[proxy:0:0@tele-lima] pmi cmd from fd 544: cmd=get_my_kvsname
[proxy:0:0@tele-lima] PMI response: cmd=my_kvsname kvsname=kvs_22356_0
[proxy:0:0@tele-lima] pmi cmd from fd 612: cmd=get_maxes
[proxy:0:0@tele-lima] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@tele-lima] pmi cmd from fd 500: cmd=get kvsname=kvs_22356_0 key=PMI_process_mapping
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,6))
[proxy:0:0@tele-lima] pmi cmd from fd 544: cmd=get kvsname=kvs_22356_0 key=PMI_process_mapping
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,6))
[proxy:0:0@tele-lima] pmi cmd from fd 612: cmd=get_appnum
[proxy:0:0@tele-lima] PMI response: cmd=appnum appnum=0
[proxy:0:0@tele-lima] pmi cmd from fd 612: cmd=get_my_kvsname
[proxy:0:0@tele-lima] PMI response: cmd=my_kvsname kvsname=kvs_22356_0
[proxy:0:0@tele-lima] pmi cmd from fd 500: cmd=put kvsname=kvs_22356_0 key=-bcast-1-0 value=4D504943485F4E454D5F32333338385F313730393134393433373035
[proxy:0:0@tele-lima] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@tele-lima] pmi cmd from fd 544: cmd=barrier_in
[proxy:0:0@tele-lima] pmi cmd from fd 612: cmd=get kvsname=kvs_22356_0 key=PMI_process_mapping
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,6))
[proxy:0:0@tele-lima] pmi cmd from fd 500: cmd=barrier_in
[proxy:0:0@tele-lima] pmi cmd from fd 612: cmd=barrier_in
[proxy:0:0@tele-lima] pmi cmd from fd 644: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@tele-lima] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@tele-lima] pmi cmd from fd 644: cmd=get_maxes
[proxy:0:0@tele-lima] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@tele-lima] pmi cmd from fd 644: cmd=get_appnum
[proxy:0:0@tele-lima] PMI response: cmd=appnum appnum=0
[proxy:0:0@tele-lima] pmi cmd from fd 644: cmd=get_my_kvsname
[proxy:0:0@tele-lima] PMI response: cmd=my_kvsname kvsname=kvs_22356_0
[proxy:0:0@tele-lima] pmi cmd from fd 644: cmd=get kvsname=kvs_22356_0 key=PMI_process_mapping
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,6))
[proxy:0:0@tele-lima] pmi cmd from fd 644: cmd=barrier_in
[proxy:0:0@tele-lima] pmi cmd from fd 676: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@tele-lima] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@tele-lima] pmi cmd from fd 676: cmd=get_maxes
[proxy:0:0@tele-lima] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@tele-lima] pmi cmd from fd 676: cmd=get_appnum
[proxy:0:0@tele-lima] PMI response: cmd=appnum appnum=0
[proxy:0:0@tele-lima] pmi cmd from fd 676: cmd=get_my_kvsname
[proxy:0:0@tele-lima] PMI response: cmd=my_kvsname kvsname=kvs_22356_0
[proxy:0:0@tele-lima] pmi cmd from fd 676: cmd=get kvsname=kvs_22356_0 key=PMI_process_mapping
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,6))
[proxy:0:0@tele-lima] pmi cmd from fd 676: cmd=barrier_in
[proxy:0:0@tele-lima] pmi cmd from fd 484: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:0@tele-lima] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@tele-lima] pmi cmd from fd 484: cmd=get_maxes
[proxy:0:0@tele-lima] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:0@tele-lima] pmi cmd from fd 484: cmd=get_appnum
[proxy:0:0@tele-lima] PMI response: cmd=appnum appnum=0
[proxy:0:0@tele-lima] pmi cmd from fd 484: cmd=get_my_kvsname
[proxy:0:0@tele-lima] PMI response: cmd=my_kvsname kvsname=kvs_22356_0
[proxy:0:0@tele-lima] pmi cmd from fd 484: cmd=get kvsname=kvs_22356_0 key=PMI_process_mapping
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,6))
[proxy:0:0@tele-lima] pmi cmd from fd 484: cmd=barrier_in
[proxy:0:1@tele_batman] pmi cmd from fd 512: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:1@tele_batman] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@tele_batman] pmi cmd from fd 576: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:1@tele_batman] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@tele_batman] pmi cmd from fd 548: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:1@tele_batman] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@tele_batman] pmi cmd from fd 512: cmd=get_maxes
[proxy:0:1@tele_batman] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:1@tele_batman] pmi cmd from fd 576: cmd=get_maxes
[proxy:0:1@tele_batman] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:1@tele_batman] pmi cmd from fd 548: cmd=get_maxes
[proxy:0:1@tele_batman] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:1@tele_batman] pmi cmd from fd 620: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:1@tele_batman] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@tele_batman] pmi cmd from fd 512: cmd=get_appnum
[proxy:0:1@tele_batman] PMI response: cmd=appnum appnum=0
[proxy:0:1@tele_batman] pmi cmd from fd 576: cmd=get_appnum
[proxy:0:1@tele_batman] PMI response: cmd=appnum appnum=0
[proxy:0:1@tele_batman] pmi cmd from fd 548: cmd=get_appnum
[proxy:0:1@tele_batman] PMI response: cmd=appnum appnum=0
[proxy:0:1@tele_batman] pmi cmd from fd 620: cmd=get_maxes
[proxy:0:1@tele_batman] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:1@tele_batman] pmi cmd from fd 512: cmd=get_my_kvsname
[proxy:0:1@tele_batman] PMI response: cmd=my_kvsname kvsname=kvs_22356_0
[proxy:0:1@tele_batman] pmi cmd from fd 576: cmd=get_my_kvsname
[proxy:0:1@tele_batman] PMI response: cmd=my_kvsname kvsname=kvs_22356_0
[proxy:0:1@tele_batman] pmi cmd from fd 548: cmd=get_my_kvsname
[proxy:0:1@tele_batman] PMI response: cmd=my_kvsname kvsname=kvs_22356_0
[proxy:0:1@tele_batman] pmi cmd from fd 620: cmd=get_appnum
[proxy:0:1@tele_batman] PMI response: cmd=appnum appnum=0
[proxy:0:1@tele_batman] pmi cmd from fd 512: cmd=get kvsname=kvs_22356_0 key=PMI_process_mapping
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,6))
[proxy:0:1@tele_batman] pmi cmd from fd 576: cmd=get kvsname=kvs_22356_0 key=PMI_process_mapping
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,6))
[proxy:0:1@tele_batman] pmi cmd from fd 548: cmd=get kvsname=kvs_22356_0 key=PMI_process_mapping
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,6))
[proxy:0:1@tele_batman] pmi cmd from fd 620: cmd=get_my_kvsname
[proxy:0:1@tele_batman] PMI response: cmd=my_kvsname kvsname=kvs_22356_0
[proxy:0:1@tele_batman] pmi cmd from fd 512: cmd=put kvsname=kvs_22356_0 key=-bcast-1-6 value=4D504943485F4E454D5F31313633325F3731363835323035303135
[proxy:0:1@tele_batman] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1@tele_batman] pmi cmd from fd 576: cmd=barrier_in
[proxy:0:1@tele_batman] pmi cmd from fd 620: cmd=get kvsname=kvs_22356_0 key=PMI_process_mapping
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,6))
[proxy:0:1@tele_batman] pmi cmd from fd 548: cmd=barrier_in
[proxy:0:1@tele_batman] pmi cmd from fd 512: cmd=barrier_in
[proxy:0:1@tele_batman] pmi cmd from fd 620: cmd=barrier_in
[proxy:0:1@tele_batman] pmi cmd from fd 656: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:1@tele_batman] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@tele_batman] pmi cmd from fd 656: cmd=get_maxes
[proxy:0:1@tele_batman] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:1@tele_batman] pmi cmd from fd 656: cmd=get_appnum
[proxy:0:1@tele_batman] PMI response: cmd=appnum appnum=0
[proxy:0:1@tele_batman] pmi cmd from fd 656: cmd=get_my_kvsname
[proxy:0:1@tele_batman] PMI response: cmd=my_kvsname kvsname=kvs_22356_0
[proxy:0:1@tele_batman] pmi cmd from fd 656: cmd=get kvsname=kvs_22356_0 key=PMI_process_mapping
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,6))
[proxy:0:1@tele_batman] pmi cmd from fd 656: cmd=barrier_in
[proxy:0:1@tele_batman] pmi cmd from fd 688: cmd=init pmi_version=1 pmi_subversion=1
[proxy:0:1@tele_batman] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@tele_batman] pmi cmd from fd 688: cmd=get_maxes
[proxy:0:1@tele_batman] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=4096
[proxy:0:1@tele_batman] pmi cmd from fd 688: cmd=get_appnum
[proxy:0:1@tele_batman] PMI response: cmd=appnum appnum=0
[proxy:0:1@tele_batman] pmi cmd from fd 688: cmd=get_my_kvsname
[proxy:0:1@tele_batman] PMI response: cmd=my_kvsname kvsname=kvs_22356_0
[proxy:0:1@tele_batman] pmi cmd from fd 688: cmd=get kvsname=kvs_22356_0 key=PMI_process_mapping
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,6))
[proxy:0:1@tele_batman] pmi cmd from fd 688: cmd=barrier_in
[proxy:0:0@tele-lima] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] PMI response: cmd=barrier_out
[proxy:0:1@tele_batman] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] PMI response: cmd=barrier_out
[proxy:0:1@tele_batman] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] PMI response: cmd=barrier_out
[proxy:0:1@tele_batman] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] PMI response: cmd=barrier_out
[proxy:0:1@tele_batman] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] PMI response: cmd=barrier_out
[proxy:0:1@tele_batman] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] pmi cmd from fd 544: cmd=get kvsname=kvs_22356_0 key=-bcast-1-0
[proxy:0:1@tele_batman] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=4D504943485F4E454D5F32333338385F313730393134393433373035
[proxy:0:1@tele_batman] pmi cmd from fd 576: cmd=get kvsname=kvs_22356_0 key=-bcast-1-6
[proxy:0:0@tele-lima] pmi cmd from fd 612: cmd=get kvsname=kvs_22356_0 key=-bcast-1-0
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=4D504943485F4E454D5F31313633325F3731363835323035303135
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=4D504943485F4E454D5F32333338385F313730393134393433373035
[proxy:0:1@tele_batman] pmi cmd from fd 548: cmd=get kvsname=kvs_22356_0 key=-bcast-1-6
[proxy:0:0@tele-lima] pmi cmd from fd 644: cmd=get kvsname=kvs_22356_0 key=-bcast-1-0
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=4D504943485F4E454D5F31313633325F3731363835323035303135
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=4D504943485F4E454D5F32333338385F313730393134393433373035
[proxy:0:1@tele_batman] pmi cmd from fd 620: cmd=get kvsname=kvs_22356_0 key=-bcast-1-6
[proxy:0:0@tele-lima] pmi cmd from fd 676: cmd=get kvsname=kvs_22356_0 key=-bcast-1-0
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=4D504943485F4E454D5F31313633325F3731363835323035303135
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=4D504943485F4E454D5F32333338385F313730393134393433373035
[proxy:0:1@tele_batman] pmi cmd from fd 656: cmd=get kvsname=kvs_22356_0 key=-bcast-1-6
[proxy:0:0@tele-lima] pmi cmd from fd 484: cmd=get kvsname=kvs_22356_0 key=-bcast-1-0
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=4D504943485F4E454D5F31313633325F3731363835323035303135
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=4D504943485F4E454D5F32333338385F313730393134393433373035
[proxy:0:1@tele_batman] pmi cmd from fd 688: cmd=get kvsname=kvs_22356_0 key=-bcast-1-6
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=4D504943485F4E454D5F31313633325F3731363835323035303135
[proxy:0:0@tele-lima] pmi cmd from fd 544: cmd=put kvsname=kvs_22356_0 key=bc-1 value=mpi#0200EF8BC0A838010000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@tele-lima] pmi cmd from fd 544: cmd=barrier_in
[proxy:0:0@tele-lima] pmi cmd from fd 484: cmd=put kvsname=kvs_22356_0 key=bc-5 value=mpi#0200EF87C0A838010000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@tele-lima] pmi cmd from fd 484: cmd=barrier_in
[proxy:0:0@tele-lima] pmi cmd from fd 676: cmd=put kvsname=kvs_22356_0 key=bc-4 value=mpi#0200EF9BC0A838010000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@tele-lima] pmi cmd from fd 676: cmd=barrier_in
[proxy:0:0@tele-lima] pmi cmd from fd 612: cmd=put kvsname=kvs_22356_0 key=bc-2 value=mpi#0200EF98C0A838010000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@tele-lima] pmi cmd from fd 612: cmd=barrier_in
[proxy:0:0@tele-lima] pmi cmd from fd 500: cmd=put kvsname=kvs_22356_0 key=bc-0 value=mpi#0200EFAAC0A838010000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@tele-lima] pmi cmd from fd 500: cmd=barrier_in
[proxy:0:0@tele-lima] pmi cmd from fd 644: cmd=put kvsname=kvs_22356_0 key=bc-3 value=mpi#0200EFADC0A838010000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@tele-lima] pmi cmd from fd 644: cmd=barrier_in
[proxy:0:1@tele_batman] pmi cmd from fd 620: cmd=put kvsname=kvs_22356_0 key=bc-9 value=mpi#0200FD679E2AC7090000000000000000$
[proxy:0:1@tele_batman] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1@tele_batman] pmi cmd from fd 620: cmd=barrier_in
[proxy:0:1@tele_batman] pmi cmd from fd 548: cmd=put kvsname=kvs_22356_0 key=bc-8 value=mpi#0200FD709E2AC7090000000000000000$
[proxy:0:1@tele_batman] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1@tele_batman] pmi cmd from fd 548: cmd=barrier_in
[proxy:0:1@tele_batman] pmi cmd from fd 688: cmd=put kvsname=kvs_22356_0 key=bc-11 value=mpi#0200FD7B9E2AC7090000000000000000$
[proxy:0:1@tele_batman] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1@tele_batman] pmi cmd from fd 688: cmd=barrier_in
[proxy:0:1@tele_batman] pmi cmd from fd 576: cmd=put kvsname=kvs_22356_0 key=bc-7 value=mpi#0200FD809E2AC7090000000000000000$
[proxy:0:1@tele_batman] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1@tele_batman] pmi cmd from fd 576: cmd=barrier_in
[proxy:0:1@tele_batman] pmi cmd from fd 656: cmd=put kvsname=kvs_22356_0 key=bc-10 value=mpi#0200FD8D9E2AC7090000000000000000$
[proxy:0:1@tele_batman] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1@tele_batman] pmi cmd from fd 512: cmd=put kvsname=kvs_22356_0 key=bc-6 value=mpi#0200FD909E2AC7090000000000000000$
[proxy:0:1@tele_batman] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:1@tele_batman] pmi cmd from fd 656: cmd=barrier_in
[proxy:0:1@tele_batman] pmi cmd from fd 512: cmd=barrier_in
[proxy:0:0@tele-lima] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] PMI response: cmd=barrier_out
[proxy:0:1@tele_batman] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] PMI response: cmd=barrier_out
[proxy:0:1@tele_batman] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] PMI response: cmd=barrier_out
[proxy:0:1@tele_batman] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] PMI response: cmd=barrier_out
[proxy:0:1@tele_batman] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] PMI response: cmd=barrier_out
[proxy:0:1@tele_batman] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] pmi cmd from fd 500: cmd=get kvsname=kvs_22356_0 key=bc-0
[proxy:0:1@tele_batman] PMI response: cmd=barrier_out
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200EFAAC0A838010000000000000000$
[proxy:0:1@tele_batman] pmi cmd from fd 512: cmd=get kvsname=kvs_22356_0 key=bc-0
[proxy:0:0@tele-lima] pmi cmd from fd 612: cmd=get kvsname=kvs_22356_0 key=bc-4
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200EFAAC0A838010000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200EF9BC0A838010000000000000000$
[proxy:0:1@tele_batman] pmi cmd from fd 620: cmd=get kvsname=kvs_22356_0 key=bc-6
[proxy:0:0@tele-lima] pmi cmd from fd 676: cmd=get kvsname=kvs_22356_0 key=bc-8
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200FD909E2AC7090000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200FD709E2AC7090000000000000000$
[proxy:0:1@tele_batman] pmi cmd from fd 688: cmd=get kvsname=kvs_22356_0 key=bc-10
[proxy:0:0@tele-lima] pmi cmd from fd 644: cmd=get kvsname=kvs_22356_0 key=bc-6
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200FD8D9E2AC7090000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200FD909E2AC7090000000000000000$
[proxy:0:1@tele_batman] pmi cmd from fd 656: cmd=get kvsname=kvs_22356_0 key=bc-8
[proxy:0:0@tele-lima] pmi cmd from fd 484: cmd=get kvsname=kvs_22356_0 key=bc-10
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200FD709E2AC7090000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200FD8D9E2AC7090000000000000000$
[proxy:0:1@tele_batman] pmi cmd from fd 576: cmd=get kvsname=kvs_22356_0 key=bc-2
[proxy:0:0@tele-lima] pmi cmd from fd 544: cmd=get kvsname=kvs_22356_0 key=bc-2
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200EF98C0A838010000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200EF98C0A838010000000000000000$
[proxy:0:1@tele_batman] pmi cmd from fd 548: cmd=get kvsname=kvs_22356_0 key=bc-4
[proxy:0:0@tele-lima] pmi cmd from fd 500: cmd=get kvsname=kvs_22356_0 key=bc-1
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200EF9BC0A838010000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200EF8BC0A838010000000000000000$
[proxy:0:1@tele_batman] pmi cmd from fd 512: cmd=get kvsname=kvs_22356_0 key=bc-1
[proxy:0:0@tele-lima] pmi cmd from fd 612: cmd=get kvsname=kvs_22356_0 key=bc-5
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200EF8BC0A838010000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200EF87C0A838010000000000000000$
[proxy:0:1@tele_batman] pmi cmd from fd 620: cmd=get kvsname=kvs_22356_0 key=bc-7
[proxy:0:0@tele-lima] pmi cmd from fd 676: cmd=get kvsname=kvs_22356_0 key=bc-9
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200FD809E2AC7090000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200FD679E2AC7090000000000000000$
[proxy:0:1@tele_batman] pmi cmd from fd 688: cmd=get kvsname=kvs_22356_0 key=bc-11
[proxy:0:0@tele-lima] pmi cmd from fd 644: cmd=get kvsname=kvs_22356_0 key=bc-7
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200FD7B9E2AC7090000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200FD809E2AC7090000000000000000$
[proxy:0:1@tele_batman] pmi cmd from fd 656: cmd=get kvsname=kvs_22356_0 key=bc-9
[proxy:0:0@tele-lima] pmi cmd from fd 484: cmd=get kvsname=kvs_22356_0 key=bc-11
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200FD679E2AC7090000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200FD7B9E2AC7090000000000000000$
[proxy:0:1@tele_batman] pmi cmd from fd 576: cmd=get kvsname=kvs_22356_0 key=bc-3
[proxy:0:0@tele-lima] pmi cmd from fd 544: cmd=get kvsname=kvs_22356_0 key=bc-3
[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200EFADC0A838010000000000000000$
[proxy:0:0@tele-lima] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200EFADC0A838010000000000000000$
[proxy:0:1@tele_batman] pmi cmd from fd 548: cmd=get kvsname=kvs_22356_0 key=bc-5

 Hello world: rank            0  of           12  running on
 tele-lima

[proxy:0:1@tele_batman] PMI response: cmd=get_result rc=0 msg=success value=mpi#0200EF87C0A838010000000000000000$
 Hello world: rank            1  of           12  running on
 tele-lima

 Hello world: rank            2  of           12  running on
 tele-lima

 Hello world: rank            3  of           12  running on
 tele-lima

 Hello world: rank            4  of           12  running on
 tele-lima

 Hello world: rank            5  of           12  running on
 tele-lima

[Kevin McGrattan]

This is from the Intel guide. Try the second suggested Troubleshooting method, for example

mpiexec ... hostname