Openmpi5.0.7 causes fatal timeout on last rank

[Achilles Vassilicos]

Hello all, new to the list.

While testing my openmpi5.0.7 installation using the simple mpi_hello_world.c code, I am experiencing an unexpected behavior where the execution on the last processor rank hangs with a "fatal unhandled timeout error", which leads to core dumps. It confirmed that it happens regardless of the compiler I use, i.e., gnu14 or intel2024.0. Moreover, it does not happen when I use mpich3.4.3-ofi. Below I am including the setting I am using and the runtime error. You will notice that the error happened on node c11, which may suggest that there is something wrong with this node. However, it turns out that any other other node that happens to execute the last processor rank leads to the same error. I must be missing something. Any Thoughts?

Sorry about the length of the post.

-----------------------------------------------------

]$ module list
Currently Loaded Modules:
  1) cmake/4.0.0        6) spack/0.23.1          11) mkl/2024.0         16) ifort/2024.0.0              21) EasyBuild/5.0.0
  2) autotools          7) oclfpga/2024.0.0      12) intel/2024.0.0     17) inspector/2024.2            22) valgrind/3.24.0
  3) hwloc/2.12.0       8) tbb/2021.11           13) debugger/2024.0.0  18) intel_ipp_intel64/2021.10   23) openmpi5/5.0.7
  4) libfabric/1.18.0   9) compiler-rt/2024.0.0  14) dpl/2022.3         19) intel_ippcp_intel64/2021.9  24) ucx/1.18.0
  5) prun/2.2          10) compiler/2024.0.0     15) icc/2023.2.1       20) vtune/2025.3

----------------------------------------------------------

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
normal*      up   infinite     10  idle* c[2-10,12]
normal*      up   infinite      3   idle c[1,11,13]
[av@sms test]$ salloc -n 24 -N 2
salloc: Granted job allocation 61
salloc: Nodes c[1,11] are ready for job
[av@c1 test]$ mpirun --display-map --map-by node -x MXM_RDMA_PORTS=mlx4_0:1 -mca btl_openib_if_include mlx4_0:1 mpi_hello_world

========================   JOB MAP   ========================
Data for JOB prterun-c1-1575@1 offset 0 Total slots allocated 24
    Mapping policy: BYNODE:NOOVERSUBSCRIBE  Ranking policy: NODE Binding policy: NUMA:IF-SUPPORTED
    Cpu set: N/A  PPR: N/A  Cpus-per-rank: N/A  Cpu Type: CORE


Data for node: c1 Num slots: 12 Max slots: 0 Num procs: 12
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 0 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 2 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 4 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 6 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 8 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 10 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 12 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 14 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 16 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 18 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 20 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 22 Bound: package[0][core:0-17]

Data for node: c11 Num slots: 12 Max slots: 0 Num procs: 12
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 1 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 3 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 5 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 7 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 9 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 11 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 13 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 15 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 17 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 19 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 21 Bound: package[0][core:0-17]
        Process jobid: prterun-c1-1575@1 App: 0 Process rank: 23 Bound: package[0][core:0-17]

=============================================================
Hello world from processor c1, rank 6 out of 24 processors
Hello world from processor c1, rank 20 out of 24 processors
Hello world from processor c1, rank 16 out of 24 processors
Hello world from processor c1, rank 12 out of 24 processors
Hello world from processor c1, rank 0 out of 24 processors
Hello world from processor c1, rank 2 out of 24 processors
Hello world from processor c1, rank 14 out of 24 processors
Hello world from processor c1, rank 10 out of 24 processors
Hello world from processor c1, rank 4 out of 24 processors
Hello world from processor c1, rank 22 out of 24 processors
Hello world from processor c1, rank 18 out of 24 processors
Hello world from processor c1, rank 8 out of 24 processors
Hello world from processor c11, rank 11 out of 24 processors
Hello world from processor c11, rank 1 out of 24 processors
Hello world from processor c11, rank 3 out of 24 processors
Hello world from processor c11, rank 13 out of 24 processors
Hello world from processor c11, rank 19 out of 24 processors
Hello world from processor c11, rank 7 out of 24 processors
Hello world from processor c11, rank 17 out of 24 processors
Hello world from processor c11, rank 21 out of 24 processors
Hello world from processor c11, rank 15 out of 24 processors
Hello world from processor c11, rank 23 out of 24 processors
Hello world from processor c11, rank 9 out of 24 processors
Hello world from processor c11, rank 5 out of 24 processors
[c11:2028 :0:2028]       ud_ep.c:278  Fatal: UD endpoint 0x1c8da90 to <no debug data>: unhandled timeout error
[c11:2035 :0:2035]       ud_ep.c:278  Fatal: UD endpoint 0x722a90 to <no debug data>: unhandled timeout error
[c11:2025 :0:2025]       ud_ep.c:278  Fatal: UD endpoint 0xc52a90 to <no debug data>: unhandled timeout error
==== backtrace (tid:   2028) ====

0  /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7fade4326ee4]
 1  /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(ucs_fatal_error_message+0xb2) [0x7fade4324292]
 2  /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(+0x2f369) [0x7fade4324369]
 3  /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/ucx/libuct_ib.so.0(+0x263f0) [0x7fade110d3f0]
 4  /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(+0x24987) [0x7fade4319987]
 5  /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucp.so.0(ucp_worker_progress+0x2a) [0x7fade43abc9a]
 6  /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(+0xa09bc) [0x7fade471b9bc]
 7  /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(opal_common_ucx_del_procs_nofence+0x6a) [0x7fade471b79a]
 8  /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(opal_common_ucx_del_procs+0x20) [0x7fade471baf0]
 9  /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libmpi.so.40(mca_pml_ucx_del_procs+0x140) [0x7fade4d1cd70]
10  /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libmpi.so.40(+0xac837) [0x7fade4b27837]
11  /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(opal_finalize_cleanup_domain+0x53) [0x7fade46aebd3]
12  /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(opal_finalize+0x2e) [0x7fade46a22be]
13  /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libmpi.so.40(ompi_rte_finalize+0x1f9) [0x7fade4b21909]
14  /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libmpi.so.40(+0xab304) [0x7fade4b26304]
15  /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libmpi.so.40(ompi_mpi_instance_finalize+0xe5) [0x7fade4b26935]
16  /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libmpi.so.40(ompi_mpi_finalize+0x3d1) [0x7fade4b1e091]
17  mpi_hello_world() [0x40258f]
18  /lib64/libc.so.6(+0x295d0) [0x7fade47b95d0]
19  /lib64/libc.so.6(__libc_start_main+0x80) [0x7fade47b9680]
20  mpi_hello_world() [0x402455]
=================================
[c11:02028] *** Process received signal ***
[c11:02028] Signal: Aborted (6)
[c11:02028] Signal code:  (-6)
[c11:02028] [ 0] /lib64/libc.so.6(+0x3ebf0)[0x7fade47cebf0]
[c11:02028] [ 1] /lib64/libc.so.6(+0x8bedc)[0x7fade481bedc]
[c11:02028] [ 2] /lib64/libc.so.6(raise+0x16)[0x7fade47ceb46]
[c11:02028] [ 3] /lib64/libc.so.6(abort+0xd3)[0x7fade47b8833]
[c11:02028] [ 4] /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(+0x2f297)[0x7fade4324297]
[c11:02028] [ 5] /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(+0x2f369)[0x7fade4324369]
[c11:02028] [ 6] /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/ucx/libuct_ib.so.0(+0x263f0)[0x7fade110d3f0]
[c11:02028] [ 7] /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(+0x24987)[0x7fade4319987]
[c11:02028] [ 8] /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucp.so.0(ucp_worker_progress+0x2a)[0x7fade43abc9a]
[c11:02028] [ 9] /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(+0xa09bc)[0x7fade471b9bc]
[c11:02028] [10] /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(opal_common_ucx_del_procs_nofence+0x6a)[0x7fade471b79a]
[c11:02028] [11] /opt/ohpc/pub/mpi/openmpi5-intel/5.0.7/lib/libopen-pal.so.80(opal_common_ucx_del_procs+0x20)[0x7fade471baf0]

c11:02028] [12] ==== backtrace (tid:   2035) ====

..................

--------------------------------------------------------------------------------

Achilles

[George Bosilca]

This error message is usually due to a misconfiguration of the network. However, I don't think this is the case here because the output contains messages from both odd and even ranks (which according to your binding policy were placed on different nodes) suggesting at least some of the processes were able to connect (and thus the network configuration is correct).

So I'm thinking about some timing issues during network setup due to the fact that you have many processes per node, and an application that does nothing except creating and then shutting down the network layer. Does this happen if you have less processes per node ? Does it happen if you add anything else in the application (such as an `MPI_Barrier(MPI_COMM_WORLD)`) ?

   George.

To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@lists.open-mpi.org.

[Achilles Vassilicos]

When I use openmpi5, I get the same behavior even with a very small number of processes per node. However, when I use mpich-ofi it runs fine (see below). That gives me confidence that the network is setup correctly. The nodes are connected via infiniband ConnectX-3 adapters, and all ib tests show no problems.

I found  an older post about ucx1.18 having possible issues with openmpi5. I have assumed that ucx1.18 is now fully compatible with openmpi5. Could this be the cause? Does anyone use ucx1.8 with openmpi5? If not ucx1.18, what version is confirmed to work with openmpi5?

My test code:

----------------------------------------------------------------------

[av@c12 test]$ cat mpi_hello_world.c
#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
    // Initialize the MPI environment
    MPI_Init(NULL, NULL);

    // Get the number of processes
    int world_size;
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);

    // Get the rank of the process
    int world_rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

    // Get the name of the processor
    char processor_name[MPI_MAX_PROCESSOR_NAME];
    int name_len;
    MPI_Get_processor_name(processor_name, &name_len);

    // Print off a hello world message
    printf("Hello world from processor %s, rank %d out of %d processors\n",
           processor_name, world_rank, world_size);

    // Finalize the MPI environment.
    MPI_Finalize();
}
-------------------------------------------------------------------------

[av@c12 test]$ which mpirun
/opt/ohpc/pub/mpi/openmpi5-gnu14/5.0.7/bin/mpirun

[av@sms test]$ mpicc -o openmpi5_hello_world mpi_hello_world.c
[av@sms test]$ salloc -n 4 -N 2
salloc: Granted job allocation 63
salloc: Nodes c[12-13] are ready for job
[av@c12 test]$ mpirun  ./openmpi5_hello_world
Hello world from processor c12, rank 0 out of 4 processors
Hello world from processor c12, rank 1 out of 4 processors
Hello world from processor c13, rank 3 out of 4 processors
Hello world from processor c13, rank 2 out of 4 processors
[c12:1709 :0:1709]       ud_ep.c:278  Fatal: UD endpoint 0x117ae80 to <no debug data>: unhandled timeout error
==== backtrace (tid:   1709) ====
 0  /opt/ohpc/pub/mpi/ucx-ohpc/1.18.0/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f200b4f3ee4]
................

-----------------------------------------------------------------------

[av@sms test]$ which mpicc
/opt/ohpc/pub/mpi/mpich-ofi-gnu14-ohpc/3.4.3/bin/mpicc
[av@sms test]$ which mpirun
/opt/ohpc/pub/mpi/mpich-ofi-gnu14-ohpc/3.4.3/bin/mpirun
[av@sms test]$ mpicc -o mpich-ofi_hello_world mpi_hello_world.c
[av@sms test]$ salloc -n 4 -N 2
salloc: Granted job allocation 66
salloc: Nodes c[12-13] are ready for job
[av@c12 test]$ mpirun ./mpich-ofi_hello_world
Hello world from processor c13, rank 2 out of 4 processors
Hello world from processor c13, rank 3 out of 4 processors
Hello world from processor c12, rank 0 out of 4 processors
Hello world from processor c12, rank 1 out of 4 processors
[av@c12 test]$
------------------------------------------------------------------------

Achilles

[George Bosilca]

UCX 1.8 or UCX 1.18 ?

Your application does not exchange any data so it is possible that MPICH behavior differs from OMPI (aka not creating connections vs creating them during MPI_Init). That's why running a slightly different version of the hello_world with a barrier would clarify the connection's status.

  George.

[Achilles Vassilicos]

I checked further using a modification to mpi_hello_world.c (that includes MPI_Barrier) and a test code that checks connectivity between all processes.

1. On the mpi_hello_world_barrier.c case, openmpi5 failed the same way as before. mpich-ofi completed without error.

2. On the connectivity_c.c case, openmpi5 failed with the same error, and did not pass connectivity. mpich-ofi completed and passed connectivity (see below).

So it boils down to openmpi/ucx is unable to communicate between processes in my network setup?

-------------------------------------------------------------------------------------

[av@sms test]$ cat mpi_hello_world_barrier.c

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
    // Initialize the MPI environment
    MPI_Init(NULL, NULL);

    // Get the number of processes
    int world_size;
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);

    // Get the rank of the process
    int world_rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

    // Get the name of the processor
    char processor_name[MPI_MAX_PROCESSOR_NAME];
    int name_len;
    MPI_Get_processor_name(processor_name, &name_len);

    // Print off a hello world message

    int i;
    for(i=0; i<world_size; i++){

        printf("Hello world from processor %s, rank %d out of %d processors\n",
           processor_name, world_rank, world_size);

        MPI_Barrier(MPI_COMM_WORLD);

    }

    // Finalize the MPI environment.
    MPI_Finalize();
}

-------------------------------------------------------------------------------------

[av@c11 ompi]$ cat connectivity_c.c
/*
 * Copyright (c) 2007      Sun Microsystems, Inc.  All rights reserved.
 */
 * Test the connectivity between all processes.
 */
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <netdb.h>
#include <unistd.h>
#include <mpi.h>
int
main(int argc, char **argv)
{
    MPI_Status  status;
    int         verbose = 0;
    int         rank;
    int         np;        /* number of processes in job */
    int         peer;
    int         i;
    int         j;
    int         length;
    char        name[MPI_MAX_PROCESSOR_NAME+1];
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &np);
    /*
     * If we cannot get the name for whatever reason, just
     * set it to unknown. */
    if (MPI_SUCCESS != MPI_Get_processor_name(name, &length)) {
        strcpy(name, "unknown");
    }
    if (argc>1 && strcmp(argv[1], "-v")==0)
        verbose = 1;

    for (i=0; i<np; i++) {
        if (rank==i) {
            /* rank i sends to and receives from each higher rank */
            for(j=i+1; j<np; j++) {
                if (verbose)
                    printf("checking connection between rank %d on %s and rank %-4d\n",
                           i, name, j);
                MPI_Send(&rank, 1, MPI_INT, j, rank, MPI_COMM_WORLD);
                MPI_Recv(&peer, 1, MPI_INT, j, j, MPI_COMM_WORLD, &status);
            }
        } else if (rank>i) {
            /* receive from and reply to rank i */
            MPI_Recv(&peer, 1, MPI_INT, i, i, MPI_COMM_WORLD, &status);
            MPI_Send(&rank, 1, MPI_INT, i, rank, MPI_COMM_WORLD);
        }
    }
    MPI_Barrier(MPI_COMM_WORLD);
    if (rank==0)
        printf("Connectivity test on %d processes PASSED.\n", np);
    MPI_Finalize();
    return 0;
}
------------------------------------------------------------

[av@sms ompi]$ mpicc -o openmpi5-connectivity_c connectivity_c.c
[av@sms ompi]$ which mpicc
/opt/ohpc/pub/mpi/openmpi5-gnu14/5.0.7/bin/mpicc
[av@sms ompi]$ salloc -n 6 -N 3
salloc: Granted job allocation 72
salloc: Nodes c[11-13] are ready for job
[av@c11 ompi]$ mpirun openmpi5-connectivity_c
[c11:1928 :0:1928]       ud_ep.c:278  Fatal: UD endpoint 0x12e1c70 to <no debug data>: unhandled timeout error

------------------------------------------------------------

[av@sms ompi]$ mpicc -o mpich-ofi-connectivity_c connectivity_c.c
[av@sms ompi]$ salloc -n 6 -N 3
salloc: Granted job allocation 71
salloc: Nodes c[11-13] are ready for job
[av@c11 ompi]$ mpirun ./mpich-ofi-connectivity_c
Connectivity test on 6 processes PASSED.
------------------------------------------------------------

Achilles.

[Achilles Vassilicos]

UCX1.18.0

[George Bosilca]

OMPI 5.x has no support for the openib BTL, all IB traffic is now going through the UCX PML. This means that `-mca btl_openib_if_include XXX` is meaningless, but you can use the UCX_NET_DEVICES to direct UCX to a specific device.

As the error happens for UD you can switch to a different transport with the environment variable `UCX_TLS=dc_x,self,sm`. You could also build UCX in debug mode and enable UCX_LOG_LEVEL to something very verbose (such as DEBUG or DIAG) to get more info about the cause of the failure.

  George.

[Achilles Vassilicos]

A lot to chew on. Thanks.

Achilles

Sent from my iPhone

On Jul 2, 2025, at 5:40 PM, George Bosilca <bos...@icl.utk.edu> wrote:



[Achilles Vassilicos]

This works:

$ mpirun -x UCX_TLS=ib,rc -x UCX_NET_DEVICES=mlx4_0:1 -x MXM_RDMA_PORTS=mlx4_0:1 ./openmpi_hello_world_barrier

 

for both openmpi5-gnu14/ucx1.18.0 and onpempi5-intel/ucx1.18.0

Thanks for guiding me to dig into it.

Achilles.