OpenMPI-5.0.5 & 5.0.6 build failure: error: expected expression before ‘struct’
6 views
Skip to first unread message
Sangam B
Mar 29, 2025, 11:08:37 AMMar 29
to Open MPI Users
Hi,
OpenMPI-5.0.5 or 5.0.6 versions fail with following error during "make" stage of the build procedure:
In file included from ../../../../../../ompi/mca/mtl/ofi/mtl_ofi.h:51,
from ../../../../../../ompi/mca/mtl/ofi/mtl_ofi.c:13:
../../../../../../ompi/mca/mtl/ofi/mtl_ofi.h: In function ‘ompi_mtl_ofi_context_progress’:
../../../../../../ompi/mca/mtl/ofi/mtl_ofi_request.h:19:5: warning: implicit declaration of function ‘container_of’ [-Wimplicit-function-declaration]
19 | container_of((_ptr_ctx), struct ompi_mtl_ofi_request_t, ctx)
| ^~~~~~~~~~~~
../../../../../../ompi/mca/mtl/ofi/mtl_ofi.h:152:27: note: in expansion of macro ‘TO_OFI_REQ’
152 | ofi_req = TO_OFI_REQ(ompi_mtl_ofi_wc[i].op_context);
| ^~~~~~~~~~
../../../../../../ompi/mca/mtl/ofi/mtl_ofi_request.h:19:30: error: expected expression before ‘struct’
19 | container_of((_ptr_ctx), struct ompi_mtl_ofi_request_t, ctx)
| ^~~~~~
../../../../../../ompi/mca/mtl/ofi/mtl_ofi.h:152:27: note: in expansion of macro ‘TO_OFI_REQ’
152 | ofi_req = TO_OFI_REQ(ompi_mtl_ofi_wc[i].op_context);
| ^~~~~~~~~~
../../../../../../ompi/mca/mtl/ofi/mtl_ofi_request.h:19:30: error: expected expression before ‘struct’
19 | container_of((_ptr_ctx), struct ompi_mtl_ofi_request_t, ctx)
| ^~~~~~
../../../../../../ompi/mca/mtl/ofi/mtl_ofi.h:200:19: note: in expansion of macro ‘TO_OFI_REQ’
200 | ofi_req = TO_OFI_REQ(error.op_context);
| ^~~~~~~~~~
make[2]: *** [Makefile:1603: mtl_ofi.lo] Error 1
from ../../../../../../ompi/mca/mtl/ofi/mtl_ofi.c:13:
../../../../../../ompi/mca/mtl/ofi/mtl_ofi.h: In function ‘ompi_mtl_ofi_context_progress’:
../../../../../../ompi/mca/mtl/ofi/mtl_ofi_request.h:19:5: warning: implicit declaration of function ‘container_of’ [-Wimplicit-function-declaration]
19 | container_of((_ptr_ctx), struct ompi_mtl_ofi_request_t, ctx)
| ^~~~~~~~~~~~
../../../../../../ompi/mca/mtl/ofi/mtl_ofi.h:152:27: note: in expansion of macro ‘TO_OFI_REQ’
152 | ofi_req = TO_OFI_REQ(ompi_mtl_ofi_wc[i].op_context);
| ^~~~~~~~~~
../../../../../../ompi/mca/mtl/ofi/mtl_ofi_request.h:19:30: error: expected expression before ‘struct’
19 | container_of((_ptr_ctx), struct ompi_mtl_ofi_request_t, ctx)
| ^~~~~~
../../../../../../ompi/mca/mtl/ofi/mtl_ofi.h:152:27: note: in expansion of macro ‘TO_OFI_REQ’
152 | ofi_req = TO_OFI_REQ(ompi_mtl_ofi_wc[i].op_context);
| ^~~~~~~~~~
../../../../../../ompi/mca/mtl/ofi/mtl_ofi_request.h:19:30: error: expected expression before ‘struct’
19 | container_of((_ptr_ctx), struct ompi_mtl_ofi_request_t, ctx)
| ^~~~~~
../../../../../../ompi/mca/mtl/ofi/mtl_ofi.h:200:19: note: in expansion of macro ‘TO_OFI_REQ’
200 | ofi_req = TO_OFI_REQ(error.op_context);
| ^~~~~~~~~~
make[2]: *** [Makefile:1603: mtl_ofi.lo] Error 1
OpenMPI-5.0.7 surpasses this error, but it is not able build cuda [GPU DIRECT] & ofi support:
Gromacs applications complains that it is not able to detect Cuda Aware MPI:
GPU-aware MPI was not detected, will not use direct GPU communication. Check the GROMACS install guide for recommendations for GPU-aware support. If you are certain about GPU-aware support in your MPI library, you can force its use by setting the GMX_FORCE_GPU_AWARE_MPI environment variable.
OpenMPI is configured like this:
'--disable-opencl' '--with-slurm' '--without-lsf'
'--without-opencl'
'--with-cuda=/opt/nvidia/hpc_sdk/Linux_x86_64/25.1/cuda/12.6'
'--without-rocm'
'--with-knem=/opt/knem-1.1.4.90mlnx3'
'--with-xpmem=/sw/openmpi/5.0.7/g133cu126_ubu2404/xpmem/2.7.3/'
'--with-xpmem-libdir=/sw/openmpi/5.0.7/g133cu126_ubu2404/xpmem/2.7.3//lib'
'--with-ofi=/sw/openmpi/5.0.7/g133cu126_ubu2404/ofi/2.0.0/c126g25xu118'
'--with-ofi-libdir=/sw/openmpi/5.0.7/g133cu126_ubu2404/ofi/2.0.0/c126g25xu118/lib'
'--enable-mca-no-build=btl-usnic'
'--without-opencl'
'--with-cuda=/opt/nvidia/hpc_sdk/Linux_x86_64/25.1/cuda/12.6'
'--without-rocm'
'--with-knem=/opt/knem-1.1.4.90mlnx3'
'--with-xpmem=/sw/openmpi/5.0.7/g133cu126_ubu2404/xpmem/2.7.3/'
'--with-xpmem-libdir=/sw/openmpi/5.0.7/g133cu126_ubu2404/xpmem/2.7.3//lib'
'--with-ofi=/sw/openmpi/5.0.7/g133cu126_ubu2404/ofi/2.0.0/c126g25xu118'
'--with-ofi-libdir=/sw/openmpi/5.0.7/g133cu126_ubu2404/ofi/2.0.0/c126g25xu118/lib'
'--enable-mca-no-build=btl-usnic'
Can somebody help me to build a successful cuda aware openmpi here?
Thanks
Gilles Gouaillardet
Mar 30, 2025, 12:51:49 AMMar 30
to us...@lists.open-mpi.org
Sangan,
The issue should have been fixed in Open MPI 5.0.6.
Anyway, are you certain Open MPI is not GPU aware and this is not cmake/GROMACS that failed to detect it?
What if you "configure" GROMACS with
cmake -DGMX_FORCE_GPU_AWARE_MPI=ON ...
If the problem persists, please open an issue at https://github.com/open-mpi/ompi/issues and do provide the required information.
Cheers,
Gilles
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@lists.open-mpi.org.
Sangam B
Mar 31, 2025, 2:48:34 AMMar 31
to us...@lists.open-mpi.org
Hello Gilles,
The gromacs-2024.4 build at cmake stage shows that CUDA AWARE MPI is detected:
-- Performing Test HAVE_MPI_EXT
-- Performing Test HAVE_MPI_EXT - Success
-- Performing Test MPI_SUPPORTS_CUDA_AWARE_DETECTION
-- Performing Test MPI_SUPPORTS_CUDA_AWARE_DETECTION - Success
-- Checking for MPI_SUPPORTS_CUDA_AWARE_DETECTION - yes
-- Performing Test HAVE_MPI_EXT - Success
-- Performing Test MPI_SUPPORTS_CUDA_AWARE_DETECTION
-- Performing Test MPI_SUPPORTS_CUDA_AWARE_DETECTION - Success
-- Checking for MPI_SUPPORTS_CUDA_AWARE_DETECTION - yes
But during runtime it is not able to detect it.
The OpenMPI MCA transports used are:
--mca btl ofi --mca coll ^hcoll -x GMX_ENABLE_DIRECT_GPU_COMM=true -x PATH -x LD_LIBRARY_PATH -hostfile s_hosts2 -np ${s_nmpi} --map-by numa --bind-to numa
This fails with following Seg-Fault error:
--------------------------------------------------------------------------
No components were able to be opened in the btl framework.
This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: g100n052
Framework: btl
--------------------------------------------------------------------------
^@[g100n052:14172:0:14172] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
[g100n052:14176:0:14176] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
No components were able to be opened in the btl framework.
This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: g100n052
Framework: btl
--------------------------------------------------------------------------
^@[g100n052:14172:0:14172] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
[g100n052:14176:0:14176] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
But 'ompi_info' shows "btl ofi" is available.
And the other notable point is, single node jobs with multiple gpus work fine, I mean on single node gromacs detects gpu aware mpi and the performance is good, as expected.
On more than 1 node only, it fails with the above seg fault error.
On a single node, is it using the XPMEM for communication?
Is there any OpenMPI env variable to show what transport is being used for communication between GPUs and between MPI ranks?
Thanks
Gilles Gouaillardet
Mar 31, 2025, 2:55:28 AMMar 31
to us...@lists.open-mpi.org
Sangam,
What if you run a simple MPI hello world program with
mpirun --mca pml ob1 --mca btl ofi ...
on one and several nodes?
Cheers,
Gilles
Sangam B
Mar 31, 2025, 8:57:48 AMMar 31
to us...@lists.open-mpi.org
Hello Gilles,
No it is not working on single node only:
mpirun --mca pml ob1 --mca btl ofi -np 4 test_ompi507stub.exe
--------------------------------------------------------------------------
No components were able to be opened in the btl framework.
This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: g100n052
Framework: btl
--------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_INIT has failed because at least one MPI process is unreachable
from another. This *usually* means that an underlying communication
plugin -- such as a BTL or an MTL -- has either not loaded or not
allowed itself to be used. Your MPI job will now abort.
You may wish to try to narrow down the problem;
* Check the output of ompi_info to see which BTL/MTL plugins are
available.
* Run your application with MPI_THREAD_SINGLE.
* Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
if using MTL-based communications) to see exactly which
communication plugins were considered and/or discarded.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):
ompi_mpi_init: ompi_mpi_instance_init failed
--> Returned "Unreachable" (-12) instead of "Success" (0)
--------------------------------------------------------------------------
[g100n052:00000] *** An error occurred in MPI_Init
[g100n052:00000] *** reported by process [901316609,0]
[g100n052:00000] *** on a NULL communicator
[g100n052:00000] *** Unknown error
[g100n052:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[g100n052:00000] *** and MPI will try to terminate your MPI job as well)
--------------------------------------------------------------------------
prterun has exited due to process rank 0 with PID 0 on node g100n052 calling
"abort". This may have caused other processes in the application to be
terminated by signals sent by prterun (as reported here).
--------------------------------------------------------------------------
MPI_INIT has failed because at least one MPI process is unreachable
from another. This *usually* means that an underlying communication
plugin -- such as a BTL or an MTL -- has either not loaded or not
allowed itself to be used. Your MPI job will now abort.
You may wish to try to narrow down the problem;
* Check the output of ompi_info to see which BTL/MTL plugins are
available.
* Run your application with MPI_THREAD_SINGLE.
* Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
if using MTL-based communications) to see exactly which
communication plugins were considered and/or discarded.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):
ompi_mpi_init: ompi_mpi_instance_init failed
--> Returned "Unreachable" (-12) instead of "Success" (0)
--------------------------------------------------------------------------
[g100n052:00000] *** An error occurred in MPI_Init
[g100n052:00000] *** reported by process [901316609,0]
[g100n052:00000] *** on a NULL communicator
[g100n052:00000] *** Unknown error
[g100n052:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[g100n052:00000] *** and MPI will try to terminate your MPI job as well)
--------------------------------------------------------------------------
prterun has exited due to process rank 0 with PID 0 on node g100n052 calling
"abort". This may have caused other processes in the application to be
terminated by signals sent by prterun (as reported here).
--------------------------------------------------------------------------
ompi_info shows:
MCA accelerator: null (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA accelerator: cuda (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA btl: self (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA btl: ofi (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA btl: sm (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA btl: tcp (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA btl: uct (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA btl: smcuda (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA mpool: hugepage (MCA v2.1.0, API v3.1.0, Component v5.0.7)
MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
v5.0.7)
MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA rcache: gpusm (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA rcache: rgpusm (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA reachable: netlink (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA smsc: cma (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA smsc: knem (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA smsc: xpmem (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA threads: pthreads (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA bml: r2 (MCA v2.1.0, API v2.1.0, Component v5.0.7)
MCA coll: adapt (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: basic (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: han (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: inter (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: libnbc (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: self (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: sync (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: tuned (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: cuda (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: ftagree (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: hcoll (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: monitoring (MCA v2.1.0, API v2.4.0, Component
v5.0.7)
MCA coll: sm (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA fs: lustre (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA hook: comm_method (MCA v2.1.0, API v1.0.0, Component
v5.0.7)
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA io: romio341 (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA mtl: ofi (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA op: avx (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v5.0.7)
MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
v5.0.7)
MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v5.0.7)
MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v5.0.7)
MCA part: persist (MCA v2.1.0, API v4.0.0, Component v5.0.7)
MCA pml: cm (MCA v2.1.0, API v2.1.0, Component v5.0.7)
MCA pml: monitoring (MCA v2.1.0, API v2.1.0, Component
v5.0.7)
MCA pml: ob1 (MCA v2.1.0, API v2.1.0, Component v5.0.7)
MCA pml: ucx (MCA v2.1.0, API v2.1.0, Component v5.0.7)
MCA pml: v (MCA v2.1.0, API v2.1.0, Component v5.0.7)
MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v5.0.7)
MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
v5.0.7)
MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA accelerator: cuda (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA btl: self (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA btl: ofi (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA btl: sm (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA btl: tcp (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA btl: uct (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA btl: smcuda (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA mpool: hugepage (MCA v2.1.0, API v3.1.0, Component v5.0.7)
MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
v5.0.7)
MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA rcache: gpusm (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA rcache: rgpusm (MCA v2.1.0, API v3.3.0, Component v5.0.7)
MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA reachable: netlink (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA smsc: cma (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA smsc: knem (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA smsc: xpmem (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA threads: pthreads (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA bml: r2 (MCA v2.1.0, API v2.1.0, Component v5.0.7)
MCA coll: adapt (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: basic (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: han (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: inter (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: libnbc (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: self (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: sync (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: tuned (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: cuda (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: ftagree (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: hcoll (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA coll: monitoring (MCA v2.1.0, API v2.4.0, Component
v5.0.7)
MCA coll: sm (MCA v2.1.0, API v2.4.0, Component v5.0.7)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA fs: lustre (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA hook: comm_method (MCA v2.1.0, API v1.0.0, Component
v5.0.7)
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA io: romio341 (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA mtl: ofi (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA op: avx (MCA v2.1.0, API v1.0.0, Component v5.0.7)
MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v5.0.7)
MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
v5.0.7)
MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v5.0.7)
MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v5.0.7)
MCA part: persist (MCA v2.1.0, API v4.0.0, Component v5.0.7)
MCA pml: cm (MCA v2.1.0, API v2.1.0, Component v5.0.7)
MCA pml: monitoring (MCA v2.1.0, API v2.1.0, Component
v5.0.7)
MCA pml: ob1 (MCA v2.1.0, API v2.1.0, Component v5.0.7)
MCA pml: ucx (MCA v2.1.0, API v2.1.0, Component v5.0.7)
MCA pml: v (MCA v2.1.0, API v2.1.0, Component v5.0.7)
MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v5.0.7)
MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v5.0.7)
MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
v5.0.7)
MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
v5.0.7)
OpenMPI is configured like this:
Configure command line: '--prefix=/sw/openmpi/5.0.7/g133cu126stubU2404/xp_minu118ofi2'
'--without-lsf'
'--with-cuda=/opt/nvidia/hpc_sdk/Linux_x86_64/25.1/cuda/12.6'
'--with-cuda-libdir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.1/cuda/12.6/targets/x86_64-linux/lib/stubs'
'--with-knem=/opt/knem-1.1.4.90mlnx3'
'--with-xpmem=/sw/openmpi/5.0.7/g133cu126stubU2404/xpmem/2.7.3/'
'--with-xpmem-libdir=/sw/openmpi/5.0.7/g133cu126stubU2404/xpmem/2.7.3//lib'
'--with-ofi=/sw/openmpi/5.0.7/g133cu126stubU2404/ofi/2.0.0/c126g25xu118'
'--with-ofi-libdir=/sw/openmpi/5.0.7/g133cu126stubU2404/ofi/2.0.0/c126g25xu118/lib'
'--enable-mca-no-build=btl-usnic'
'--without-lsf'
'--with-cuda=/opt/nvidia/hpc_sdk/Linux_x86_64/25.1/cuda/12.6'
'--with-cuda-libdir=/opt/nvidia/hpc_sdk/Linux_x86_64/25.1/cuda/12.6/targets/x86_64-linux/lib/stubs'
'--with-knem=/opt/knem-1.1.4.90mlnx3'
'--with-xpmem=/sw/openmpi/5.0.7/g133cu126stubU2404/xpmem/2.7.3/'
'--with-xpmem-libdir=/sw/openmpi/5.0.7/g133cu126stubU2404/xpmem/2.7.3//lib'
'--with-ofi=/sw/openmpi/5.0.7/g133cu126stubU2404/ofi/2.0.0/c126g25xu118'
'--with-ofi-libdir=/sw/openmpi/5.0.7/g133cu126stubU2404/ofi/2.0.0/c126g25xu118/lib'
'--enable-mca-no-build=btl-usnic'
UCX-1.18.0 is configured like this:
../../configure --prefix=${s_pfix} \
--enable-mt \
--without-rocm \
--with-cuda=${cuda_path} \
--with-knem=${knem_path} \
--with-xpmem=${xpmem_path}
--enable-mt \
--without-rocm \
--with-cuda=${cuda_path} \
--with-knem=${knem_path} \
--with-xpmem=${xpmem_path}
OFI [libfabric-2.0.0] is configured:
./configure \
--prefix=${s_pfix} \
--enable-shm=dl \
--enable-sockets=dl \
--enable-udp=dl \
--enable-tcp=dl \
--enable-rxm=dl \
--enable-rxd=dl \
--enable-verbs=dl \
--enable-psm2=no \
--enable-psm3=no \
--enable-ucx=dl:${ucx_path} \
--prefix=${s_pfix} \
--enable-shm=dl \
--enable-sockets=dl \
--enable-udp=dl \
--enable-tcp=dl \
--enable-rxm=dl \
--enable-rxd=dl \
--enable-verbs=dl \
--enable-psm2=no \
--enable-psm3=no \
--enable-ucx=dl:${ucx_path} \
--enable-gdrcopy-dlopen --with-gdrcopy=/usr \
--enable-cuda-dlopen --with-cuda=${cuda_path} \
--enable-xpmem=${xpmem_path} > config.out
With some verbose settings, It is not able to initialize btl component ofi:
[g100n052:24169] mca: base: components_register: component ofi register function successful
[g100n052:24169] mca: base: components_open: opening btl components
[g100n052:24169] mca: base: components_open: found loaded component ofi
[g100n052:24169] mca: base: components_open: component ofi open function successful
[g100n052:24172] select: initializing btl component ofi
[g100n052:24171] select: initializing btl component ofi
[g100n052:24170] select: initializing btl component ofi
[g100n052:24169] select: initializing btl component ofi
[g100n052:24172] select: init of component ofi returned failure
[g100n052:24171] select: init of component ofi returned failure
[g100n052:24169] select: init of component ofi returned failure
[g100n052:24170] select: init of component ofi returned failure
[g100n052:24172] mca: base: close: component ofi closed
[g100n052:24169] mca: base: components_open: opening btl components
[g100n052:24169] mca: base: components_open: found loaded component ofi
[g100n052:24169] mca: base: components_open: component ofi open function successful
[g100n052:24172] select: initializing btl component ofi
[g100n052:24171] select: initializing btl component ofi
[g100n052:24170] select: initializing btl component ofi
[g100n052:24169] select: initializing btl component ofi
[g100n052:24172] select: init of component ofi returned failure
[g100n052:24171] select: init of component ofi returned failure
[g100n052:24169] select: init of component ofi returned failure
[g100n052:24170] select: init of component ofi returned failure
[g100n052:24172] mca: base: close: component ofi closed
Thanks
Jeff Squyres
Mar 31, 2025, 9:56:34 AMMar 31
to Open MPI User's List
Try running with --mca btl_base_verbose 100 — that will enable much more verbose output and hopefully show what’s going wrong.
--
{+} Jeff Squyres
Keller, Rainer
Mar 31, 2025, 10:05:23 AMMar 31
to us...@lists.open-mpi.org
Dear Sangam,
as Gilles suggested, please try add self for loopback:
mpirun --mca pml ob1 --mca btl ofi,self …
since the error is:
[g100n052:00000] *** An error occurred in MPI_Init
[g100n052:00000] *** reported by process [901316609,0]
[g100n052:00000] *** on a NULL communicator
[g100n052:00000] *** Unknown error
Hope, this helps.
Best regards,
Rainer
as Gilles suggested, please try add self for loopback:
mpirun --mca pml ob1 --mca btl ofi,self …
since the error is:
[g100n052:00000] *** An error occurred in MPI_Init
[g100n052:00000] *** reported by process [901316609,0]
[g100n052:00000] *** on a NULL communicator
[g100n052:00000] *** Unknown error
Best regards,
Rainer
Gilles Gouaillardet
Mar 31, 2025, 10:08:05 AMMar 31
to us...@lists.open-mpi.org
Nice catch Rainer!
I absolutely forgot to include the btl/self component.
Cheers,
Gilles
Reply all
Reply to author
Forward
0 new messages