Fwd: [upc-users] Environment variables ?
17 views
Skip to first unread message
Paul H. Hargrove
Jul 14, 2023, 6:39:43 PM7/14/23
to UPC++
I am forwarding the post below to a more appropriate list.
-Paul
Hi upc-users,
--
You received this message because you are subscribed to the Google Groups "upc-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to upc-users+...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/upc-users/4c28e7d2-05e1-4971-bc32-be5a3a55ae69n%40lbl.gov.
--
---------- Forwarded message ---------
From: Tore Heide Larsen <to...@simula.no>
Date: Fri, Jul 14, 2023 at 7:29 AM
Subject: [upc-users] Environment variables ?
To: upc-users <upc-...@lbl.gov>
From: Tore Heide Larsen <to...@simula.no>
Date: Fri, Jul 14, 2023 at 7:29 AM
Subject: [upc-users] Environment variables ?
To: upc-users <upc-...@lbl.gov>
Hi upc-users,
Is it possible to specify active interface/HCA should be used with NETWORK='ibv' ? My nodes are multi-homed in various networks, multiple rails.
And is it possible to disable CUDA and HIP probing using environment variables for nodes lacking GPUs?
Thanks.
torel@srl-login1:~$ module show upcxx/gcc/2023.3.0
-------------------------------------------------------------------
/cm/shared/modulefiles/upcxx/gcc/2023.3.0:
module-whatis {UPC++ version 2023.3.0}
conflict upcxx
module load openmpi/gcc/64/4.1.5
module load valgrind/3.21.0
append-path PATH /cm/shared/apps/upcxx/gcc/2023.3.0/bin
prepend-path LD_LIBRARY_PATH /cm/shared/apps/upcxx/gcc/2023.3.0/lib:/cm/shared/apps/upcxx/gcc/2023.3.0/lib/valgrind
prepend-path LIBRARY_PATH /cm/shared/apps/upcxx/gcc/2023.3.0/lib:/cm/shared/apps/upcxx/gcc/2023.3.0/lib/valgrind
prepend-path INCLUDE /cm/shared/apps/upcxx/gcc/2023.3.0/include
prepend-path C_INCLUDE_PATH /cm/shared/apps/upcxx/gcc/2023.3.0/include
prepend-path CPLUS_INCLUDE_PATH /cm/shared/apps/upcxx/gcc/2023.3.0/include
prepend-path INCLUDE /cm/shared/apps/upcxx/gcc/2023.3.0/include
prepend-path INFOPATH /cm/shared/apps/upcxx/gcc/2023.3.0/share/doc/upcxx
setenv CMAKE_ROOT /cm/shared/apps/upcxx/gcc/2023.3.0/share/cmake/UPCXX
setenv UPCXX_NETWORK ibv
-------------------------------------------------------------------
/cm/shared/modulefiles/upcxx/gcc/2023.3.0:
module-whatis {UPC++ version 2023.3.0}
conflict upcxx
module load openmpi/gcc/64/4.1.5
module load valgrind/3.21.0
append-path PATH /cm/shared/apps/upcxx/gcc/2023.3.0/bin
prepend-path LD_LIBRARY_PATH /cm/shared/apps/upcxx/gcc/2023.3.0/lib:/cm/shared/apps/upcxx/gcc/2023.3.0/lib/valgrind
prepend-path LIBRARY_PATH /cm/shared/apps/upcxx/gcc/2023.3.0/lib:/cm/shared/apps/upcxx/gcc/2023.3.0/lib/valgrind
prepend-path INCLUDE /cm/shared/apps/upcxx/gcc/2023.3.0/include
prepend-path C_INCLUDE_PATH /cm/shared/apps/upcxx/gcc/2023.3.0/include
prepend-path CPLUS_INCLUDE_PATH /cm/shared/apps/upcxx/gcc/2023.3.0/include
prepend-path INCLUDE /cm/shared/apps/upcxx/gcc/2023.3.0/include
prepend-path INFOPATH /cm/shared/apps/upcxx/gcc/2023.3.0/share/doc/upcxx
setenv CMAKE_ROOT /cm/shared/apps/upcxx/gcc/2023.3.0/share/cmake/UPCXX
setenv UPCXX_NETWORK ibv
torel@srl-login1:~$ upcxx-info
--------------------------------------------------------------------
Software Version Info:
UPC++ version 2023.3.0 / gex-2023.3.0-0-g1675232
Citing UPC++ in publication? Please see: https://upcxx.lbl.gov/publications
Copyright (c) 2023, The Regents of the University of California,
through Lawrence Berkeley National Laboratory.
https://upcxx.lbl.gov
g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
--------------------------------------------------------------------
Configuration settings:
prefix: /cm/shared/apps/upcxx/gcc/2023.3.0
UPCXX_CODEMODE: opt
UPCXX_NETWORK: ibv
UPCXX_THREADMODE: seq
CONFIG_CC: /usr/bin/gcc
CONFIG_CXX: /cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx
UPCXX_BASH: /bin/bash
UPCXX_PYTHON: /usr/bin/python3.11
UPCXX_CUDA: 1
UPCXX_CUDA_NVCC: /cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/nvcc
UPCXX_CUDA_CPPFLAGS: -I/cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/../targets/x86_64-linux/include
UPCXX_CUDA_LIBFLAGS: -Wl,--start-group -L/cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/../targets/x86_64-linux/lib/stubs -L/cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/../targets/x86_64-linux/lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl -Wl,--end-group -lcuda
UPCXX_HIP: 1
UPCXX_HIP_CPPFLAGS: -I/cm/shared/apps/amd/rocm/5.2.3/include -isystem /cm/shared/apps/amd/rocm/5.2.3/include -D__HIP_PLATFORM_AMD__= -D__HIP_PLATFORM_HCC__=
UPCXX_HIP_LIBFLAGS: -L/cm/shared/apps/amd/rocm/5.2.3/lib -lamdhip64
UPCXX_ZE: <null>
UPCXX_ZE_CPPFLAGS: <null>
UPCXX_ZE_LIBFLAGS: <null>
UPCXX_VALGRIND: 1
UPCXX_DISCONTIG: <null>
UPCXX_FORCE_LEGACY_RELOCATIONS: <null>
UPCXX_MPSC_QUEUE: UPCXXI_MPSC_QUEUE_ATOMIC
GASNet RELEASE_VERSION: 2023.3.0
GASNet EX_SPEC_VERSION_MAJOR: 0
GASNet EX_SPEC_VERSION_MINOR: 16
GASNet CONFIGURE_ARGS: '--enable-kind-cuda-uva' '--enable-kind-hip' '--enable-valgrind' '--enable-ucx' '--enable-ofi' '--with-cc=/usr/bin/gcc' '--with-cflags=' '--with-cxx=/cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx' '--with-cxxflags=' '--with-hip-home=/cm/shared/apps/amd/rocm/5.2.3' '--disable-parsync' '--enable-seq' '--enable-par' '--enable-pthreads' '--disable-segment-everything' '--disable-ibv-rcv-thread' '--disable-debug'
GASNet CONDUITS: smp ofi udp mpi ucx ibv
PLATFORM_COMPILER_IDSTR: |COMPILER_FAMILY:GNU|COMPILER_VERSION:7.5.0|COMPILER_FAMILYID:1|STD:__STDC__,__STDC_VERSION__=201112L|misc:7.5.0|
PLATFORM_CXX_IDSTR: |COMPILER_FAMILY:GNU|COMPILER_VERSION:7.5.0|COMPILER_FAMILYID:1|STD:__STDC__,__cplusplus=201402L|misc:7.5.0|
PLATFORM_MPI_CC_IDSTR: |COMPILER_FAMILY:GNU|COMPILER_VERSION:7.5.0|COMPILER_FAMILYID:1|STD:__STDC__,__STDC_VERSION__=201112L|misc:7.5.0|
BUILD_ID: Thu Jul 13 17:00:29 CEST 2023 torel
SYSTEM_TUPLE: x86_64-pc-linux-gnu
SYSTEM_NAME: srl-login1
--------------------------------------------------------------------
UPC++ configure output:
UPC++ configure: ../configure --prefix=/cm/shared/apps/upcxx/gcc/2023.3.0 --enable-cuda --enable-hip --with-python=/usr/bin/python3.11 --enable-valgrind --disable-ze --enable-ucx --enable-ofi --with-cxx=/cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx
Unpacking /global/D1/homes/torel/workspace/UPCXX/upcxx-2023.3.0/src/GASNet-2023.3.0.tar.gz
System: Linux srl-login1 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic
Date: Thu Jul 13 16:54:47 CEST 2023
Current directory: /home/torel/workspace/UPCXX/upcxx-2023.3.0/Build-x86_64
Install directory: /cm/shared/apps/upcxx/gcc/2023.3.0
Configure command: ../configure --prefix=/cm/shared/apps/upcxx/gcc/2023.3.0 --enable-cuda --enable-hip --with-python=/usr/bin/python3.11 --enable-valgrind --disable-ze --enable-ucx --enable-ofi --with-cxx=/cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx
Configure environment:
GASNET='/global/D1/homes/torel/workspace/UPCXX/upcxx-2023.3.0/src/GASNet-2023.3.0.tar.gz'
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
/cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx
g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
/usr/bin/gcc
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
/usr/bin/python3.11: Python 3.11.3
/usr/bin/make
GNU Make version 4.1
/cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/nvcc:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
--------------------------------------------------------------------
GASNet configure warning summary:
ucx-conduit is still experimental, and may have performance and correctness bugs.
The verbs;ofi_rxm OFI provider was detected on your system and the ofi-conduit has been
configured for that provider. If that is not your desired provider, override this by using the
--with-ofi-provider=<provider name> configure flag. See the ofi-conduit README for details.
----------------------------------------------------------------------
GASNet configuration:
Portable conduits:
-----------------
Portable SMP-loopback conduit (smp) ON (auto)
OpenFabrics Interfaces conduit (ofi) ON (enabled)
Portable UDP/IP conduit (udp) ON (auto)
Portable MPI conduit (mpi) ON (auto)
Native, high-performance conduits:
---------------------------------
Unified Communication X conduit (ucx) ON (enabled)
InfiniBand IB Verbs conduit (ibv) ON (auto)
Cray XC Aries conduit (aries) OFF (not found)
Memory kinds:
------------
GPUs with NVIDIA CUDA API (cuda-uva) ON (enabled)
GPUs with AMD HIP API (hip) ON (enabled)
Some conduits and memory kinds require --enable-XXX configure flags and/or
additional variables providing the install location of vendor drivers.
See the GASNet documentation for details.
Misc Settings
-------------
MPI compatibility: yes
Pthreads support: yes
Segment config: fast
PSHM support: posix
Atomics support: native
PMI support: no
----------------------------------------------------------------------
torel@srl-login1:~$
--------------------------------------------------------------------
Software Version Info:
UPC++ version 2023.3.0 / gex-2023.3.0-0-g1675232
Citing UPC++ in publication? Please see: https://upcxx.lbl.gov/publications
Copyright (c) 2023, The Regents of the University of California,
through Lawrence Berkeley National Laboratory.
https://upcxx.lbl.gov
g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
--------------------------------------------------------------------
Configuration settings:
prefix: /cm/shared/apps/upcxx/gcc/2023.3.0
UPCXX_CODEMODE: opt
UPCXX_NETWORK: ibv
UPCXX_THREADMODE: seq
CONFIG_CC: /usr/bin/gcc
CONFIG_CXX: /cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx
UPCXX_BASH: /bin/bash
UPCXX_PYTHON: /usr/bin/python3.11
UPCXX_CUDA: 1
UPCXX_CUDA_NVCC: /cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/nvcc
UPCXX_CUDA_CPPFLAGS: -I/cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/../targets/x86_64-linux/include
UPCXX_CUDA_LIBFLAGS: -Wl,--start-group -L/cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/../targets/x86_64-linux/lib/stubs -L/cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/../targets/x86_64-linux/lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl -Wl,--end-group -lcuda
UPCXX_HIP: 1
UPCXX_HIP_CPPFLAGS: -I/cm/shared/apps/amd/rocm/5.2.3/include -isystem /cm/shared/apps/amd/rocm/5.2.3/include -D__HIP_PLATFORM_AMD__= -D__HIP_PLATFORM_HCC__=
UPCXX_HIP_LIBFLAGS: -L/cm/shared/apps/amd/rocm/5.2.3/lib -lamdhip64
UPCXX_ZE: <null>
UPCXX_ZE_CPPFLAGS: <null>
UPCXX_ZE_LIBFLAGS: <null>
UPCXX_VALGRIND: 1
UPCXX_DISCONTIG: <null>
UPCXX_FORCE_LEGACY_RELOCATIONS: <null>
UPCXX_MPSC_QUEUE: UPCXXI_MPSC_QUEUE_ATOMIC
GASNet RELEASE_VERSION: 2023.3.0
GASNet EX_SPEC_VERSION_MAJOR: 0
GASNet EX_SPEC_VERSION_MINOR: 16
GASNet CONFIGURE_ARGS: '--enable-kind-cuda-uva' '--enable-kind-hip' '--enable-valgrind' '--enable-ucx' '--enable-ofi' '--with-cc=/usr/bin/gcc' '--with-cflags=' '--with-cxx=/cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx' '--with-cxxflags=' '--with-hip-home=/cm/shared/apps/amd/rocm/5.2.3' '--disable-parsync' '--enable-seq' '--enable-par' '--enable-pthreads' '--disable-segment-everything' '--disable-ibv-rcv-thread' '--disable-debug'
GASNet CONDUITS: smp ofi udp mpi ucx ibv
PLATFORM_COMPILER_IDSTR: |COMPILER_FAMILY:GNU|COMPILER_VERSION:7.5.0|COMPILER_FAMILYID:1|STD:__STDC__,__STDC_VERSION__=201112L|misc:7.5.0|
PLATFORM_CXX_IDSTR: |COMPILER_FAMILY:GNU|COMPILER_VERSION:7.5.0|COMPILER_FAMILYID:1|STD:__STDC__,__cplusplus=201402L|misc:7.5.0|
PLATFORM_MPI_CC_IDSTR: |COMPILER_FAMILY:GNU|COMPILER_VERSION:7.5.0|COMPILER_FAMILYID:1|STD:__STDC__,__STDC_VERSION__=201112L|misc:7.5.0|
BUILD_ID: Thu Jul 13 17:00:29 CEST 2023 torel
SYSTEM_TUPLE: x86_64-pc-linux-gnu
SYSTEM_NAME: srl-login1
--------------------------------------------------------------------
UPC++ configure output:
UPC++ configure: ../configure --prefix=/cm/shared/apps/upcxx/gcc/2023.3.0 --enable-cuda --enable-hip --with-python=/usr/bin/python3.11 --enable-valgrind --disable-ze --enable-ucx --enable-ofi --with-cxx=/cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx
Unpacking /global/D1/homes/torel/workspace/UPCXX/upcxx-2023.3.0/src/GASNet-2023.3.0.tar.gz
System: Linux srl-login1 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic
Date: Thu Jul 13 16:54:47 CEST 2023
Current directory: /home/torel/workspace/UPCXX/upcxx-2023.3.0/Build-x86_64
Install directory: /cm/shared/apps/upcxx/gcc/2023.3.0
Configure command: ../configure --prefix=/cm/shared/apps/upcxx/gcc/2023.3.0 --enable-cuda --enable-hip --with-python=/usr/bin/python3.11 --enable-valgrind --disable-ze --enable-ucx --enable-ofi --with-cxx=/cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx
Configure environment:
GASNET='/global/D1/homes/torel/workspace/UPCXX/upcxx-2023.3.0/src/GASNet-2023.3.0.tar.gz'
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
/cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx
g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
/usr/bin/gcc
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
/usr/bin/python3.11: Python 3.11.3
/usr/bin/make
GNU Make version 4.1
/cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/nvcc:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
--------------------------------------------------------------------
GASNet configure warning summary:
ucx-conduit is still experimental, and may have performance and correctness bugs.
The verbs;ofi_rxm OFI provider was detected on your system and the ofi-conduit has been
configured for that provider. If that is not your desired provider, override this by using the
--with-ofi-provider=<provider name> configure flag. See the ofi-conduit README for details.
----------------------------------------------------------------------
GASNet configuration:
Portable conduits:
-----------------
Portable SMP-loopback conduit (smp) ON (auto)
OpenFabrics Interfaces conduit (ofi) ON (enabled)
Portable UDP/IP conduit (udp) ON (auto)
Portable MPI conduit (mpi) ON (auto)
Native, high-performance conduits:
---------------------------------
Unified Communication X conduit (ucx) ON (enabled)
InfiniBand IB Verbs conduit (ibv) ON (auto)
Cray XC Aries conduit (aries) OFF (not found)
Memory kinds:
------------
GPUs with NVIDIA CUDA API (cuda-uva) ON (enabled)
GPUs with AMD HIP API (hip) ON (enabled)
Some conduits and memory kinds require --enable-XXX configure flags and/or
additional variables providing the install location of vendor drivers.
See the GASNet documentation for details.
Misc Settings
-------------
MPI compatibility: yes
Pthreads support: yes
Segment config: fast
PSHM support: posix
Atomics support: native
PMI support: no
----------------------------------------------------------------------
torel@srl-login1:~$
--Tore
You received this message because you are subscribed to the Google Groups "upc-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to upc-users+...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/upc-users/4c28e7d2-05e1-4971-bc32-be5a3a55ae69n%40lbl.gov.
Computer Languages & Systems Software (CLaSS) Group
Computer Science Department
Lawrence Berkeley National Laboratory
Paul H. Hargrove
Jul 14, 2023, 6:54:15 PM7/14/23
to UPC++
Tore,
The `GASNET_IBV_PORTS` environment variable is what you need, and it is documented in ibv-conduit/README of the GASNet-EX sources.
The answer to your first question about InfiniBand interfaces is relatively simple:
The `GASNET_IBV_PORTS` environment variable is what you need, and it is documented in ibv-conduit/README of the GASNet-EX sources.
The question regarding "CUDA and HIP probing" is less simple:
If a UPC++ application does not attempt to use GPU memory kinds, then nothing in UPC++ or the underlying GASNet-EX communications library will make any calls to the CUDA or HIP libraries. So, I am not entirely sure what "probing" you are referring to.
My best guess is that you might be experiencing problems launching applications on non-GPU nodes due to lack of (for instance) libcuda.so or libamdhip64.so. Those are shared library dependencies of the executable when the corresponding support was enabled at configure time. So the need for these libraries is not something subject to control via UPC++ or GASNet-EX environment variables. If that is the problem you need to solve, then your options include installing the libraries on all nodes (even where the associated hardware is not present), or building multiple installs of UPC++ (including one for use on non-GPU nodes in which `--enable-cuda` and `--enable-hip` were *not* passed to UPC++'s configure script).
I hope that helps. Feel free to let us know if it does not.
-Paul
Reply all
Reply to author
Forward
0 new messages