Environment variables ?

[Tore Heide Larsen]

Hi upc-users,

     Is it possible to specify active interface/HCA should be used with NETWORK='ibv' ?  My nodes are multi-homed in various networks, multiple rails.

     And is it possible to disable CUDA and HIP probing using environment variables for nodes lacking GPUs?

Thanks.

torel@srl-login1:~$ module show upcxx/gcc/2023.3.0
-------------------------------------------------------------------
/cm/shared/modulefiles/upcxx/gcc/2023.3.0:

module-whatis   {UPC++ version 2023.3.0}
conflict        upcxx
module          load openmpi/gcc/64/4.1.5
module          load valgrind/3.21.0
append-path     PATH /cm/shared/apps/upcxx/gcc/2023.3.0/bin
prepend-path    LD_LIBRARY_PATH /cm/shared/apps/upcxx/gcc/2023.3.0/lib:/cm/shared/apps/upcxx/gcc/2023.3.0/lib/valgrind
prepend-path    LIBRARY_PATH /cm/shared/apps/upcxx/gcc/2023.3.0/lib:/cm/shared/apps/upcxx/gcc/2023.3.0/lib/valgrind
prepend-path    INCLUDE /cm/shared/apps/upcxx/gcc/2023.3.0/include
prepend-path    C_INCLUDE_PATH /cm/shared/apps/upcxx/gcc/2023.3.0/include
prepend-path    CPLUS_INCLUDE_PATH /cm/shared/apps/upcxx/gcc/2023.3.0/include
prepend-path    INCLUDE /cm/shared/apps/upcxx/gcc/2023.3.0/include
prepend-path    INFOPATH /cm/shared/apps/upcxx/gcc/2023.3.0/share/doc/upcxx
setenv          CMAKE_ROOT /cm/shared/apps/upcxx/gcc/2023.3.0/share/cmake/UPCXX
setenv          UPCXX_NETWORK ibv

torel@srl-login1:~$ upcxx-info
--------------------------------------------------------------------
Software Version Info:

UPC++ version 2023.3.0  / gex-2023.3.0-0-g1675232
Citing UPC++ in publication? Please see: https://upcxx.lbl.gov/publications
Copyright (c) 2023, The Regents of the University of California,
through Lawrence Berkeley National Laboratory.
https://upcxx.lbl.gov

g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

--------------------------------------------------------------------
Configuration settings:

prefix:                             /cm/shared/apps/upcxx/gcc/2023.3.0
UPCXX_CODEMODE:                     opt
UPCXX_NETWORK:                      ibv
UPCXX_THREADMODE:                   seq
CONFIG_CC:                          /usr/bin/gcc
CONFIG_CXX:                         /cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx
UPCXX_BASH:                         /bin/bash
UPCXX_PYTHON:                       /usr/bin/python3.11
UPCXX_CUDA:                         1
UPCXX_CUDA_NVCC:                    /cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/nvcc
UPCXX_CUDA_CPPFLAGS:                -I/cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/../targets/x86_64-linux/include
UPCXX_CUDA_LIBFLAGS:                -Wl,--start-group -L/cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/../targets/x86_64-linux/lib/stubs -L/cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/../targets/x86_64-linux/lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl -Wl,--end-group -lcuda
UPCXX_HIP:                          1
UPCXX_HIP_CPPFLAGS:                 -I/cm/shared/apps/amd/rocm/5.2.3/include -isystem /cm/shared/apps/amd/rocm/5.2.3/include  -D__HIP_PLATFORM_AMD__= -D__HIP_PLATFORM_HCC__=
UPCXX_HIP_LIBFLAGS:                 -L/cm/shared/apps/amd/rocm/5.2.3/lib -lamdhip64
UPCXX_ZE:                           <null>
UPCXX_ZE_CPPFLAGS:                  <null>
UPCXX_ZE_LIBFLAGS:                  <null>
UPCXX_VALGRIND:                     1
UPCXX_DISCONTIG:                    <null>
UPCXX_FORCE_LEGACY_RELOCATIONS:     <null>
UPCXX_MPSC_QUEUE:                   UPCXXI_MPSC_QUEUE_ATOMIC
GASNet RELEASE_VERSION:             2023.3.0
GASNet EX_SPEC_VERSION_MAJOR:       0
GASNet EX_SPEC_VERSION_MINOR:       16
GASNet CONFIGURE_ARGS:              '--enable-kind-cuda-uva' '--enable-kind-hip' '--enable-valgrind' '--enable-ucx' '--enable-ofi' '--with-cc=/usr/bin/gcc' '--with-cflags=' '--with-cxx=/cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx' '--with-cxxflags=' '--with-hip-home=/cm/shared/apps/amd/rocm/5.2.3' '--disable-parsync' '--enable-seq' '--enable-par' '--enable-pthreads' '--disable-segment-everything' '--disable-ibv-rcv-thread' '--disable-debug'
GASNet CONDUITS:                    smp ofi udp mpi ucx ibv
PLATFORM_COMPILER_IDSTR:            |COMPILER_FAMILY:GNU|COMPILER_VERSION:7.5.0|COMPILER_FAMILYID:1|STD:__STDC__,__STDC_VERSION__=201112L|misc:7.5.0|
PLATFORM_CXX_IDSTR:                 |COMPILER_FAMILY:GNU|COMPILER_VERSION:7.5.0|COMPILER_FAMILYID:1|STD:__STDC__,__cplusplus=201402L|misc:7.5.0|
PLATFORM_MPI_CC_IDSTR:              |COMPILER_FAMILY:GNU|COMPILER_VERSION:7.5.0|COMPILER_FAMILYID:1|STD:__STDC__,__STDC_VERSION__=201112L|misc:7.5.0|
BUILD_ID:                           Thu Jul 13 17:00:29 CEST 2023 torel
SYSTEM_TUPLE:                       x86_64-pc-linux-gnu
SYSTEM_NAME:                        srl-login1

--------------------------------------------------------------------
UPC++ configure output:

UPC++ configure: ../configure --prefix=/cm/shared/apps/upcxx/gcc/2023.3.0 --enable-cuda --enable-hip --with-python=/usr/bin/python3.11 --enable-valgrind --disable-ze --enable-ucx --enable-ofi --with-cxx=/cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx
Unpacking /global/D1/homes/torel/workspace/UPCXX/upcxx-2023.3.0/src/GASNet-2023.3.0.tar.gz

System: Linux srl-login1 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic
 
Date: Thu Jul 13 16:54:47 CEST 2023
Current directory: /home/torel/workspace/UPCXX/upcxx-2023.3.0/Build-x86_64
Install directory: /cm/shared/apps/upcxx/gcc/2023.3.0
Configure command: ../configure --prefix=/cm/shared/apps/upcxx/gcc/2023.3.0 --enable-cuda --enable-hip --with-python=/usr/bin/python3.11 --enable-valgrind --disable-ze --enable-ucx --enable-ofi --with-cxx=/cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx
Configure environment:
    GASNET='/global/D1/homes/torel/workspace/UPCXX/upcxx-2023.3.0/src/GASNet-2023.3.0.tar.gz'
 
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
 
/cm/shared/apps/openmpi/gcc/64/4.1.5/bin/mpicxx
g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

/usr/bin/gcc
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

 
/usr/bin/python3.11: Python 3.11.3

/usr/bin/make
GNU Make version 4.1

/cm/shared/apps/cuda11.8/toolkit/11.8.0/bin/nvcc:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

--------------------------------------------------------------------
GASNet configure warning summary:
 
ucx-conduit is still experimental, and may have performance and correctness bugs.
 

The verbs;ofi_rxm OFI provider was detected on your system and the ofi-conduit has been
configured for that provider. If that is not your desired provider, override this by using the
--with-ofi-provider=<provider name> configure flag. See the ofi-conduit README for details.
 
----------------------------------------------------------------------
GASNet configuration:

 Portable conduits:
 -----------------
  Portable SMP-loopback conduit (smp)                ON     (auto)
  OpenFabrics Interfaces conduit (ofi)               ON     (enabled)
  Portable UDP/IP conduit (udp)                      ON     (auto)
  Portable MPI conduit (mpi)                         ON     (auto)

 Native, high-performance conduits:
 ---------------------------------
  Unified Communication X conduit (ucx)              ON     (enabled)
  InfiniBand IB Verbs conduit (ibv)                  ON     (auto)
  Cray XC Aries conduit (aries)                      OFF    (not found)

 Memory kinds:
 ------------
  GPUs with NVIDIA CUDA API (cuda-uva)               ON     (enabled)
  GPUs with AMD HIP API (hip)                        ON     (enabled)

 Some conduits and memory kinds require --enable-XXX configure flags and/or
 additional variables providing the install location of vendor drivers.
 See the GASNet documentation for details.

 Misc Settings
 -------------
  MPI compatibility:      yes
  Pthreads support:       yes
  Segment config:         fast
  PSHM support:           posix
  Atomics support:        native
  PMI support:            no
----------------------------------------------------------------------
torel@srl-login1:~$

 

--Tore

[Paul H. Hargrove]

Tore,

Thanks for your questions.

This is actually a list for UPC, not UPC++.

However, since they are monitored by the same staff, I am forwarding your message to the up...@googlegroups.com list (aka https://groups.google.com/g/upcxx) where I will provide you with some answers.

-Paul

--
You received this message because you are subscribed to the Google Groups "upc-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to upc-users+...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/upc-users/4c28e7d2-05e1-4971-bc32-be5a3a55ae69n%40lbl.gov.

--

Paul H. Hargrove <PHHar...@lbl.gov>
Pronouns: he, him, his

Computer Languages & Systems Software (CLaSS) Group

Computer Science Department

Lawrence Berkeley National Laboratory