GASNET Error on multiple nodes

[Clément Fontenaille]

Hi,

I downloaded the last version of UPC and UPC2C, and compiled it in order to run the UPC implementation of the UTS benchmark on an intel Xeon + OmniPath cluster.

I am very new with UPC as well as GASNET.

Everything is compiled using icc, conduit used is ibverbs.

srun is used to launch the program.

Setup is 1 process per core, i set  GASNET_PHYSMEM_MAX='1 GB' for the sake of getting this out of the way (physical memory is actually about 3GB per process), with 28 core per compute node.

Everything works fine up to 28 processes, but when using 2 compute nodes, i get the following output :

WARNING: GASNET_USE_XRC disabled because HCA lacks support.
         To suppress this message set environment variable
         GASNET_USE_XRC=0 or reconfigure with --disable-ibv-xrc.
UPCR: UPC thread 39 of 56 on my130 (pshm node 1 of 2, process 39 of 56, pid=170288)
UPCR: UPC thread 44 of 56 on my130 (pshm node 1 of 2, process 44 of 56, pid=170293)
UPCR: UPC thread 34 of 56 on my130 (pshm node 1 of 2, process 34 of 56, pid=170283)
UPCR: UPC thread 45 of 56 on my130 (pshm node 1 of 2, process 45 of 56, pid=170294)
UPCR: UPC thread 35 of 56 on my130 (pshm node 1 of 2, process 35 of 56, pid=170284)
UPCR: UPC thread 31 of 56 on my130 (pshm node 1 of 2, process 31 of 56, pid=170280)
UPCR: UPC thread 32 of 56 on my130 (pshm node 1 of 2, process 32 of 56, pid=170281)
UPCR: UPC thread 36 of 56 on my130 (pshm node 1 of 2, process 36 of 56, pid=170285)
UPCR: UPC thread 47 of 56 on my130 (pshm node 1 of 2, process 47 of 56, pid=170296)
UPCR: UPC thread 33 of 56 on my130 (pshm node 1 of 2, process 33 of 56, pid=170282)
UPCR: UPC thread 51 of 56 on my130 (pshm node 1 of 2, process 51 of 56, pid=170300)
UPCR: UPC thread 38 of 56 on my130 (pshm node 1 of 2, process 38 of 56, pid=170287)
UPCR: UPC thread 49 of 56 on my130 (pshm node 1 of 2, process 49 of 56, pid=170298)
UPCR: UPC thread 48 of 56 on my130 (pshm node 1 of 2, process 48 of 56, pid=170297)
UPCR: UPC thread 43 of 56 on my130 (pshm node 1 of 2, process 43 of 56, pid=170292)
UPCR: UPC thread 50 of 56 on my130 (pshm node 1 of 2, process 50 of 56, pid=170299)
UPCR: UPC thread 37 of 56 on my130 (pshm node 1 of 2, process 37 of 56, pid=170286)
UPCR: UPC thread 24 of 56 on my129 (pshm node 0 of 2, process 24 of 56, pid=79765)
UPCR: UPC thread  0 of 56 on my129 (pshm node 0 of 2, process  0 of 56, pid=79741)
UPCR: UPC thread  6 of 56 on my129 (pshm node 0 of 2, process  6 of 56, pid=79747)
UPCR: UPC thread 15 of 56 on my129 (pshm node 0 of 2, process 15 of 56, pid=79756)
UPCR: UPC thread  5 of 56 on my129 (pshm node 0 of 2, process  5 of 56, pid=79746)
UPCR: UPC thread  3 of 56 on my129 (pshm node 0 of 2, process  3 of 56, pid=79744)
UPCR: UPC thread 26 of 56 on my129 (pshm node 0 of 2, process 26 of 56, pid=79767)
UPCR: UPC thread  9 of 56 on my129 (pshm node 0 of 2, process  9 of 56, pid=79750)
UPCR: UPC thread 20 of 56 on my129 (pshm node 0 of 2, process 20 of 56, pid=79761)
UPCR: UPC thread 27 of 56 on my129 (pshm node 0 of 2, process 27 of 56, pid=79768)
UPCR: UPC thread 25 of 56 on my129 (pshm node 0 of 2, process 25 of 56, pid=79766)
UPCR: UPC thread  4 of 56 on my129 (pshm node 0 of 2, process  4 of 56, pid=79745)
UPCR: UPC thread 18 of 56 on my129 (pshm node 0 of 2, process 18 of 56, pid=79759)
UPCR: UPC thread  2 of 56 on my129 (pshm node 0 of 2, process  2 of 56, pid=79743)
UPCR: UPC thread 13 of 56 on my129 (pshm node 0 of 2, process 13 of 56, pid=79754)
UPCR: UPC thread 16 of 56 on my129 (pshm node 0 of 2, process 16 of 56, pid=79757)
UPCR: UPC thread  8 of 56 on my129 (pshm node 0 of 2, process  8 of 56, pid=79749)
UPCR: UPC thread 11 of 56 on my129 (pshm node 0 of 2, process 11 of 56, pid=79752)
UPCR: UPC thread 23 of 56 on my129 (pshm node 0 of 2, process 23 of 56, pid=79764)
UPCR: UPC thread 19 of 56 on my129 (pshm node 0 of 2, process 19 of 56, pid=79760)
UPCR: UPC thread  1 of 56 on my129 (pshm node 0 of 2, process  1 of 56, pid=79742)
UPCR: UPC thread 10 of 56 on my129 (pshm node 0 of 2, process 10 of 56, pid=79751)
UPCR: UPC thread 22 of 56 on my129 (pshm node 0 of 2, process 22 of 56, pid=79763)
UPCR: UPC thread  7 of 56 on my129 (pshm node 0 of 2, process  7 of 56, pid=79748)
UPCR: UPC thread 12 of 56 on my129 (pshm node 0 of 2, process 12 of 56, pid=79753)
UPCR: UPC thread 17 of 56 on my129 (pshm node 0 of 2, process 17 of 56, pid=79758)
UPCR: UPC thread 21 of 56 on my129 (pshm node 0 of 2, process 21 of 56, pid=79762)
UPCR: UPC thread 14 of 56 on my129 (pshm node 0 of 2, process 14 of 56, pid=79755)
UPCR: UPC thread 40 of 56 on my130 (pshm node 1 of 2, process 40 of 56, pid=170289)
UPCR: UPC thread 52 of 56 on my130 (pshm node 1 of 2, process 52 of 56, pid=170301)
UPCR: UPC thread 29 of 56 on my130 (pshm node 1 of 2, process 29 of 56, pid=170278)
UPCR: UPC thread 46 of 56 on my130 (pshm node 1 of 2, process 46 of 56, pid=170295)
UPCR: UPC thread 41 of 56 on my130 (pshm node 1 of 2, process 41 of 56, pid=170290)
UPCR: UPC thread 54 of 56 on my130 (pshm node 1 of 2, process 54 of 56, pid=170303)
UPCR: UPC thread 28 of 56 on my130 (pshm node 1 of 2, process 28 of 56, pid=170277)
UPCR: UPC thread 30 of 56 on my130 (pshm node 1 of 2, process 30 of 56, pid=170279)
UPCR: UPC thread 42 of 56 on my130 (pshm node 1 of 2, process 42 of 56, pid=170291)
UPCR: UPC thread 55 of 56 on my130 (pshm node 1 of 2, process 55 of 56, pid=170304)
UPCR: UPC thread 53 of 56 on my130 (pshm node 1 of 2, process 53 of 56, pid=170302)
UPC Runtime warning: Requested shared memory (64 MB) > available (18 MB) on node 0 (my129): using 18 MB per thread instead
UTS - Unbalanced Tree Search 2.1 (UPC)
Tree type:  0 (Binomial)
Tree shape parameters:
  root branching factor b_0 = 2000.0, root seed = 7
  BIN parameters:  q = 0.200014, m = 5, E(n) = 1.000070, E(s) = -14285.71
Random number generator: SHA-1 (state size = 20B)
Compute granularity: 1
Execution strategy:  Parallel search using 56 threads
   Load balance by work stealing, chunk size = 20 nodes
  CBarrier Interval: 1
   Polling Interval: 1
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 40/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 49/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 38/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 35/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 37/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 30/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 45/56
*** Caught a fatal signal: SIGABRT(6) on node 41/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 32/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 36/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 33/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 28/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 39/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 54/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 55/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 47/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 31/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 34/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 43/56
*** Caught a fatal signal: SIGABRT(6) on node 29/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 50/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 52/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 42/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 46/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 51/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 44/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 53/56
*** FATAL ERROR: ibv_reg_mr failed in firehose_move_callback errno=14 (Bad address)
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
NOTICE: We recommend linking the debug version of GASNet to assist you in resolving this application issue.
*** Caught a fatal signal: SIGABRT(6) on node 48/56
srun: error: my130: tasks 28-55: Aborted
srun: Terminating job step 456379.0
slurmstepd: error: *** STEP 456379.0 ON my129 CANCELLED AT 2018-09-28T15:03:08 ***
*** Caught a signal: SIGTERM(15) on node 0/56
*** Caught a signal: SIGTERM(15) on node 1/56
*** Caught a signal: SIGTERM(15) on node 2/56
*** Caught a signal: SIGTERM(15) on node 3/56
*** Caught a signal: SIGTERM(15) on node 4/56
*** Caught a signal: SIGTERM(15) on node 5/56
*** Caught a signal: SIGTERM(15) on node 6/56
*** Caught a signal: SIGTERM(15) on node 7/56
*** Caught a signal: SIGTERM(15) on node 8/56
*** Caught a signal: SIGTERM(15) on node 9/56
*** Caught a signal: SIGTERM(15) on node 10/56
*** Caught a signal: SIGTERM(15) on node 11/56
*** Caught a signal: SIGTERM(15) on node 12/56
*** Caught a signal: SIGTERM(15) on node 13/56
*** Caught a signal: SIGTERM(15) on node 14/56
*** Caught a signal: SIGTERM(15) on node 15/56
*** Caught a signal: SIGTERM(15) on node 16/56
*** Caught a signal: SIGTERM(15) on node 17/56
*** Caught a signal: SIGTERM(15) on node 18/56
*** Caught a signal: SIGTERM(15) on node 19/56
*** Caught a signal: SIGTERM(15) on node 20/56
*** Caught a signal: SIGTERM(15) on node 21/56
*** Caught a signal: SIGTERM(15) on node 22/56
*** Caught a signal: SIGTERM(15) on node 23/56
*** Caught a signal: SIGTERM(15) on node 24/56
*** Caught a signal: SIGTERM(15) on node 25/56
*** Caught a signal: SIGTERM(15) on node 26/56
*** Caught a signal: SIGTERM(15) on node 27/56
application called MPI_Abort(comm=0x84000002, 0) - process 7
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 7)
application called MPI_Abort(comm=0x84000002, 0) - process 21
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 21)
application called MPI_Abort(comm=0x84000002, 0) - process 3
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 3)
application called MPI_Abort(comm=0x84000002, 0) - process 16
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 16)
application called MPI_Abort(comm=0x84000002, 0) - process 4
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 4)
application called MPI_Abort(comm=0x84000002, 0) - process 11
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 11)
application called MPI_Abort(comm=0x84000002, 0) - process 14
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 14)
application called MPI_Abort(comm=0x84000002, 0) - process 20
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 20)
application called MPI_Abort(comm=0x84000002, 0) - process 22
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 22)
application called MPI_Abort(comm=0x84000002, 0) - process 25
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 25)
application called MPI_Abort(comm=0x84000002, 0) - process 27
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 27)
application called MPI_Abort(comm=0x84000002, 0) - process 19
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 19)
application called MPI_Abort(comm=0x84000002, 0) - process 10
In: PMI_Abort(0, application called MPI_Abort(comm=0x84000002, 0) - process 10)
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: my129: tasks 0-2,5-6,8-9,12-13,15,17-18,23-24,26: Killed

What can i attempt to solve the issue ?

Best regards,

Clément

[Paul Hargrove]

Clément,

Sorry to hear that you are having problems.
The errors you are seeing are consistent with the GASNet communications layer not being able to register enough memory with the NIC.

The first evidence of this I see is the message

UPC Runtime warning: Requested shared memory (64 MB) > available (18 MB) on node 0 (my129): using 18 MB per thread instead

That says that only 18MB of memory could be registered for use as shared memory on each compute node before hitting some limit (though which limit is not clear).  Given that (512 / 28) = 18 is seems likely that a 512MB limit is present.

By setting GASNET_PHYSMEM_MAX=1G you are allowing GASNet to use up to 1GB of memory for communication, including temporaries.  Since the space for the UPC heap appears to already have hit the limits on your system, I suspect this setting is leading to errors when trying to register temporary buffers for communication.

Please try your run again with the addition of GASNET_PHYSMEM_PROBE=1 in your environment.

This will tell GASNet to verify the 1GB setting at the beginning of the run.

Please share the resulting output and success or failure of your run.

Based on what you see, we can decide on the next steps.

To understand the source of the low limit, it would also be helpful to have the outputs from the following to gather some information about a compute node:

1) srun -n1 bash -c 'ulimit -a'

2) srun -n1 bash -c 'ulimit -a -H'

3) srun -n1 ibv_devinfo

4) srun -n1 df -h /dev/shm

-Paul

--
You received this message because you are subscribed to the Google Groups "upc-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to upc-users+...@lbl.gov.
To post to this group, send email to upc-...@lbl.gov.
Visit this group at https://groups.google.com/a/lbl.gov/group/upc-users/.

--

Paul H. Hargrove <PHHar...@lbl.gov>

Computer Languages & Systems Software (CLaSS) Group

Computer Science Department

Lawrence Berkeley National Laboratory

[Paul Hargrove]

Clément,

Please disregard most of what I said in my previous reply.

A colleague alerted me to the fact that you are using Omni-Path, which I had missed.

We do not support (nor advise) using ibv-conduit with Omni-Path hardware.

In our experience, use of either ofi-conduit or psm-conduit will provide as much as 10x better performance.

This may be as simple as passing --with-default-network=ofi to the configure script.

However, if you need any assistance configuring to use either conduit, just let us know.

-Paul

[Jeff Hammond]

Based on this, would it be prudent to greylist the IBV conduit on OPA systems?

I’m not sure what the best implementation is but always choosing PSM over IBV would work in many cases.

If OPA is plugged into the login node, lspci works but I guess that’s not always the case.

Jeff

[Dan Bonachea]

Jeff -

The BUPC + GASNet configure script already places psm-conduit at a higher priority than ibv-conduit, so on a system where both sets of drivers are detected at configure time, the default should be to use psm-conduit. Of course this is only the default, it doesn't prevent users from selecting a particular network on the UPC compile line with `-network=ibv`. Also, this doesn't help if only one set of drivers is detected in the user environment. Also (as you mention) even a theoretical hardware-level detection would be inadequate at configure/install/build time since the frontend node may lack any of the relevant HPC network cards.

Consequently, we currently handle this "grey-listing" in the ibv-conduit documentation : From http://gasnet.lbl.gov/dist/ibv-conduit/README :

    While ibv-conduit may work on Intel Omni-Path (OPA) HCAs, we instead
    recommend use of psm-conduit on such systems.

We have an existing RFE to expand ibv-conduit startup routines to detect and warn about OPA hardware:

  https://upc-bugs.lbl.gov/bugzilla/show_bug.cgi?id=3609

but it has not been a high priority for development effort. You are welcome to append suggestions there, or better yet donate a patch ;-)

-D

[Clément Fontenaille]

Hi Paul,

Thank you for pointing out the ibv issue.

In order to do so, i used the following configure command

cd build && ../berkeley_upc-2.28.0/multiconf.pl 'CC=mpicc' 'CXX=mpicxx' 'MPI_CC=mpicc' '--prefix=$HOME/UTS/UPC/install' 'BUPC_TRANS=$HOME/UTS/UPC/upc2c/install/targ' '--with-default-network=psm'

I reconfigured with psm as default conduit but it didnt really solve the issue.

So i ran the hello world test on the login node, in a single-node (exclusive) reservation and a 2-node reservation.

Here is an overview of the results :

grep -c "Hello from thread" *libupcr-*.out
loginnode_libupcr-ibv-par-test.out:2
loginnode_libupcr-ibv-seq-test.out:2
loginnode_libupcr-ibv-thr-test.out:2
loginnode_libupcr-mpi-par-test.out:2
loginnode_libupcr-mpi-seq-test.out:2
loginnode_libupcr-mpi-thr-test.out:2
loginnode_libupcr-psm-par-test.out:0
loginnode_libupcr-psm-seq-test.out:0
loginnode_libupcr-psm-thr-test.out:0
loginnode_libupcr-smp-par-test.out:2
loginnode_libupcr-smp-seq-test.out:2
loginnode_libupcr-smp-thr-test.out:2
loginnode_libupcr-udp-par-test.out:0
loginnode_libupcr-udp-seq-test.out:0
loginnode_libupcr-udp-thr-test.out:0
multinode_libupcr-ibv-par-test.out:4
multinode_libupcr-ibv-seq-test.out:2
multinode_libupcr-ibv-thr-test.out:2
multinode_libupcr-mpi-par-test.out:4
multinode_libupcr-mpi-seq-test.out:2
multinode_libupcr-mpi-thr-test.out:2
multinode_libupcr-psm-par-test.out:0
multinode_libupcr-psm-seq-test.out:0
multinode_libupcr-psm-thr-test.out:0
multinode_libupcr-smp-par-test.out:4
multinode_libupcr-smp-seq-test.out:2
multinode_libupcr-smp-thr-test.out:2
multinode_libupcr-udp-par-test.out:0
multinode_libupcr-udp-seq-test.out:0
multinode_libupcr-udp-thr-test.out:0
singlenode_libupcr-ibv-par-test.out:4
singlenode_libupcr-ibv-seq-test.out:2
singlenode_libupcr-ibv-thr-test.out:2
singlenode_libupcr-mpi-par-test.out:4
singlenode_libupcr-mpi-seq-test.out:2
singlenode_libupcr-mpi-thr-test.out:2
singlenode_libupcr-psm-par-test.out:0
singlenode_libupcr-psm-seq-test.out:0
singlenode_libupcr-psm-thr-test.out:0
singlenode_libupcr-smp-par-test.out:4
singlenode_libupcr-smp-seq-test.out:2
singlenode_libupcr-smp-thr-test.out:2
singlenode_libupcr-udp-par-test.out:0
singlenode_libupcr-udp-seq-test.out:0
singlenode_libupcr-udp-thr-test.out:0

I attach an archive of the detailed output.

what do you think should be the next step ?

thank you for your support.

Clement

[Paul Hargrove]

Clement,

If I am following correctly, your results show no problems with mpi-conduit or ibv-conduit, but you are unable to spawn multinode udp or psm jobs. I believe I can explain both failures (and hopefully help you to resolve them).

The psm spawning problem is explained (sort of) by the error message you see:

  reason: psm2_ep_open failure: Exceeded supported amount of endpoints

This is libpsm2 reporting that GASNet could not open an endpoint because the limit (1 per process) has already been reached.  It has been reached because, due to integration with SLURM, MPI has taken the one endpoint when srun was used for job launch.  This problem of MPI taking the only psm2 endpoint can be addressed in a few ways.

If you have a sufficiently new libpsm2 (Aug 2017 or newer, I believe), then Intel's documentation indicates that setting PSM2_MULTI_EP=1 in your environment will remove the single-open restriction.  However, I had not had any opportunity to test that functionality until I began composing this message.  My initial testing shows this works as advertised, passing the tests provided with GASNet on one system I have access to (with Intel MPI and an OPA network, but not SLURM).  So, you may be the first to test this capability with any "real" application and the first to try with srun.  We would very much like to know of your success or failure if you are able to try this approach.

If libpsm2's multi-endpoit support is not available (or does not work) our recommendation is to set MPI-specific environment variable(s) to prevent MPI's use of psm2, usually by forcing use of TCP for the few MPI communications used to launch the job.  You can find recommendations for various MPIs in share/doc/GASNet/README-mpi-spawner of the Berkeley UPC install directory (or gasnet/other/mpi-spawner/README in the source directory).   I am going to guess you are using Intel MPI, in which case I_MPI_DEVICE=sock is appropriate.

Alternatively one can try to spawn GASNet without the use of MPI, using either ssh or pmi for job launch.  If you use "gasnetrun_psm -spawner=ssh -n <proc> <app> <args>", then ssh will be used for launch and should automatically use the nodes allocated to you by SLURM.  Alternatively, "gasnetrun_psm -spawner=pmi -n <proc> <app> <args>" may also work, but only if SLURM's libpmi and corresponding header were found at configure time.  You should be able to find the gasnetrun_psm script in the opt/bin directory of the Berkeley UPC install directory (or opt/gasnet/psm-conduit/contrib in the build directory).

I am concerned, however, by the fact that you have set CC=mpicc at configure time.  This makes me believe that you intend to run hybrid UPC+MPI jobs.  If that is the case and PSM2_MULTI_EP=1 does not work as hoped, then you are left with the unfortunate consequences of libpsm's single-open restriction: you cannot have both GASNet and MPI using psm2 as the transport library.  If that is the case, then let me know and we can discuss your options.

I would not recommend use of udp-conduit when you have a high-speed network.  However, for completeness, I believe the udp-conduit spawning problem can be addressed using the following instructions from udp-conduit's README (substitute "export var=val" for "setenv var val" if using bash):

$ setenv GASNET_SPAWNFN 'C'

$ setenv GASNET_CSPAWN_CMD 'srun -n %N %C'

$ ./libupcr-udp-seq-test 2

Where 2 is the number of processes to run.  The srun command will be executed transparently.

-Paul

--

You received this message because you are subscribed to the Google Groups "upc-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to upc-users+...@lbl.gov.
To post to this group, send email to upc-...@lbl.gov.
Visit this group at https://groups.google.com/a/lbl.gov/group/upc-users/.

[Clément Fontenaille]

Hi again,

tldr : it works thanks to your advices, thank you. Now how do you expand max threads ?

So i got it to work, here is what i observed.

Firstly, regarding the hello test case in login node, single node reservation and 2-nodes reservation.

When running the test program on login node I do not use srun, but gasnetrun_psm, gasnetrun_ibv, mpirun and ./xxx for udp.

Using srun with PSM2_MULTI_EP=1 works just fine.

I_MPI_DEVICE=sock did not do the trick and the error output is similar (yes i use intel mpi, but i am not sure what slurm uses)

Gasnetrun_psm works as intended, and gasnetrun_ibv works fine too for this test.

Finally, 

export GASNET_SPAWNFN='C'
export GASNET_CSPAWN_CMD='srun -n %N %C'

effectively allows to run udp jobs in single and multiple nodes, but does not work on login node due to srun complaining.

I am concerned with the output of "par" versions of the test : they spawn 4 threads and output 4 "Hello" lines even though they were launched with "-n 2" . Is this working as intended ?

Regarding CC=mpicc, i just use mpicc as a wrapper to whatever compiler i currently use, reducing just a bit the amount of script editing I have to do when i switch compiler. I do not intend to run UPC+MPI programs.

However, when using srun to run an actual program compiled with "upcc -network=psm", i get the following error *** FATAL ERROR: PMI2_Job_GetId() failed

For completeness, here is the job script, which does not do anything special.

#!/bin/bash
#SBATCH --exclusive
# Partition (submission class)
#SBATCH --partition somepartitionclass
# Job time (hh:mm:ss)
#SBATCH --time 01:00:00
# ----------------------------
# MPI tasks number
#SBATCH --ntasks 8
# MPI task maximum memory (MB)
# SBATCH --mem-per-cpu 3000 
# ----------------------------
#move files from home directory to scratch directory, replace it with a simlink
cd $LOCAL_WORK_DIR 
mv $HOME/UTS/WORKDIR/uts-upc_8_T1L/* .
rm -rf $HOME/UTS/WORKDIR/uts-upc_8_T1L
ln -s $LOCAL_WORK_DIR /home/2012006/cfonte01/UTS/WORKDIR/uts-upc_8_T1L
# ----------------------------
export GASNET_PHYSMEM_MAX='1 GB' 
export PSM2_MULTI_EP=1
env > env.txt

source $UTS_HOME/sample_trees.sh
srun -n 8 $UTS_HOME/uts-upc $T1L > uts-upc.out 2>&1 

replacing srun with "$UPC_HOME/opt/bin/gasnetrun_psm -spawner=ssh" works fine. Using  "-spawner=pmi" makes upc startup very slow, so i just stick with ssh.

Now, i am limited by UPCR_MAX_THREADS=1024. I looked it up in the sources and searched in the doc and online resources, but could not figure out how to change the representation size of thread ids properly, i guess i have to input something to configure or so ?

Thanks again for your help.

Clément

[Paul Hargrove]

Clément,

I am glad to hear things are now working for you.

Thank you for your patience.

The output with the par cases is as expected.

The "par" versions use pthreads to implement multiple UPC threads per process (2 per proc by default).

So, when you launch N processes using srun or another launch script you are launching 2*N UPC threads.

This is handled a bit better with the `upcrun` launch script, which manages the number of pthreads per process and number of processes to give the expected results. 

I had not recommended `upcrun` earlier because you I was concerned you were writing hybrid UPC+MPI applications.  Since you have indicated that is not the case, I suggest you try $UPC_HOME/bin/upcrun which should automatically use the appropriate gasnetrun_{ibv,psm,ofi} scripts, as well as the right commands for smp and udp.  Additionally, it will provide additional command line options for things like increasing the size of the UPC shared heap, which must otherwise be done using environment variables.  A man page is available in $UPCHOME/share/man/man1/upcrun.1 or online here.

The only thing you probably need to setup in that case is to set GASNET_{IBV,OFI,PSM}_SPAWNER=ssh if you want ssh-based spawning (in place of the -spawner=* options).  However, that can be made the default configure time as well, for instance by using --with-psm-spawner=ssh.

Increasing the 1024-thread limit is, as you guessed, done at configure time.  The details are covered under the heading "TRADING-OFF MAXIMUM 'THREADS', BLOCKSIZE, AND HEAP SIZE" in the file INSTALL.TXT .  

I believe that if you are using ssh-based spawning for psm-conduit (and not running UPC+MPI hybrid apps), then it should not be necessary to set PSM2_MULTI_EP=1 in your environment.  Since that support is off by default in libpsm, I would suggest not enabling it without an actual need.

-Paul

[Jeff Hammond]

Thanks for the details, Dan.  I should have known that you all would have done quite a bit about this already.

I looked at the RFE and might be able to help there, but it will likely be late December before I have cycles for this.

Best,

Jeff

--

Jeff Hammond
jeff.s...@gmail.com
http://jeffhammond.github.io/