upcrun spawns all mpi processes in a single node when using slurm

[stur...@gmail.com]

Hello all,

While running Hipmer, I noticed the following warning:

WARNING: Using GASNet's mpi-conduit, which exists for portability convenience.
WARNING: Support was detected for native GASNet conduits: ibv, psm
WARNING: You should *really* use the high-performance native GASNet conduit
WARNING: if communication performance is at all important in this program run.
WARNING: Host node06.cluster running more threads (48) than there are physical CPU's (24)
         enabling "polite", low-performance synchronization algorithms
[Th0 INFO 2019-04-10 16:35:20 mach.h:273]: Found 24 cores_per_node

which was concerning since I submit the script via slurm with 2 nodes and 24 cores per node.

A response in the hipmer mailing list suggested I run with `upcrun -v` to see what exactly is happening. After the upcrun command I see the attached log. As you can see (and as Rob suggested) upcrun is spawning all tasks in a single node instead of the two available. Any suggestions on why this might be?

The message in Hipmer's mailing list suggested the following:

1. " recompile upc with bindings to the slurm configuration (it should have autodetected this, but might not have) It is also possible that slurm is configured differently from our machines and srun is behaving differently than the hipmer scripts are expecting."
2. you can set the -config=FILE option to upcrun or create a $HOME/.upcrunrc file

For 1., is that documented anywhere? For 2. what kind of settings would I need to put in .upcrunrc?

Thanks for any guidance!

[Paul Hargrove]

Salomon,

As the first 4 lines of the warning say, you should probably not be using mpi-conduit.

I encourage you to try use of psm-conduit, but you should probably avoid ibv-conduit (IB Verbs emulation on Intel Omni-Path is pretty slow in my experience).  However, the job launch problem (the rest of the warning you report) is mostly independent of the choice of network (since they all use mpirun as the default job launcher).

I would have suspected that in the slurm job you describe, mpirun would have been able to honor the 2node X 24process request.  Since it is not, I think it may be necessary to provide the information explicitly.  So, I would suggest "upcrun -N2 -n48 ..."

Even when you add the "-N2", I suspect you may encountering a bug in the integration between GASNet and Open MPI when the latter is used as a job launcher.  This was reported previously at https://upc-bugs.lbl.gov/bugzilla/show_bug.cgi?id=3803, and I believe the second attachment to that bug report should be sufficient to get "-N2" working correctly for you.

If you use the default mpirun to launch psm-conduit jobs, you will need to set "OMPI_MCA_btl=^psm2" in your environment to prevent a conflict (GASNet and MPI cannot both use psm/psm2 as their transport library).

If after following the steps above, job launch works as intended, then you can stop reading here.

Otherwise, we can pursue PMI (slurm) support.

Regarding the information you mention from the HipMer mailing list:

Berkeley UPC (and GASNet which is the network library below it) does not, in general, support launching mpi-conduit jobs using anything other than mpirun.  If the Open MPI build was properly integrated with slurm, then I don't think you would have experienced any problem (it would have detected the 2node X 24core jobs geometry).  So PMI/slurm support become relevant only if BOTH: you switch to psm-conduit (which I do recommend) AND cannot launch correctly following the advice I gave above.

It is possible that PMI/slurm support was detected at configure time.  Since the steps to configure that are non-trivial, I suggest you first look for

    checking for PMI spawner support... yes

in the output of the configure step.  If it is present then we can proceed with setting up upcrun.conf.

If it is not present, then we can pursue locating your pmi.h and libpmi.

-Paul

--
You received this message because you are subscribed to the Google Groups "upc-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to upc-users+...@lbl.gov.
To post to this group, send email to upc-...@lbl.gov.
Visit this group at https://groups.google.com/a/lbl.gov/group/upc-users/.

--

Paul H. Hargrove <PHHar...@lbl.gov>

Computer Languages & Systems Software (CLaSS) Group

Computer Science Department

Lawrence Berkeley National Laboratory

[Salomon Turgman]

Dear Paul,

Thanks again for the detailed help (way beyond just upc support at this point).

I attempted to add the -N 2 flag to upcrun and got the following error:

--------------------------------------------------------------------------
Your job has requested more processes than the ppr for
this topology can support:

  App: /home/faculty/sturgmancohen/install/hipmer/hipmer-install-Linux/bin/main-64
  Number of procs:  48
  PPR: 24:node

Please revise the conflict and try again.
--------------------------------------------------------------------------

the command that spawned this error was:

somepath/bin/mpirun -np 48 -npernode 24 -x UPC_SHARED_HEAP_SIZE -x UPC_NODES -x UPC_PTHREADS_OPT -x GASNET_SSH_SERVERS -x GASNET_NODEFILE -x GASNET_RM_NODEFILE -x GASNET_VERBOSEENV -x GASNET_ENVCMD -x GASNET_PSHM_ENABLED -x GASNET_PHYSMEM_MAX -x GASNET_MAX_SEGSIZE -x GASNET_PHYSMEM_NOPROBE -x GASNET_SPAWN_CONTROL -x GASNET_PLATFORM -x GASNET_SPAWN_CONDUIT /someotherpath/bin/main-64 -f validation.config -N 24 -B -H -C

That said, I think you are exactly right in your diagnosis of my problem. I was not aware that I need to build mpi with slurm support or anything else. Since I had to use the install script provided in Hipmer's distribution to install bupc and upc2c, I had to use a different compiler/mpi combo than the one provided by my system (my system's seems to be too old). When I built the new openmpi version, I did not specify anything to do with slurm or pmi or psm.

I now definitely see why clusters typically have close to full-time administrators. This feels way past my abilities (especially without a list like this for support).

All that said, I am re-building mpi with slurm support and pmi support. Then I will re-install bupc and upc2c with the new settings and try hipmer again. The only concern is that the install script in Hipmer seems to default to the mpi-conduit.

Thanks again for all the help! I hope I am not taking too much of your time.

-s-

[Salomon Turgman]

Hey Paul,

Also... when looking a the psm-conduit readme it very emphatically suggests not to use psm and to instead use ofi.. is that something you recommend against?

-s-

[Paul Hargrove]

Salomon,

I replied as I did because the wrining output you had provided indicated that psm and ibv libraries were found at configure time.  However, in light of your new question, it occurs to me that ofi-conduit would not appear in that warning because it is categorized as a "portable" conduit rather than a "native" one.  So, you are right to ask.

If your build supports ofi-conduit, then it is likely to be a more stable choice than psm-conduit, with nearly identical performance.  Note, however, that it will raise the same concern I mentioned regarding MPI and GASNet using psm/psm2 simultaneously (since GASNet will be using psm indirectly through ofi).  So mpirun as a spawner (if srun does not work out) may need to be instructed not to use psm, psm2 OR ofi as its transport.  I can provide more detail if/when you reach that point.

-Paul

[stur...@gmail.com]

Dear Paul,

Success!!! I am running Hipmer on multiple nodes like it was meant to be. Thank you for all the help throughout this process! If I obtain any additional insight on the original error, I'll post back here.