OpenMP thread count handling appears to be broken with 2023.2

[tibo...@gmail.com]

Dear developers,

I have noticed some odd behaviour related to OpenMP thread count handling in Molpro 2023.2. It looks like by default Molpro is always trying to use all threads that a computer has (as OMP threads), so 2x the number of cores if HT/SMT is enabled on the machine. This excessive threading results in roughly 10-20% slower execution than purely serial, so there is a lot of wasted CPU usage for no benefit. The situation is even worse if most of the CPU cores are already being used by other programs.

The previous version (2022.3) behaved as expected, serial unless the user asks for parallelism.

This problem is happening both on our own hardware (Ubuntu 20.04), and on a HPC cluster that uses RHEL, so it does not seem like an OS or environment issue.

Please see attached outputs of running "OMP_DISPLAY_ENV=TRUE molpro -v" with 2022.3 and 2023.2. As you can see the newest version starts up as "(24 PROC) 64 bit serial version" instead of just "64 bit serial version".

I have tried using "-t 1" to turn off OpenMP threading, but it had zero effect. The only workaround that I can confirm to be working is setting OMP_NUM_THREADS=1 before starting Molpro.

Best wishes,

Tibor Győri

[Tibor Győri]

One more observation, running the latest binary release of MRCC from Molpro 23.2 results in the following warnings being printed to stderr:

OMP: Warning #96: Cannot form a team with 24 threads, using 1 instead.
OMP: Hint Consider unsetting KMP_DEVICE_THREAD_LIMIT (KMP_ALL_THREADS), KMP_TEAMS_THREAD_LIMIT, and OMP_THREAD_LIMIT (if any are set).

These do not show up if the same latest MRCC is used with Molpro 22.3.

--
You received this message because you are subscribed to a topic in the Google Groups "molpro-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/molpro-user/cDOEyACh6vY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to molpro-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/molpro-user/1c0ddbc8-ca26-4844-9f40-900f08af1ab9n%40googlegroups.com.

[Andy May]

Dear Tibor,

Firstly, a general note that whilst we've done our best to make the 'molpro' shell script work in most scenarios, it's simply not possible to know about all queue systems, MPI etc., so you are welcome if required to modify it, eg. to explicitly set OMP_NUM_THREADS there if that's what is needed on your system.

The 'OMP: Warning #96' warning message is something that comes from the new intel compiler (ifx), I can only assume that the new binary version of MRCC is built with this compiler. It is harmless and just tells you that you will be running with a single thread, see https://community.intel.com/t5/Intel-Fortran-Compiler/OMP-Warning-96/td-p/922762 . It is annoying, and there are various posts asking how to disable the warning.

In the previous version of Molpro we used OMP_NUM_THREADS to limit the total number of OMP threads, but in the latest version we switched to OMP_THREAD_LIMIT, which is supposed to exist for this purpose: https://www.ibm.com/docs/en/xl-c-aix/13.1.3?topic=openmp-omp-thread-limit . If I run on my system I see:

  [host] OMP_NUM_THREADS = '1'

...

  [host] OMP_THREAD_LIMIT = '1'

It seems then that because OMP_NUM_THREADS is 24 on your system, despite OMP_THREAD_LIMIT being set to 1 you are getting a threaded run. Is the value of OMP_NUM_THREADS set somewhere in your environment, or is that really a default value determined by the openmp library when you run? Even so it would be nice to know why OMP_NUM_THREADS is able to override OMP_THREAD_LIMIT. If you have any further insight please feel free to send it to me directly and we will discuss this in our next developer meeting later this week. For now though, please feel free to modify the molpro shell wrapper to get the behaviour you want.

 

Best wishes,

Andy

You received this message because you are subscribed to the Google Groups "molpro-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to molpro-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/molpro-user/CALL7z21ZYJwYf8EJWjF0NFWPF9jgqeHHn6biw6hgZy4kfaKLXg%40mail.gmail.com.

[tibo...@gmail.com]

Dear Andy,

Thank you for the swift response. The HPC cluster in have mentioned uses SLURM, but on our own hardware we have no queuing system. They are just a bunch of servers running Ubuntu 20.04 we can SSH into directly.

I can confirm that OMP_NUM_THREADS is not set on our hardware before Molpro is run:

gytibor @ CRD-ROME-1:~$ echo $O

$OPENBLAS_INC  $OPENBLAS_LIB  $OPTERR        $OPTIND        $OSTYPE

This is happening both on machines with Intel and AMD CPUs, so it is not a case of MKL being confused on AMD hardware.

I have no explanation for why this is not happening on your system. Perhaps (unlike us) you are running a cutting-edge Linux distribution and that somehow makes a difference?

I will let you know if I find out anything interesting, but probably I will just follow your suggestion and try replacing OMP_THREAD_LIMIT with OMP_NUM_THREADS in the shell wrapper.

Best,

Tibor