NWPW requesting more cpus than assigned?

[SAcompuser]

So I am running a dynamics simulation using plane wave density functional theory using the NWPW directive, and I am submitting it as a job to a cluster which uses the PBSpro has it's job controller. I have tested my nwchem input script using a single core and it runs fine but I've been running into an issue with running on multiple cores.

 I submit the job with 'ncpus=8,mpiprocs=8' on the assumption that each MPI task will use a single core and my command to use mpirun is 'mpirun -np 8', but soon after the job starts it returns an error that PBS killed the job because 'ncpus 22.99 exceeded limit 8' I assume then that each non-controller MPI process requests 3 cpus each which then exceeds the CPU limit of the job causing PBS to kill.

 Is there a command in NWChem which controls the number of cores that an MPI process can request or is this something I have to fix on the job submission script side? 

[Edoardo Aprà]

A possible explanation for this behavior is that OpenMP threading kicks in.
Try disable it by setting

export OMP_NUM_THREADS=1

You should be able to check the number of threads NWChem is using by looking for the string

"NWChem w/ OpenMP: maximum threads"

in the output file

[SAcompuser]

This does seem to have fixed the issue, although I should note the environmental variable is actually OMP_NUM_THREADS and not OMP_NUMTHREADS. I think the issue comes about from an issue mentioned here in the NWChem compilation guide where it mentions that "NWChem itself automatically detects the available coprocessors in the system and properly partitions them for optimal use" and the system I was running on had 24 cores, but I was only wanting to run on 8 cores. Therefore NWChem was generating OpenMP threads for 24 cores even though it was only allowed to run on 8 cores at most. 

Thanks again!

On Thursday, October 15, 2020 at 7:15:03 PM UTC+2 Edoardo Aprà wrote:

A possible explanation for this behavior is that OpenMP threading kicks in.

Try disable it by setting

export OMP_NUMTHREADS=1

[Eric Bylaska]

Note you can compile without openmp by compiling without setting the USE_OPENMP environment variable.

---

From: nwchem...@googlegroups.com <nwchem...@googlegroups.com> on behalf of SAcompuser <tom...@gmail.com>
Sent: Monday, October 19, 2020 1:48:32 AM
To: NWChem Forum <nwchem...@googlegroups.com>
Subject: [nwchem-forum] Re: NWPW requesting more cpus than assigned?

 

--
You received this message because you are subscribed to the Google Groups "NWChem Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nwchem-forum...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nwchem-forum/993c0214-d687-495e-bddb-168522472efcn%40googlegroups.com.