Does ATS (v1.6) Support Multi-Thread by Default

[Olawale Quadri Ogunsola]

Dear ATS users,

I am currently using HPC to run ATS with SLURM as the task manager for job submission. I wanted to use 7 nodes, each with 48 cores. However, ATS is using only 1 core for runs, and when I asked the technical person at the back end. This is a substantial underuse of the node resources, which were intended to drastically decrease the runtime. I would like to know whether ATS supports multithreading of cores by default or if I need to specify it in the XML files.
Below are the relevant SLURM lines I used for the resource allocation and ATS run.

#SBATCH --nodes=7
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=48 # I intend to use all these resources, but each run is currently using 1

...

export OMP_NUM_THREADS=48
export OMP_PROC_BIND=true
export OMP_PLACES=cores
srun -n 7 --cpus-per-task=48 ats

I have also tried, and it gave the same result as above.

srun -n 7 ats

Best regards,

Olawale

[Moulton, J. David]

 

Hi Olawale,

 

     Your setup is confusing tasks within an openmp thread and mpi ranks.  ATS doesn’t make use of openmp, just MPI, for parallelism.  So definitely drop the “cpus-per-task” and the environment variables about OMP. 

 

If you want to run on 48 cores per node, over 7 nodes (336 mpi ranks in total) you could specify this in a couple of ways:

 

#SBATCH --ntasks=336

 

Slurm will distribute these across available nodes automatically unless you specify otherwise.

 

If you want to control this (i.e., in the case you chose to run on few cores than each node has to get better performance from the memory system), you can specify –nodes an –ntasks-per-node, like this

 

#SBATCH --nodes=7

#SBATCH --ntasks-per-node=48

 

Then your srun command should specify the total number of mpi ranks,

 

srun -n 336 ats

 

Hope that helps, good luck, David

--
You received this message because you are subscribed to the Google Groups "Amanzi-ATS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ats-users+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ats-users/98d147c2-636e-46f9-8e68-690d2e38f0bdn%40googlegroups.com.

[Olawale Quadri Ogunsola]

Hi David,

Thank you very much for the prompt response. 

I have tried the two suggestions you gave, but I am running into errors. I have pasted the lines of code and the relevant error message from the ATS error log file.

#SBATCH --ntasks=336

...

#SBATCH --nodes=7
#SBATCH --ntasks-per-node=48

srun -n 336 ats # I also tried without -n 336


srun: error: c31: tasks 240-287: Terminated
srun: error: c28: tasks 144-191: Terminated
srun: error: c25: tasks 0-47: Terminated
srun: error: c30: tasks 192-239: Terminated
srun: error: c27: tasks 96-143: Terminated
srun: error: c26: tasks 49-95: Terminated
srun: error: c32: tasks 288-335: Terminated

Best regards,

Olawale

[Olawale Quadri Ogunsola]

Hi David,

I tried varying the number of nodes with #SBATCH --nodes, and the ATS crashed when the number exceeded 2. However, my current simulations use all available cores, for instance, for two nodes: 2 nodes * 48 cores = 96 CPU cores, which is what I want.

Thank you very much for the continued support.

Best regards,

Olawale

[Bo Gao]

Hi Olawale,

If you still have this problem when asking for more than 2 nodes, try this (using your 7 nodes as example):

 

............

#SBATCH --nodes=7
#SBATCH --ntasks-per-node=48

srun -n 336 --cpu-bind=cores ats ......

Best,

Bo Gao

To view this discussion visit https://groups.google.com/d/msgid/ats-users/545af1fd-6ab5-4ae9-8aa0-126260844b60n%40googlegroups.com.

[Olawale Quadri Ogunsola]

Hi Bo,

Thank you very much for the follow-up. I tried what you suggested, and it is still giving the same error. I have attached the ATS error log file.

Best regards,

Olawale

--

_______________________

Best regards,

Olawale Quadri Ogunsola

Graduate Research Assistant 

Department of Environmental Science

The University of Toledo

Bowman-Oddy Lab., Rm. 3045

Mobile: +1 419 205 3009

[Bo Gao]

OK, I had the same issue on COMPASS when asking for more than 3 nodes, but adding --cpu-bind=cores solved my problem. Just FYI.

Best, 

Bo Gao

[Olawale Quadri Ogunsola]

Hi Bo,

That is good to know. It is not currently working for my runs. However, two nodes, each fully maximised (2*48 CPU cores), are enough for what I am currently running.

Best regards,

Olawale