maximum number of calls to pflotran executable file

[Kewei Chen]

Dear all,

I have multiple(100,000) independent cases(1d model) running on a HPC. I plan to request 1000 cores and run 100 cases on each core. However, I run into a weird problem that pflotran executable file is called successfully on only part of the cores. I don't know if it is because pflotran are called too many times(1000) simultaneously. Any suggestions are appreciated. 

Thanks,

Kewei  

[Kris Kuhlman]

PFLOTRAN needs at least one processor per simulation (and additionally a simulation with only 10 elements cannot be run on more than 10 processors). Also, two simulations can’t be run in the same directory unless the input/output files are named differently. They will step on one other’s output files. 

--
You received this message because you are subscribed to the Google Groups "pflotran-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pflotran-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pflotran-users/71f50bf7-82ed-4232-80e3-1d68d77ed339n%40googlegroups.com.

[Park, Heeho]

Hi Kewei,

It looks like you are trying to run everything in an embarrassingly parallel fashion for this 1D model.

Are you using slurm for the request? can I see one of your execution scripts? There may be some pause statements that are needed.

Kris's point about the directory structure for all individual and unique 100,000 directories or file names are required.

Best,

Heeho Daniel Park

! ------------------------------------ !
Sandia National Laboratories

Org: 08844, R&D

Work: 505-844-1319
! ------------------------------------ !

---

From: pflotra...@googlegroups.com <pflotra...@googlegroups.com> on behalf of Kris Kuhlman <kristophe...@gmail.com>
Sent: Saturday, July 30, 2022 10:58 AM
To: pflotra...@googlegroups.com <pflotra...@googlegroups.com>
Subject: [EXTERNAL] Re: [pflotran-users: 6780] maximum number of calls to pflotran executable file

 

To view this discussion on the web visit https://groups.google.com/d/msgid/pflotran-users/CACQ2i_T8K588jXkjKM17wmPzVzCMKoK-kR-aHv35z%2BG0uiJ61Q%40mail.gmail.com.

[Kewei Chen]

Thanks for your explanations, Kris and Park.

Kris: I created 100,000 folders and each folder contained an individual case. The initial and boundary conditions are all different for the cases, so the Stochastic mode does not work in my case.

Park: I used LSF for the request. All scripts are attached. bsub_1.sh is the script submitted to LSF. The python script ctr_1.py takes the cases number and then call the script pflotran.sh to run all the cases. It worked well in a small test that requested 400 cores with one case running in each core. However, when the number of requested core increased to 1000, many cores just did nothing. I don't know if it is because the job submissions was delayed if the number of pending jobs reached the limit. I received the message "User <ese-chenkw>: Pending job threshold reached. Retrying in 60 seconds..."  when I submit too many jobs at a time.  Thanks.