[slurm-users] Different max number of jobs in individual and array jobs

[Shaohao Chen]

Hi,

 

We use Slurm on our cluster and set a max number of all jobs. But we would want to implement this: N concurrent individual jobs and M concurrent jobs in all job arrays (e.g. m job arrays with n jobs in each and M=n*m), and M is much larger than N. I have read the Slurm documentation, but cannot find a way to separate the individual jobs and jobs in arrays. Does anyone know how to do that?

 

Thank you.

 

Shaohao

[Brian Andrus]

Array jobs are individual jobs that have been grouped. Underneath, they each have their own jobid besides the grouped array jobid.

Not sure there is an easy way to pull what you are looking to do.

Brian Andrus

[Sebastian T Smith]

Hi,

This doesn't solve your problem but might be an option:

In similar cases, we instruct our users to create `n` Jobs of `m` Steps.  Some experimentation may be required to determine the number of Steps to maximize Job run time without hitting your limits.  Our max limit is 14 days, so this pattern has been easy for users to implement.

Sebastian

--

Sebastian Smith High-Performance Computing Engineer Office of Information Technology 1664 North Virginia Street MS 0291 work-phone: 775-682-5050 email: sts...@unr.edu website: http://rc.unr.edu

---

From: slurm-users <slurm-use...@lists.schedmd.com> on behalf of Brian Andrus <toom...@gmail.com>
Sent: Thursday, June 3, 2021 10:16 AM
To: slurm...@lists.schedmd.com <slurm...@lists.schedmd.com>
Subject: Re: [slurm-users] Different max number of jobs in individual and array jobs

 

[Prentice Bisbal]

II know I'm a few weeks late with this response. I actually looked into this 4-6 weeks ago. According to the Slurm documenntation, an individual job step counts as a job when evaluating job limits. Pay attention to the note in the documenation below. From https://slurm.schedmd.com/slurm.conf.html:

MaxJobCount

The maximum number of jobs Slurm can have in its active database at one time. Set the values of MaxJobCount and MinJobAge to ensure the slurmctld daemon does not exhaust its memory or other resources. Once this limit is reached, requests to submit additional jobs will fail. The default value is 10000 jobs. NOTE: Each task of a job array counts as one job even though they will not occupy separate job records until modified or initiated.

Prentice 

[Prentice Bisbal]

... to complete my thought, I don't think what you want to do is possible. If M is the number of job steps, and N is total jobs, M cannot be greater than N.

Prentice 

[Paul Brunk]

Hi:

 

If you (Shaohao) mean you want to limit all running jobs to the sum of

up to N non-array-jobs and up to M array jobs, could you have N

"local" licenses of LicenseName 'nonarray' (e.g.) and M "local"

licenses of LicenseName 'array', and cause job_submit lua to add a

request for a license of the relevant type to each submission?

 

--

Flailing wildly at the keyboard,

Paul Brunk, system administrator

Georgia Advanced Computing Resource Center

Enterprise IT Svcs, the University of Georgia

 

From: slurm-users <slurm-use...@lists.schedmd.com> On Behalf Of Prentice Bisbal
Sent: Thursday, June 17, 2021 15:30
To: slurm...@lists.schedmd.com
Subject: Re: [slurm-users] [External] Different max number of jobs in individual and array jobs

 

[EXTERNAL SENDER - PROCEED CAUTIOUSLY]