[slurm-users] Slurm commands for launching tasks: salloc and sbatch

[Татьяна Озерова]

Hello! I have question about way of launching tasks in Slurm. I use the service in cloud and submit an application with sbatch or salloc. As far as I am concerned, the commands are similar: they allocate resources for counting users tasks and run them. However, I have received different results in cluster performance for the same task (task execution time is too long in case of salloc). So my question is what is the difference between these two commands, that can affect on task performance? Thank you beforehand.

[Mike Mikailov]

They should not affect the task performance.

May be the cluster configuration allocated slow machines for salloc. 

salloc and sbatch have different purposes:

• salloc is used to allocate a set of resources to a job. Once the resources have been allocated, the user can run a command or script on the allocated resources.
• sbatch is used to submit a batch script to Slurm. The batch script contains a list of commands or scripts that will be executed on the allocated resources.

In general, salloc is used for jobs that need to be run interactively, such as jobs that require a shell or jobs that need to be debugged. sbatch is used for jobs that can be run in the background, such as long-running jobs or jobs that are submitted by a queuing system.

Here is a table that summarizes the key differences between salloc and sbatch:

Feature	salloc	sbatch
Purpose	Allocate resources and run a command or script	Submit a batch script
Interactive	Yes	No
Background	No	Yes
Queuing system	No	Yes

Here are some examples of how to use salloc and sbatch:

• To allocate 2 nodes with 4 CPUs each and run the command ls, you would use the following command:

Code snippet

salloc -N 2 -c 4 ls

• To submit a batch script called my_job.sh that contains the command python my_script.py, you would use the following command:

Code snippet

sbatch my_job.sh

For more information on salloc and sbatch, please see the following documentation:

• salloc documentation: https://slurm.schedmd.com/salloc.html
• sbatch documentation: https://slurm.schedmd.com/sbatch.html

Sent from my iPhone

On Jul 4, 2023, at 8:22 AM, Татьяна Озерова <tanyaoze...@gmail.com> wrote:



[Татьяна Озерова]

Thank you for your answer! And if slurm workers are identical, what can be the reason? Can interactive mode affect the performance? I have submitted the task with the help of "srun {{ name_of_task }} --pty bash", and the result is the same as for launching with salloc. Thanks in advance!

вт, 4 июл. 2023 г. в 15:51, Mike Mikailov <mmik...@gmail.com>:

[Loris Bennett]

Mike Mikailov <mmik...@gmail.com> writes:

> They should not affect the task performance.
>
> May be the cluster configuration allocated slow machines for salloc.
>
> salloc and sbatch have different purposes:
>

> * salloc is used to allocate a set of resources to a job. Once the resources have been allocated, the user can run a command or script on the allocated
> resources.
> * sbatch is used to submit a batch script to Slurm. The batch script contains a list of commands or scripts that will be executed on the allocated

> resources.
>
> In general, salloc is used for jobs that need to be run interactively, such as jobs that require a shell or jobs that need to be debugged. sbatch is used
> for jobs that can be run in the background, such as long-running jobs or jobs that are submitted by a queuing system.
>
> Here is a table that summarizes the key differences between salloc and sbatch:
>
> Feature salloc sbatch
> Purpose Allocate resources and run a command or script Submit a batch script
> Interactive Yes No
> Background No Yes
> Queuing system No Yes

I am not sure what you mean with the last point, since 'salloc' is also
handled by the queueing system. If the resources requested are
currently not available, 'salloc' will wait until they are.

Cheers,

Loris

> Here are some examples of how to use salloc and sbatch:
>

> * To allocate 2 nodes with 4 CPUs each and run the command ls, you would use the following command:

>
> Code snippet
> salloc -N 2 -c 4 ls
>

> * To submit a batch script called my_job.sh that contains the command python my_script.py, you would use the following command:

>
> Code snippet
> sbatch my_job.sh
>
> For more information on salloc and sbatch, please see the following documentation:
>

> * salloc documentation: https://slurm.schedmd.com/salloc.html
> * sbatch documentation: https://slurm.schedmd.com/sbatch.html

>
> Sent from my iPhone
>
> On Jul 4, 2023, at 8:22 AM, Татьяна Озерова <tanyaoze...@gmail.com> wrote:
>
> Hello! I have question about way of launching tasks in Slurm. I use the service in cloud and submit an application with sbatch or salloc. As far as I
> am concerned, the commands are similar: they allocate resources for counting users tasks and run them. However, I have received different
> results in cluster performance for the same task (task execution time is too long in case of salloc). So my question is what is the difference
> between these two commands, that can affect on task performance? Thank you beforehand.
>

--
Dr. Loris Bennett (Herr/Mr)
ZEDAT, Freie Universität Berlin

[Mike Mikailov]

Performance depends only on the worker node capabilities along with IO. If the worker nodes are the same then may be the nodes under salloc use network drive (main storage) for IO which may slow down the tasks.

There are many tools available to localize the bottleneck in the task performance. You may also measure bandwidth between the local node and the main storage using “dd” command for instance.

Sent from my iPhone

On Jul 4, 2023, at 9:04 AM, Татьяна Озерова <tanyaoze...@gmail.com> wrote:



[Mike Mikailov]

About the last point. In the case of sbatch the jobs wait in the queue as long as it takes until the resources are available. In the case of interactive jobs (at least using Son of Grid Engine) they fail after a short time if no resources available. 

Sent from my iPhone

On Jul 4, 2023, at 9:11 AM, Loris Bennett <loris....@fu-berlin.de> wrote:

[Mike Mikailov]

Nodes for salloc could also be allowed to be oversubscribed or overloaded. 

There are a number of tools that can be used to study task performance bottlenecks on HPC clusters. Some of these tools include:

• SLURM Profiler: The SLURM Profiler is a tool that can be used to collect performance data for SLURM jobs. This data can be used to identify bottlenecks in the job execution process, such as slow nodes, I/O bottlenecks, and memory contention.
• Ganglia: Ganglia is a monitoring system that can be used to collect performance data for HPC clusters. This data can be used to identify bottlenecks in the cluster, such as overloaded nodes, high network traffic, and slow storage devices.
• PerfTools: PerfTools is a set of tools that can be used to collect performance data for Linux systems. This data can be used to identify bottlenecks in the system, such as CPU usage, memory usage, and I/O activity.
• VTune Amplifier: VTune Amplifier is a tool from Intel that can be used to collect performance data for Intel processors. This data can be used to identify bottlenecks in the processor, such as cache misses, branch mispredictions, and memory latency.
• HPCToolkit: HPCToolkit is a suite of tools that can be used to collect performance data for HPC applications. This data can be used to identify bottlenecks in the application, such as inefficient algorithms, memory leaks, and threading issues.

The best tool for a particular situation will depend on the specific needs of the user. However, all of the tools listed above can be used to identify task performance bottlenecks on HPC clusters.

In addition to these tools, there are a number of other things that can be done to study task performance bottlenecks on HPC clusters. These include:

• Reviewing the job submission scripts: The job submission scripts can be reviewed to ensure that they are using the correct resources and that they are submitting the jobs to the correct nodes.
• Monitoring the job execution: The job execution can be monitored to track the progress of the jobs and to identify any potential problems.
• Analyzing the performance data: The performance data can be analyzed to identify the specific bottlenecks that are impacting the performance of the jobs.
• Tuning the jobs: The jobs can be tuned to improve their performance. This may involve changing the parameters of the jobs, using different algorithms, or using different libraries.

By taking these steps, it is possible to identify and address task performance bottlenecks on HPC clusters. This can help to improve the performance of the jobs and to get the most out of the HPC resources.

Sent from my iPhone

On Jul 4, 2023, at 9:04 AM, Татьяна Озерова <tanyaoze...@gmail.com> wrote:



[Loris Bennett]

Mike Mikailov <mmik...@gmail.com> writes:

> About the last point. In the case of sbatch the jobs wait in the queue as long as it takes until the resources are available. In the case of interactive jobs
> (at least using Son of Grid Engine) they fail after a short time if no resources available.

But you were referring to 'salloc' and not something SGE does. The
default for 'salloc' is to wait indefinitely. You can change this
behaviour with the option '--deadline':

--deadline=<OPT>
remove the job if no ending is possible before this deadline
(start > (deadline - time[-min])). Default is no deadline.
Valid time formats are:
HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]]
now[+count[seconds(default)|minutes|hours|days|weeks]]

[snip (36 lines)]

>
> Queuing system No Yes
>
> I am not sure what you mean with the last point, since 'salloc' is also
> handled by the queueing system. If the resources requested are
> currently not available, 'salloc' will wait until they are.

[snip (42 lines)]

[Mike Mikailov]

Thank you Loris, for the further clarifications. The only question is who will wait forever in interactive mode? And how practical is it?

Interactive mode is what its name implies - interactive, not queueing.

It would make more sense if the default setting for deadline would be set to a reasonable time not indefinite in Slurm.


Sent from my iPhone

> On Jul 5, 2023, at 1:43 AM, Loris Bennett <loris....@fu-berlin.de> wrote:

[Loris Bennett]

Mike Mikailov <mmik...@gmail.com> writes:

> Thank you Loris, for the further clarifications. The only question is
> who will wait forever in interactive mode? And how practical is it?
>
> Interactive mode is what its name implies - interactive, not queueing.

To me, "interactive" is the alternative to "batch" - queueing happens
with both. Whether I submit an interactive job or a batch job, the job
can only start if the requested resources are available and the job has
a sufficiently high priority.

If you have enough nodes, you might be able to spare some to be
essentially idle most of the time, so that jobs sent to these nodes
will, in general, start quickly. If you have a small cluster, this
might not be an option.

> It would make more sense if the default setting for deadline would be set to a reasonable time not indefinite in Slurm.

One person's "reasonable" may be another person's "ridiculously long".
A "reasonable time" may even vary for the same person depending on the
circumstances. If you submit an interactive job before lunch, you might
be prepared to wait longer than if the building is going to close in 30
minutes and you will have to leave. With '--deadline' you can decide
case by case.

Cheers,

Loris

[Mike Mikailov]

Thank you Loris, for the further feedback.

“Reasonable” for SGE is within a few minutes, would be nice if it could be adjusted.

Still interactive means the user has almost immediate access to the system, not queued.

Sent from my iPhone

[Paul H. Hargrove]

Mike,

I think your definitions are probably in the minority on this list.

To be clear, I am not saying you (or SGE) are wrong, just that the folk here use different terms for what you are asking for.

I think of it like dialects of English where the same food might be a "cookie" or a "biscuit" depending on the local dialect.

To me, "interactive" means "keyboard interactive".

If I want the most immediate access then "immediate" is the term I use.

Fwiw, the `srun` and `salloc` commands have an `--immediate=time` option to bound how long they wait for resources.

It is possible to define an "immediate" or "interactive" (or any other name you may choose) QOS to ensure that interactive jobs are scheduled on a distinct partition of the cluster and/or are given higher priority in scheduling of nodes which are also used for non-interactive jobs.

I've just confirmed that LSF and PBS documentation use the term "interactive" in the same way as Slurm.

-Paul

--

Paul H. Hargrove <PHHar...@lbl.gov>
Pronouns: he, him, his

Computer Languages & Systems Software (CLaSS) Group

Computer Science Department

Lawrence Berkeley National Laboratory

[Mike Mikailov]

Hi Paul,

Thank you for the explanations. Actually this was not the main point of the question asked. I think we can close the discussions. 

The main point was: why a job is running more efficiently using sbatch than salloc.

Thank you all for the contributions. 

- Mike

Sent from my iPhone

On Jul 5, 2023, at 12:33 PM, Paul H. Hargrove <phhar...@lbl.gov> wrote: