[slurm-users] Get original script of a job

[Alberto Morillas, Angelines]

Hi,

 

I would like to know if it will be possible to get the script that was used to send a job.

I know that when I send a job with scontroI can get the path and the name of the script used to send this job, but normally the users change theirs scripts and sometimes all was wrong after that, so is there any possibility to reproduce the script of an old job???

 

Thanks in advance

 

Angelines

 

[Ole Holm Nielsen]

The Slurm database doesn't store information about old jobs' Command
(i.e., the job script) information, let alone store the script itself!
See "man sacct" and look at the list below --helpformat (Print a list of
fields that can be specified with the --format option).

This was also discussed in a previous mailing list thread:
https://lists.schedmd.com/pipermail/slurm-users/2020-April/005307.html

/Ole

[Ward Poelmans]

Hi,

On 5/03/2021 11:29, Alberto Morillas, Angelines wrote:

> I know that when I send a job with scontroI can get the path and the
> name of the script used to send this job, but normally the users change
> theirs scripts and sometimes all was wrong after that, so is there any
> possibility to reproduce the script of an old job???

It's not stored by default. You can have a look at
https://github.com/itkovian/sarchive for archiving jobs scripts.

Ward

[Carl Ponder]

I put this line in my job-control file (written in bash) to capture the original as part of the run:

cp $0 $RUNDIR/$SLURM_JOB_NAME

The $0 gives the full path to the working copy of the script, so it expands to this for example:

/fs/slurm/var/spool/job67842/slurm_script

It depends on this spooling directory being readable by all users.
This may not be the case on all systems.
If not, there might be a way to make it readable to just the owner of the job, you'd have to talk to the SLURM developers about that.

Also I'm not setting the jobname, so the $SLURM_JOB_NAME defaults to the name of the original script.
Otherwise, you should also be able to extract this from the "command" field output from squeue.

Note that this only works because the job is still running.
But it enables me to edit-and-submit the script in rapid-fire and capture the original script with each of the runs.
So I can put it in the same directory where all the other output files go, and have a complete description of the run conditions.

---

Subject:	[slurm-users] Get original script of a job
Date:	Fri, 5 Mar 2021 10:29:58 +0000
From:	Alberto Morillas, Angelines <angeline...@ciemat.es>
Reply-To:	Slurm User Community List <slurm...@lists.schedmd.com>
To:	slurm...@lists.schedmd.com <slurm...@lists.schedmd.com>

External email: Use caution opening links or attachments

[Prentice Bisbal]

I have this same problem - a user writes to me about a problem they're having with Slurm, and instead of pasting the script into the support ticket or e-mail, give me the path to it. When finally get to looking at it, it's obviously they've changed it since reporting the issue, so I have no forensic information for my investigation.

Once a job completes, Slurm doesn't store that information anywhere. The working directory is stored in the sacct data in the field WorkDir, but the actual command information isn't saved anywhere.

Prentice

-- 
Prentice 

[Alberto Morillas, Angelines]

Thanks for your help!

________________________________________________

Dra. Angelines Alberto Morillas

Unidad de Arquitectura Informática
Despacho: 22.1.32
Telf.: +34 91 346 6119
Fax: +34 91 346 6537

skype: angelines.alberto

CIEMAT
Avenida Complutense, 40
28040 MADRID
________________________________________________


El 5/3/21 14:14, "slurm-users en nombre de slurm-use...@lists.schedmd.com" <slurm-use...@lists.schedmd.com en nombre de slurm-use...@lists.schedmd.com> escribió:

Send slurm-users mailing list submissions to
slurm...@lists.schedmd.com

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users
or, via email, send a message with subject or body 'help' to
slurm-use...@lists.schedmd.com

You can reach the person managing the list at
slurm-us...@lists.schedmd.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of slurm-users digest..."


Today's Topics:

1. Re: slurm-users Digest, Vol 41, Issue 13
(Alberto Morillas, Angelines)
2. Re: Get original script of a job (Ward Poelmans)
3. Re: Get original script of a job (Carl Ponder)
4. Use nodes exclusive and shared simultaneously (Heckes, Frank)


----------------------------------------------------------------------

Message: 1
Date: Fri, 5 Mar 2021 12:15:21 +0000

From: "Alberto Morillas, Angelines" <angeline...@ciemat.es>

To: "slurm...@lists.schedmd.com" <slurm...@lists.schedmd.com>
Subject: Re: [slurm-users] slurm-users Digest, Vol 41, Issue 13
Message-ID:
<DB8PR04MB66357D5614...@DB8PR04MB6635.eurprd04.prod.outlook.com>

Content-Type: text/plain; charset="us-ascii"

Thanks!

________________________________
De: slurm-users <slurm-use...@lists.schedmd.com> en nombre de slurm-use...@lists.schedmd.com <slurm-use...@lists.schedmd.com>
Enviado: Friday, March 5, 2021 1:00:01 PM
Para: slurm...@lists.schedmd.com <slurm...@lists.schedmd.com>
Asunto: slurm-users Digest, Vol 41, Issue 13

Send slurm-users mailing list submissions to
slurm...@lists.schedmd.com

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users
or, via email, send a message with subject or body 'help' to
slurm-use...@lists.schedmd.com

You can reach the person managing the list at
slurm-us...@lists.schedmd.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of slurm-users digest..."


Today's Topics:

1. Re: Get original script of a job (Ole Holm Nielsen)


----------------------------------------------------------------------

Message: 1
Date: Fri, 5 Mar 2021 11:56:05 +0100
From: Ole Holm Nielsen <Ole.H....@fysik.dtu.dk>
To: <slurm...@lists.schedmd.com>
Subject: Re: [slurm-users] Get original script of a job
Message-ID: <61c47956-5fc9-5be5...@fysik.dtu.dk>
Content-Type: text/plain; charset="utf-8"; format=flowed

On 05-03-2021 11:29, Alberto Morillas, Angelines wrote:
> I would like to know if it will be possible to get the script that was
> used to send a job.
>
> I know that when I send a job with scontroI can get the path and the
> name of the script used to send this job, but normally the users change
> theirs scripts and sometimes all was wrong after that, so is there any
> possibility to reproduce the script of an old job???

The Slurm database doesn't store information about old jobs' Command
(i.e., the job script) information, let alone store the script itself!
See "man sacct" and look at the list below --helpformat (Print a list of
fields that can be specified with the --format option).

This was also discussed in a previous mailing list thread:
https://lists.schedmd.com/pipermail/slurm-users/2020-April/005307.html

/Ole

End of slurm-users Digest, Vol 41, Issue 13
*******************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210305/3b9b478c/attachment-0001.htm>

------------------------------

Message: 2
Date: Fri, 5 Mar 2021 13:28:03 +0100
From: Ward Poelmans <ward.p...@vub.be>
To: slurm...@lists.schedmd.com
Subject: Re: [slurm-users] Get original script of a job
Message-ID: <d1c0fbab-2686-e3c8...@vub.be>
Content-Type: text/plain; charset=utf-8

Hi,

On 5/03/2021 11:29, Alberto Morillas, Angelines wrote:

> I know that when I send a job with scontroI can get the path and the
> name of the script used to send this job, but normally the users change
> theirs scripts and sometimes all was wrong after that, so is there any
> possibility to reproduce the script of an old job???

It's not stored by default. You can have a look at
https://github.com/itkovian/sarchive for archiving jobs scripts.

Ward

------------------------------

Message: 3
Date: Fri, 5 Mar 2021 06:58:05 -0600
From: Carl Ponder <cpo...@nvidia.com>
To: "Alberto Morillas, Angelines" <angeline...@ciemat.es>
Cc: Slurm User Community List <slurm...@lists.schedmd.com>
Subject: Re: [slurm-users] Get original script of a job
Message-ID: <266b8b0d-29d4-4673...@nvidia.com>
Content-Type: text/plain; charset="utf-8"; Format="flowed"

I put this line in my job-control file (written in bash) to capture the
original as part of the run:

cp $0 $RUNDIR/$SLURM_JOB_NAME

The $0 gives the full path to the working copy of the script, so it
expands to this for example:

/fs/slurm/var/spool/job67842/slurm_script

It depends on this spooling directory being readable by all users.
This may not be the case on all systems.
If not, there might be a way to make it readable to just the owner of
the job, you'd have to talk to the SLURM developers about that.

Also I'm not setting the jobname, so the $SLURM_JOB_NAME defaults to the
name of the original script.
Otherwise, you should also be able to extract this from the "command"
field output from squeue.

Note that this only works because the job is still running.
But it enables me to edit-and-submit the script in rapid-fire and
capture the original script with each of the runs.
So I can put it in the same directory where all the other output files
go, and have a complete description of the run conditions.

------------------------------------------------------------------------

Subject: [slurm-users] Get original script of a job
Date: Fri, 5 Mar 2021 10:29:58 +0000
From: Alberto Morillas, Angelines <angeline...@ciemat.es>
Reply-To: Slurm User Community List <slurm...@lists.schedmd.com>
To: slurm...@lists.schedmd.com <slurm...@lists.schedmd.com>

*External email: Use caution opening links or attachments*

Hi,

I would like to know if it will be possible to get the script that was
used to send a job.

I know that when I send a job with scontroI can get the path and the
name of the script used to send this job, but normally the users change
theirs scripts and sometimes all was wrong after that, so is there any
possibility to reproduce the script of an old job???

Thanks in advance


Angelines

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210305/008584f1/attachment-0001.htm>

------------------------------

Message: 4
Date: Fri, 5 Mar 2021 13:13:43 +0000
From: "Heckes, Frank" <hec...@mps.mpg.de>
To: "slurm...@lists.schedmd.com" <slurm...@lists.schedmd.com>
Subject: [slurm-users] Use nodes exclusive and shared simultaneously
Message-ID: <bfec4752b9db47a1...@mps.mpg.de>
Content-Type: text/plain; charset="utf-8"

Hi all,

Sorry if this has been ask and answered before.



Resulting from a user/owner requirement I would need to set-up a subset of a
nodes in a partition be used as common shared resources still, but these nodes
should be available with smallest latency possible in case the owner wants run
a job to them.



The only way to accomplish this within the scope of single partition set-up
would be to allow preemption for the subset of nodes? Or are there other
options? Many thanks in advance.

Cheers,

-Frank Heckes

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210305/8d9cb510/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6142 bytes
Desc: not available
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210305/8d9cb510/attachment.bin>

End of slurm-users Digest, Vol 41, Issue 14
*******************************************