automatically set slurm --job-name = suite ID

[Jonny Williams]

Hi there

I'm using slurm to submit my Cylc suites which show up with the `squeue -u $USER` like this...

   JOBID     USER ACCOUNT           NAME  ST REASON    START_TIME                TIME  TIME_LEFT NODES CPUS

    5289 williams niwa000         medusa   R None      2018-10-11T01:49:50    1:29:19   10:30:41    32 2560

    5305 williams niwa000          GA7.1   R None      2018-10-11T03:01:48      17:21   11:42:39    23 1840

    5308 williams niwa000         nudged   R None      2018-10-11T03:04:37      14:32   11:45:28    23 1840

What I want to do is automatically set the `NAME` to be equal to the suite ID.

I've tried this in the suite.rc...

[[[directives]]]

    --partition = foo

    --job-name = $(basename "$PWD")

... but this gives...

5312 williams niwa000     $(basename  PD QOSMaxJob N/A                       0:00 21-00:00:00     1    1

This is wrong and is limited to 10 characters it seems.

This would be a nice feature to include in me and my colleague's suites.

Cheers

Jonny

[David Sutherland]

Hi Jonny,

I usually use the following formatting option with squeue:
--Format=jobid:10s,partition:12s,name:45s,username:12s,statecompact:5s,timeused:10s,numnodes:8s,nodelist:20s

The by default the name is equal to <suite_id>.<task_id> ..
But as you've discovered the values of the directives (and environment vars) are passed to the job script as strings..

Jinja2 maybe?

David

[Jonny Williams]

thanks a lot david

i'll check this out and maybe even delve into jinja2... :)

cheers

jonny

[David Sutherland]

Hi,

So there's two ways you can do this:

1) Manually
{% set SUITE_NAME = 'My Suite Name' %}

--job-name = {{SUITE_NAME}}

2) Via a Jinja2 filter:
- In a folder called Jinja2Filters next to your suite.rc in your definition create a python script say "job_name.py", with the contents:

#!/usr/bin/env python

# Get suite name

def get_suite_name(SUITE_NAME):
    import os
    from subprocess import Popen, PIPE

    try:
       SUITE_NAME = Popen('echo -n $(basename $PWD)', shell=True, stdout=PIPE).communicate()[0]
    except:
       SUITE_NAME = os.environ['CYLC_SUITE_NAME']

    return SUITE_NAME
 

Then in the suite.rc:

{% SUITE_NAME = 'SomethingElse' | job_name %}

--job-name = {{SUITE_NAME}}


Maybe there's an easier way XD

David

[Hilary Oliver]

Hi Jonny,

For task jobs submitted to Slurm, Cylc automatically sets the job name to <suite-name>.<task-name>.<cycle-point>.  E.g. for the "20181011T00Z" instance of task "model" in suite "nwp", you would see this in the generated task job script:

...

# DIRECTIVES:

#SBATCH --job-name=nwp.model.20181011T00Z

...

If you change this to just the suite name, you won't be able to distinguish different jobs within the same suite (by interrogating slurm).

So it seems to me you just need `squeue` to print the full Job ID instead of a truncated version of it.  You can do presumably do that with the squeue formatting option in David's first reply.  The way I do it is by setting $SQUEUE_FORMAT in my environment, e.g.:

export SQUEUE_FORMAT="%.10i %.20P %.30j %.8u %.2t %.10M %.6D %.20R %q"

 

("%.30j" means print job names up to 30 characters long).

Hilary

p.s. The reason your attempted use of "$(basename "$PWD")" failed is that slurm directives are embedded in bash comments (#SBATCH --job-name=...), so bash does not evaluate them when the job runs.  But even if bash did evaluate them as shell expressions, you would not get the result you want because Cylc executes jobs in a task-specific "work directory", not in the top level suite run directory! 

--

---
You received this message because you are subscribed to the Google Groups "cylc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cylc+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hilary Oliver]

> p.s. The reason your attempted use of "$(basename "$PWD")" failed ...

Actually, there's another reason: Slurm presumably parses its directives from the job script before it executes the script, so setting the job name at run time from within the job script would be too late anyway.

[Matt Shin]

Not sure why you want to do this, because you will end up having the same job name for every tasks in your suite?

Shell syntax does not get evaluated in job directives section. If you really need to do this, you can try using "cylc broadcast" to broadcast this setting to all relevant tasks.

[Jonny Williams]

ok brilliant, thanks guys, i've got exactly what i wanted now by using a format statement as per david's suggestion, thanks david!

@hilary i didn't know about the default name which is actually very helpful indeed and basically solved my issue on its own!

@matt in climate model runs we very often only have one task running at a time and so the perceived problem you mention doesn't arise most of the time.

thanks again

jonny