[slurm-users] Is there any public scientific-workflow example that can be run through Slurm?

[Alper Alimoglu]

In slurm we can build pipelines using [slurm dependencies][1], which allows us to run workflows.

In my work, I have stuck in a point regarding finding a workflow that I can run using Slurm.

As an example, I have to use a workflow benchmark like in here https://pegasus.isi.edu/workflow_gallery/ , but all of them are implemented inside Pegasus.

I was wondering is there any public scientific-workflow examples that can be run through Slurm?

Any suggestion is highly appreciated.

  [1]: https://www.hpc.caltech.edu/documentation/faq/dependencies-and-pipelines

-Alper

[Ole Holm Nielsen]

Hi Alper,

My colleagues at Technical University of Denmark are heavy users of
workflows through Slurm on our cluster, and this accounts for most of
our usage. They have developed an Open Source workflow system:

> MyQueue is a frontend for SLURM/PBS/LSF that makes handling of tasks easy. It has a command-line interface called mq with a number of Sub-commands and a Python interface for managing Workflows. Simple to set up: no system administrator or database required.

See https://myqueue.readthedocs.io/en/latest/

I'm personally not involved in MyQueue, but you might take a look to see
if it's useful in your environment.

Best regards,
Ole

[Bernstein, Noam CIV USN NRL (6393) Washington DC (USA)]

I'm the lead developer of another workflow system, wfl (github.com/libAtoms/workflow), which works with slurm using an abstraction layer we also developed, ExPyRe (github.com/libAtoms/Expyre), but in writing a recent paper about it we looked at other systems, and ones that we know of that use queuing systems include:

ASR atomic simulation recipes (which uses MyQueue)

Atomate/Fireworks

PyIron/Pysqua

AiiDA

icolos (https://github.com/MolecularAI/Icolos)

qmpy (part of OQMD)

Note that I'm not promising that they currently support slurm, but it's a list to start your research from.

Noam

[Loris Bennett]

"Bernstein, Noam CIV USN NRL (6393) Washington DC (USA)"

On our system we have a number of people, mainly from the life sciences,
who use Nextflow

https://www.nextflow.io/

It is slightly problematic from our point of view, as it does not yet
support job arrays. However, there is development activity going on to
address this:

https://github.com/nextflow-io/nextflow/issues/1477

Cheers,

Loris

--
Dr. Loris Bennett (Herr/Mr)
ZEDAT, Freie Universität Berlin