slurm without interactive session

[Mathieu Perrin-Terrin]

Hello,

I would like to use makeflow to send jobs to a slurm batch system.

The system is such that I cannot stay interactively logged in for more than few hours.

In these conditions how would you suggest to send jobs? Is there something like condor-submit_makeflow for slurm?

thanks for your help,

mathieu

[Ben Tovar]

Mathieu,

If you are using makeflow with the -Tslurm option, then submitting makeflow itself as a slurm job may not work for you. The issue is that makeflow needs to generate and submit slurm jobs, and this usually can only be done from the frontend machine you log in to the cluster. Thus, a first step would be to check with your sysadmin if slurm jobs can be submitted from the compute nodes.

When compute nodes cannot submit jobs, then we recommend using makeflow with Work Queue, -Twq. With this, you can submit makeflow as a slurm job. The difference is that you also need to submit as slurm jobs a set of work queue workers. These workers then connect to your running makeflow to execute your workflow. The makefow file may have to be slightly rewritten when switching from -Tslurm to -Twq, as Work Queue does not assume a shared file system.

Here are some notes on how to use makeflow+wq: https://cctools.readthedocs.io/en/stable/makeflow/#using-work-queue

(You would use slurm_submit_workers)

Ben

--
You received this message because you are subscribed to the Google Groups "Cooperative Computing Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cctools-nd+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/189b88ff-4869-4276-aedd-56773bdcbd2fn%40googlegroups.com.