[slurm-users] Submitting jobs from machines outside the cluster

[Steven Swanson]

Can I submit jobs with a computer/docker container that is not part of the slurm cluster?

I'm trying to set up slurm as the backend for a system with Jupyter Notebook-based front end.  

The jupyter notebooks are running in containers managed by Jupyter Hub, which is a mostly turnkey system for providing docker containers that users can access via jupyter.

I would like the jupyter containers to be able to submit jobs to slurm, but making them part of the cluster doesn't seem to make sense because:

1.  They are dynamically created and don't have known hostnames.

2.  They aren't supposed to run jobs.

Is there a way to do this?  I tried just running slurmd in the jupyter containers, but it complained about not being able to figure out its name (I think because the container's hostname is not listed in slurm.conf).

My fall back solution is to use ssh to connect to the slurm head node and run jobs there, but that seems kludgy.

-steve

[mohammed shambakey]

Hi

May be slurm rest api be useful (https://slurm.schedmd.com/rest.html)? But I think you will need to generate a token to be able to communicate with the cluster.

Regards

[William Brown]

The machine that runs the CLI isn't usually a cluster node or run slurmd.

The control node has to accept the user is who they claim and AFAIK that is the job of munge.  And your onboard and external firewalls must allow the requisite ports.

We used Galaxy (see galaxy.eu if unfamiliar) and could submit jobs to various job runners including Slurm.  The galaxy node definitely didn't run any slurm daemons.

I think you do need a common authentication system between the submitting node and the cluster, but that may just be what I'm used to.

William Brown

[Ward Poelmans]

Hi Steven,

On 27/08/2023 08:17, Steven Swanson wrote:
>
> I'm trying to set up slurm as the backend for a system with Jupyter Notebook-based front end.
>
> The jupyter notebooks are running in containers managed by Jupyter Hub, which is a mostly turnkey system for providing docker containers that users can access via jupyter.

We use a similar setup. It's documented on https://github.com/vub-hpc/notebook-platform

Ward

[Loris Bennett]

I'm not sure what your use-case is, so the following might not be
relevant.

Our Jupyter Notebook users don't want to submit jobs from their
notebooks. Instead they want to run their notebooks interactively, but
make use of cluster resources. To do this, they submit a job on the
login node which sets up an SSH tunnel between the compute node and
their local machine and then starts jupyter-lab. The users can then can
point the browsers on their local machines to a local port and be
connected to the session on the compute node.

Cheers,

Loris

--
Dr. Loris Bennett (Herr/Mr)
ZEDAT, Freie Universität Berlin

[Hagdorn, Magnus Karl Moritz]

Hi Steve,

we are running a jupyterhub that uses our cluster as a backend. You need to be able to submit jobs from the hub node, ie you need to configure slurm and have the slurm binaries. You don't need to run slurmd. We are using the batchspawner to launch the jobs on the cluster. 

Regards

magnus

-- 

Magnus Hagdorn

Charité – Universitätsmedizin Berlin

Geschäftsbereich IT | Scientific Computing

 

Campus Charité Virchow Klinikum

Forum 4 | Ebene 02 | Raum 2.020

Augustenburger Platz 1

13353 Berlin

 

magnus....@charite.de

https://www.charite.de

HPC Helpdesk: sc-hpc-...@charite.de