[slurm-users] SLURM reservations with MAGNETIC flag

[Bas van der Vlies]

We have installed slurm 20.02.5 and I am trying to use this new reservation flag
MAGNETIC:
* https://slurm.schedmd.com/reservations.html

From this page I understand that the job will land in the reservation even if we
did not specify the
reservation name. I tested it on our cluster setup but it does not seems to
work as expected

I create a reservation for 1 node for user bas with flag magnetic. I submit a
job and to my surprise the job is scheduled on a free node and not on the node
in the reservation. It is only scheduled in the reservation if all nodes are
occupied. Is this the default behaviour or did a miss a setting?

I have set all available nodes offline except the reservation node and then I
see this:
```
bas@batch2:~/src$ srun -N1 --pty /bin/bash
srun: Required node not available (down, drained or reserved)
srun: job 1713 queued and waiting for resources
srun: job 1713 has been allocated resources
```

From this I see that the "magentic" reservation is considered as last.


regards

--
--
Bas van der Vlies
| Operations, Support & Development | SURF | Science Park 140 | 1098 XG
Amsterdam
| T +31 (0) 20 800 1300 | bas.van...@surf.nl | www.surf.nl

[Bas van der Vlies]

Are people using the MAGNETIC reservation flag? My question would be how? because to my it would be more useful if the reservation is tried first and then the free nodes.
That is what I expected with the MAGNETIC flag.


Bas van der Vlies
| Operations, Support & Development | SURFsara | Science Park 140 | 1098 XG Amsterdam

| T +31 (0) 20 800 1300 | bas.van...@surf.nl | www.surf.nl |

[Baer, Troy]

I've been looking at it for classroom type reservations, but I ran into a bug where jobs that weren't eligible to access the reservation were being attracted to it anyway. That's supposed to be fixed in 20.02.6. See https://bugs.schedmd.com/show_bug.cgi?id=9593 for details.

--Troy

On 9/25/20, 10:49 AM, "slurm-users on behalf of Bas van der Vlies" <slurm-use...@lists.schedmd.com on behalf of bas.van...@surf.nl> wrote:

Are people using the MAGNETIC reservation flag? My question would be how? because to my it would be more useful if the reservation is tried first and then the free nodes.
That is what I expected with the MAGNETIC flag.


Bas van der Vlies
| Operations, Support & Development | SURFsara | Science Park 140 | 1098 XG Amsterdam

| T +31 (0) 20 800 1300 | bas.van...@surf.nl | https://urldefense.com/v3/__http://www.surf.nl__;!!KGKeukY!mRT6FSlug4OBODiqDnysihd41efy6vwp4wT9pmdFkOUqPDR0jy3l6d-gzG2P$ |

> On 24 Sep 2020, at 12:16, Bas van der Vlies <bas.van...@surf.nl> wrote:
>
> We have installed slurm 20.02.5 and I am trying to use this new reservation flag
> MAGNETIC:

> * https://urldefense.com/v3/__https://slurm.schedmd.com/reservations.html__;!!KGKeukY!mRT6FSlug4OBODiqDnysihd41efy6vwp4wT9pmdFkOUqPDR0jy3l6VfveQgB$

>
> From this page I understand that the job will land in the reservation even if we
> did not specify the
> reservation name. I tested it on our cluster setup but it does not seems to
> work as expected
>
> I create a reservation for 1 node for user bas with flag magnetic. I submit a
> job and to my surprise the job is scheduled on a free node and not on the node
> in the reservation. It is only scheduled in the reservation if all nodes are
> occupied. Is this the default behaviour or did a miss a setting?
>
> I have set all available nodes offline except the reservation node and then I
> see this:
> ```
> bas@batch2:~/src$ srun -N1 --pty /bin/bash
> srun: Required node not available (down, drained or reserved)
> srun: job 1713 queued and waiting for resources
> srun: job 1713 has been allocated resources
> ```
>
> From this I see that the "magentic" reservation is considered as last.
>
>
> regards
>
> --
> --
> Bas van der Vlies
> | Operations, Support & Development | SURF | Science Park 140 | 1098 XG
> Amsterdam

> | T +31 (0) 20 800 1300 | bas.van...@surf.nl | https://urldefense.com/v3/__http://www.surf.nl__;!!KGKeukY!mRT6FSlug4OBODiqDnysihd41efy6vwp4wT9pmdFkOUqPDR0jy3l6d-gzG2P$

[Bas van der Vlies]

Thanks Troy,

That is our intention also for course/training purposes if there are course days.

regards

--
Bas van der Vlies

| Operations, Support & Development | SURFsara | Science Park 140 | 1098 XG Amsterdam

| T +31 (0) 20 800 1300 | bas.van...@surf.nl | www.surf.nl |

[Bas van der Vlies]

Still have this question. Sometime we have free nodes and users that are
allowed to run in the MAGNETIC reservation are first scheduled on the
free nodes instead of reservation nodes. Dit I forgot an option or is
this the expected behavior?

| HPCV Supercomputing | Internal Services | SURF |
https://userinfo.surfsara.nl |
| Science Park 140 | 1098 XG Amsterdam | Phone: +31208001300 |
| bas.van...@surf.nl

[Bas van der Vlies]

The problem has been solved. We tested it with ’srun’ and that did not work the same is for ’salloc’ To make it work with these commands apply one of the solutions:
1. Add ‘defer’ to ScheduleParameters
2. srun --begin=now+1


sbatch works out of the box. Thanks to schedmd (Ben) for the solution, see:
* https://bugs.schedmd.com/show_bug.cgi?id=12350

Regards