[slurm-users] remote license

[Davide DelVento]

I am a bit confused by remote licenses.

https://lists.schedmd.com/pipermail/slurm-users/2020-September/006049.html
(which is only 2 years old) claims that they are just a counter, so
like local licenses. Then why call them remote?

Only a few days after, this
https://lists.schedmd.com/pipermail/slurm-users/2020-September/006081.html
appeared to imply (but not clearly stated) that the remote license are
not simply a counter, but then it's not clear how they are different.

The current documentation (and attempts to run the "add resource"
command) says that one must use the license count, which seems to
imply they are just a simple counter (but then what do they need the
server for)?

So what is what?

In my cursory past experience with this, it seemed that it were
possible to query a license server (at least some of them) to get the
actual number of available licenses and schedule (or let jobs pending)
accordingly. Which would be very helpful for the not-too-uncommon
situation in which the same license server provides licenses for both
the HPC cluster and other non-slurm-controlled resources, such a
user's workstations. Was that impression wrong, or perhaps somebody
scripted it in some way? If the latter, does anybody know if those
scripts are publicly available anywhere?

Thanks

[Brian Andrus]

So if you follow the links to: https://slurm.schedmd.com/licenses.html
you should see the difference.

Local licenses are just a counter that is setup in slurm.conf
Remote liceneses are a counter in a database (the database is "remote"),
so you can change/update it dynamically. So, you could change their
allocation with a sacctmgr command. It is especially useful when you are
managing multiple clusters that share licenses. You can allocate that a
certain number are allowed by each cluster and change that if needed.

If you got creative, you could keep the license count that is in the
database updated to match the number free from flexlm to stop license
starvation due to users outside slurm using them up so they really
aren't available to slurm.

Brian Andrus

[Davide DelVento]

So if I understand correctly, this "remote database" is something that
is neither part of slurm itself, nor part of the license server per
se, correct?

Regarding the "if you got creative", has anybody on this list done
that already? I can't believe I'm the first one wanting this feature!
Matching the number in that database with the actual number the
license server knows would be extremely helpful! We use various
license servers (for various licensed software), so each one of them
would be useful. I can probably script/develop one of these myself,
but I am not sure I've got the time...

Thanks!

[Brian Andrus]

Davide,

You have it pretty correct. While the database itself is not part of the
slurm suite, slurmdbd (which would access the database) is.

As far as writing something that keeps things updated, I'm sure many
have done this. However, it would be unique to your installation. The
specific number of licenses, naming them, what license server is being
used, etc.
All of that could easily be a few lines in a script that you have in a
cron job or other trigger (eg prolog/epilog). You would just:

1) Read/parse current licenses/use (eg: if you are using flexlm, lmutil
lmstat output)
2) Update the database (sacctmgr command)

As you can see, that 1st step would be highly dependent on you and your
environment. The 2nd step would be dependent on what things you are
tracking within that.

Brian Andrus

[Davide DelVento]

Thanks Brian.

I am still perplexed. What is a database to install, administer,
patch, update, could break, be down, etc buying us? I see limited use
cases, e.g. a license server which does not provide the license
count/use in a parsable way, and that someone wants to use with
multiple SLURM installations (if it's on a single one, the local
license is perfect). Wouldn't it much, much easier for everybody if
one could specify a script (the bullet 1. you mentioned) inside SLURM,
and use the license server ITSELF as the authoritative source of
license count? Sure, it won't be perfect, e.g. race conditions in
license acquisition can still cause failures, but the database won't
be fixing that....
I must be missing something....

Alternatively, can one update the license count of local license with
a scontrol command, rather than changing the slurm.conf and
reconfigure? That could make what I say possible....

Thanks

[Brian Andrus]

Davide,

I'll not engage on this. If you want a feature, pay SchedMD for support
and they will prioritize it and work on it.  You are already using a
very impressive bit of software for free.

As far as local license updates, yes, you can do the local license and
reconfigure regularly. Feel free to do that. It is not something that
scales well, but it looks like you have a rather beginner cluster that
would never be impacted by such choices.

Brian Andrus

[Davide DelVento]

Hi Brian,

From your response, I speculate that my wording sounded harsh or
unrespectful. That was not my intention and therefore I sincerely
apologize for it.

In fact my perplexity is certainly due to my ignorance (as it must be
very clear by the number and "quality" of queries that I am posting on
this mailing list). It seemed to me that what is currently available
is a special edge case, whereas the simpler one is not covered, so I
was (perhaps still am) convinced that I must have misunderstood how
things work. Perhaps my "am I missing something?" sounded rhetorical
rather than sincere which was the spirit I wrote it with. Sorry about
that.

I find it a nice coincidence that you suggested paying SchedMD for
this, because after the clarifications which I am trying to get in
this thread, I thought to ask my management to do exactly that!!!

For the local license, are you suggesting to programmatically change
slurm.conf and reconfigure e.g. in a cron?

Thanks a lot for your help and have a great weekend

Davide

[Brian Andrus]

Davide,

No worries. I like to see folks engage SchedMD, especially if they are
seeing value in all the work that has gone into creating the software
that runs the majority of fastest clusters on the planet.

To your question: yes, you can update slurm.conf and do 'scontrol
reconfigure' to change the number of local licenses available. You could
do that fairly regularly. I am not sure what it would do if you lowered
the number to something smaller than the total in use. It should allow
jobs to continue and drain down.

It is a much simpler way to achieve your goal, however, it affects the
system more to run 'scontrol reconfigure' than it does to use the remote
license method.
For a large scale system, that impact is very noticeable. For a small
cluster, it is negligible. Thus, it may well be the best method for you
and your environment. If you find yourself working on a large cluster
sometime in your career, I would not recommend using it there.

Brian Andrus