[slurm-users] Prolog and job_submit

[Davide DelVento]

My problem: grant licensed software availability to my users only if
they request it on slurm; for now with local licenses.

I wrote a job_submit lua script which checks job_desc.licenses and if
it contains the appropriate strings it sets an appropriate
SOMETHING_LICENSE_REQ environmental variable.

This part works, I can see the environmental variable correctly set in
the jobs that require the license.

Now this licensed software is a bit tricky to manage, so the way that
I thought to use it is simply to make its binary disappear from the
nodes when not requested, with a prolog and epilog scripts which copy
it from a location in the shared filesystem accessible only by root.
Simply a copy during the prolog and a delete during the epilog.

Something like this:

if [[ $SOMETHING_LICENSE_REQ == 1 ]]; then
# copy the binary
fi

After banging my head to make the prolog run (executable bit and full
path required, and not said so in the documentation and error logs
being cryptic about it) I am finally able to see it running.... only
to find out that the SOMETHING_LICENSE_REQ environmental variable is
not set, despite the documentation at
https://slurm.schedmd.com/prolog_epilog.html stating

> The task prolog is executed with the same environment as the user tasks to be initiated.

Now, I'd be very happy to do this copy from the job_submit, but that
is run on the head node (I checked) and so I can't do that. It would
seem strange that the job_submit is run after the prolog, since the
latter runs on the compute node (I checked that too).

Moreover, I also verified with additional environmental variables
which I set at submit time are available for the user job correctly,
but not in the prolog.

So either I misinterpreted that "same environment as the user tasks"
or there is something else that I am doing wrong.

Does anybody have any insight?

[Sarlo, Jeffrey S]

Not sure if this will help.  It has which user will execute the scripts

https://slurm.schedmd.com/prolog_epilog.html

Maybe the variable isn't set for the user executing the prolog/epilog/taskprolog

Jeff

---

From: slurm-users <slurm-use...@lists.schedmd.com> on behalf of Davide DelVento <davide....@gmail.com>
Sent: Saturday, October 29, 2022 9:37 AM
To: slurm...@schedmd.com <slurm...@schedmd.com>
Subject: [slurm-users] Prolog and job_submit

 

My problem: grant licensed software availability to my users only if
they request it on slurm; for now with local licenses.

I wrote a job_submit lua script which checks job_desc.licenses and if
it contains the appropriate strings it sets an appropriate
SOMETHING_LICENSE_REQ environmental variable.

This part works, I can see the environmental variable correctly set in
the jobs that require the license.

Now this licensed software is a bit tricky to manage, so the way that
I thought to use it is simply to make its binary disappear from the
nodes when not requested, with a prolog and epilog scripts which copy
it from a location in the shared filesystem accessible only by root.
Simply a copy during the prolog and a delete during the epilog.

Something like this:

if [[ $SOMETHING_LICENSE_REQ == 1 ]]; then
    # copy the binary
fi

After banging my head to make the prolog run (executable bit and full
path required, and not said so in the documentation and error logs
being cryptic about it) I am finally able to see it running.... only
to find out that the SOMETHING_LICENSE_REQ environmental variable is
not set, despite the documentation at

https://urldefense.com/v3/__https://slurm.schedmd.com/prolog_epilog.html__;!!LkSTlj0I!GeYFMK5LTz38C-Y5efzywysYqnEPMq4Rq9Nj77WE3gOEYPJ-rRbnBR6alpC8cPi6XJbsBhkAgFOFbEIzzx4ItgH_Ssba$   stating

[Davide DelVento]

Thanks Jeff.
That's exactly the documentation that I looked and quoted, and yes, I
know that the user running the prolog is a different one (root) from
the one which will be running the job (regular user submitting the
job).
I speculated that the sentence I quoted (again: prolog is executed
with the same environment as the user tasks to be initiated) meant
that root had the user environment, besides PATH for security reasons
as written in that page. Apparently that is not the case, and as such
I am stuck and can't solve my problem: if even I find an appropriate
prolog which runs as a regular user, then it won't have the necessary
permissions to unhide the licensed binary. One step forward and two
steps backward, so frustrating!
Thanks again and have a nice weekend

[Chris Samuel]

On 29/10/22 7:37 am, Davide DelVento wrote:

> So either I misinterpreted that "same environment as the user tasks"
> or there is something else that I am doing wrong.

Slurm has a number of different prologs that can run which can cause
confusion, and I suspect that's what's happening here.

The "Prolog" in your configuration runs as root, but its the
"TaskProlog" that runs as the user and so has access to the jobs
environment (including the environment variable you are setting).

Unfortunately it looks like the license request information doesn't get
propagated into any prologs from what I see from a scan of the
documentation. :-(

Best of luck,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA

[Chris Samuel]

On 30/10/22 10:23 am, Chris Samuel wrote:

> Unfortunately it looks like the license request information doesn't get
> propagated into any prologs from what I see from a scan of the

> documentation. 🙁

This _may_ be fixed in the next major Slurm release (February) if I'm
reading this right:

https://github.com/SchedMD/slurm/commit/3c6c4c08d8deb89aa2c992a65964f53663097d26

All the best,

[Davide DelVento]

Hi Chris,

> Unfortunately it looks like the license request information doesn't get
> propagated into any prologs from what I see from a scan of the
> documentation. :-(

Thanks. If I am reading you right, I did notice the same thing and in
fact that's why I wrote that job_submit lua script which gets the
license information and sets an environmental variable, in the hope
that such a variable would be inherited by the prolog script.

But if I understand correctly your Prolog vs TaskProlog distinction,
the latter would have the environmental variable and run as user,
whereas the former runs as root and doesn't get the environment, not
even from the job_submit script. The problem with a TaskProlog
approach is that what I want to do (making a non-accessible file
available) would work best as root. As a workaround is that I could
make that just obscure but still user-possible. Not ideal, but better
than nothing as it is now.

Alternatively, I could use another way to let the job_submit lua
script communicate with the Prolog, not sure exactly what (temp
directory on the shared filesystem, writeable only by root??)

Thanks for pointing to that commit. I bit too down the road but good to know.

Cheers,
Davide

[Chris Samuel]

On 30/10/22 12:27 pm, Davide DelVento wrote:

> But if I understand correctly your Prolog vs TaskProlog distinction,
> the latter would have the environmental variable and run as user,
> whereas the former runs as root and doesn't get the environment,

That's correct. My personal view is that injecting arbitrary input from
a user (such as these environment variables) would make life hazardous
from a security point of view for a root privileged process such as a
prolog.

> not even from the job_submit script.

That is correct, all the job_submit will do is inject the environment
variable into the jobs environment, just as if a user had done so.

> The problem with a TaskProlog
> approach is that what I want to do (making a non-accessible file
> available) would work best as root. As a workaround is that I could
> make that just obscure but still user-possible. Not ideal, but better
> than nothing as it is now.
>
> Alternatively, I could use another way to let the job_submit lua
> script communicate with the Prolog, not sure exactly what (temp
> directory on the shared filesystem, writeable only by root??)

My only other thought is that you might be able to use node features &
job constraints to communicate this without the user realising.

For instance you could declare the nodes where the software is installed
to have "Feature=mysoftware" and then your job submit could spot users
requesting the license and add the constraint "mysoftware" to their job.
The (root privileged) Prolog can see that via the SLURM_JOB_CONSTRAINTS
environment variable and so could react to it.

Then when 23.02 comes out you could use the new SLURM_JOB_LICENSES
environment variable in addition and retire the old way once jobs using
the old method have completed.

> Thanks for pointing to that commit. I bit too down the road but good to know.

No worries, best of luck!

All the best,

[Davide DelVento]

Thanks for helping me find workarounds.

> My only other thought is that you might be able to use node features &
> job constraints to communicate this without the user realising.

I am not sure I understand this approach.

> For instance you could declare the nodes where the software is installed
> to have "Feature=mysoftware" and then your job submit could spot users
> requesting the license and add the constraint "mysoftware" to their job.
> The (root privileged) Prolog can see that via the SLURM_JOB_CONSTRAINTS
> environment variable and so could react to it.

Are you saying that if the job_submit.lua can't directly add an
environmental variable that the prolog can see, but can add the
constraint which will become an environmental variable that the prolog
can see?
Would that work if that feature is available in all nodes?

[Christopher Samuel]

On 10/31/22 5:46 am, Davide DelVento wrote:

> Thanks for helping me find workarounds.

No worries!

>> My only other thought is that you might be able to use node features &
>> job constraints to communicate this without the user realising.
>
> I am not sure I understand this approach.

I was just trying to think of things that could get into the Prolog that
runs as root that you could use as a signal to it. Job constraints
seemed the most reasonable choice.

> Are you saying that if the job_submit.lua can't directly add an
> environmental variable that the prolog can see, but can add the
> constraint which will become an environmental variable that the prolog
> can see?

That's correct - the difference being that Slurm, not the user, is in
control of its presence and the possible values it can have (as it's
constrained by what you've chosen for the name of the node feature).

> Would that work if that feature is available in all nodes?

Yes, that should work just fine I believe.