[slurm-users] Slurm webhooks

[Davide DelVento via slurm-users]

Happy Monday everybody,

I've gotten a request to have Slurm notify users for the typical email things (job started, completed, failed, etc) with a REST API instead of email. This would allow notifications in MS Teams, Slack, or log stuff in some internal websites and things like that.

As far as I can tell, Slurm does not support that, for example there was somebody who was looking for that on Galaxy and did not find a solution: https://help.galaxyproject.org/t/web-hook-post-to-external-url-when-job-begins-and-completes-running/4017

Is that indeed the case, as searching the web indicates?

If Slurm does not support this, is there a workaround? For example, I'm thinking of installing a local SMTP server, or an alternative/dummy mailx program which instead of relaying emails as requested would post an encrypted web url thing using the information from the email. I am sure I could actually write such a software myself, but I don't have enough time to dedicate to the design, maintenance and debugging of such, so I am looking for something decent already in existence. A cursory web search did not find anything suitable, but perhaps I did not look in the appropriate places, because my gut feeling is that somebody must have already had such an itch to scratch!

Any other ideas about alternative ways to accomplish this?

Thanks

[Kilian Cavalotti via slurm-users]

Hi Davide,

Two things you may want to look into:

1. some (most?) web services have "email-to-service" mechanisms of
some sort: for instance, you can send an email to a Slack channel,
which will create a message from it:
https://slack.com/help/articles/206819278-Send-emails-to-Slack

2. Slurm has a trigger mechanism, which can act upon a variety of events:
https://slurm.schedmd.com/strigger.html
"Triggers include events such as a node failing, a job reaching its
time limit or a job terminating."
Registering a trigger for each job will probably be a bit on the
heavy-side on busy environments, but at least, Slurm can do it.

Cheers,
--
Kilian

> --
> slurm-users mailing list -- slurm...@lists.schedmd.com
> To unsubscribe send an email to slurm-us...@lists.schedmd.com



--
Kilian

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com

[Bjørn-Helge Mevik via slurm-users]

Davide DelVento via slurm-users <slurm...@lists.schedmd.com> writes:

> I've gotten a request to have Slurm notify users for the typical email
> things (job started, completed, failed, etc) with a REST API instead of
> email. This would allow notifications in MS Teams, Slack, or log stuff in
> some internal websites and things like that.

We are just in the process of implementing this on one of our clusters.
(The ReST API is already there, what we are implementing is Slurm using
it instead of sending emails.) For us, it is quite easy: Simply write a
bash script that uses SLURM_* environment variables to get information
about the message and user, and then uses curl to issue the required
ReST API calls. Then we set the MailProg parameter in slurm.conf to
point to this script.

Here is our current test version for this script (so far, it simply logs
what it would do instead of actually contacting the ReST API, together
with some debug output):

#!/bin/bash

exec &>> /tmp/mail.log

echo $(date +%FT%T): Starting
SUBJECT="$2"
echo Args:
while [[ $# > 0 ]]; do
echo "$1"
shift
done
echo
echo Envs:
env | grep SLURM | sort
echo

case $SLURM_JOB_MAIL_TYPE in
Began) ACTION="started";;
Ended) if [[ $SLURM_JOB_STATE == COMPLETED ]]; then
ACTION="completed"
elif [[ $SLURM_JOB_STATE == CANCELLED ]]; then
ACTION="been cancelled"
else
ACTION="ended"
fi;;
Requeued) ACTION="been requeued";;
*) ACTION="unknwon action";;
esac

BODY="Your job $SLURM_JOB_ID ($SLURM_JOB_NAME) on $SLURM_CLUSTER_NAME has $ACTION.
"

echo Recipient: $SLURM_JOB_USER
echo Subject: $SUBJECT
echo Body:
echo $BODY
echo
echo Done.

--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo

[Davide DelVento via slurm-users]

Thank you all. I had thought of writing my own, but I suspected it would be too large of a time sink. Your nudges (and example script) have convinced me otherwise, and in fact this is what I will do!

Thanks again!