[slurm-dev] slurm-dev Re: Pro/Epi-Log scripts not running

[Joan Arbona]

In my understanding the export command
makes that variable available only to child processes of your
shell. Consequently I doubt this way you can access to variables
exported in a prolog script from any shell that slurm provides.
However, in the documentation I've seen there's a way to do this:

- Create a prolog script and print the command that exports your
variables:
prolog.sh:
/
#!/bin/bash
echo "export VAR1=\"VALUE1\""/
/echo "export VAR2=\"VALUE2\""
...

/- In your slurm.conf file, use the TaskProlog directive
instead of Prolog/:
/Slurm.conf/:
...
TaskProlog=<dir>/prolog.sh
.../

- And execute:
/$srun env/

And you should see VAR1 and VAR2 variables.

Note that this prolog only gets executed in jobsteps (when
executing srun), that means that when you run an sbatch or an
salloc it won't work.

However, the best way to check if your prolog/epilog scripts are
working is to cast something to a file somewhere and then go to
check it manually.

Greetings,
Joan
/
/On 18/06/14 16:13, Brian Baughman wrote:
Hello Joan,

The scripts basically just echo stuff and set some environmental variables. If I set TESTVAL in my user prolog file:

export TESTVAL="this is a test variable"

Then run a test job that does something like:

echo ">>>${TESTVAL}<<<"

I get the output:
<<<
So the variable isn’t set.

Regards,
Brian
--

Joan Francesc Arbona
Ext. 2582
Centre de Tecnologies de la Informació
Universitat de les Illes Balears

http://jfdeu.wordpress.com
http://guifisoller.wordpress.com

[Lyn Gerner]

Hi Brian,

This bites a lot of people, so sorry if I'm misunderstanding your problem, but I give it good odds that this is relevant.   The requirement that bites people is in the prolog_epilog doc; see the paragraph that begins "The task prolog ..." immediately below the second table here: http://slurm.schedmd.com/prolog_epilog.html.  (It's a bit more subtle in the slurm.conf page, alluded to by the statement "Standard output from this program can be used to control the environment variables and output for the user program" in the TaskProlog definition.)

So, as Joan pointed out, you have to put the echo in front of any export command, to actually get the variable into your job's environment.  This is from a working TaskProlog:

echo "export LOCAL_TMPDIR=/lscratch/tdirs/slurm.${SLURM_JOBID}.${SLURM_User}"

Note that in the case of this particular site, many users haven't adopted use of srun, and the desired env vars still are being properly set in their environments.

Best,

Lyn

On Wed, Jun 18, 2014 at 10:19 PM, Joan Arbona <joan....@uib.es> wrote:

In my understanding the export command makes that variable available only to child processes of your shell. Consequently I doubt this way you can access to variables exported in a prolog script from any shell that slurm provides. However, in the documentation I've seen there's a way to do this:

- Create a prolog script and print the command that exports your variables:
prolog.sh:

#!/bin/bash
echo "export VAR1=\"VALUE1\""

echo "export VAR2=\"VALUE2\""
...

- In your slurm.conf file, use the TaskProlog directive instead of Prolog:
Slurm.conf:
...
TaskProlog=<dir>/prolog.sh
...

- And execute:
$srun env

And you should see VAR1 and VAR2 variables.

Note that this prolog only gets executed in jobsteps (when executing srun), that means that when you run an sbatch or an salloc it won't work.

However, the best way to check if your prolog/epilog scripts are working is to cast something to a file somewhere and then go to check it manually.

Greetings,
Joan

On 18/06/14 16:13, Brian Baughman wrote:

Hello Joan,

The scripts basically just echo stuff and set some environmental variables. If I set TESTVAL in my user prolog file:

export TESTVAL="this is a test variable"

Then run a test job that does something like:

echo ">>>${TESTVAL}<<<"

I get the output:

<<<

So the variable isn’t set. 

Regards,
Brian

--
Joan Francesc Arbona
Ext. 2582
Centre de Tecnologies de la Informació
Universitat de les Illes Balears

http://jfdeu.wordpress.com
http://guifisoller.wordpress.com

[Brian Baughman]

Hello Lyn,

I think my issue is more subtle than that. In my TaskProlog script i have:

echo "export QTMPDIR=${QTMPDIR}"

echo "print QTMPDIR=${QTMPDIR}"

In my job script I have:


echo ">>>${QTMPDIR}<<<“


In my TaskEpilog script I have:

echo "print Deleted QTMPDIR"

In the output of my job I have:

QTMPDIR=/tmp/172
>>>/tmp/172<<<


Without the expected “Deleted QTMPDIR.” The initial problem I posted disappeared after I changed the scripts from Task* to Srun* and then back to Task*. I do not know why that worked but suddenly the output was displayed for at least the Prolog scripts. I have confirmed that the Epilog scripts are running they just aren’t sending things via the “print” command to the stdout of the job.


Regards,
Brian

[Lyn Gerner]

Hi Brian,

I can understand why you'd think that TaskEpilog also has the same capabilities as TaskProlog, but I've not seen anything to confirm that in the doc.

Best,

Lyn

[Brian Baughman]

Hello Lyn,

Ah! I see (or rather don’t see) now. Thank you.

Regards,
Brian