[slurm-dev] Getting account information in the JobComp log
Lennart Karlsson
needed to count project usage on a cluster, as I want to know
which project allocated how many nodes for how long time.
One piece of information is missing, though: Account, given with
a "-U" or "--account" flag at job start. Because a single user
may be a member of more than one project, I need also this
information.
Please, how do I add this information to the log, or get SLURM
to write some other log that contains this information?
Cheers,
-- Lennart Karlsson <Lennart....@nsc.liu.se>
National Supercomputer Centre in Linkoping, Sweden
http://www.nsc.liu.se
Danny Auble
You can add the fields to your slurm.conf...
- JobAcctType
- Specifies which plugin should be used.
- JobAcctFrequency
- Let the plugin know how long between pollings.
- JobAcctLogFile
- Let the plugin the name of the logfile to use.
JobAcctType=jobacct/linux
JobAcctFrequency=5
JobAcctLogFile=/var/slurm/jobacct.log
Mark A. Grondona
> You might want to look at the job accounting plugin. This has the
> information I think you are looking for. You can also look at the data
> with sacct.
What if you want this information without the overhead of the whole
accounting framework?
The "jobcomp/script" plugin will set the ACCOUNT environment variable
accordingly when the job completion script is invoked for each completing
job. (This is as of slurm-1.2.10 or so)
Set
JobCompType=jobcomp/script
JobCompLog=/path/to/script
in your slurm.conf.
The script in question will be invoked once for each completing job,
with the following env vars set by SLURM:
JOBID, UID, START, END, SUBMIT, PROCS, BATCH, NODES, ACCOUNT,
JOBNAME, JOBSTATE, PARTITION, LIMIT (and PATH).
you can log this information, store it in a database, or perform some
up front analysis (though I wouldn't do anything that takes too long)
mark
> --------------060608030504080106050308
> Content-Type: text/html; charset=ISO-8859-1
> Content-Transfer-Encoding: 7bit
>
> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
> <html>
> <head>
> <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
> </head>
> <body bgcolor="#ffffff" text="#000000">
> You might want to look at the job accounting plugin. This has the
> information I think you are looking for. You can also look at the data
> with sacct.<br>
> <br>
> You can add the fields to your slurm.conf...<br>
> <br>
> <dl>
> <span class="commandline"><dt><span class="commandline">JobAcctType</span>
> </dt>
> <dd>Specifies which plugin should be used.
> </dd>
> <dt><span class="commandline">JobAcctFrequency</span>
> </dt>
> <dd>Let the plugin know how long between pollings.
> </dd>
> <dt><span class="commandline">JobAcctLogFile</span>
> </dt>
> <dd>Let the plugin the name of the logfile to use.</dd>
> </span>
> </dl>
> i.e.<br>
> JobAcctType=jobacct/linux<br>
> JobAcctFrequency=5<br>
> JobAcctLogFile=/var/slurm/jobacct.log<br>
> <br>
> <br>
> Lennart Karlsson wrote:
> <blockquote cite="mid:2007071720440...@papput.nsc.liu.se"
> type="cite">
> <pre wrap="">I notice that the JobComp log contains most of the information
> needed to count project usage on a cluster, as I want to know
> which project allocated how many nodes for how long time.
>
> One piece of information is missing, though: Account, given with
> a "-U" or "--account" flag at job start. Because a single user
> may be a member of more than one project, I need also this
> information.
>
> Please, how do I add this information to the log, or get SLURM
> to write some other log that contains this information?
>
> Cheers,
> -- Lennart Karlsson <a class="moz-txt-link-rfc2396E" href="mailto:Lennart.Kar
> ls...@nsc.liu.se"><Lennart....@nsc.liu.se></a>
> National Supercomputer Centre in Linkoping, Sweden
> <a class="moz-txt-link-freetext" href="http://www.nsc.liu.se">http://www.n
> sc.liu.se</a>
>
> </pre>
> </blockquote>
> </body>
> </html>
>
> --------------060608030504080106050308--
>
Lennart Karlsson
> > You might want to look at the job accounting plugin. This has the
> > information I think you are looking for. You can also look at the data
> > with sacct.
>
> What if you want this information without the overhead of the whole
> accounting framework?
>
> The "jobcomp/script" plugin will set the ACCOUNT environment variable
> accordingly when the job completion script is invoked for each completing
> job. (This is as of slurm-1.2.10 or so)
>
> Set
>
> JobCompType=jobcomp/script
> JobCompLog=/path/to/script
>
> in your slurm.conf.
>
> The script in question will be invoked once for each completing job,
> with the following env vars set by SLURM:
>
> JOBID, UID, START, END, SUBMIT, PROCS, BATCH, NODES, ACCOUNT,
> JOBNAME, JOBSTATE, PARTITION, LIMIT (and PATH).
>
> you can log this information, store it in a database, or perform some
> up front analysis (though I wouldn't do anything that takes too long)
>
> mark
Thank you Mark, just what I needed! I include my script, if someone
else needs inspiration.
Best regards,
-- Lennart Karlsson <Lennart....@nsc.liu.se>
National Supercomputer Centre in Linkoping, Sweden
http://www.nsc.liu.se
slurm.conf:JobCompType=jobcomp/script
slurm.conf:JobCompLoc=/root/bin/slurm_jobcomp_logger
#! /usr/bin/perl -w
use strict;
my $version = "2007-07-18";
# Job completion logging for SLURM.
# Lennart....@nsc.liu.se
my $logdir = "/var/log/slurm/accounting";
my $logfile;
my @env_vars = ("JOBSTATE", "JOBID", "UID", "ACCOUNT", "START", "END", "SUBMIT", "NODES", "PROCS", "BATCH", "JOBNAME", "PARTITION", "LIMIT");
my ($sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst) = localtime(time);
my ($date, $time);
$mon++; # From zero-counting to one-counting
$year += 1900; # From internal year numbers to real-world year numbers
mkdir $logdir, 0755 if ! -d $logdir;
chdir $logdir or die "Error: Can not go to directory $logdir.\n";
$time = sprintf "%2.2d:%2.2d:%2.2d", $hour, $min, $sec;
$date = sprintf "%d-%2.2d-%2.2d", $year, $mon, $mday;
$logfile = $date; # Give log file date as name, does not need rotation
if (! exists $ENV {"JOBID"}) {
die "Error: Misssing environment for job completion.\n";
}
open LOGFILE, ">> $logfile" or die "Error: Cannot append to file $logfile.\n";
printf LOGFILE "$date $time";
foreach my $var (@env_vars) {
if (exists $ENV {$var}) {
my $var_lowcase = lc $var;
if ($var_lowcase eq "uid") { # Want usernames, not uids
my $username = getpwuid $ENV {$var};
if ($username =~ /^[a-z]/) {
print LOGFILE " username=", $username;
}
else {
print LOGFILE " username=", $ENV {$var};
}
}
else {
print LOGFILE " $var_lowcase=", $ENV {$var};
}
}
else {
print LOGFILE " -";
}
}
print LOGFILE "\n";
close LOGFILE;