Load-Leveler

[Jost]

Hi everybody,

MiG supports PBS (and its descendants that speak the same language) by
scripts to wrap job submission and job control. These scripts are in
folder mig/resource/pbs-scripts.

Are there comparable scripts for LoadLeveler around somewhere? They
are at least not checked in.

A related question is, LoadLeveler supports job priorities. What
options do you see in order to use these from MiG?
I believe MiG can only use job priorities through calling the submit
script with one of several priority parameters, fixed per execution
node of the resource (by specifying them with the submit command).
Ideas are welcome.

Cheers
Jost

[Jonas Bardino]

Hi Jost / list

LoadLeveler support was planned a long time ago, but it was never
implemented due to lack of actual LL resources.
You are of course welcome to implement them based on the pbs-scripts
you found - feel free to ask about the passing of MIG_X env vars and
so on.

The job priority support could be made dynamic based on MiG scheduling
order, but I doubt it is worth the effort. Most resource admins will
likely more or less dictate the MiG priority to use anyway.
Some setups might provide, say, two priority levels to choose between
for short and long jobs and that can easily be supported by having X
resource exe nodes using matching priorities and cputimes.

Cheers, Jonas

[Jost]

OK, so I will put together the basic scripts, not a hard problem.
And yes, the priority-per-resource-exe is what is possible and what I
tried to describe above.
However,
a) I do not like the limit on CPUTime: it means that the slots
reserved for short jobs will never take a longer job, even if they are
idle. But the other way round, "long" resources refusing short jobs,
is actually worse.
b) Do we have priorities for resource exe nodes? Then please explain.
Even nicer would be support for passing a user-chosen priority from
MiG to the queueing system to loadleveler when submitting the job.
This would cater for the fact that some people use MiG, others use
llsubmit directly. But it seems the priorities are always set to
maximum by all users on this particular system anyway :-)

So I go the straightforward way for now, and will put together the LL
scripts.

/Jost

[Jonas Bardino]

My point was actually that only short jobs should be allowed with high
priority - that is a quite typical use case for quick testing. In that
case the short exe should never take long jobs so a short cputime is
necessary.
The long exe may still end up getting a short job and thus give it low
priority, but the user can specifically target the short exe in the
job description (RESOURCE keyword), so that should not be a big
problem.

If you add different exes with the same specs but different back end
priorities it is completely left to the user to select the appropriate
target exe, but that will likely result in inefficient use as I
haven't yet heard about users not wanting their jobs executed
first :-)

No sleep jobs are queued in the LRMS, so there is no queue penalty on
the resource when no jobs fit.

We do not have exe node priorities as such in MiG, it all depends on
the job request order (and the MiG scheduler filling out the execution
slots). Users can target a specific resource and exe as mentioned
above, though.

For LRMS specific dynamic priority setting we could:
a) add a PRIORITY keyword in job mRSL and use it for optionally
ranking jobs in the central queue (if the scheduler implements it) and
for such LRMS priorities.
b) modify the LRMS submit script to read an optional priority from the
job script. PBS-style that would lead to the user adding e.g. a
"#PRIORITY=X" in the EXECUTE field and the submit script falling back
to a default priority if no such line is found.

Option (a) is obviously the cleaner solution, whereas (b) would be a
quick hack: I think both solutions will be okay considering the
expected small intended audience.

With my previous remark about sane user priorities I would suggest
that the MiG scheduler should not weigh such priorities much and maybe
not even care about them when comparing jobs from different users.

Cheers, Jonas