MiG interface module

[Benjamin Sedoc]

I have written a python mig interface module that extends the
miglib.py
module to simplify and speedup grid application development. It hides
mRSL configuration and introduces a more direct user-to-resource job
flow. Along with error detection to ensure a quick, more precise
response, it also provides local mode execution, where a local backend
of miglib emulates mig server operations. This makes debugging
considerably faster.

The goal is to have a standard grid developer API for broad use, so I
am
interested in comments, requirements, ideas you may have...

The code is available via:

svn checkout
http://migrid.googlecode.com/svn/trunk/user-projects/miginterface

There is a tutorial on the MiG google code Wiki here
http://code.google.com/p/migrid/wiki/MigInterfaceModule

Benjamin Sedoc

[Jonas Bardino]

On Mar 9, 10:17 am, Benjamin Sedoc <bse...@gmail.com> wrote:
> I have written a python mig interface module that extends the
> miglib.py
> module to simplify and speedup grid application development. It hides
> mRSL configuration and introduces a more direct user-to-resource job
> flow. Along with error detection to ensure a quick, more precise
> response, it also provides local mode execution, where a local backend
> of miglib emulates mig server operations. This makes debugging
> considerably faster.
>
> The goal is to have a standard grid developer API for broad use, so I
> am
> interested in comments, requirements, ideas you may have...
>
> The code is available via:
>

> svn checkouthttp://migrid.googlecode.com/svn/trunk/user-projects/miginterface

>
> There is a tutorial on the MiG google code Wiki herehttp://code.google.com/p/migrid/wiki/MigInterfaceModule
>
> Benjamin Sedoc

Hi Benja

It looks good so far, but I do have some questions and comments.

Why not a list or dictionary for output_files and such list arguments
in create_job()?
They should allow multiple values with optional (src, dst) format.

You should always explicitly use $PYTHON in execute when using PYTHON-
X runtime envs. The ./my_script.py implicitly uses the python
interpreter path from the '#!/path/to/python' header in the script.
That is likely to cause problems with custom/multiple python
installations.

Did you consider merging e.g. job_info() and jobs_info() and just
always require a list as input? It looks like less redundancy and
maintenance without loss of functionality.
Similarly job_finished() may be a bit to specific. You could consider
a more general job_wait(job_id, state, timeout) that waits for the job
to reach the provided state but with a time out. Then users could
still use something like:
try:
finished = job_wait(job_id, FINISHED, 0)
except JobWaitTimeout:
finished = False
to check a state without delay and use the function to wait for jobs
to finish in general.
Negative (or None) as timeout could disable timeout and thus would
also simplify your first job example.

The local mode idea is very cool!

Maybe consider using an OS independent
os.path.join(os.path.expanduser('~'), '.mig', 'miginterface')
path for log and other files currently in what sounds like hard coded /
tmp/ . If you consistently use the tempfile module for getting the tmp
path, this is less important.

typo on the wiki and in example3.py: parameter_sweet_script ->
parameter_sweep_script
...and on a 'personal taste' note I would prefer more telling names
instead of exampleX.py :-)

Technical note to the README file: Is it generated from a t2t file
(the table of contents looks and should be auto-generated)?
Feel free to include it somewhere for easy future updates if so.

README change: ensure anonymity -> allow anonymity

local mode: consider wrapping the job commands in a ulimit'ed job
script like we do for real jobs to prevent job problems with to small
resource request entries when later submitting to real grid.
You can search for ulimit in http://code.google.com/p/migrid/source/browse/trunk/mig/server/genjobscriptsh.py
for details.
Similarly such a job wrapper script would allow setting the MIG_X env
variables that some jobs may rely on.

local mode: you use subprocess and a unix specific 'ps' command to
check process status. Wouldn't it be possible to use only the
multiprocess functions like is_alive() or similar to make it simpler
and less platform dependent? The same applies for explicit kill
command vs terminate().

I have tested the basic examples and they ran without problems both
with local and grid execution - good job!

Cheers, Jonas

[Benjamin Sedoc]

Thanks for the comments, keep them coming. I agree on most of your
points.

Although job_finished() may a little specific, i ended up creating it
to make the code more readable/intuitive to new users.
job_wait() would make a nice addition to the API to cut down on loops
in user code.

Regards,
Benjamin

> You can search for ulimit inhttp://code.google.com/p/migrid/source/browse/trunk/mig/server/genjob...

[Benjamin Sedoc]

Thanks for the comments, keep them coming. I agree on most of your
points.

Although job_finished() may a little specific, i ended up creating it
to make the code more readable/intuitive to new users.
job_wait() would make a nice addition to the API to cut down on loops
in user code.

Regards,
Benjamin

On Mar 9, 10:51 am, Jonas Bardino <jonas.bard...@gmail.com> wrote:

> You can search for ulimit inhttp://code.google.com/p/migrid/source/browse/trunk/mig/server/genjob...