XgridDRMAA for PyXG

[Barry]

Has anyone thought about adopting the XgridDRMAA work done this summer
by Ed Baskerville in place of the command line wrapping it currently
has? If not, I may have a crack at it but didn't want to duplicate
anyone's efforts...

[Brian Granger]

Barry,

I have been watching Ed's DRMAA work curiously as well as reading up
on DRMAA in general. Currently it is not clear to me that it would be
an improvement to have pyxg implemented on top of XgridDRMAA rather
than the command line. The main reason is that I am not sure what is
going on with XgridDRMAA "underneath the hood." The main issue is
whether or not, XGridDRMAA requires the Cocoa runloop to be running.
If Ed is using XgridFoundation then it most definitely does need to be
running and it won't be useful for PyXG.

I think you will remember that one of my original design requirements
was that PyXG could be used from an interactive Python prompt - not
just in static python scripts (which can run the NSRunloop just fine).

I do however think it would be a fantastic idea to simply have a
Python XGridDRMAA binding. In fact, using ctypes, I would bet you
could immediately call Ed's XGridDRMAA framework from python. The
Python API might need some minor wrapping to make it more pythonic. A
good starting point would be the python DRMAA bindings that have been
developed for Sun Grid Engine:

http://blogs.sun.com/templedf/entry/python_binding_for_drmaa

For now though, I am sort of waiting to see what happens to Xgrid in
10.5. To date, I still don't know of a better way of implementing
pyxg other than wrapping the CLI. In the long run though, I would
love to see a really nice Twisted interface to Xgrid. While you
couldn't use it interactively, Twisted is such a nice way of
abstracting the asynchronous stuff.

Let me know what you think

Brian

--
Brian E. Granger, Ph.D.
Research Scientist
Tech-X Corporation
phone: 720-974-1850
bgra...@txcorp.com
elli...@gmail.com

[Barry Wark]

Brian,

I may have gotten my head all mixed up on how Ed's code worked. It
looked to me like he was running an NSRunLoop in a separate thread and
using mutex locks to block for synchronous operations. This is an
approach that I've played around with for my matlabxg package and has
some success with, though Ed's code is far cleaner than mine.

I agree that one of the (maybe the) most important feature of pyxg was
that it could be used from the interpreter without a runloop.

I think your suggestion is a good one. I will look into a python
wrapper for Ed's code, both by directly wrapping the DRMAA interface
(or what's exposed in Ed's framework) and in adapting my xgf package
to use his DRMAA instead of Twisted under the hood. We'll just see
what comes out better.

I think moving my xgf to using DRMAA would be nice because it would
give a better separation between Twisted style code and synchronous
style code. A Twisted interface for Xgrid would be really nice,
especially if it was well done, but crufting on synchronous wrappers
for it (like what i'm doing now) just feels wrong.

I will send something by you on the Twisted end when I have it cleaned
up a bit. Along those lines, I haven't been keeping great track of the
ipython1 development. What is the status of using it on an Xgrid?

Thanks,
Barry

[Brian Granger]

Barry,

See me replies inline.

On 10/31/06, Barry Wark <barr...@gmail.com> wrote:
>
> Brian,
>
> I may have gotten my head all mixed up on how Ed's code worked. It
> looked to me like he was running an NSRunLoop in a separate thread and
> using mutex locks to block for synchronous operations. This is an
> approach that I've played around with for my matlabxg package and has
> some success with, though Ed's code is far cleaner than mine.

Hmm, that is an interesting idea. Can the NSRunloop be run in a
secondary thread or does it have to be run in the main thread? Most
GUI toolkits (like QT and WX) need to be in the main thread. I would
like to know more about how he does this though.

> I agree that one of the (maybe the) most important feature of pyxg was
> that it could be used from the interpreter without a runloop.
>
> I think your suggestion is a good one. I will look into a python
> wrapper for Ed's code, both by directly wrapping the DRMAA interface
> (or what's exposed in Ed's framework) and in adapting my xgf package
> to use his DRMAA instead of Twisted under the hood. We'll just see
> what comes out better.
>
> I think moving my xgf to using DRMAA would be nice because it would
> give a better separation between Twisted style code and synchronous
> style code. A Twisted interface for Xgrid would be really nice,
> especially if it was well done, but crufting on synchronous wrappers
> for it (like what i'm doing now) just feels wrong.

I am not sure I am fully following you. I do think the best way of
developing Twisted stye code for using Xgrid, is to directly use
XgridFramework though PyObjC. That everything is properly
asynchronous.

> I will send something by you on the Twisted end when I have it cleaned
> up a bit. Along those lines, I haven't been keeping great track of the
> ipython1 development. What is the status of using it on an Xgrid?

I would love to see that.

About IPython1. I have been working on IPython1 a lot this summer -
had a student working full time on it. We are close to making an
initial public release but it is already *very* usable. Folks here at
my work are using it daily on the NERSC supercomputers now (earlier
this year I moved back to Boulder CO to work for a research company).

Here is our main page on IPython1's parallel computing capabilities if
you haven't seen it:

http://ipython.scipy.org/moin/Parallel_Computing

Currently, there is not explicit support for using this through Xgrid,
but it would be *absolutely trivial* to get this working. The only
thing that has to be done is it to use pyxg (or even the CLI) to start
the IPython Engines as XGrid Jobs. Something like

xgrid submit ipengine

should work. The only things that have to be set up on a cluster
specific basis are:

1. Open the needed firewall ports.
2. Tell the engines where the IPython Controller is running.

I no longer have access to a large Xgrid cluster so it is a little
more difficult for me to test things. But I would be more than glad
to help you get this setup on your cluster.

Brian

[Barry Wark]

Brian,

My replies inline too...

On 10/31/06, Brian Granger <elli...@gmail.com> wrote:
>
> Barry,
>

> See me replies inline.
>
> On 10/31/06, Barry Wark <barr...@gmail.com> wrote:
> >
> > Brian,
> >
> > I may have gotten my head all mixed up on how Ed's code worked. It
> > looked to me like he was running an NSRunLoop in a separate thread and
> > using mutex locks to block for synchronous operations. This is an
> > approach that I've played around with for my matlabxg package and has
> > some success with, though Ed's code is far cleaner than mine.
>
> Hmm, that is an interesting idea. Can the NSRunloop be run in a
> secondary thread or does it have to be run in the main thread? Most
> GUI toolkits (like QT and WX) need to be in the main thread. I would
> like to know more about how he does this though.

As I understand, each thread in OS X has its own NSRunLoop (really a
CFRunLoop). This is necessary to allow Mach messaging between threads,
among other things. The reason that most GUI toolkits have to be run
on the main thread is that much of AppKit is not thread safe, so most
UI updates (drawing, really) have to happen on the main thread. If
you're not using AppKit, most of the rest of CoreFoundation/Foundation
et al. are thread safe, and there'd be no reason to restrict yourself
to the main thread for NSRunLoop based code.

So, if you spawn a new thread to do some NSRunLoop based stuff, you
can start the run loop, let it manage itself, and then let it exit
when you've removed all ports etc. from the run loop. In the case of
Twisted, you can stop the reactor (and use that to stop the run loop)
for each thread, so you don't have to manage the run loop or twisted
across invocations of a synchronous API.

>
> > I agree that one of the (maybe the) most important feature of pyxg was
> > that it could be used from the interpreter without a runloop.
> >
> > I think your suggestion is a good one. I will look into a python
> > wrapper for Ed's code, both by directly wrapping the DRMAA interface
> > (or what's exposed in Ed's framework) and in adapting my xgf package
> > to use his DRMAA instead of Twisted under the hood. We'll just see
> > what comes out better.
> >
> > I think moving my xgf to using DRMAA would be nice because it would
> > give a better separation between Twisted style code and synchronous
> > style code. A Twisted interface for Xgrid would be really nice,
> > especially if it was well done, but crufting on synchronous wrappers
> > for it (like what i'm doing now) just feels wrong.
>
> I am not sure I am fully following you. I do think the best way of
> developing Twisted stye code for using Xgrid, is to directly use
> XgridFramework though PyObjC. That everything is properly
> asynchronous.

Yes, that's what I was trying to say, but munged. Right now, to make
something like pyxg using XgridFoundation (with or without Twisted), I
had to cruft some ugly code on top to make things appear synchronous.
Pulling out all that cruft should leave something that looks like
proper Twisted code.

>
> > I will send something by you on the Twisted end when I have it cleaned
> > up a bit. Along those lines, I haven't been keeping great track of the
> > ipython1 development. What is the status of using it on an Xgrid?
>
> I would love to see that.

You bet.

>
> About IPython1. I have been working on IPython1 a lot this summer -
> had a student working full time on it. We are close to making an
> initial public release but it is already *very* usable. Folks here at
> my work are using it daily on the NERSC supercomputers now (earlier
> this year I moved back to Boulder CO to work for a research company).
>
> Here is our main page on IPython1's parallel computing capabilities if
> you haven't seen it:
>
> http://ipython.scipy.org/moin/Parallel_Computing
>
> Currently, there is not explicit support for using this through Xgrid,
> but it would be *absolutely trivial* to get this working. The only
> thing that has to be done is it to use pyxg (or even the CLI) to start
> the IPython Engines as XGrid Jobs. Something like
>
> xgrid submit ipengine
>
> should work. The only things that have to be set up on a cluster
> specific basis are:
>
> 1. Open the needed firewall ports.
> 2. Tell the engines where the IPython Controller is running.
>
> I no longer have access to a large Xgrid cluster so it is a little
> more difficult for me to test things. But I would be more than glad
> to help you get this setup on your cluster.

Very cool. You guys have done a lot of work this summer! Looks very
nice. I will pull it out of svn and have a peek. It may take me a week
or two to get to it, but I'll try to get it running on our cluster
here (mixed PPC and Intel at the moment so would be a good test case).
At that point, anything I can do to help test would be my pleasure.

barry

[clip]

[Brian Granger]

> As I understand, each thread in OS X has its own NSRunLoop (really a
> CFRunLoop). This is necessary to allow Mach messaging between threads,
> among other things. The reason that most GUI toolkits have to be run
> on the main thread is that much of AppKit is not thread safe, so most
> UI updates (drawing, really) have to happen on the main thread. If
> you're not using AppKit, most of the rest of CoreFoundation/Foundation
> et al. are thread safe, and there'd be no reason to restrict yourself
> to the main thread for NSRunLoop based code.
>
> So, if you spawn a new thread to do some NSRunLoop based stuff, you
> can start the run loop, let it manage itself, and then let it exit
> when you've removed all ports etc. from the run loop. In the case of
> Twisted, you can stop the reactor (and use that to stop the run loop)
> for each thread, so you don't have to manage the run loop or twisted
> across invocations of a synchronous API.

Interesting. I am not that familiar with this stuff so this is good to know.

> Very cool. You guys have done a lot of work this summer! Looks very
> nice. I will pull it out of svn and have a peek. It may take me a week
> or two to get to it, but I'll try to get it running on our cluster
> here (mixed PPC and Intel at the moment so would be a good test case).
> At that point, anything I can do to help test would be my pleasure.

That would be great. We would love to get more people testing and
using this stuff before we make our public release. I am more than
willing to help you get this going. I would even be open to writing
some code in IPython1 that automate things using Xgrid. Let me know
when you are ready to move in this direction.

Brian

[Barry Wark]

On 10/31/06, Brian Granger <elli...@gmail.com> wrote:
>

Sounds great. I will get in touch as soon as I'm ready to move on it
(like I said it may be a couple of weeks). We've just re-written much
of our lab's data acquisition software, so the next step will be the
analysis end, and I'd like to write it with ipython1 in mind, if
possible. It sounds like the timing couldn't be better.

Thanks,
barry