Announcing Gproc cluster management tool

[John Floren]

After months of development, we're finally ready to announce an
initial public release of Gproc, a cluster management tool written
entirely in Go. From the README:

Gproc is a new system, written in Go, that combines features of the
LANL version of bproc (http://sourceforge.net/projects/bproc/) and the
xcpu software (xcpu.org). Unlike bproc, gproc requires no kernel
patch. Like both of them, it provides a process startup mechanism for
lightweight cluster nodes which have only a small ramdisk as the root
file system, with no local disks or NFS root at all. Lightweight
cluster nodes can be very easy to manage if configured correctly (see,
e.g., http://portal.acm.org/citation.cfm?id=1132314).

Gproc provides a system for running programs across clusters. A
command and a set of nodes upon which to run it are specified at the
command line; the binary, any required libraries, and any additional
files specified at the command line are packaged up and sent out to
the selected nodes for execution. The libraries need for the binary
are determined by gproc programatically. The outputs are then
forwarded back to the control node. Currently, we do not support stdin
for the tree of processes as bproc did; it's coming but we have not
needed it yet.

The source is available at http://bitbucket.org/rminnich/gproc. We
have updated the code to work with changeset 8101:6aa7ab6b6a9d (a
recent weekly release) of Go; see the INSTALL file for information on
getting it compiled.

The README file should help you get started using gproc, if you have a
cluster or even a set of computers which might pass as a cluster. You
can even test it on just one computer, using the "local" locale, like
so:

% sudo rm -rf /tmp/g && ./gproc_linux_amd64 -locale=local m
master : 2011/04/20 17:21:40 -cmdport=0.0.0.0:39563

# open a new terminal
# look at the port in the "cmdport=" output from the previous command,
use that as our -cmdport argument
# we have to do this because everything is running on one machine,
it's not necessary otherwise
% sudo ./gproc_linux_amd64 -locale=local -cmdport=39563 s
slave : 2011/04/20 17:21:53 -cmdport=0.0.0.0:41412

# open a third terminal
% ./gproc_linux_amd64 e . /bin/date
Wed Apr 20 17:21:59 PDT 2011

As you might have guessed, gproc is a huge security hole, so you
should be careful when running it. If you look... this allows any user
to run any program as root. Useful for a supercomputer, but something
to be wary of on a regular machine.

The authors of gproc are: Ron Minnich, Noah Evans, Jason Dreisbach,
and John Floren. Be gentle, this was our first major Go project!

John Floren

[Fango]

I just prayed hard for this kind of cluster thing last night and it
happened. Thanks for four of you.

Anyway, seems the example above is not quite working on OSX, for slave
I have:

mini:gproc go$ sudo ./gproc_darwin_amd64 -locale=local -cmdport=52780
s
slave : 2011/04/21 11:13:53 -cmdport=0.0.0.0:52798
newgproc : 2011/04/21 11:14:35 privatemount called on OSX

and for 3rd term I have:
mini:gproc go$ ./gproc_darwin_amd64 e . /bin/date

but no output.

Will dig into codes.

Cheers,
Fango

[ron minnich]

On Wed, Apr 20, 2011 at 8:22 PM, Fango <fan.h...@gmail.com> wrote:
> I just prayed hard for this kind of cluster thing last night and it
> happened. Thanks for four of you.
>
> Anyway, seems the example above is not quite working on OSX, for slave
> I have:
>
> mini:gproc go$ sudo ./gproc_darwin_amd64 -locale=local -cmdport=52780
> s
> slave : 2011/04/21 11:13:53 -cmdport=0.0.0.0:52798
> newgproc : 2011/04/21 11:14:35 privatemount called on OSX

First problem: we don't really support OSX as a slave node -- we have
no OSX clusters. (BTW, this master/slave terminology is from bproc. I
hope it doesn't upset anyone, if it does we can change it). But you
might be able to make it work ...
set -p=false and -localbin=true on all your commands.
Also, note, I said something wrong: it's not that everything runs as
root, it runs as the user who starts the gproc s. Easy to fix, we just
have not had the time.

> Will dig into codes.

Very kind of you! If you want to fork it at bitbucket we will gladly
take pull requests. We also take programming advice :-)

There is a gproc developers list; contact john if you would like to be on it.

Also,please note, some of these design decisions may seem unusual,
just contact us if you're wondering but believe me, there is a reason
for them :-)

ron

[Jason Dreisbach]

If I remember correctly gproc can't determine library dependencies in
mach-o executables. Gproc ldd's the binary then copies the linked libs
and the binary to the slave nodes so they may set up an environment
for it to run. The problem is that ldd only works with the elf
binaries...

I tried a while back to add support but it ended up as an ugly hack.

A analog for linux's ldd for OS X is otool.
jtdreisb@zephyr:gostream$ otool -L /bin/ls
/bin/ls:
/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current
version 5.4.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 125.2.0)



OS X support is mainly there so a user may launch processes from their
mac across a cluster. But to do this you need to have a local copy of
a filesystem containing the binaries and libraries you wish to push to
the slaves.

Also, private mounts are not supported in OS X(to disable -p=false).

[ron minnich]

On Wed, Apr 20, 2011 at 8:59 PM, Jason Dreisbach <jtdr...@gmail.com> wrote:
> If I remember correctly gproc can't determine library dependencies in
> mach-o executables. Gproc ldd's the binary then copies the linked libs
> and the binary to the slave nodes so they may set up an environment
> for it to run. The problem is that ldd only works with the elf
> binaries...

Juat a note here, we're not running the ldd program, rather the ldd
is done with bitbucket.org/npe/ldd.

I wonder if we could extend that code to support equivalent functions on OSX?

And, even better, if we could get npe/ldd type capability integrated
into standard go libraries :-)

thanks

ron

[John Floren]

On Apr 20, 9:03 pm, ron minnich <rminn...@gmail.com> wrote:

Actually, we're using bitbucket.org/floren/ldd, because I needed to
update the libraries for recent changes in Go. I could issue a pull
request to Noah and switch it back, but at this time I've been doing
most of the gproc development and decided I'd maintain the
dependencies myself. If you want to hack on ldd, I recommend forking
my repo.

John

[John Floren]

On Apr 20, 8:59 pm, ron minnich <rminn...@gmail.com> wrote:
> Also, note, I said something wrong: it's not that everything runs as
> root, it runs as the user who starts the gproc s. Easy to fix, we just
> have not had the time.

To expand on this, you should be able to run as a non-root user by
setting -p=false and running gproc s as that user; make sure that /tmp/
xproc (or whatever you set -binRoot to) is either owned by you or does
not yet exist.

John