Running distcc on a SiCortex? (aka Department of dumb questions II)
Danny
these days.
It strikes me that it might be a fantastic mipsel compile farm machine
[1], but I have zero experience in HPC, so have what is probably a
dumb question:
Would there be an (obvious / easy) way to just run http://distcc.org
on each of the 12 nodes / use all 72 processors for compiling on a
SC072-PDS?
distcc does not require all machines to share a filesystem, have
synchronized clocks, or to have the same libraries or header files
installed.
distcc runs as a TCP/IP daemon, so each of the 12 nodes would need to
be reachable with a separate IP address.
Thanks,
--
Daniel JB Clark | Sys Admin, Free Software Foundation
pobox.com/~dclark | http://www.fsf.org/about/staff#danny
Win Treese
On Dec 26, 2009, at 2:08 PM, Danny wrote:
> The SiCortex SC072-PDS seems to be going for less than $2000 on ebay
> these days.
>
> It strikes me that it might be a fantastic mipsel compile farm machine
> [1], but I have zero experience in HPC, so have what is probably a
> dumb question:
>
> Would there be an (obvious / easy) way to just run http://distcc.org
> on each of the 12 nodes / use all 72 processors for compiling on a
> SC072-PDS?
>
> distcc does not require all machines to share a filesystem, have
> synchronized clocks, or to have the same libraries or header files
> installed.
>
> distcc runs as a TCP/IP daemon, so each of the 12 nodes would need to
> be reachable with a separate IP address.
Not a dumb question at all.
It's quite straightforward to set up distcc to run on all the nodes. Each
node has its own IP address on the internal network. Normally those
addresses are visible inside the box, but with a little bit of work, you
can make those addresses routable from the outside as well.
You more or less just follow the distcc setup instructions; just make
sure the files end up in the right places.
- Win
Win Treese
Serissa Research, Inc. / Sector 9 Software
tre...@serissa.com
+1 508 314 4359
Danny
> On Dec 26, 2009, at 2:08 PM, Danny wrote:
> > The SiCortex SC072-PDS seems to be going for less than $2000 on ebay
> > these days.
>
> > It strikes me that it might be a fantastic mipsel compile farm machine
> > [1], but I have zero experience in HPC, so have what is probably a
> > dumb question:
>
> > on each of the 12 nodes / use all 72 processors for compiling on a
> > SC072-PDS?
>
> > distcc does not require all machines to share a filesystem, have
> > synchronized clocks, or to have the same libraries or header files
> > installed.
>
> > distcc runs as a TCP/IP daemon, so each of the 12 nodes would need to
> > be reachable with a separate IP address.
>
> Not a dumb question at all.
>
> It's quite straightforward to set up distcc to run on all the nodes. Each
> node has its own IP address on the internal network. Normally those
> addresses are visible inside the box, but with a little bit of work, you
> can make those addresses routable from the outside as well.
Thanks a lot; I got an account on a SiCortex system and am testing
this out now.
The first problem I've run into is that SiCortex does not support the
o32 MIPS ABI, which Debian (and probably many other GNU/Linux
distributions) use because in the past it has been "compatible with
everything". However (from the SiCortex Programmer's Guide):
"""
Programmers who have used other MIPS systems may be familiar with
the o32 MIPS ABI. This is a 32-bit ABI and predates the availability
of
64-bit registers in the MIPS architecture family. The System does not
sup-
port the o32 ABI.
"""
So at the moment I'm contacting the people who were involved with a
Debian n32 port [1] (someone did performance tests and for non-
scientific workloads n32 is more efficient than n64), and reading up
on their published work [2].
[1] Debian "loongson2f"-optimized N32 ABI port
http://wiki.debian.org/SummerOfCode2009/Debian_loongson2f_N32_port
[2] Final report for Create A MIPS3 Port for Debian
https://alioth.debian.org/docman/view.php/100380/4074/FinalReport.odt
"C. Bergström"
This is not an authoritative answer, but it seems we have abi code for
both mipsn32 and mips64. I have a machine I can test with, but this is
something we should pursue offlist and then report the results back.
./C
Danny
> Danny wrote:
> > The first problem I've run into is that SiCortex does not support the
> > o32 MIPS ABI, which Debian (and probably many other GNU/Linux
> > distributions) use because in the past it has been "compatible with
> > everything". However (from the SiCortex Programmer's Guide):
>
> > """
> > Programmers who have used other MIPS systems may be familiar with
> > the o32 MIPS ABI. This is a 32-bit ABI and predates the availability
> > of 64-bit registers in the MIPS architecture family. The System does
> > """
>
> > So at the moment I'm contacting the people who were involved with a
> > Debian n32 port [1] (someone did performance tests and for non-
> > scientific workloads n32 is more efficient than n64), and reading up
> > on their published work [2].
>
> Why not contact the people who are still maintaining the compiler?
>
> This is not an authoritative answer, but it seems we have abi code for
> both mipsn32 and mips64. I have a machine I can test with, but this is
> something we should pursue offlist and then report the results back.
My reading of the above was that o32 abi binaries was something that
wasn't supported either at the silicon level, or at the linux level
(eg the option to support the abi would have to be compiled in; for
example, a.out support is sometimes not compiled in to x86 linux). So
I don't see how compilers would be related to the problem.
It's possible that the compilers could produce o32 binaries that could
not then run on sicortex systems but could on other mipsel systems;
that may be useful, but isn't what I was talking about. As I'm
interested in getting n32 into Debian anyway, whether or not this
works isn't of long-term concern for me.
BTW it looks like the Debian n32 project is alive and there is an
actual repo available [1], so I'll try doing a fakechroot debboostrap
with that [2] and see how it works on the SiCortex.
[1] Blog post that has link to Debian n32 repo in comments
http://sandyleo26.wordpress.com/2009/10/25/create-a-mips-3-port-for-debian-update-20091025
[2] My notes on creating and using fakechroot environments, for when
you don't have root.
1. (from a debian system as nonroot)
PATH=/sbin:/usr/sbin:$PATH fakeroot -s ${DIST}-fakechroot.save
fakechroot debootstrap --arch=mipsel --variant=fakechroot $DIST ./$
{DIST}-fakechroot $URL
2. (rsync over to sicortex system in $HOME/fakechroots/)
3. (from sicortex system as nonroot)
PATH=$HOME/local/bin:$PATH fakeroot -i $HOME/fakechroots/${DIST}-
fakechroot.save fakechroot chroot $HOME/fakechroots/${DIST}-
fakechroot /bin/bash