Blue Mountain / Origin 2000

62 views
Skip to first unread message

Bjorn Lindgren

unread,
Oct 8, 1999, 3:00:00 AM10/8/99
to
Hi, I wonder if someone knows what kind of interconnect
SGI uses betwwen the O2000 "nodes" in LANL Blue Mountain
supercomputer, and that bandwidth it have (GByte/s).
I know they use CrayLink between the system boards inside
the O2000.

I'm also intrested to know what kind of "design" the
supercomputer has. I know O2000 is SMP/ccNUMA.
(single kernel image), but SMP/ccNUMA on that many CPUs?
I guess its a mix of MPP and SMP/ccNUMA. Someone can
maybe clarify for an layman. :)

- bln

John Stephens

unread,
Oct 8, 1999, 3:00:00 AM10/8/99
to
On 8 Oct 1999 17:09:30 GMT, Bjorn Lindgren <b...@sunphocus.upp.se.pnu.com>
wrote:

>Hi, I wonder if someone knows what kind of interconnect
>SGI uses betwwen the O2000 "nodes" in LANL Blue Mountain
>supercomputer, and that bandwidth it have (GByte/s).
>I know they use CrayLink between the system boards inside
>the O2000.

HIPPI.

>
>I'm also intrested to know what kind of "design" the
>supercomputer has. I know O2000 is SMP/ccNUMA.
>(single kernel image), but SMP/ccNUMA on that many CPUs?
>I guess its a mix of MPP and SMP/ccNUMA. Someone can
>maybe clarify for an layman. :)
>
>- bln

--
John Stephens (remove spaces for legal email address)
s t e p h e n s @ p o b o x . c o m

Nick Maclaren

unread,
Oct 9, 1999, 3:00:00 AM10/9/99
to
In article <OaX+N=MU4unTlZ2ib...@4ax.com>,

John Stephens <step...@no-fixed-abode.com> wrote:
>On 8 Oct 1999 17:09:30 GMT, Bjorn Lindgren <b...@sunphocus.upp.se.pnu.com>
>wrote:
>
>>Hi, I wonder if someone knows what kind of interconnect
>>SGI uses betwwen the O2000 "nodes" in LANL Blue Mountain
>>supercomputer, and that bandwidth it have (GByte/s).
>>I know they use CrayLink between the system boards inside
>>the O2000.
>
>HIPPI.

Isn't it GSN (aka HiPPI 6400), rather than plain HiPPI 800? Or is
that the next upgrade?

>>I'm also intrested to know what kind of "design" the
>>supercomputer has. I know O2000 is SMP/ccNUMA.
>>(single kernel image), but SMP/ccNUMA on that many CPUs?
>>I guess its a mix of MPP and SMP/ccNUMA. Someone can
>>maybe clarify for an layman. :)

You have just done it! The system is an MPP cluster of SMP/ccNUMA
nodes. One way of programming such beasts is to use MPI between
nodes and OpenMP within the nodes - giving some delightfully complex
algorithmic and programming problems :-)

The Hitachi SR8000 and several other new supercomputers have similar
designs.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QG, England.
Email: nm...@cam.ac.uk
Tel.: +44 1223 334761 Fax: +44 1223 334679

Dan Nagle

unread,
Oct 9, 1999, 3:00:00 AM10/9/99
to
Hello,

Nick Maclaren wrote:
>
<snip ongoing discussion>


>
> You have just done it! The system is an MPP cluster of SMP/ccNUMA
> nodes. One way of programming such beasts is to use MPI between
> nodes and OpenMP within the nodes - giving some delightfully complex
> algorithmic and programming problems :-)

Why not use Co-array Fortran and leave the "delightfully complex
algorithmic and programming problems" to the compiler?

<snip rest and sig>

--

Cheers!
Dan Nagle dna...@erols.com
Purple Sage Computing Solutions, Inc.

Nick Maclaren

unread,
Oct 9, 1999, 3:00:00 AM10/9/99
to
In article <37FF5FA7...@erols.com>, Dan Nagle <dna...@erols.com> wrote:

>
>Nick Maclaren wrote:
>>
>> You have just done it! The system is an MPP cluster of SMP/ccNUMA
>> nodes. One way of programming such beasts is to use MPI between
>> nodes and OpenMP within the nodes - giving some delightfully complex
>> algorithmic and programming problems :-)
>
>Why not use Co-array Fortran and leave the "delightfully complex
>algorithmic and programming problems" to the compiler?

Nice one. It was a joke, wasn't it?

Greg Lindahl

unread,
Oct 9, 1999, 3:00:00 AM10/9/99
to
nm...@cus.cam.ac.uk (Nick Maclaren) writes:

> >Why not use Co-array Fortran and leave the "delightfully complex
> >algorithmic and programming problems" to the compiler?
>
> Nice one. It was a joke, wasn't it?

Only problem is, nobody's laughing.

-- g

John Stephens

unread,
Oct 9, 1999, 3:00:00 AM10/9/99
to
On 9 Oct 1999 10:27:25 GMT, nm...@cus.cam.ac.uk (Nick Maclaren) wrote:

>In article <OaX+N=MU4unTlZ2ib...@4ax.com>,
>John Stephens <step...@no-fixed-abode.com> wrote:
>>On 8 Oct 1999 17:09:30 GMT, Bjorn Lindgren <b...@sunphocus.upp.se.pnu.com>
>>wrote:
>>
>>>Hi, I wonder if someone knows what kind of interconnect
>>>SGI uses betwwen the O2000 "nodes" in LANL Blue Mountain
>>>supercomputer, and that bandwidth it have (GByte/s).
>>>I know they use CrayLink between the system boards inside
>>>the O2000.
>>
>>HIPPI.
>
>Isn't it GSN (aka HiPPI 6400), rather than plain HiPPI 800? Or is
>that the next upgrade?

The next upgrade, but pretty soon now. It's possible some GSN links are
already in place, but it would have been quite recent.



>
>>>I'm also intrested to know what kind of "design" the
>>>supercomputer has. I know O2000 is SMP/ccNUMA.
>>>(single kernel image), but SMP/ccNUMA on that many CPUs?
>>>I guess its a mix of MPP and SMP/ccNUMA. Someone can
>>>maybe clarify for an layman. :)
>

>You have just done it! The system is an MPP cluster of SMP/ccNUMA
>nodes. One way of programming such beasts is to use MPI between

>nodes and OpenMP within the nodes - giving some delightfully complex
>algorithmic and programming problems :-)
>
>The Hitachi SR8000 and several other new supercomputers have similar
>designs.
>
>

>Regards,
>Nick Maclaren,
>University of Cambridge Computing Service,
>New Museums Site, Pembroke Street, Cambridge CB2 3QG, England.
>Email: nm...@cam.ac.uk
>Tel.: +44 1223 334761 Fax: +44 1223 334679

--

Per Ekman

unread,
Oct 12, 1999, 3:00:00 AM10/12/99
to
Bjorn Lindgren <b...@sunphocus.upp.se.pnu.com> writes:

> Hi, I wonder if someone knows what kind of interconnect
> SGI uses betwwen the O2000 "nodes" in LANL Blue Mountain
> supercomputer, and that bandwidth it have (GByte/s).
> I know they use CrayLink between the system boards inside
> the O2000.

HIPPI 6400 AFAIK. (6400 Mb/s)

> I'm also intrested to know what kind of "design" the
> supercomputer has. I know O2000 is SMP/ccNUMA.
> (single kernel image), but SMP/ccNUMA on that many CPUs?
> I guess its a mix of MPP and SMP/ccNUMA. Someone can
> maybe clarify for an layman. :)

I guess it's just a big cluster, you submit your jobs to a
scheduler and use MPI or whatever to communicate between
nodes. The O2K scales to 256 CPUs, although I think that
the Blue Mountain machines are 128 CPU ones. So they have
a number if SSI O2Ks in a fast network. If that should count as
one machine or not is another question entirely.

Pek


Bjorn Lindgren

unread,
Oct 12, 1999, 3:00:00 AM10/12/99
to
Per Ekman <p...@pdc.kth.se> wrote:
: Bjorn Lindgren <b...@sunphocus.upp.se.pnu.com> writes:

: I guess it's just a big cluster, you submit your jobs to a


: scheduler and use MPI or whatever to communicate between
: nodes. The O2K scales to 256 CPUs, although I think that
: the Blue Mountain machines are 128 CPU ones. So they have
: a number if SSI O2Ks in a fast network. If that should count
: as one machine or not is another question entirely.

There are serval kernel images so it should count as serval
machines (think IBM SP2). I was a bit unsure about that part,
becoz I'd tought they'd (SGI) use some sort of special
shrink-warped version of IRIX (Celluar IRIX or what name is)
to build single kernel image systems out of serval Origion
2000 nodes. (SMP on a cluster)

- bln

Michael Woodacre

unread,
Oct 12, 1999, 3:00:00 AM10/12/99
to
In article <mjen1tp...@toby.pdc.kth.se>, Per Ekman <p...@pdc.kth.se> writes:
|> Bjorn Lindgren <b...@sunphocus.upp.se.pnu.com> writes:
|>
|> > Hi, I wonder if someone knows what kind of interconnect
|> > SGI uses betwwen the O2000 "nodes" in LANL Blue Mountain
|> > supercomputer, and that bandwidth it have (GByte/s).
|> > I know they use CrayLink between the system boards inside
|> > the O2000.
|>
|> HIPPI 6400 AFAIK. (6400 Mb/s)
|>
|> > I'm also intrested to know what kind of "design" the
|> > supercomputer has. I know O2000 is SMP/ccNUMA.
|> > (single kernel image), but SMP/ccNUMA on that many CPUs?
|> > I guess its a mix of MPP and SMP/ccNUMA. Someone can
|> > maybe clarify for an layman. :)
|>
|> I guess it's just a big cluster, you submit your jobs to a
|> scheduler and use MPI or whatever to communicate between
|> nodes. The O2K scales to 256 CPUs, although I think that

Actually, the Origin ccNUMA architecture scales to 1024 CPU's. Currently the
largest configurations that have actually been built are 512P.

Cheers,
Mike


|> the Blue Mountain machines are 128 CPU ones. So they have
|> a number if SSI O2Ks in a fast network. If that should count as
|> one machine or not is another question entirely.
|>

|> Pek
|>

--
Michael S. Woodacre
UUCP: wood...@sgi.com
Phone: +44 1118 925 7846 (work)
Home: +44 1249 721536 (home)
M/S: IGB-3110
Address : 1530 Lakeside
Arlington Business Park
Theale, Berkshire
Reading
RG7 4SB
United Kingdom

Marco Eschelwek

unread,
Nov 2, 1999, 3:00:00 AM11/2/99
to
On 12 Oct 1999, Bjorn Lindgren wrote:

[snip]

> becoz I'd tought they'd (SGI) use some sort of special
> shrink-warped version of IRIX (Celluar IRIX or what name is)

Cellular IRIX ? As far as I know there ist still no final release
available. (But I'm not 100% sure) Because we've been waiting for this
OS for our Origin2000 for ages.

regards
marco

--
_________________________________
UNIX - using the computer smarter
_________________________________

John Stephens

unread,
Nov 3, 1999, 3:00:00 AM11/3/99
to
On Tue, 2 Nov 1999 14:12:10 +0100, Marco Eschelwek <ma...@rz.tu-ilmenau.de>
wrote:

>On 12 Oct 1999, Bjorn Lindgren wrote:
>
>[snip]
>
>> becoz I'd tought they'd (SGI) use some sort of special
>> shrink-warped version of IRIX (Celluar IRIX or what name is)
>
>Cellular IRIX ? As far as I know there ist still no final release
>available. (But I'm not 100% sure) Because we've been waiting for this
>OS for our Origin2000 for ages.

You're going to wait a long time. The Cellular IRIX project was abandoned
at least 9 months ago.

Bjorn Lindgren

unread,
Nov 3, 1999, 3:00:00 AM11/3/99
to
John Stephens <step...@no-fixed-abode.com> wrote:
: On Tue, 2 Nov 1999 14:12:10 +0100, Marco Eschelwek <ma...@rz.tu-ilmenau.de>

: wrote:
:>On 12 Oct 1999, Bjorn Lindgren wrote:
:>
:>[snip]
:>
:>> becoz I'd tought they'd (SGI) use some sort of special
:>> shrink-warped version of IRIX (Celluar IRIX or what name is)
:>
:>Cellular IRIX ? As far as I know there ist still no final release
:>available. (But I'm not 100% sure) Because we've been waiting for this
:>OS for our Origin2000 for ages.

: You're going to wait a long time. The Cellular IRIX project was abandoned
: at least 9 months ago.

Do you know the reason why it was abandoned?

- bln

John Stephens

unread,
Nov 3, 1999, 3:00:00 AM11/3/99
to

Not sure whether they figured they didn't know how to do it, or if the "rush
to Linux" got in the way.

SGI's planning has not been very consistent over the past year,
unfortunately. :-(

Dave Olson

unread,
Nov 4, 1999, 3:00:00 AM11/4/99
to
John Stephens <step...@no-fixed-abode.com> writes:
| >: You're going to wait a long time. The Cellular IRIX project was abandoned
| >: at least 9 months ago.
| >
| >Do you know the reason why it was abandoned?
|
| Not sure whether they figured they didn't know how to do it, or if the "rush
| to Linux" got in the way.
|
| SGI's planning has not been very consistent over the past year,
| unfortunately. :-(

It was killed (as a major effort) about 15 months ago, well before any
of the earliest serious talk about moving towards linux.

It became clear that the customers who would benefit (a fairly
small percentage of origin customers) didn't justify the work, when
there was so much other work to do that was more broadly applicable,
and a limited number of people to do the work.

Some of the work did end up in irix 6.5 in various ways, CXFS,
HA, etc., so it wasn't completely abandoned/dropped. And some of
it may show up in future cluster work.

And some of the more general work shaped the origin follow-on
hardware and software as well.

And finally, IRIX 6.5 (as with 6.4) is still officially called
"cellular irix", even though the "true" cell work was killed,
just to confuse things further.
--

Dave Olson
ol...@bengaltech.com
http://www.bengaltech.com

Reply all
Reply to author
Forward
0 new messages