SC 5832 Compute Blades

[LM SiCortex]

To the Group:

Does anyone know of any spare, unused, compute blades for the SC 5832?  I'm looking at possible upgrade options for our SC 5832.

If you do, then please let me know where and the cost.

Thank you,
Jason

--
Jason R. Smith
Lockheed Martin
Systems Integrator
AFRL/RCM, Bldg 676
2435 5th Street
WPAFB  OH  45433
Jason...@wpafb.af.mil
Jason....@lmco.com
937-255-0941 (desk)
937-238-6471 (cell)

[Narayan Desai]

Which clock rate? We might be able to part with a few of ours, if we
can figure out the organizational gymnastics to do so.
-nld

> --
> You received this message because you are subscribed to the Google Groups
> "SiCortex Users" group.
> To post to this group, send email to sicorte...@googlegroups.com.
> To unsubscribe from this group, send email to
> sicortex-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/sicortex-users?hl=en.
>

[LM SiCortex]

Ours is 700 MHz.  The compute nodes have 1.5 Volt memory sticks.  We have four (4) boards with 1.8 Volt memory sticks, but it appears we will have to modify the boot script to accommodate those boards.  Larry said he'd look into it.

We have 8 blank slots in our machine.

Thanks!
Jason

[Narayan Desai]

all of our node boards are 633s. As far as I know, you can't mix and
match. Is that right?
-nld

[Michael Di Domenico]

We've mixed 700Mhz and 500Mhz (i think, might be 633Mhz) boards in our
648. the system sees the slower boards and revs the whole system to
the slowest speed board

[Narayan Desai]

OK. I'll check if it is possible to sell things outside of DOE or not.
I think that we have more spares than we will reasonably need at this
point. Just be forewarned that this will likely take quite a while to
get done.
-nld

[Lawrence Stewart]

I think Jason has 4 700 MHz boards, which would bring the system to 32 boards.
If you then plug in 4 633 MHz boards, bringing the system to 36 boards, but running
all of them at 633, what you get is the equivalent of a 633/700 * 36, or 32.5 board machine.

Probably the boot code could be hacked to let them run at independent speeds, and
to disable the global clock synchronization. That would cost about a factor of 2 in
the performance of very large collectives, but if the job mix is embarassingly parallel that might be OK.

-L

[Narayan Desai]

And you were saying at SC that you can run any number of boards in a
system when you get that close to full, right?
-nld

[Larry Stewart]

yes, I've dusted off the buildroute program. It doesn't optimize the module placement but it can evaluate them. There are perfectly good 32 module configs. For example. The choice of offering 12, 20, 28, and 36 module versions was mostly marketing and only slightly technical.

I also discovered that the guidelines for disabling links that are "too long" in the presence of placeholders are wrong, although not by much. I've worked out the correct set for 28 boards.

-Larry

[LM SiCortex]

Narayan,

Did you all ever receive a Statement of Volatility (SOV) from SiCortex?  We were not able to obtain one.  That I know of, the only volatile parts on each blade is the memory.  For security reasons, we have to make sure that no customer data can be retained on the boards.  That may be another determining factor if parts could be sold/bought.  Or does anyone have one??


Others,

Yes, our system is 28 boards of 700 MHz.  I'm starting to wonder if the 700 MHz was too new to have many like-spares available.

Memory Voltages:
Has anyone run different memory voltages in their SiCortex system?  Our system runs 1.5 Volt memory.  The 4 spare 700 MHz blades we have are 1.8 Volt.  Larry said the boot script would probably have to be modified for that difference... or the system may default to the lowest common denominator... I'm not sure if that's good for the system stability & performance.

- Jason

[Lawrence Stewart]

Regarding dimm voltage

On Jan 12, 2011, at 12:39 PM, LM SiCortex wrote:

Others,

Yes, our system is 28 boards of 700 MHz.  I'm starting to wonder if the 700 MHz was too new to have many like-spares available.

Memory Voltages:
Has anyone run different memory voltages in their SiCortex system?  Our system runs 1.5 Volt memory.  The 4 spare 700 MHz blades we have are 1.8 Volt.  Larry said the boot script would probably have to be modified for that difference... or the system may default to the lowest common denominator... I'm not sure if that's good for the system stability & performance.

- Jason

I asked Kem Stewart about this, and he said

They say the system has all "1.5 volt modules" and that the spares are 1

1.5v and 3 "1.8v".

What does that refer to?  What happens if they get plugged in?  WP says they

were told it would work, but I thought they all had to be the same.

This refers to the DIMM power supply voltage.  Has to be consistent on
a board, but can vary from board to board.

My recollection is that scboot polls the modules, and sets them at least to

all the same

clock frequency, but I don't know what happens with the voltages.

The very latest version of the software and diags certainly does the
right thing:  each board runs its DIMMs at the right voltage for them,
and any voltage margining you do is differential, wrt to each board's
nominal DIMM voltage.  Plug in the spare boards and see what
happens--you know, measure the voltages with the diags or through your
back-door commands to the MSP.  My recollection is that the 1.5V DIMMs
will take 1.8V without damage for an unspecified period, basically
indefinitely if you don't run anything that exercises them.

I think older versions of the software may simply barf if multiple
DIMM voltages are detected, or they may run but not allow for proper
voltage margining.

-Larry

[Win Treese]

On Jan 12, 2011, at 12:39 PM, LM SiCortex wrote:

> Narayan,
>
> Did you all ever receive a Statement of Volatility (SOV) from SiCortex? We were not able to obtain one. That I know of, the only volatile parts on each blade is the memory. For security reasons, we have to make sure that no customer data can be retained on the boards. That may be another determining factor if parts could be sold/bought. Or does anyone have one??

The only non-volatile memory on a SiCortex board is the flash memory for the Module Service Processor (MSP), which is where its boot code is stored. This memory is only written by special tools from the SSP, and it's a rather convoluted process to do it. Normally, it would only be modified if there were a software update from SiCortex. I don't recall that the production MSP software was ever updated.

Some security-sensitive customers were sufficiently satisfied with this that they were willing to return broken boards to SiCortex for refurbishing.

Best regards,

Win Treese

Win Treese
Sector 9 Software / Serissa Research, Inc.
tre...@sector9software.com / tre...@serissa.com
+1 508 314 4359