are parallel arrays of general-purpose CPUs obsolete?

[Mark Thorson]

Supercomputers were made obsolete by parallel arrays
of CPUs. Are those arrays now obsolete?

I notice that in Bitcoin mining, the evolution has
been CPUs -> GPUs -> FPGAs -> custom ASICs.

http://gizmodo.com/5994446/digital-drills-the-monster-machines-that-mine-bitcoin

If you wanted to do some big simulation, wouldn't
you debug it on a CPU, then run it on parallel
arrays of GPUs or FPGAs? I suppose custom ASICs
are too expensive to develop and too inflexible
for anything except algorithms that are not going
to change, like Bitcoin mining and some cryptology
problems. What are HPC people buying these days?

[Quadibloc]

On May 3, 2:45 pm, Mark Thorson <nos...@sonic.net> wrote:

> Supercomputers were made obsolete by parallel arrays
> of CPUs.

I'd tend to dispute the phrasing of that. It wasn't really the
parallel arrays of CPUs that made the supercomputers obsolete.
Parallel arrays of computers had been known - thanks to ILLIAC IV - to
be terribly inefficient. So parallelism in itself wasn't a powerful
new tool that made things better.

Instead, it was a last resort.

What made supercomputers obsolete - and forced the use of parallel
arrays of commodity CPUs - was the larger scale of integrated
circuits, so that microprocessors could now be complete large CPUs
(i.e. the Pentium, equivalent roughly to a 360/195 in basic technology
- cache plus pipeline plus Wallace Tree).

At this point, the small market for supercomputers eventually made it
impractical to make custom chips for them.

> I notice that in Bitcoin mining, the evolution has
> been CPUs -> GPUs -> FPGAs -> custom ASICs.

True, but that's because it's a specialized problem.

> What are HPC people buying these days?

CPUs and GPUs. Because they want to do something standardized - double-
precision floating-point arithmetic. So FPGAs would be too
inefficient, and ASICs would be too expensive.

John Savard

[timca...@aol.com]

Or you add an instruction that does the processing
in question (like AESNI) and completely eradicate
any custom ASIC market for that problem....

- Tim

[Stephen Fuld]

On 5/3/2013 2:39 PM, Quadibloc wrote:
> On May 3, 2:45 pm, Mark Thorson <nos...@sonic.net> wrote:
>
>> Supercomputers were made obsolete by parallel arrays
>> of CPUs.
>
> I'd tend to dispute the phrasing of that. It wasn't really the
> parallel arrays of CPUs that made the supercomputers obsolete.
> Parallel arrays of computers had been known - thanks to ILLIAC IV - to
> be terribly inefficient. So parallelism in itself wasn't a powerful
> new tool that made things better.

Agreed.

> Instead, it was a last resort.

Here I disagree. See below

> What made supercomputers obsolete - and forced the use of parallel
> arrays of commodity CPUs - was the larger scale of integrated
> circuits, so that microprocessors could now be complete large CPUs
> (i.e. the Pentium, equivalent roughly to a 360/195 in basic technology
> - cache plus pipeline plus Wallace Tree).

Sort of. A, and perhaps The major factor was the changing relationship
between the speeds of CPU instruction execution and memory latency and
bandwidth. Let's take as an example, the Cray-1. It used 16
interleaved banks of high speed SRAM to provide enough bandwidth to be
able to load vectors two 64 bit words per cycle with no extra latency to
the cacheless CPU, which ran at 80 MHz.

Now fast forward to current time. You can't put enough pins on a chip
to get enough memory bandwidth to keep up with wide vector operations on
a say 3 GHz chip. Not to mention the increased latency of DRAMs makes
caching on the CPU chip pretty much a necessity and caching doesn't
"play nicely" with long vectors. It was this change more than anything
else that made long vector CPUs obsolete.

> At this point, the small market for supercomputers eventually made it
> impractical to make custom chips for them.

And the loss of advantage of a custom chip over a commodity one reduced
the advantage of a custom chip for supercomputer applications.



--
- Stephen Fuld
(e-mail address disguised to prevent spam)

[John D. McCalpin]

Caches have no trouble with long vectors (or short ones, for that matter).
They don't provide any advantages, but they are not a problem.

Caches do have trouble with indirect vectors that have poor spatial locality.

Perhaps surprisingly, many of the prototypical examples of algorithms that
make heavy use of indirect vectors perform reasonably well on cached systems.
In unstructured mesh codes, for example, the indirection vector contains
information about "nearby" points, so with good node numbering there is an
excellent chance that most of the data elements in a cache line will be used
before the line is evicted.

Bandwidth is a different problem, of course, as noted above.

Example 1: for the STREAM benchmark running on a Xeon E5-2680 (Sandy
Bridge EP), one socket sustains about 38 GB/s. The STREAM Triad kernel
requires 12 Bytes of memory traffic per FP operation (24 Bytes per multiply-
add), so this corresponds to 3.167 GFLOPS. The Sandy Bridge core can
execute four 64-bit multiply-add pairs per cycle, so one core running at
0.8 GHz could handle the arithmetic for the whole chip.

Example 2: The Intel Xeon Phi SE10P can sustain up to 174 GB/s on STREAM
Triad, corresponding to 14.5 GFLOPS. Each of the 61 cores on the Xeon Phi
SE10P has a peak performance of 17.6 GFLOPS for 64-bit multiply-add operations,
so only one core would be required to handle all the arithmetic.

STREAM is intended to be an extreme, but not pathological, test case.
All of the data in the cache lines is used, but used only once.
The "balance" parameter is 2 FLOPS / 3 Words = 0.67.
The ocean models that I was running when I developed STREAM tend to have
a little bit of re-use, with "balance" parameters in the range of
2 FLOPS / 1 Word = 2.0 --- about 3x the computational intensity of the
STREAM Triad kernel.
(Typical hardware platforms have corresponding "balance" parameters of
30-50 FLOPS / Word.)

Latency has scaled much more slowly than bandwidth. On the Xeon E5-2680,
I measure about 77 ns local memory latency, which corresponds to ~208
cycles at the nominal 2.7 GHz, or ~1660 64-bit floating-point operations.


As far as supercomputers are concerned, a major factor in the demise of
the traditional supercomputers was price. Cray vector systems had prices
in the $1M per processor range. These were largely displaced in the 1990's
by RISC-based servers with prices in the range of $25k-$50k per processor.
Since ~2004 the market has been dominated by clusters of systems based on
x86-64 processors, with prices that have stayed fairly flat at $2000-$3500
per socket.

[Ivan Godard]

Amen. We did a paper system for LLNL a couple of years ago. To feed a
4-core Mill running Livermore Loops-type loads we needed 0.5TB/s
bandwidth. Our partner, LSI Logic, could in fact get that much across
the pins - all 2700 of them. We decided to focus on HPC apps that
weren't bandwidth limited. It was an interesting exercise :-)

Ivan

[John D. McCalpin]

With select DDR3 DRAM parts operating in a single-rank configuration
(point to point with no active write termination) it is possible to
get DRAM power efficiency for open page accesses in the range of
20 mW/Gbs. Multi-rank systems (requiring active read and write
termination) and closed-page access can increase this to over
100 mW/Gbs.

Taking this as a reasonable range of DRAM power efficiencies, your
0.5 TB/s would require a minimum of 80W of DRAM power and a maximum
of over 400W of DRAM power.

This does not include the decidedly non-trivial power required in
your chip to drive the outbound commands, drive the outbound data,
sink the inbound data, and manage all the queueing required.
Note that 500 GB/s * 60 ns latency = 30,000 Bytes in flight,
or about 500 64-Byte cache lines.

For a real chip with bandwidth in the same ballpark (0.352 TB/s peak),
I am just starting to look at power measurements on Xeon Phi SE10P.
My first measurement showed a power increase of ~90 Watts above nominal
when running the STREAM benchmark at ~155 GB/s (average of the 4 kernels).
This is about 70 mW/Gbs, but this does include the power in the cores,
the uncore, and the memory subsystem. I need to find more documentation
to try to understand the details of the power breakdown....