Use of Mips/W/$ as a measure for architectural comparison

[Ivan Godard]

There has been a somewhat heated discussion on reddit
(http://www.reddit.com/r/programming/comments/1fz4v9/video_drinking_from_the_firehose_how_the_mill_cpu/)
about the merits of this measure.

Many posters seem to have misunderstood what I was comparing, which
points to a bug in the presentation that I will try to fix. Here's my
explanation, reposted from there:

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I'm the speaker in the video.

All the comments about the inaptness of Mips/W/$ for any particular
usage or purchase decision are totally correct. The measure is not a
measure of chips; it is a measure of design spaces.

There are many tradeoffs in chip design.

Some applications care only about performance; power consumption and
price are irrelevant. An example is the data-gather at the experiment
nodes in CERN, where you must keep up with very complex events that
happen very very fast, and if you can't keep up nothing else matters.

Some applications care only about power; performance and price are
irrelevant. "Smart dust" nano-processor sensors are an example; you only
have the power you can harvest from the antenna, and if you can't do
that then nothing else matters.

Some applications care only about price (front price, roughly equal to
area in mm2, not TCO); performance and power are irrelevant. Some toys
are examples; if you cannot beat the retail price point of a
non-computer alternative then nothing else matters.

In most applications all of performance, power, and price matter, with
varying weights.

When designing a particular chip, the implementation can skew in favor
of performance, price or power at the expense of the others. There is no
one-size-fits-all optimum; you tune to different markets. However, in
any particular architecture there is a limit to tuning in any of the
dimensions; the collective limits are called a design space. In a CPU
family, like an x86 generation, the IBM 360, or the Mill, the marketing
guys pick what they think is a market-optimum point within the design
space and the engineers tune to that point.

By picking three particular chips (one still vapor) as examples, I was
trying to give a sense of where the respective design spaces lie. At
best this is good only for an intuitive feeling for the shape of things.
To properly compare the architectures I would need to select at least
one chip optimized for performance, power, or price, for each family,
and probably more. That would have been an interesting talk, but that
wasn't what the actual talk was about.

Because the particular chips shown are only points in design space, if
your interest is a different point that the selection is inapt. If your
interest is in comparing spaces (which is all that matters when
comparing architectures as opposed to chips, and the talk was about
architecture, albeit having to use particular chip examples), then
Mips/W/$ is a reasonable measure.

In future presentations I may emphasize that the comparison is of
architectures; I had thought I had done so, but it appears not enough
for some listeners.

Ivan

[Quadibloc]

The reason for complaint is that the measure is dimensionally flawed,
and thus not independent of scale.

I don't see how it makes any difference if you're comparing
architectures instead of comparing specific implementations; in fact,
since you can't benchmark architectures, but only implementations, I'm
even having trouble making sense of that claim.

John Savard

[Ivan Godard]

In the design space there are three dimensions of interest: performance,
power, and area (which becomes cost at the fab, and price after markup).
There are other dimensions of course, but these three seem the most
important. These three dimensions form a vector space. For ease of
insight, we make "bigger is better" on all three dimensions and set the
scales to "Mips", "1/W", and "1/$".

Any given architecture at any given fab process has hard limits on each
of these dimensions. For the performance dimension, you can't run a
circuit at zero Hz, and you can't get it over some maximum rate either.
The Mill is necessarily a 64-bit design and must have at least four
function pipelines, which sets lower limits on area and power; the upper
limits are more fuzzy, being defined by diminishing returns when
scaling, but are present nonetheless. Other architectures have similar
limits, set by the nature of the architecture. These limits define a 3-D
solid in the design space; points within the solid can be implemented,
and points outside the solid cannot.

The dimensions interact in their limits; we call such interactions
"tradeoffs". Because of the interactions, the design space is not a
regular Euclidean solid, nor even necessarily convex. I think it will
always be strongly connected and topologically equivalent to a sphere
though; thinking about toroidal design spaces makes my head hurt.

As a 3-D solid, the design space has a volume, which has the
dimensionality of the product of the dimensions of the component
vectors, i.e. Mips/W/$. Each architecture occupies such a volume in the
space, which may or may not overlap with the volumes of other
architectures. Because we have defined the dimensions so that "bigger is
better", we can compare the volumes of different architectures by
judging that a larger volume is "better" than a smaller one, and one
further from the origin is "better" than one closer to the origin.
Depending on the actual shapes of the volumes and their degree of
overlap, "better" may be ambiguous.

Particular points in such a volume, i.e. a particular implementation of
the enclosing architecture, have the same dimensionality, defining a
vector rather than a volume. One can get a sense of the size, location
and shape of the design space of an architecture by looking at a set of
such points. Particularly informative are points furthest from the
origin in each dimension (what we call "high end") because the shapes
tend to be less fuzzy there than on the close-to-the-origin boundaries.

Design spaces like these, and the points within them, are useful only to
compare architectures in general. They are not suitable for comparing
individual chips for some particular need. To compare chips, say for a
buying decision, the customer must also define a "utility space" which
defines, at each point of the architecture space, the utility function
(i.e. actual value) of that combination of qualities for that particular
use for that particular user. The product of the design space and the
utility space maps each implementation point into a linear value
dimension, and the higher value wins.

The general utility space is not necessarily smooth or differentiable,
but in practice can be approximated with three functions, one for each
of the dimensions. "I need 120 Mips and a price point of $15, but past
that point then power is three times as important as price and eight
times as important as additional performance" is an example of a
functionally defined utility space.

The Mill is an architecture, not a single chip, and so we have been
concerned with creating an architectural space, not a single point. As a
commercial enterprise, our goal has been to stake out a portion of the
architecture space that is not already occupied by other architectures.
We have tried to place the Mill in the overall space such that, when
combined with utility functions typical of large markets, it will map to
higher value points than the offerings of our competitors. We call it a
"general purpose architecture" because we try to satisfy the centroid of
utility functions, at the cost of being less able for utility functions
that are further away.

The concept of a utility function is well-developed in economics,
although not commonly formalized in engineering design. For more on the
subject see http://en.wikipedia.org/wiki/Utility_function

Ivan

[MitchAlsup]

On Monday, June 10, 2013 9:22:28 PM UTC-5, Ivan Godard wrote:

MIPS/Watt seems to be a reasonable metric,
MIPS/$ also seems to be a reasonable metric,
MIPS/AREA seems to be a reasonable metric,
MIPS/Watt/$ does not--too many divisors.

I suspect that you are making the assumption that $==AREA which
anyone who has been in this industry for more than a few years
will tell you:: most of the time the relationship is tenuous at
best.

Mitch

[timca...@aol.com]

On Monday, June 10, 2013 10:22:28 PM UTC-4, Ivan Godard wrote:
>As a 3-D solid, the design space has a volume, which has the
>dimensionality of the product of the dimensions of the component
>vectors, i.e. Mips/W/$.

Wouldn't that be: Mips/(W*$) ?

- Tim

[Scot]

On Monday, June 10, 2013 10:22:28 PM UTC-4, Ivan Godard wrote:

> There has been a somewhat heated discussion on reddit
>
> (http://www.reddit.com/r/programming/comments/1fz4v9/video_drinking_from_the_firehose_how_the_mill_cpu/)
>
> about the merits of this measure.
>

The best I can grant this measure is to see it as the price of efficiency, which might be of interest to a rapidly growing company building data centers.

A builder of a data center would like to minimize energy consumed, but would also like to minimize capital expenditure.

Does this measure suggest where the sweet spot might be? If a typical chip/blade/server can be expected to last 10 years, how can this measure be altered in a way to help determine how to minimize total cost?

Maybe instead of Mips/W/$ you should simply give the price per operation assuming a reasonable price per KWH and expected chip lifetime in terms of total reliable ops.

[Quadibloc]

But that equals (Mips/W)/$.

John Savard

[MitchAlsup]

On Wednesday, June 12, 2013 7:13:13 PM UTC-5, Quadibloc wrote:
> But that equals (Mips/W)/$.

Not necessarily in FP arithmetic.

Mitch

[Terje Mathisen]

Mitch, we _are_ talking about simulations and ballpark numbers, not 0.5
ulp differences! :-)

Terje

--
- <Terje.Mathisen at tmsw.no>
"almost all programming can be viewed as an exercise in caching"

[Zeste]

You cannot use Mips/W/$ since $ is not a fixed unit: its value is
continuously changing, so you cannot measure something according to a
non fixed unit...

[John D. McCalpin]

On Tuesday, June 11, 2013 2:42:59 PM UTC-5, Ivan Godard wrote:
>
> The general utility space is not necessarily smooth or differentiable,
>
> but in practice can be approximated with three functions, one for each
>
> of the dimensions. "I need 120 Mips and a price point of $15, but past
>
> that point then power is three times as important as price and eight
>
> times as important as additional performance" is an example of a
>
> functionally defined utility space.

It is sort of entertaining and sort of sad to see how little progress the
computer architecture community has made in conceptualizing the basics
of cost and utility functions. As humans, we make complex trade-offs
many times every day based on combinations of boolean, linear, non-
linear, and "fuzzy" cost and utility functions.

The good news is that our brains can be quite adept at multidimensional
pattern recognition, bound analysis (typically thought of as "common
sense"), and analogical reasoning pertaining to these trade-offs.

The bad news is that these are not conscious activities and are therefore
not amenable to either personal or third-party inspection, review,
quantification, or validation.

The worse news is that these are also subject to bias due to psychological,
sociological, and biological factors that can be extremely difficult to
identify.

As noted below, "utility functions" are well-developed in some fields.
Unfortunately, quantitative utility functions that are easy to understand
are very seldom useful in practice, while quantitative utility functions
that are complex enough to represent real systems are either too complex
to understand or too specialized to be generalizable.

I still like the idea of working with quantifiable utility functions for
computer architecture studies (I think that the first talk I gave on this
was at the "First NASA Workshop on Performance-Engineered Information
Systems" in 1998), but not necessarily because I am going to use them
directly. The benefit that keeps bringing me back to these models is
that the act of making the list of constraints and trade-offs can trigger
"insight" -- that frustrating, but immensely useful, neurological function
that seems so difficult to replace....

john