A BIG problem with IEEE Floating Point...
Steve Richfie1d
dimensionality. However, the IEEE floating point standards arbitrarily
discard significance, even though the methodology of manipulating
significance was worked out in 1959. These methods are fairly simple -
in many cases simply omitting the post-normalization operation. However,
this can't be done when the position of the most significant bit is
assumed as in the IEEE standards.
Now, this universal absence of capability plays havoc on financial
predictive applications and some neural network applications, and may be
partially responsible for the present world economic problems via
short-circuiting predictions by losing the computable significance of
results.
I am a co-author of a peer-reviewed paper explaining these issues and
discussing some work-arounds at the upcoming AI conference in Las Vegas.
If you would like an advance draft of this paper, to get your early
comments in on, then please email me and I'll email you one.
It is my hope that something can be worked out to expand the IEEE
standards to incorporate significance, to avoid the ugly prospect of
having to completely replace the standard.
Steve Richfield
408-230-4935
Brian Inglis
Richfie1d <St...@NOSPAM.smart-life.net> wrote:
>All quantities have the properties of value, significance, and
>dimensionality. However, the IEEE floating point standards arbitrarily
>discard significance, even though the methodology of manipulating
>significance was worked out in 1959. These methods are fairly simple -
>in many cases simply omitting the post-normalization operation. However,
>this can't be done when the position of the most significant bit is
>assumed as in the IEEE standards.
If you don't like the properties of floating point arithmetic, feel
free to use fixed point arithmetic.
>It is my hope that something can be worked out to expand the IEEE
>standards to incorporate significance, to avoid the ugly prospect of
>having to completely replace the standard.
Consider commenting on IBM's IEEE decimal floating point arithmetic
proposal.
--
Thanks. Take care, Brian Inglis Calgary, Alberta, Canada
Brian....@CSi.com (Brian dot Inglis at SystematicSw dot ab dot ca)
fake address use address above to reply
Dale Morris
The IEEE currently has a technical committee which is considering future
extensions to the IEEE 754 floating-point standard. If you seriously wish to
work towards acceptance of the approach you mention here, the IEEE 754R
technical committee would likely be a much more efficacious forum in which
to present your arguments. The committee is also considering proposals for
decimal floating point and other things which might be relevant to your
interests.
I think that there's almost zero likelihood of an alternate, competing
floating-point standard arising and displacing the IEEE standard from the
market. So, I think that trying to influence the future direction of the
IEEE standard would be your best bet, if you are in earnest about wanting to
have some influence on what gets built.
Best of luck!
- Dale Morris
Itanium processor architect
Hewlett Packard
"Steve Richfie1d" <St...@NOSPAM.smart-life.net> wrote in message
news:d09622c276be9907...@news.teranews.com...
Steve Richfie1d
> The IEEE currently has a technical committee which is considering future
> extensions to the IEEE 754 floating-point standard. If you seriously wish to
> work towards acceptance of the approach you mention here, the IEEE 754R
> technical committee would likely be a much more efficacious forum in which
> to present your arguments. The committee is also considering proposals for
> decimal floating point and other things which might be relevant to your
> interests.
Yes, I've discussed this with some of those folks and read the new
decimal standard. They appear to have been laboring under some invalid
logic, that goes something like:
1. You can write library routines for many languages like C++ and Java
to do arithmetic any way you want. WRONG, because comparisons must
disrespect guard bits, that are needed in significance implementation.
Also, the I/O conversion routines all need to work differently,
exceptions need to be handled differently, etc. However, the BIGGEST
problem is that people writing predictive programs seldom use C++ or
Java, but instead prefer using proprietary packages and BASIC that stop
immediately whenever there is an arithmetic exception.
2. Interval arithmetic is much better than "significant digit" methods,
so why come up with a less-than-the-best standard based on "significant
digit" methods? What no one realized is that variations in the
correlations between argument errors is BIGGER than the problems with
the worst of the significance methods, so that in effect all
significance methods work about equally well.
3. Providing for un-normalized representation in the proposed decimal
standard was all that was really needed, without realizing that proper
guard digit handling in comparisons and I/O, special significance
handling in multiply and divide operations, etc., are also needed.
> I think that there's almost zero likelihood of an alternate, competing
> floating-point standard arising and displacing the IEEE standard from the
> market.
It has already happened in some narrow settings. IEEE is probably about
as good as you can do in 32 bits. However, I have hope for a much better
standard for double precision. Obviously, this would be much better WITH
the IEEE than against the IEEE.
> So, I think that trying to influence the future direction of the
> IEEE standard would be your best bet, if you are in earnest about wanting to
> have some influence on what gets built.
I agree. However, it appears that the best of ideas will die unless a
little heat is also applied.
> Best of luck!
Thanks. I'll obviously need it.
> Itanium processor architect
Hey, maybe you can answer an Itanium question for me that no one else
has been able to answer: Why are Itanium processors that are made on the
same FAB lines as Pentium processors only able to clock at half Pentium
speed (though with about the same MFLOPS due to architectural
advantages)? Theories abound, e.g. sloppy layout, too much cross-chip
communication from the architectural complexity, too much fanout from
all those registers, intentional slowdown to avoid direct competition
with the Pentium, etc. What is the REAL answer?
Also, have you looked at the antique IBM STRETCH computer. The
architecture is nearly identical to the Itanium, with some problems
(like the lack of a "guess bit" on conditional instructions) fixed by
the presence of a cache, but other problems (like the high cost of store
instructions) about the same on both processors. They match in countless
details, e.g. they even both have a floating multiply-accumulate
instruction - and with the same mnemonic! Never have I seen two designs
so separated in time come out so identically!
> Hewlett Packard
OK, why is an Itanium processor architect at HP?!
Thanks for your comments.
Steve Richfield
================
Steve Richfie1d
> If you don't like the properties of floating point arithmetic, feel
> free to use fixed point arithmetic.
Unfortunately, there are a LOT of details to deal with to make
significance methods work - too many to deal with in any one
application. Also, predictive programs that have these problems tend to
be large, so high-level language support is also needed. There ARE some
alternatives, like interval arithmetic that is possible in some
languages like C++ and Java, but these languages still mess up some
important details, like I/O.
> Consider commenting on IBM's IEEE decimal floating point arithmetic
> proposal.
Unfortunately, it has most of the same problems - only in decimal. While
they have an un-normalized representation, they don't respect guard
bits, and they don't do what they need to do to make significance work.
Even in decimal, 1.0/3.0 =! 0.333333333333... instead of 0.33 as it should.
Steve Richfield
Andrew Reilly
> Unfortunately, it has most of the same problems - only in decimal. While
> they have an un-normalized representation, they don't respect guard
> bits, and they don't do what they need to do to make significance work.
> Even in decimal, 1.0/3.0 =! 0.333333333333... instead of 0.33 as it should.
What field of endeavor requires the latter answer?
I've got a nice fixed-point library that could probably be tweaked
to be pressed into that sort of service, but it's certainly not what I
designed it for. (and it doesn't do division at all, at the moment...)
--
Andrew
Steve Richfie1d
>>Unfortunately, it has most of the same problems - only in decimal. While
>>they have an un-normalized representation, they don't respect guard
>>bits, and they don't do what they need to do to make significance work.
>>Even in decimal, 1.0/3.0 =! 0.333333333333... instead of 0.33 as it should.
> What field of endeavor requires the latter answer?
All computation should work this way, but sadly it doesn't. In the above
case, if you only know the arguments to 2 significant digits, then the
result can only be computed to two significant digits.
For example, your tank just ran out of fuel but there is another broken
down tank nearby that has fuel. You found a damaged 1-gallon container
that probably still holds about a gallon, give or take 1/10 of a gallon.
You transfer one container of fuel to your tank. Your tank gets pretty
close to ~1/3 of a mile per gallon. How far can you go on your ~1 gallon
of fuel? The answer is of course about 1/3 of a mile. No, there is no
need to break out the laser ranging equipment to measure this to the
nearest micron, as neither the fuel nor the MPG were known all that
well. Numerically, the answer is ~0.33 miles.
We all know how to manipulate significance like this in our daily lives,
and college physics and chemistry lab courses require such math to pass
them. However, neither computers nor calculators know how to preserve
significance, and so our society gets into a LOT of trouble when the
GIGO (Garbage In, Garbage Out) phenomenon sets in and we compute
something to 16 places when even the first digit is in doubt.
The area where this problem is at its worst is predictive computations
like financial and ecological prediction. There, the formulas are
available to predict things centuries into the future, but then
significance rapidly deteriorates with projection time so that the
predictions become worthless after a while. Wouldn't it be nice if
computers only output what could actually be reliably computed from
their inputs? THAT is what significance computation is all about. No,
your tank almost certainly will NOT go exactly 0.333333333333333333 miles.
Steve Richfield
Brian Inglis
Richfie1d <St...@NOSPAM.smart-life.net> wrote:
>Brian,
>
>> If you don't like the properties of floating point arithmetic, feel
>> free to use fixed point arithmetic.
>
>Unfortunately, there are a LOT of details to deal with to make
>significance methods work - too many to deal with in any one
>application. Also, predictive programs that have these problems tend to
>be large, so high-level language support is also needed. There ARE some
>alternatives, like interval arithmetic that is possible in some
>languages like C++ and Java, but these languages still mess up some
>important details, like I/O.
If you don't like the I/O in C++ and Java, you can override it by
providing your own.
>> Consider commenting on IBM's IEEE decimal floating point arithmetic
>> proposal.
>
>Unfortunately, it has most of the same problems - only in decimal. While
>they have an un-normalized representation, they don't respect guard
>bits, and they don't do what they need to do to make significance work.
>Even in decimal, 1.0/3.0 =! 0.333333333333... instead of 0.33 as it should.
Have you had a look at Mike Cowlishaw's decimal arithmetic software
library?
Perhaps if you used that as a (design) base, defined the additional
properties you require, and added implementations for them, released
it as an Open Source library implementation, and got some of the
applications developers who need this facility to use it and propagate
it, you might be able to get IEEE, hardware, and compiler folks
interested.
Criticism that some standard, software, or hardware, does not do what
you want it to do, rather than failing in what it is intended to do,
is not likely to draw support.
Discussing what properties need to be changed, how that may be
implemented effectively, and changes that would be required to the
standard, software, or hardware, may garner some interest.
Steve Richfie1d
> If you don't like the I/O in C++ and Java, you can override it by
> providing your own.
Yes, but then there are the comparison operators, etc. Then, you finally
get a language working that no self-respecting financial analyst would
dare associate with - or even knows how to program. YES, you can fix ALL
of the reasons that C++ and Java don't work in a financial environment,
but then you must retrain people to program in these. Of course, that
isn't so hard, but first you must motivate them. Hmmm. Politics.
> Have you had a look at Mike Cowlishaw's decimal arithmetic software
> library?
> Perhaps if you used that as a (design) base, defined the additional
> properties you require, and added implementations for them, released
> it as an Open Source library implementation, and got some of the
> applications developers who need this facility to use it and propagate
> it, you might be able to get IEEE, hardware, and compiler folks
> interested.
Now, there IS an interesting idea. My first "mainframe" computer was an
IBM-650, a vacuum tubed decimal computer, and I've always retained a
like for decimal arithmetic, due in part to its "nothing hidden" aspect.
However, this'll have to wait for an application that can support the
additional overhead of developing and debugging a complete significance
arithmetic system.
> Criticism that some standard, software, or hardware, does not do what
> you want it to do, rather than failing in what it is intended to do,
> is not likely to draw support.
There is a "prime directive" statement within the IEEE spec, that the
spec itself clearly fails to achieve due to the lack of significance. My
applications have nothing special about them, except for a greater
sensitivity to these shortcomings.
> Discussing what properties need to be changed, how that may be
> implemented effectively, and changes that would be required to the
> standard, software, or hardware, may garner some interest.
Good points that are well taken. All of these approaches start with a
modified spec, which is what appears to be needed to proceed. I should
probably prepare one.
BTW, I and two other people (Russ Hansberry and Wayne Albin) tried and
failed to get some of the more horrendous bugs out of the "C" spec
before it first came out. Perhaps you remember that the original ANSI
standard was a year late - that was us. In the end, 100,000 too-early
printed copies of the spec overcame all logic.
Now, C and its descendant languages (like C++) STILL have the wrong
order of operations, so that if you want logical expressions to work
correctly, you must fill them with should-be redundant parenthesis.
Early in the one-year delay, Russ offered Denis Ritchie to drop our
objections to the proposed C standard if Denis could find one single
statement, in any code, that required the defective order of operations
to execute correctly. Denis was unable to do so.
The bottom line here is that there is something broken behind the
eyeballs of every tournament chess player and standards person that I
have yet met (many names withheld). They do what they do for ego and
self-agrandisement, and definitely NOT for the good of others.
Since then, I have abandoned trying to work "logically" in such
environments, where logic seems to be the LAST concern.
A few years ago, someone approached me about working on the IEEE FP
standards committee. "Oh good, then now maybe I can get some of its many
bugs fixed." "Bugs! What bugs?" I then discussed significance arithmetic
and some of the missing non-numeric special forms. "I'll get back to you
later on this." Of course, I've never heard a thing more about this.
I used to maintain a folder of the people who had since been killed from
the bugs in the ANSI C standard that we tried to get fixed, but this was
a depressing exercise and really, no one else was interested.
So, I'm waiting for some evidence of "a light in the attic" and a
willingness to address IEEE FP shortcomings by those who can affect it.
Until then, all I can do is to periodically apply a little heat to the
wound.
Steve Richfield
Dale Morris
> > The IEEE currently has a technical committee which is considering future
> > extensions to the IEEE 754 floating-point standard. If you seriously
wish to
> > work towards acceptance of the approach you mention here, the IEEE 754R
> > technical committee would likely be a much more efficacious forum in
which
> > to present your arguments. The committee is also considering proposals
for
> > decimal floating point and other things which might be relevant to your
> > interests.
>
> Yes, I've discussed this with some of those folks and read the new
> decimal standard. They appear to have been laboring under some invalid
> logic, that goes something like:...
I agree with you that it's hard to get as much control over exceptions and
conversions as you'd like. And I don't mean to discourage you from
discussing these issues on comp.arch - far from. I think this is a good and
useful place for such discussions. I only wanted to suggest that in terms
of making the rubber meet the road, it will also be important to take your
arguments to the folks who are doing the definitional work for the
appropriate standards. IEEE 754R is one of them, although I suspect you
would also do well to get yourself hooked into language standards
discussions as well.
> > I think that there's almost zero likelihood of an alternate, competing
> > floating-point standard arising and displacing the IEEE standard from
the
> > market.
>
> It has already happened in some narrow settings. IEEE is probably about
> as good as you can do in 32 bits. However, I have hope for a much better
> standard for double precision. Obviously, this would be much better WITH
> the IEEE than against the IEEE.
I guess it depends on what you would like to accomplish. If you only want
to influence somebody to build something that embodies the directions you
suggest, then that may not need to involve the IEEE standard. But, there's
much more leverage behind 754, and if you'd like this sort of capability in
a specific kind of machine (a PC for individual research, or a huge HPC
cluster, or something else specific), then influencing 754R is your best
bet, I think.
> > So, I think that trying to influence the future direction of the
> > IEEE standard would be your best bet, if you are in earnest about
wanting to
> > have some influence on what gets built.
>
> I agree. However, it appears that the best of ideas will die unless a
> little heat is also applied.
Certainly true.
> > Best of luck!
>
> Thanks. I'll obviously need it.
>
> > Itanium processor architect
>
> Hey, maybe you can answer an Itanium question for me that no one else
> has been able to answer: Why are Itanium processors that are made on the
> same FAB lines as Pentium processors only able to clock at half Pentium
> speed (though with about the same MFLOPS due to architectural
> advantages)? Theories abound, e.g. sloppy layout, too much cross-chip
> communication from the architectural complexity, too much fanout from
> all those registers, intentional slowdown to avoid direct competition
> with the Pentium, etc. What is the REAL answer?
I'll be happy to give you some of my thoughts on this. If this spawns a
whole thread on analysing my comments, etc., I'm afraid I won't have the BW
to get into an involved discussion. But here's some of my thinking.
First off, although frequency is certainly important as one technology
driver of performance, it's really performance that we all should really
care about. Unfortunately, performance is a fuzzier concept than GHz, and
the human condition being what it is, people prefer (whenever possible and
even when it's not possible) to substitute simpler, easier-to-understand
concepts for more complex ones. (This isn't a bad thing, in general - a
good simplification is one of the tools that helps us to reason about
increasingly complex systems.)
Intel, in particular, has fueled this frequency==performance equation, in
their marketing messages, and it has served them very well. Or until
recently, perhaps I should say.
If you want to get a really high GHz number that you can crow about, there
are some fairly straight-forward implications for your design.
Fundamentally, frequency is determined by the process technology and by how
much "work" you try to accomplish in a single cycle. (By "work" here, I
mean number of levels of gate delays.) So, if you design your pipeline with
relatively few levels of logic in each cycle (and hence a fairly long
pipeline), and then do the normal good work of tuning paths, etc., you'll
get to a higher frequency than if you'd opted for a shorter pipe with more
levels of logic per cycle.
Partly because of Intel's long-standing marketing message of the importance
of frequency, their design teams have been pushed into doing longer pipes
with thinner stages to get to higher frequencies. Again, this has not been
bad for them, since the higher frequencies have been a great selling point
for them.
But, if you focus on delivered performance instead, then the tradeoffs
you're faced with are different. How much work per cycle is the optimal
choice in terms of delivered performance? If there were no downsides to
making thin pipe stages, then the answer would clearly be the thinnest pipe
stage you can make. But, there are downsides - branch misprediction
penalties, stall implementation (which is complex to pipeline), power
efficiency (the clock tree alone consumes between 10 and 40% of power in
modern, high-end processors), clock skew grows as a percentage of the cycle
time, etc.
So, for the current Itanium processors, a point on this tradeoff curve was
chosen, mostly based on peformance (and not on GHz bragging rights). Within
Intel's x86 processors, the Pentium 4 and Pentium M teams obviously chose
different points on this tradeoff curve as well.
Interestingly, I think that the top GHz choice on this curve, which used to
be a pretty good choice (especially if you marketed the heck out of it), has
become much less attractive. This is mostly due to power efficiency and
scalability (in my view), and I think these things are likely central in
Intel's choice to move strategically to the M core.
Besides pipe length vs. pipe stage thickness, there's also the execution
width of current Itanium processors, in terms of numbers of parallel
functional units. (Really you can think of this as the same sort of
frequency vs. how much work you get done per cycle, except now the "work"
we're talking about has to do with length of wires for data bypass, etc.)
And, there's also the issue of how many times the core has been tweaked.
When we started the Itanium program, there was a lot of new stuff. So, that
means that you can't hyper-optimize absolutely everything on the first
processor - there's just too much to do. So, not surprisingly, the first
revisions were not as highly tuned as the Pentium 4 core is, since they've
had a lot of time to wring every bit of frequency from it. As we go forward
with the Itanium program, you'll see the results of having more time to tune
stuff. In fact, if you look at the frequency/time curve for Merced to
McKinley to Madison, it's a lot faster than Moore's law. There's always
headroom in a brand new design for this kind of tuning - especially when the
architecture is new, and there's much learning headroom about how best to
implement it as well.
The results of all this have been the real proof, in terms of actual
delivered performance. And, for HP anyway, our customers buy machines from
us because they solve their problems better, and not so they can feel good
about the frequency of the crystals in the box. Even despite having some
as-yet-unused headroom in terms of tuning opportunities, we've still done
pretty darn well in terms of performance. And, of course, there's more to
come along these lines.
> Also, have you looked at the antique IBM STRETCH computer. The
> architecture is nearly identical to the Itanium, with some problems
> (like the lack of a "guess bit" on conditional instructions) fixed by
> the presence of a cache, but other problems (like the high cost of store
> instructions) about the same on both processors. They match in countless
> details, e.g. they even both have a floating multiply-accumulate
> instruction - and with the same mnemonic! Never have I seen two designs
> so separated in time come out so identically!
Sure. I don't know that I would characterize them as so identical. Itanium
takes a huge amount from VLIW and superscalar learnings that have accrued
since the time of the Stretch machine.
> > Hewlett Packard
>
> OK, why is an Itanium processor architect at HP?!
Itanium processor architecture was co-developed by HP and Intel, and is
still being co-developed. Much of the text you read in the Software
Developer's Manual for Itanium was written by HP architects, including Jim
Hull and me.
> Thanks for your comments.
You're welcome. Again, best of luck!
Andrew Reilly
Thanks for the explanation.
I agree that that sort of analysis is useful (essential). In fact I do it
pretty frequently, myself, when designing audio signal processing
algorithms.
Maybe floating-point-with-significance will work for you. Maybe interval
arithmetic will do too. Can't you manage either of those in
object-oriented languages by defining an appropriate new type that
contains the additional information?
My main argument against these approaches is that for many
algorithms, the significance of the numbers is statically determinable,
and *independent* of the values of the numbers. Certainly, all of the
ones that I generally deal with. That is, I can know that I have 16
significant bits in my input data, and my algorithm corrodes nine bits but
by using 24-bit words I can achieve 15 significant digits in the result.
In audio processing we usually talk about all of that in terms of SNR, but
that's just the nature of the endeavor, I think. (Often elaborated into
worst-case SNR and average (or RMS) SNR, if these differ.)
That is, I don't need to tag every input value with its significance, and
I don't need to compute the significance of every result, because these
are determined by the data acquisition method/format and the algorithm,
respectively.
Are your algorithms really such that you can't make similar determinations
independent of the data values?
--
Andrew
Steve Richfie1d
> Maybe floating-point-with-significance will work for you. Maybe interval
> arithmetic will do too. Can't you manage either of those in
> object-oriented languages by defining an appropriate new type that
> contains the additional information?
I'd almost be willing to bet that there is not one financial analysis
person who is familiar with object oriented programming! There's
probably one hiding somewhere, but this guy wouldn't make a market.
> My main argument against these approaches is that for many
> algorithms, the significance of the numbers is statically determinable,
> and *independent* of the values of the numbers. Certainly, all of the
> ones that I generally deal with. That is, I can know that I have 16
> significant bits in my input data, and my algorithm corrodes nine bits but
> by using 24-bit words I can achieve 15 significant digits in the result.
> In audio processing we usually talk about all of that in terms of SNR, but
> that's just the nature of the endeavor, I think. (Often elaborated into
> worst-case SNR and average (or RMS) SNR, if these differ.)
>
> That is, I don't need to tag every input value with its significance, and
> I don't need to compute the significance of every result, because these
> are determined by the data acquisition method/format and the algorithm,
> respectively.
>
> Are your algorithms really such that you can't make similar determinations
> independent of the data values?
Most of the time, the significance is known and fixed as you say, but
not always. The problem is that you deal with large lists of numbers to
crunch, whereas the financial analysis guys deal in countless pages of
formulas dealing with thousands of input values. It would be a
horrendous imposition to make them code all of these significances,
rather than just having them enter as much as they know.
Of course, if you do the significance analysis on the formulas then you
could figure out how many digits to display, but if you are going to do
all of this analysis by hand, then who needs a computer?!
Significance analysis is NOT simple. It is much easier to do it along
with the computations than aside from them. Note that many functions
gain and/or lose significance, e.g. Cos(~0) has very high significance,
while (~0)^2 has very low significance. Taking the sqrt of a number
doubles the significance of the argument, etc.
Steve Richfield
Steve Richfie1d
Thanks for your Itanium discussion. It was exactly what I was looking
for and it answered my questions.
Your explanation of the HP/Intel relationship gave me an idea:
If some company, e.g. AMD, Intel, HP, etc., were to jump on the
significance bandwagon, then it would give them a tremendous feature
(other than speed) to sell their products that wouldn't be enjoyed by
other companies unless/until they eventually caught up, which could take
quite a while.
The entire 3-man design team from Remote Time-Sharing Corp is still
around and looking for work in this dreadful economy. They are all quite
experienced with the ins and outs of significance arithmetic in the real
world, as they moved from development to support when their significance
system was completed and commercialized. Hiring/contracting these three
people would give a company a complete monopoly on experience in this area.
1. Do you think that anyone at HP/Intel might be interested in doing
this? Who?
2. Which of your competition do you think might be interested?
Thanks for your many helpful comments and suggestions.
Steve Richfield
Brian Inglis
Richfie1d <St...@NOSPAM.smart-life.net> wrote:
>The entire 3-man design team from Remote Time-Sharing Corp is still
>around and looking for work in this dreadful economy. They are all quite
>experienced with the ins and outs of significance arithmetic in the real
>world, as they moved from development to support when their significance
>system was completed and commercialized. Hiring/contracting these three
>people would give a company a complete monopoly on experience in this area.
Who were their customers? Are none of them interested?
Dale Morris
> Dale,
>
> Thanks for your Itanium discussion. It was exactly what I was looking
> for and it answered my questions.
You're most welcome.
> Your explanation of the HP/Intel relationship gave me an idea:
>
> If some company, e.g. AMD, Intel, HP, etc., were to jump on the
> significance bandwagon, then it would give them a tremendous feature
> (other than speed) to sell their products that wouldn't be enjoyed by
> other companies unless/until they eventually caught up, which could take
> quite a while.
I'm not really an expert in numerics, but I'm not really aware of any
significance bandwagon. Are there a lot of customers who understand this
and would base some significant portion of their buying decisions on such
capabilities?
I agree that it's interesting to look for differentiators that help solve
real problems. However, I'm not sure that the market really has much
appreciation for these types of capabilities. Most folks are happy just to
compute some answer and be blissfully unaware of the signficance of the
result (or even if the algorithm is buggy!)
Itanium systems (for example), because the FP datapath is based on extended
precision and not just double, are able to compute intrinsic functions (like
sin, cos, power, etc.) more accurately (closer to a maximum error of 1/2
unit-in-last-place) than competiting systems. However, I'm not sure that
this capability has helped us sell one more Itanium system than we would
have otherwise. (Perhaps I'm wrong there, but I've had a hard time hearing
any music emanating from the 1/2 ULP bandwagon.)
I don't mean to argue that it's not useful for folks to work on things like
computing significance. Just that I'm not sure one can look to the market
to provide the motivation for such work. And, again, without a pent-up
market, and given the market's love of the standardization with 754, I think
that standards groups are the best place to take your argument.
> The entire 3-man design team from Remote Time-Sharing Corp is still
> around and looking for work in this dreadful economy. They are all quite
> experienced with the ins and outs of significance arithmetic in the real
> world, as they moved from development to support when their significance
> system was completed and commercialized. Hiring/contracting these three
> people would give a company a complete monopoly on experience in this
area.
>
> 1. Do you think that anyone at HP/Intel might be interested in doing
> this? Who?
> 2. Which of your competition do you think might be interested?
>
> Thanks for your many helpful comments and suggestions.
You're very welcome.
Steve Richfie1d
>>The entire 3-man design team from Remote Time-Sharing Corp is still
>>around and looking for work in this dreadful economy. They are all quite
>>experienced with the ins and outs of significance arithmetic in the real
>>world, as they moved from development to support when their significance
>>system was completed and commercialized. Hiring/contracting these three
>>people would give a company a complete monopoly on experience in this area.
> Who were their customers? Are none of them interested?
That was 30 years ago - so I suspect that most of our customers are dead
by now. I've remained friends with one, a former VP from Tally Corp.,
but his wife set up her own corporate empire that my friend is now VP
of. Unfortunately they don't do financial projections of any complexity
there.
Bill Gates and Paul Allen as students handled our business with Lakeside
School. When I first met Paul, I thought that he was a teacher, as he
had grown sideburns at a very young age. Bill was always VERY quiet. I
had promoted/demoted myself from developer to salesman, and Lakeside
School was only a mile or so from my home in North Seattle, so I spent a
bunch of spare time there helping those kids out.
8 years ago, Bill paid the way to Seattle for me and the rest of the
team to meet with the members of his R$D division. There was a major
team-wide chemistry mismatch, as our event horizon was a lot further out
than theirs. They were working on whiz-bangs, many of which are now well
known, whereas we were more interested in the core structure
(significance and lots of other things) to support MUCH more ambitious
projects.
I suspect that I'll hear from Bill again if/when significance ever
starts to really take off. However, until then, "pushing" on Microsoft
is like pushing on a rope.
Steve Richfield
Steve Richfie1d
> I'm not really an expert in numerics, but I'm not really aware of any
> significance bandwagon. Are there a lot of customers who understand this
> and would base some significant portion of their buying decisions on such
> capabilities?
It's not just how many, but who. I believe that there IS an important
minority of PC purchasers, maybe 5%, who would view this as an absolute
necessity if it were available. I suppose that the next step would be to
measure this subset. Hmmm, don't you have some marketing folks there at
HP whose job it is to track such things?! Perhaps you could have one of
them contact me? Some questions on a periodic routine survey should
answer this question to everyone's satisfaction.
> I agree that it's interesting to look for differentiators that help solve
> real problems. However, I'm not sure that the market really has much
> appreciation for these types of capabilities. Most folks are happy just to
> compute some answer and be blissfully unaware of the signficance of the
> result (or even if the algorithm is buggy!)
I agree that the majority of users don't care if it doesn't affect their
video games and MS WORD usage. However, it WOULD affect Excel in some
important ways that might be noticed.
I think that to most users, this will be like the now-famous Floating
Point Divide Bug, that never hurt anyone, but everyone was very careful
NOT to purchase a processor with the bug. Once it becomes well-known
that Intel's (or whoever's) processors are the only acceptable
processors for government agencies and financial gurus, that other
processors will become like the processors with the Divide Bug - they
won't hurt most people, but no one will want them.
> Itanium systems (for example), because the FP datapath is based on extended
> precision and not just double, are able to compute intrinsic functions (like
> sin, cos, power, etc.) more accurately (closer to a maximum error of 1/2
> unit-in-last-place) than competiting systems. However, I'm not sure that
> this capability has helped us sell one more Itanium system than we would
> have otherwise. (Perhaps I'm wrong there, but I've had a hard time hearing
> any music emanating from the 1/2 ULP bandwagon.)
Anyone who cares about getting the right answer has already gone to
double precision, and there, such trivial errors can't be noticed in any
application that I know of. I used to maintain and migrate the CAD
systems at Boeing. There, if they could get 0.001" accuracy at a wingtip
they were ecstatic. 36-bit IBM-7094 arithmetic was just barely good
enough to use (but not quite 0.001" at a wingtip), 32-bit IBM-360 was
completely worthless, but 64-bit IBM-360 produced excellent results,
even with its hexadecimal normalization and complete lack of rounding or
guard bits. Unfortunately, 64-bit IBM-360 was just too slow to ever get
anything useful done. Hence, for years, Boeing had to maintain two
groups of computers, IBMs to run their business, and CDCs to design
airplanes.
>
> I don't mean to argue that it's not useful for folks to work on things like
> computing significance. Just that I'm not sure one can look to the market
> to provide the motivation for such work. And, again, without a pent-up
> market, and given the market's love of the standardization with 754, I think
> that standards groups are the best place to take your argument.
From my experience with several standards groups, they seem to see that
part of their place in life is to "defend" their standards from
essential improvements once they start to be used. I and two others
attempted to get the worst of the bugs in the C standard fixed, e.g. the
wrong order of operations, but there was no way that such reason could
ever prevail against the glut of products that already incorporated the
bugs. Since then, several people have died in connection with the order
of operations bugs in C, yet still the standard specifies the
dysfunctional order of operations.
Again, maybe if I could talk to a marketing guy at HP, we might be able
to close on a good estimation of market size and close on this issue.
Thanks yet again for some good comments.
Steve Richfield
Andrew Reilly
On Thu, 27 May 2004 20:36:10 +0000, Steve Richfie1d wrote:
> I agree that the majority of users don't care if it doesn't affect their
> video games and MS WORD usage. However, it WOULD affect Excel in some
> important ways that might be noticed.
>
> I think that to most users, this will be like the now-famous Floating
> Point Divide Bug, that never hurt anyone, but everyone was very careful
> NOT to purchase a processor with the bug. Once it becomes well-known
> that Intel's (or whoever's) processors are the only acceptable
> processors for government agencies and financial gurus, that other
> processors will become like the processors with the Divide Bug - they
> won't hurt most people, but no one will want them.
Now here you've stepped into a weird place, which relates to not just the
ability to store and process significance at the hardware level, but the
interaction of that with the language and application software levels.
There is currently no notion of significance in C, C#, C++ or Pascal (the
languages that have historically been used to code Excel, I believe),
although an appropriate type could be written as a class for C++, as I
mentioned before. In any of these non-significance languages,
double foo = 1.0/3;
double bar = 0.1/3;
or the language equivalent, had better continue to produce the binary
representation closest to 0.3333... and 0.033333... respectively, or else
all the existing code will simply stop working at all. It won't work
"better" in a significance sense unless the application code (and perhaps
the languages) are modified around that new paradigm. This is not like
the FDIV bug at all.
So if you have to re-write the application software to give access to this
supposed new hardware feature, you can also re-write (with the aid of the
appropriate library) to provide the application feature on the
contemporary hardware that is available today. The demand for that
application feature can then drive demand for hardware acceleration of the
feature: your significance FPU extension.
Compare this to support for other numeric types, like large integers,
which have always been possible, sometimes available via library,
sometimes in the language (lisp), sometimes helped along by appropriate
hardware changes, hardly ever mainstream until now. Now high performance
extended precision integers are essential, thanks to ubiquitous use of
cryptography.
Cheers,
--
Andrew
Steve Richfie1d
> Now here you've stepped into a weird place, which relates to not just the
> ability to store and process significance at the hardware level, but the
> interaction of that with the language and application software levels.
>
> There is currently no notion of significance in C, C#, C++ or Pascal (the
> languages that have historically been used to code Excel, I believe),
> although an appropriate type could be written as a class for C++, as I
> mentioned before. In any of these non-significance languages,
> double foo = 1.0/3;
> double bar = 0.1/3;
> or the language equivalent, had better continue to produce the binary
> representation closest to 0.3333... and 0.033333... respectively, or else
> all the existing code will simply stop working at all.
Not true! The new mode of operation would be activated with a special
instruction. Until then, things would work as at present. Then, when
activated...
While the absence of significance is a mathematical weirdity, I've seen
very little code that actually depends on it, except for mathematical
library and I/O functions that often do bit manipulation on the floating
point numbers.
My favorite programming trick is to search for the leftmost bit in a
string by doing a normalized floating point subtraction of the bit
pattern needed to remove the "hidden" bit. The remainder of the mantissa
will then be normalized, with the most significant bit disappearing into
the hidden position. You can then find out which bit it was by examining
the exponent byte of the result. What wonders we can do when we use
floating point arithmetic instruction for smart bit manipulation!
> It won't work
> "better" in a significance sense unless the application code (and perhaps
> the languages) are modified around that new paradigm.
Done right, most applications will work OK/better. However, the
languages will definitely need modification. To illustrate, suppose you
are displaying with an F10.4 format, and there aren't 4 significant
digits after the decimal point. Then, only the significant digits should
be displayed.
Comparisons may or may not need repair, depending on how they are done,
as >0, =0, and <0 are no longer exclusive conditions.
> This is not like
> the FDIV bug at all.
Since the processors would work EXACTLY as they do now unless/until an
instruction is issued to change modes, current software would continue
to work just as badly as it now does. Hence, the impact would be pretty
much as with the FDIV bug.
>
> So if you have to re-write the application software to give access to this
> supposed new hardware feature, you can also re-write (with the aid of the
> appropriate library) to provide the application feature on the
> contemporary hardware that is available today. The demand for that
> application feature can then drive demand for hardware acceleration of the
> feature: your significance FPU extension.
Many/most of the applications are already stressed for performance.
Further, this is NOT a casual thing - it is a significant development
project. I don't know of anyone (certainly not me) who is willing to do
this "for free" or "on the come" without the pieces being in place to
propagate this everywhere.
> Compare this to support for other numeric types, like large integers,
> which have always been possible, sometimes available via library,
> sometimes in the language (lisp), sometimes helped along by appropriate
> hardware changes, hardly ever mainstream until now. Now high performance
> extended precision integers are essential, thanks to ubiquitous use of
> cryptography.
I already have a long list of shortcomings in the IEEE standard. Briefly:
1. No guard digit.
2. Missing NAN (Not A Number) forms.
3. Nothing to indicate when a result is fundamentally non-numeric, e.g.
the square root of a negative number.
4. No short (16-bit) logarithmic form for DSP and NN applications.
5. No format or version indication to allow for future growth. THIS is
the critical missing piece that makes all major repairs onerous.
Hmmm, maybe I should add long integers to this list?!
There have been a couple of computers in history, the IBM 1401 and 1620
computers, that were hard-wired to handle indefinite-length numbers. The
decimal arithmetic instruction on current IBM mainframe computers can
almost (but not quite) do this. Further, if your floating point works
properly, there is no reason that you can't do integer computations in
FP, except that so far no FP has held big enough numbers for practical
crypto use. Long integer support sounds like a good thing, but I'm not
sure how best to work it in to the present (defective) FP structure. Any
ideas?
Thanks for your thoughts.
Steve Richfield
Fred J. Tydeman
>
> > It won't work
> > "better" in a significance sense unless the application code (and perhaps
> > the languages) are modified around that new paradigm.
As I recall (it has been 15 years, so I am fuzzy), PL/I had some
"strange"
rules for computing significance of computations using the basic
(+,-,*,/)
operators. I recall reading a few books saying how bad it was and
trying
to explain it.
> I already have a long list of shortcomings in the IEEE standard. Briefly:
> 1. No guard digit.
I believe IEEE-754 FP has guard, round, and sticky bits.
> 2. Missing NAN (Not A Number) forms.
What forms?
> 3. Nothing to indicate when a result is fundamentally non-numeric, e.g.
> the square root of a negative number.
The FPU raises invalid and returns a NaN. A NaN indicates non-numeric.
> 4. No short (16-bit) logarithmic form for DSP and NN applications.
> 5. No format or version indication to allow for future growth. THIS is
> the critical missing piece that makes all major repairs onerous.
> There have been a couple of computers in history, the IBM 1401 and 1620
> computers, that were hard-wired to handle indefinite-length numbers.
IBM 1620 floating-point is variable length, but is limited to 99 digits
(if I remember correctly).
---
Fred J. Tydeman Tydeman Consulting
tyd...@tybor.com Programming, testing, numerics
+1 (775) 287-5904 Vice-chair of J11 (ANSI "C")
Sample C99+FPCE tests: ftp://jump.net/pub/tybor/
Savers sleep well, investors eat well, spenders work forever.
Steve Richfie1d
> Vice-chair of J11 (ANSI "C")
Hey, did they ever do anything about the many issues that held up the
original ANSI C ratification, or were these all conveniently forgotten?!
As you may recall, three people, including Russ Hansberry, Wayne Albin,
and myself, sought to fix 19 of the worst bugs in the original ANSI C
spec, which held it up for a year. However, no power on earth could
overcome 100,000 too-early printed copies of the spec sitting in the
warehouse.
The worst of the bugs was the incorrect order of operations, which
forced people to put redundant parentheses into many expressions to get
them to work right.
Denis Ritchie expressed concerned that fixing the spec might break
existing code, so Russ offered to drop all of our objections and allow
the spec to go through without further delay if Denis could produce one
single statement from production code that actually required the
defective order of operations to function correctly, but Denis was
unable to do this.
Russ then proposed a VERY reasonable compromise - to keep the defective
spec, but issue a warning message whenever a statement was encountered
that impinged on its bugs, but that was also rejected. Everyone then dug
their heels in - and THAT is how things got held up for a year.
This has for years been my "poster child" for everything that is wrong
with the standards process - a story that I have probably told a hundred
times. It would indeed be interesting to see what, if anything, ever
happened regarding these many issues, or have they just remained
out-of-sight, out-of-mind, with the spec still having an erroneous order
of operations and the other problems?
I've BCC'd (to hide their email addresses from the span engines) this
message to Russ and Wayne, just to advise them of our contact. They may
choose to join in on this discussion.
I'll have to do my homework to respond accurately to some of the several
interesting points on IEEE FP you raised in your posting, so it may be a
bit before I get back to you on them.
Terje Mathisen
> Dale,
>
>> I'm not really an expert in numerics, but I'm not really aware of any
>> significance bandwagon. Are there a lot of customers who understand this
>> and would base some significant portion of their buying decisions on such
>> capabilities?
>
> It's not just how many, but who. I believe that there IS an important
> minority of PC purchasers, maybe 5%, who would view this as an absolute
> necessity if it were available. I suppose that the next step would be to
You can make this available quite easily, although at a performance cost
of up to 10X for heavy fp apps, by writing a library that carries
significance in bits or digits along with the operands, and does
rounding after every operation.
Write it in C++, and you could use operator overloading to compile
existing code with it.
Should be an easy afternoon job.
Afterwards, show the results to some of those 5% people and check their
reactions. My bet would be that when you demonstrate that i.e. their
spreadsheet results have no significant digits at all, they'll be less
interested.
Terje
PS. I do agree that C really messed up operator priorities, my personal
fix has been to _always_ use () around every sub-expression just to make
the desired result absolutely clear.
--
- <Terje.M...@hda.hydro.com>
"almost all programming can be viewed as an exercise in caching"
Steve Richfie1d
> As I recall (it has been 15 years, so I am fuzzy), PL/I had some
> "strange"
> rules for computing significance of computations using the basic
> (+,-,*,/)
> operators. I recall reading a few books saying how bad it was and
> trying
> to explain it.
I tried Google, but didn't find anything among the countless hits. They
probably did something wrong. Significance worked great as the ONLY
arithmetic on a commercial time-sharing service for the two years that
the system was in operation.
>>I already have a long list of shortcomings in the IEEE standard. Briefly:
>>1. No guard digit.
> I believe IEEE-754 FP has guard, round, and sticky bits.
NOT in the representational format, but only in the ALU. When you print
out a number, every 3.3 bits becomes another decimal digit, right to the
end of the number. Hence, the last digit gets pretty ragged from
computational noise. This isn't bad if there weren't very many
computations, but if there were ...
>>2. Missing NAN (Not A Number) forms.
> What forms?
Infinity with unknown sign, + indefinite, - indefinite, indefinite with
unknown sign, underflow with unknown sign.
All but the last are now coded as the very same NAN.
>>3. Nothing to indicate when a result is fundamentally non-numeric, e.g.
>>the square root of a negative number.
> The FPU raises invalid and returns a NaN. A NaN indicates non-numeric.
Yet another use of the same old NAN.
By failing to provide important differentiating information in the NAN,
even though there are 23 bits to do this with, it is often not possible
to tell whether a bug is just an artifact of the arithmetic, or a real
genuine programming error like taking the square root of a negative
number. Hence, most current use of double precision is NOT because the
precision is needed, but to avoid potential numerical artifacts that the
ubiquitous NAN has made so hard to debug.
>>4. No short (16-bit) logarithmic form for DSP and NN applications.
>>5. No format or version indication to allow for future growth. THIS is
>>the critical missing piece that makes all major repairs onerous.
>>There have been a couple of computers in history, the IBM 1401 and 1620
>>computers, that were hard-wired to handle indefinite-length numbers.
> IBM 1620 floating-point is variable length, but is limited to 99 digits
> (if I remember correctly).
I'm sure that we could do MUCH better with newly designed hardware. The
point was that with work marks, things can be as long as you like.
Steve Richfield
==================
Steve Richfie1d
> You can make this available quite easily, although at a performance cost
> of up to 10X for heavy fp apps, by writing a library that carries
> significance in bits or digits along with the operands, and does
> rounding after every operation.
>
> Write it in C++, and you could use operator overloading to compile
> existing code with it.
>
> Should be an easy afternoon job.
>
> Afterwards, show the results to some of those 5% people and check their
> reactions. My bet would be that when you demonstrate that i.e. their
> spreadsheet results have no significant digits at all, they'll be less
> interested.
Unfortunately, doing this will NOT affect their spreadsheet programs,
and we can't affect those without access to the source code for Excel,
etc. I also doubt whether there are ANY financial planners who code in C++.
I do appreciate your ideas. Please keep them coming until one works.
> PS. I do agree that C really messed up operator priorities, my personal
> fix has been to _always_ use () around every sub-expression just to make
> the desired result absolutely clear.
That's the usual way that most people survive this bug. Note that these
extraneous parentheses short-circuit most compiler optimizations. A
pretty stupid system, isn't it? If you try mixing bit-logical and
boolean operators, the stupid "compliant" C compilers first do the
boolean operations on the bit-logical arguments, and then do the
bit-logical operations on the boolean results, all without so much as a
warning message that sanity was being violated, even though it was in
the spec! Such a system is just too dangerous to use for any critical
application.
It seems totally incredible to me that the ANSI C committee was
perfectly willing to do battle and delay the standard for a year just to
stick the world with this crap.
It will indeed be interesting to see if Fred can shed any light in this
VERY dark place.
Terje Mathisen
> Unfortunately, doing this will NOT affect their spreadsheet programs,
> and we can't affect those without access to the source code for Excel,
> etc. I also doubt whether there are ANY financial planners who code in C++.
Check the people doing automated trading.
>
> I do appreciate your ideas. Please keep them coming until one works.
Steve, please!
You would _of course_ have to download one of the open source
spreadsheets and recompile it with your new fp library.
I think you have at least a couple of alternatives that are sufficiently
Excel compatible that you could expect the original spreadsheet to load
and run.
Terje
Walter Banks
Steve Richfie1d wrote:
> > PS. I do agree that C really messed up operator priorities, my personal
> > fix has been to _always_ use () around every sub-expression just to make
> > the desired result absolutely clear.
>
> That's the usual way that most people survive this bug. Note that these
> extraneous parentheses short-circuit most compiler optimizations.
Why?
w..
Ken Hagan
>
> That's the usual way that most people survive this bug. Note that
> these extraneous parentheses short-circuit most compiler
> optimizations.
I fail to see how. The as-if rule gives optimisers scope to do
anything that cannot *portably* be detected.
> A pretty stupid system, isn't it? If you try mixing bit-logical and
> boolean operators, the stupid "compliant" C compilers first do the
> boolean operations on the bit-logical arguments, and then do the
> bit-logical operations on the boolean results, all without so much as
> a warning message that sanity was being violated, even though it was
> in the spec! Such a system is just too dangerous to use for any
critical
> application.
Which is why tools exist which flag such things. They don't cost
anything like as much as the legal bill when you are sued for
killing someone, so I can't see any reason not to use them.
If operator precedence were the only unsafe part of C then it would
indeed be a pity to have "left this bug" in the standard.
Returning more to the original subject: Last time interval arithmetic
came up here (on comp.arch, rather than this group) the feeling seemed
to be that hardware support would follow if anyone demonstrated a
workable set of primitives. Those primitives would have to go well
beyond
the basic operators, or else many calculations would emerge with
unhelpfully large intervals. Do significance methods suffer similar
problems? If not, why not?
Jeff Kenton
> Dale,
>
>> I'm not really an expert in numerics, but I'm not really aware of any
>> significance bandwagon. Are there a lot of customers who understand this
>> and would base some significant portion of their buying decisions on such
>> capabilities?
>
>
> It's not just how many, but who. I believe that there IS an important
> minority of PC purchasers, maybe 5%, who would view this as an absolute
> necessity if it were available. I suppose that the next step would be to
> measure this subset.
This would be interesting to measure. Until then, I guess that you're off by
a factor of 100, and that no more than 1 in 1000, or worse, would care at all,
much less consider it an "absolute necessity".
[ Note: I'm not saying this isn't interesting nor useful. Just that most
people don't care. And that means you have little chance of getting hardware
support for it. You'll need to do it with software, with the current IEEE-754
support. ]
--
-------------------------------------------------------------------------
= Jeff Kenton Consulting and software development =
= http://home.comcast.net/~jeffrey.kenton =
-------------------------------------------------------------------------
Jeff Kenton
> Terje,
...
>> PS. I do agree that C really messed up operator priorities, my
>> personal fix has been to _always_ use () around every sub-expression
>> just to make the desired result absolutely clear.
>
>
> That's the usual way that most people survive this bug. Note that these
> extraneous parentheses short-circuit most compiler optimizations.
Compilers have come a long way. I wouldn't expect parentheses to affect
optimizations at all in any good compiler.
Fred J. Tydeman
>
> Fred,
>
> > Vice-chair of J11 (ANSI "C")
>
> Hey, did they ever do anything about the many issues that held up the
> original ANSI C ratification, or were these all conveniently forgotten?!
>
> As you may recall, three people, including Russ Hansberry, Wayne Albin,
> and myself, sought to fix 19 of the worst bugs in the original ANSI C
> spec, which held it up for a year. However, no power on earth could
> overcome 100,000 too-early printed copies of the spec sitting in the
> warehouse.
>
> The worst of the bugs was the incorrect order of operations, which
> forced people to put redundant parentheses into many expressions to get
> them to work right.
I joined ANSI C around 1990. This is after the 1989 version of the
standard was finalized. This was my first standards committee.
All I know about the 1989 standardization process was that there was a
year long appeal. I have never heard what was appealed. I did not
know
there was 19 bugs in the spec trying to be fixed. Therefore, I do not
know if anything was ever done. Also, Hansberry is the only name
mentioned.
I did a quick look in
http://anubis.dkuug.dk/jtc1/sc22/wg14/www/olddocuments.txt
and I did not find anything about the appeal.
If you go to:
http://anubis.dkuug.dk/jtc1/sc22/wg14/www/docs/summary.htm
you will find a summary of the Defect Reports (DR) against C99.
And, here is the list of Defects against C89:
http://anubis.dkuug.dk/jtc1/sc22/wg14/www/docs/dr.htm
http://anubis.dkuug.dk/jtc1/sc22/wg14/www/docs/defect-reports.tar.gz
Aside: I think that SNOBOL is the only language that got the
order of * and / correct. 3/4*5 is done as 3/(4*5) in SNOBOL.
Since this is comp.arch.arithmetic, let's take the discussion
elsewhere.
I suggest email (you have my address below). I would be interested in
the list of 19 bugs.
Steve Richfie1d
>> It's not just how many, but who. I believe that there IS an important
>> minority of PC purchasers, maybe 5%, who would view this as an
>> absolute necessity if it were available. I suppose that the next step
>> would be to measure this subset.
>
> This would be interesting to measure. Until then, I guess that you're
> off by a factor of 100, and that no more than 1 in 1000, or worse, would
> care at all, much less consider it an "absolute necessity".
The BIG issue (for me, not so much for significance) here is that there
are maybe only ~100 or so people who are using their computers to form
US foreign policy, e.g. on outsourcing, and in predicting the effects of
various potential intervention on climate changes, e.g. global warming.
Unless these ~100 or so people get this capability, we're all going to
be in a VERY dreadful way that dwarfs tiny considerations, like
providing hardware support for IEEE-754
For example, have you been watching what has been happening to the oil
supply and its application to fertilizer manufacture that is critical to
farming. By the best of estimates, most of the population of the US must
starve in coming decades. However, due to significance problems, they
can't agree on WHICH decade we are due to starve on.
Not implementing significance issue destroys our ability to test
alternatives before they test us.
> [ Note: I'm not saying this isn't interesting nor useful. Just that
> most people don't care. And that means you have little chance of
> getting hardware support for it.
We are certainly in agreement here.
You'll need to do it with software,
> with the current IEEE-754 support.
Unfortunately, that wouldn't fix things for the ~100 users, as they are
probably using ~50 different platforms.
Steve Richfield
Steve Richfie1d
> You would _of course_ have to download one of the open source
> spreadsheets and recompile it with your new fp library.
>
> I think you have at least a couple of alternatives that are sufficiently
> Excel compatible that you could expect the original spreadsheet to load
> and run.
Gee, I didn't realize that there was any sort of open source
Excel-compatible spreadsheet program. Where do I find it?
Thanks.
Steve Richfield
Steve Richfie1d
>> That's the usual way that most people survive this bug. Note that
>> these extraneous parentheses short-circuit most compiler optimizations.
>
> Compilers have come a long way. I wouldn't expect parentheses to affect
> optimizations at all in any good compiler.
Having written and maintained several commercial compilers, including a
year at CDC chasing the bugs out of their supercomputer optimizer and
vectorizer code - Hmmm, let me put my compiler hat back on...
The difficulty of optimizing to a given degree is proportional to the
square of the number of semantic language elements. Hence, for example,
C compilers tend to be worse at optimizing than FORTRAN compilers
because of their much richer language, though if you throw enough money
to pay for enough man-lifetimes, you can optimize the hell out of anything.
Quality commercial compilers, especially those used to compile operating
systems, recognize that unintended consequences and hidden dependencies
may be present, but nearly or absolutely invisible to the compiler.
Wide-scoped variables that another thread may be affecting, variables
used for "mailboxes" to control multi-threading, etc.
A typical case is the handling of the ++ operator, which must increment
a variable WITHOUT allowing another thread to intervene between the
access and the incrementation if you are going to use the compiler to
compile operating systems code, as the ++ operator is the usual means of
affecting such mailboxes.
Hence, when a programmer redundantly specifies a particular order of
operations, a compiler can only optimize to the extent that it is
absolutely impossible for such unintended consequences to step in, which
pretty much means that the variables must all be non-static local
variables, with no embedded variable-modifying operators, etc.
While extremely few expressions may actually have unintended
consequences, the really BIG question is whether the optimizer-writer
fully understood this issue and went to the considerable trouble to
consider the impossibility of unintended consequences.
I've worked on some VERY nasty bugs in this area, which are extremely
hard to find because by their nature they tend to be low-probability and
timing-dependent. Compiler-writers understand that these are really
extreme hazards, and so usually opt to just abandon anything that might
possibly induce such nasty bugs. Somewhere between the best (you're
right) and worst (parentheses short-circuit all optimizations) that
could possibly be done is what the compiler writer actually does.
Hence, I don't think that any broad accurate statement could be made
about optimizations, beyond that redundant parentheses could cause an
optimization problem for some compilers - a softer statement than my
original.
Steve Richfield
glen herrmannsfeldt
> Gee, I didn't realize that there was any sort of open source
> Excel-compatible spreadsheet program. Where do I find it?
I don't know if it is C or not.
There are versions for windows, and many unix systems,
and it reads/writes the popular file formats, as well
as its own format.
-- glen
Dale Morris
> > I'm not really an expert in numerics, but I'm not really aware of any
> > significance bandwagon. Are there a lot of customers who understand
this
> > and would base some significant portion of their buying decisions on
such
> > capabilities?
>
> It's not just how many, but who. I believe that there IS an important
> minority of PC purchasers, maybe 5%, who would view this as an absolute
> necessity if it were available. I suppose that the next step would be to
> measure this subset. Hmmm, don't you have some marketing folks there at
> HP whose job it is to track such things?! Perhaps you could have one of
> them contact me? Some questions on a periodic routine survey should
> answer this question to everyone's satisfaction.
I'm a little confused here. Today's machines are Turing-complete, which
means that they can compute any result (including computation of
significance) that you like. Discussion of specialized hardware is a
discussion of accelerating some interesting common case.
For PC purchasers, is there an argument that a significant percentage of
them would value this capability AND need to use it so much that it would
warrant hardware acceleration? This is what I really don't see.
> > I agree that it's interesting to look for differentiators that help
solve
> > real problems. However, I'm not sure that the market really has much
> > appreciation for these types of capabilities. Most folks are happy just
to
> > compute some answer and be blissfully unaware of the signficance of the
> > result (or even if the algorithm is buggy!)
>
> I agree that the majority of users don't care if it doesn't affect their
> video games and MS WORD usage. However, it WOULD affect Excel in some
> important ways that might be noticed.
How would additional HW capability affect EXCEL? Microsoft would need to
modify EXCEL to use such capability, and they can already do so, at some
performance loss. But, do the users of EXCEL need computation of
significance at a performance level that could only be attained with
hardware acceleration? I've made a lot of spreadsheets myself, and probably
none of them would have much of a performance problem if the FP calculations
were slowed down by a few orders of magnitude.
> I think that to most users, this will be like the now-famous Floating
> Point Divide Bug, that never hurt anyone, but everyone was very careful
> NOT to purchase a processor with the bug. Once it becomes well-known
> that Intel's (or whoever's) processors are the only acceptable
> processors for government agencies and financial gurus, that other
> processors will become like the processors with the Divide Bug - they
> won't hurt most people, but no one will want them.
I doubt this very much. Situations like the FDIV problem in Pentium are
seen by the public as "bugs" because they are anomalies. A bug, for most
people, is defined as a result that differs from the results on other kinds
of machines (calculators, other computers, etc.)
For the problem with loss of significance, there's no expectation (for most
people) of anything different than the way a calculator works. If you were
to build a machine that reports result + significance, and it reports that a
particular calculation has lost all significance and hence there is no
answer of any value, most people would see THAT as a bug, and say "just give
me the darn answer".
> > Itanium systems (for example), because the FP datapath is based on
extended
> > precision and not just double, are able to compute intrinsic functions
(like
> > sin, cos, power, etc.) more accurately (closer to a maximum error of 1/2
> > unit-in-last-place) than competiting systems. However, I'm not sure
that
> > this capability has helped us sell one more Itanium system than we would
> > have otherwise. (Perhaps I'm wrong there, but I've had a hard time
hearing
> > any music emanating from the 1/2 ULP bandwagon.)
>
> Anyone who cares about getting the right answer has already gone to
> double precision, and there, such trivial errors can't be noticed in any
> application that I know of. I used to maintain and migrate the CAD
> systems at Boeing. There, if they could get 0.001" accuracy at a wingtip
> they were ecstatic. 36-bit IBM-7094 arithmetic was just barely good
> enough to use (but not quite 0.001" at a wingtip), 32-bit IBM-360 was
> completely worthless, but 64-bit IBM-360 produced excellent results,
> even with its hexadecimal normalization and complete lack of rounding or
> guard bits. Unfortunately, 64-bit IBM-360 was just too slow to ever get
> anything useful done. Hence, for years, Boeing had to maintain two
> groups of computers, IBMs to run their business, and CDCs to design
> airplanes.
I know people who would disagree with you that improvements to accuracy of
transcedentals are valueless.
But, my point was merely that improvements to numerics are pretty much
ignored by most of the market. I understand you to feel that the
computation of significance would be much more important and more valued
than the increase in accuracy for operations like sin and cos. But, from my
experience, I don't see a huge difference in how the market would value the
two.
> > I don't mean to argue that it's not useful for folks to work on things
like
> > computing significance. Just that I'm not sure one can look to the
market
> > to provide the motivation for such work. And, again, without a pent-up
> > market, and given the market's love of the standardization with 754, I
think
> > that standards groups are the best place to take your argument.
>
> From my experience with several standards groups, they seem to see that
> part of their place in life is to "defend" their standards from
> essential improvements once they start to be used. I and two others
> attempted to get the worst of the bugs in the C standard fixed, e.g. the
> wrong order of operations, but there was no way that such reason could
> ever prevail against the glut of products that already incorporated the
> bugs. Since then, several people have died in connection with the order
> of operations bugs in C, yet still the standard specifies the
> dysfunctional order of operations.
I agree with you that changing the standards (and you need to change more
than just the FP hardware standard to accomplish what you're aiming at)
would by no means be easy. I'm just saying that in my view (and you're
welcome to place whatever value on my advice you like) your most likely path
is through the standards.
However, that said, I'm not really convinced that hardware acceleration and
language semantics changes are the best first steps. I'd recommend finding
someone who wanted to do a SW implementation of what you have in mind, and
use it in some important applications. If EXCEL is key, then I think that
Microsoft (and not any hardware or systems company) should be top of your
list for influencing.
> Again, maybe if I could talk to a marketing guy at HP, we might be able
> to close on a good estimation of market size and close on this issue.
From HP's perspective, as chief processor architect for HP's Itanium server
business, I don't yet see a good argument as to why this should yet be an
open issue. I'd really like to see some SW implementation that's used in
some application, and see that the market is really interested.
> Thanks yet again for some good comments.
You're welcome. It's an interesting topic, especially since it spans so
many disciplines!
Dale Morris
> The BIG issue (for me, not so much for significance) here is that there
> are maybe only ~100 or so people who are using their computers to form
> US foreign policy, e.g. on outsourcing, and in predicting the effects of
> various potential intervention on climate changes, e.g. global warming.
> Unless these ~100 or so people get this capability, we're all going to
> be in a VERY dreadful way that dwarfs tiny considerations, like
> providing hardware support for IEEE-754
Although I agree with your concern, I believe that the loss of significance
in FP calculations is a very tiny fraction of the risk of
computation/interpretation errors. For example, consider these articles
about EXCEL spreadsheet bugs (which would not have been helped by any
computation of significance):
http://www.theregister.co.uk/2003/06/19/excel_snafu_costs_firm_24m/
http://www.eusprig.org/eusprig.pdf
I agree that it's a scary proposition to have people doing computational
analysis without any real understanding of what they're doing.
Steve Richfie1d
> I'm a little confused here. Today's machines are Turing-complete, which
> means that they can compute any result (including computation of
> significance) that you like. Discussion of specialized hardware is a
> discussion of accelerating some interesting common case.
>
> For PC purchasers, is there an argument that a significant percentage of
> them would value this capability AND need to use it so much that it would
> warrant hardware acceleration? This is what I really don't see.
The issue isn't so much a purely hardware one, but more software
compatibility with BOTH whatever hardware is on the computer AND with
significance computations. Some here have proposed a purely software
approach, which would require recreating much of Microsoft's line of
products to provide significance to everyone - a little beyond what I
can do in my scarce spare time. Yes, in theory it could be done entirely
in software, given some small but not completely insignificant fraction
of Microsoft's 20,000 employees to do it.
> How would additional HW capability affect EXCEL? Microsoft would need to
> modify EXCEL to use such capability, and they can already do so, at some
> performance loss. But, do the users of EXCEL need computation of
> significance at a performance level that could only be attained with
> hardware acceleration? I've made a lot of spreadsheets myself, and probably
> none of them would have much of a performance problem if the FP calculations
> were slowed down by a few orders of magnitude.
There are some users who really twist Excel to do some outrageous
things. However, I suspect that you are right, that the financial people
mostly use a variety of other tools. I've heard that VB is popular.
>>I think that to most users, this will be like the now-famous Floating
>>Point Divide Bug, that never hurt anyone, but everyone was very careful
>>NOT to purchase a processor with the bug. Once it becomes well-known
>>that Intel's (or whoever's) processors are the only acceptable
>>processors for government agencies and financial gurus, that other
>>processors will become like the processors with the Divide Bug - they
>>won't hurt most people, but no one will want them.
>
>
> I doubt this very much. Situations like the FDIV problem in Pentium are
> seen by the public as "bugs" because they are anomalies. A bug, for most
> people, is defined as a result that differs from the results on other kinds
> of machines (calculators, other computers, etc.)
... or the correct answer?
> For the problem with loss of significance, there's no expectation (for most
> people) of anything different than the way a calculator works. If you were
> to build a machine that reports result + significance, and it reports that a
> particular calculation has lost all significance and hence there is no
> answer of any value, most people would see THAT as a bug, and say "just give
> me the darn answer".
... and THAT is why our economy and ecology is in the mess it is in.
> I know people who would disagree with you that improvements to accuracy of
> transcedentals are valueless.
I'm ALWAYS up for a rousing mathematical argument! Perhaps you could
give me a glimpse of the other side of this issue?
> But, my point was merely that improvements to numerics are pretty much
> ignored by most of the market. I understand you to feel that the
> computation of significance would be much more important and more valued
> than the increase in accuracy for operations like sin and cos. But, from my
> experience, I don't see a huge difference in how the market would value the
> two.
You obviously live in a world of Taylor series, pipeline construction,
transistor counts, etc. You are speculating as to what a competent
market analyst might determine about two entirely different market
questions.
I've worked in marketing and sales in the past. One thing I learned is
that it is REALLY difficult to guess accurately at marketing issues -
that it is MUCH better to get real data. We are of differing opinions on
an issue, but face it, neither of us has any sort of real data to back
up our opinions, so if I were a 3rd party, I would have to flip a coin
to guess who was right.
However, I'd guess that you have some marketing guys running around
there, and that they periodically circulate surveys to test such issues,
to figure out where to spend their next million dollars of development
money. What I think should happen is for you/I to (try to) get one of
those guys to include this question in their next marketing survey, and
let the chips fall where they may.
Hey, you might just be right, but what is the cost/benefit of testing this?
>> From my experience with several standards groups, they seem to see that
>>part of their place in life is to "defend" their standards from
>>essential improvements once they start to be used. I and two others
>>attempted to get the worst of the bugs in the C standard fixed, e.g. the
>>wrong order of operations, but there was no way that such reason could
>>ever prevail against the glut of products that already incorporated the
>>bugs. Since then, several people have died in connection with the order
>>of operations bugs in C, yet still the standard specifies the
>>dysfunctional order of operations.
>
>
> I agree with you that changing the standards (and you need to change more
> than just the FP hardware standard to accomplish what you're aiming at)
> would by no means be easy. I'm just saying that in my view (and you're
> welcome to place whatever value on my advice you like) your most likely path
> is through the standards.
I'm trying, but the prospects are pretty grim without applying
additional heat.
> However, that said, I'm not really convinced that hardware acceleration and
> language semantics changes are the best first steps. I'd recommend finding
> someone who wanted to do a SW implementation of what you have in mind, and
> use it in some important applications. If EXCEL is key, then I think that
> Microsoft (and not any hardware or systems company) should be top of your
> list for influencing.
The problem with this is that the applications that are
significance-critical are spread across a LOT of platforms, so fixing
any one of them isn't going to make a lot of difference. Besides,
Microsoft, like HP, sells to demand, and as you pointed out, there is
not at present any demand, because there isn't any product, because
there isn't any demand, because...
> From HP's perspective, as chief processor architect for HP's Itanium server
> business, I don't yet see a good argument as to why this should yet be an
> open issue.
THAT explains a LOT for me! If new ideas must get over some considerable
threshold even to collect data to see if they are worth investing
anything at all in them, then obviously nearly all ideas will die
without ever being really considered, which fully explains the really
sedentary state of development in this business.
I've tried to get interest in developing a new sort of "smart memory". I
have a memory-process-compatible architecture for logarithmic vector
processor cores, so that you could sprinkle dozens/hundreds throughout a
large memory chip. This would provide somewhere around 3 orders of
magnitude increase in the short-precision computing capability of a PC.
Here's the kicker. For complex reasons I don't want to post, these would
actually be as cheap or cheaper than making just the bare memory without
the processors! As you have probably already guessed, there weren't any
currently big applications needing that much crunching power, and there
was no market demand for these non-existent devices, so no semi maker
wanted to develop these. Hence, we are now stuck slogging with our
clunky Pentiums rather than flying with these memories that would
actually be cheaper to make.
Can you see how the "open issue" filter, which appears to be standard in
the industry, needlessly stifles development?
I'd really like to see some SW implementation that's used in
> some application, and see that the market is really interested.
This has come and gone 30 years ago. You're welcome to talk to various
developers and users of that system to get an idea as to its strengths
and weaknesses. I could email you some phone numbers if you are interested.
Unfortunately, "the market" is now so dispersed across platforms that
any SW implementation would only affect some very small minority of
them, so I don't see how this could be a legitimate test.
Besides, there are many more subtle design issues in a significance
implementation than in a straight FP implementation, though the
resulting hardware is no more complex. I don't see how to separate out
the impact of residual bugs and weaknesses from the concept itself in a
small-scale test.
When we implemented the system in the '70s, there was the usual tail of
minor bugs in response to user complaints that we had to work out,
including one that was VERY similar to the FDIV bug that was found
months after the others. Updates were MUCH easier on a time-sharing
computer system back then, because users didn't need to download
anything to get the latest versions of all the system components. Doing
this successfully in a modern setting would be a MAJOR challenge, and
require a BIG user who REALLY needs this DESPERATELY.
I'll be in Las Vegas in 3 weeks, presenting and discussing this. Maybe
there the "key" will present itself.
Thanks again.
Steve Richfield
Steve Richfie1d
Dale,
>>The BIG issue (for me, not so much for significance) here is that there
>>are maybe only ~100 or so people who are using their computers to form
>>US foreign policy, e.g. on outsourcing, and in predicting the effects of
>>various potential intervention on climate changes, e.g. global warming.
>>Unless these ~100 or so people get this capability, we're all going to
>>be in a VERY dreadful way that dwarfs tiny considerations, like
>>providing hardware support for IEEE-754
> Although I agree with your concern, I believe that the loss of significance
> in FP calculations is a very tiny fraction of the risk of
> computation/interpretation errors.
But, the fundamental maximum time limit of a projection is determined
ONLY by the loss of significance, that can be VERY data-dependent.
Without significance, there really aren't any good alternatives to tell
just how far into the future your predictions are good for, except
through experience. Unfortunately, there are many high-value things to
predict, like the economy and ecology, where such experience will
probably be probably suicidal.
> For example, consider these articles
> about EXCEL spreadsheet bugs (which would not have been helped by any
> computation of significance):
>
> http://www.theregister.co.uk/2003/06/19/excel_snafu_costs_firm_24m/
>
> http://www.eusprig.org/eusprig.pdf
Actually, these are NOT bugs in the usual sense - but fundamental
conceptual weaknesses in the spreadsheet concept. I like this simile,
because it is VERY parallel with the current FP discussion.
There ARE some expensive proprietary spreadsheet products, which don't
allow the sort of cut-and-paste without checking that Excel and other
"dumb" spreadsheets do. However, they aren't "standard". They don't come
with the computer, and so few people use them.
Here, Microsoft set the standard, and like most standards efforts, they
dropped the ball. I could easily propose corrections to Excel that are
like Foresight, one of those expensive proprietary spreadsheet products
that I have some experience with, but what would be the use? Microsoft
would still just keep on doing whatever they want without regard to such
problems.
I also have a skeletal BASIC program where you can just drop your
formulas in between the appropriate comments, and it produces a good
solid numerically-consistent spreadsheet. But, the UI isn't as fancy, so
few people are interested in using it.
> I agree that it's a scary proposition to have people doing computational
> analysis without any real understanding of what they're doing.
We're sure in agreement on this point.
Steve Richfield
Dale Morris
I'm going to have to bow out pretty soon, to keep up with regular work. But
a few thoughts for you.
I'm still not understanding how making some change to HW would allow
applications like EXCEL to factor in signicance in some way that would not
involve making any changes to the application. If we're talking about
making changes to the application, then I don't see what's essential about
special HW to accelerate significance computation. This is the key thing
I'm not understanding.
With respect to your thoughts on thresholds for considering changes, here
are some thoughts.
First off, I think it's quite obvious that there must be some threshold.
(If not, I have a googleplex of extremely useless problems you might work
on - interested? ;-)
What we're talking about right now is something that would require very
large amounts of investment, would require very large coordination with the
standards that we'd be defacto extending, and would be very intertwined with
relationships with many other partners and other companies. So, the
investment would be very large. And, I (in my role as processor architect
for HP's server business) still do not have any reasonable sense of whether
there might be any sort of return. So, given that, my evaluation is that it
would not be proper for me to recommend any action within HP.
Now, it's certainly possible that I'm simply misunderstanding what you're
proposing (which is why I've been staying in the discussion). It may also
turn out that I make a mistake, and fail to see the potential for a real
opportunity, and hence HP (at least through my agency) might miss out on
that opportunity. But there are also risks on the other side, of
distracting marketing folks from doing something that will really increase
revenue. No-one's time is free. It's part of my job to evaluate that
balance of risks and potential returns when considering some new, proposed
technology or approach.
That said, I'll point out that I entered this conversation only to suggest
some avenues that might best serve your aims. You may well know much more
about how to achieve your aims than I. (In fact, that seems quite likely to
me.) So, my advice may not be of much value to you.
Again, though, I have found this to be an interesting discussion, and I
thank you for that.
- Dale Morris
Itanium processor architect
Hewlett Packard
"Steve Richfie1d" <St...@NOSPAM.smart-life.net> wrote in message
news:5bd736388817ba67...@news.teranews.com...
Steve Richfie1d
> I'm still not understanding how making some change to HW would allow
> applications like EXCEL to factor in signicance in some way that would not
> involve making any changes to the application. If we're talking about
> making changes to the application, then I don't see what's essential about
> special HW to accelerate significance computation. This is the key thing
> I'm not understanding.
Here's my thought: If someone like Intel came out with a new proposed
standard for some arithmetic enhancement, much like happened with the
8087, then new products would support it, and big players like Microsoft
would roll support for it into the next version of their existing
products like Office, etc. Without at least the implied promise of
hardware to support, I doubt if hardly anyone would make such a change,
because they would have BOTH the significance interfacing AND software
emulation issues to deal with, not to mention getting their product to
work right.
In short, I was just suggesting to follow the same path pointed out by
the 8087, that was sold even before the standard was officially
released. I know, it sounds a bit backwards, but it looked like a
workable path.
Alternatively, a "paper tiger" approach might work even better, where a
significance standard is announced by a big company like HP, but no work
proceeds on any hardware until the paper standard gains some reasonable
level of acceptance in the rest of the world. This would certainly
reduce the "up front" money required. I'd be willing to discuss some
deal with HP to produce such a standard, if HP would agree to back it
once produced and accepted by you and other experts within HP (so any
pipelining and microcode compatibility concerns would be fully
addressed). For a significant later piece of the action (if any), I
could do this for no up-front cost to HP! Would THAT solve your economic
and market acceptance concerns - zero cost until it proves itself?!
> With respect to your thoughts on thresholds for considering changes, here
> are some thoughts.
>
> First off, I think it's quite obvious that there must be some threshold.
> (If not, I have a googleplex of extremely useless problems you might work
> on - interested? ;-)
Are there any of them that you think are REALLY important? Any that our
economy or ecology might be hanging on? If so, sure, I'd be interested.
> What we're talking about right now is something that would require very
> large amounts of investment, would require very large coordination with the
> standards that we'd be defacto extending, and would be very intertwined with
> relationships with many other partners and other companies. So, the
> investment would be very large. And, I (in my role as processor architect
> for HP's server business) still do not have any reasonable sense of whether
> there might be any sort of return. So, given that, my evaluation is that it
> would not be proper for me to recommend any action within HP.
STOP! You've taken a wrong turn here! It would take an HP marketing guy
less time than we've spent typing here to include yet another question
or two about this in their next customer survey. If the results show
that there is a billion bucks to be made here, then it is a good
decision to go with it. If the survey is a flop, then this whole
exercise will have probably cost less than one man-day (including our
typing here).
I've been associated with some marketing (as opposed to sales) flops in
the past. These have taught me to always check the marketing
considerations FIRST, BEFORE dumping a bunch of money into development.
The obvious way to do this is to work with a company (like HP) that is
already doing similar marketing, as this reduces an otherwise expensive
exercise to nil.
> Now, it's certainly possible that I'm simply misunderstanding what you're
> proposing (which is why I've been staying in the discussion). It may also
> turn out that I make a mistake, and fail to see the potential for a real
> opportunity, and hence HP (at least through my agency) might miss out on
> that opportunity. But there are also risks on the other side, of
> distracting marketing folks from doing something that will really increase
> revenue. No-one's time is free. It's part of my job to evaluate that
> balance of risks and potential returns when considering some new, proposed
> technology or approach.
We're in agreement here that whatever HP might do at the beginning
should be CHEAP - SO cheap that HP can initially evaluate anything that
looks possibly promising without worrying about the cost of initial
evaluation. If HP were to send me an existing customer survey, then I
could suggest a question or two that fit in with the rest. The
incremental cost of including another question or two on a customer
survey would be nil.
> That said, I'll point out that I entered this conversation only to suggest
> some avenues that might best serve your aims. You may well know much more
> about how to achieve your aims than I. (In fact, that seems quite likely to
> me.) So, my advice may not be of much value to you.
A good idea is a good idea, regardless of who suggests it.
> Again, though, I have found this to be an interesting discussion, and I
> thank you for that.
You're very welcome.
BTW, what part of the world are you in? We'll have to meet sometime if
we ever get to the same place at the same time.
Steve Richfield
=================
Mike Cowlishaw
> Steve Richfie1d wrote:
>>> I'm not really an expert in numerics, but I'm not really aware of
>>> any significance bandwagon. Are there a lot of customers who
>>> understand this and would base some significant portion of their
>>> buying decisions on such capabilities?
>>
>> It's not just how many, but who. I believe that there IS an important
>> minority of PC purchasers, maybe 5%, who would view this as an
>> absolute necessity if it were available. I suppose that the next
>> step would be to measure this subset.
>
> This would be interesting to measure. Until then, I guess that
> you're off by a factor of 100, and that no more than 1 in 1000, or
> worse, would care at all, much less consider it an "absolute
> necessity".
I agree .. when I investiated this a few years ago, the only place I
found significance arithmetic used regularly is among chemists.
> [ Note: I'm not saying this isn't interesting nor useful. Just that
> most people don't care. And that means you have little chance of
> getting hardware support for it. You'll need to do it with software,
> with the current IEEE-754 support. ]
It's an interesting discussion, but I don't think it is very useful.
For a discussion on why the decimal arithmetic in the proposed
IEEE 754 revision is not 'significance arithmetic', see:
http://www2.hursley.ibm.com/decimal/decifaq4.html#signif
Mike Cowlishaw [traveling at the moment; erratic net access]
Dik T. Winter
...
> A typical case is the handling of the ++ operator, which must increment
> a variable WITHOUT allowing another thread to intervene between the
> access and the incrementation if you are going to use the compiler to
> compile operating systems code, as the ++ operator is the usual means of
> affecting such mailboxes.
Hrm, if you are doing multithreaded programming, there is *no* way to
ensure that another thread will not step in between the access and the
store of the incremented value. Unless you machine has either things
like barrier instructions or an atomic read and increment. If you are
not doing multithreaded programming there is no problem at all.
> Hence, when a programmer redundantly specifies a particular order of
> operations, a compiler can only optimize to the extent that it is
> absolutely impossible for such unintended consequences to step in, which
> pretty much means that the variables must all be non-static local
> variables, with no embedded variable-modifying operators, etc.
Wrong. Take a read of the C standard about sequence points.
> While extremely few expressions may actually have unintended
> consequences, the really BIG question is whether the optimizer-writer
> fully understood this issue and went to the considerable trouble to
> consider the impossibility of unintended consequences.
Give an example of such unintended consequences, please.
> Hence, I don't think that any broad accurate statement could be made
> about optimizations, beyond that redundant parentheses could cause an
> optimization problem for some compilers - a softer statement than my
> original.
Well, gcc seems to have no problems with it, but that is not a commercial
compiler...
Have you ever been concerned about such problems with the Fortran standard?
--
dik t. winter, cwi, kruislaan 413, 1098 sj amsterdam, nederland, +31205924131
home: bovenover 215, 1025 jn amsterdam, nederland; http://www.cwi.nl/~dik/
Dik T. Winter
...
> Here's my thought: If someone like Intel came out with a new proposed
> standard for some arithmetic enhancement, much like happened with the
> 8087,
Some historical enlightment here. There was already an IEEE committee
trying to set up a floating-point standard. At about the same time
Intel wished to create their floating-point unit, so they hired in
Kahan (one of the committee members) to assist in implementing the
forthcoming standard. The 8087 did not implement the final standard,
but some intermediate form of it, that is why there are differences
between the 8087 and the 80187 (which implements the final standard).
(The differences are related to boundary problems, the use of infinities,
etc.)
I think you do not understand the IEEE standard well enough. You complained
that in a NaN 23 bits are unused. This is false. The standard
specifically allows to encode the reason of NaN generation in those
bits (but that part is not standardised). Also, one bit is reserved
to distinguish between a signalling NaN and a non-signalling NaN.
Another complaint from you: that IEEE only did have signed infinities.
Take a look at the 8087. It had only one unsigned infinity. There
is, mathematically, a distinction between a set of numbers with two
signed infinities and one unsigned infinity. There were reasons
(coming from numerical mathematics) to have the two signed version.
I wonder in what case you want to generate an unsigned infinity.
And finally, you complained about the lack of indefinitives. (This
definitively shows your CDC background ;-).) Indefinitives are
renamed NaN's in IEEE.
Significance arithmetic is a great tool in some fields of mathematics,
but it has the same problem as interval arithmetic. Correlation
between errors is not included. A really good numerical error analysis
gets you much better results.
Steve Richfie1d
> I agree .. when I investiated this a few years ago, the only place I
> found significance arithmetic used regularly is among chemists.
Financial and other predictive systems are fundamentally
significance-limited, in that you can only predict the future until you
lose all of your significance, after which your predictions are PURE
garbage. In a sense, the significance is equally or perhaps even more
important than the value being computed in these applications! However,
people doing this do NOT currently use significance! Why? Because,
unlike chemists, their computations are WAY too complex to do by hand,
and IEEE-754 has deprived them of any opportunity to do this by machine.
The results to our economy and ecology are all too tragic.
> It's an interesting discussion, but I don't think it is very useful.
> For a discussion on why the decimal arithmetic in the proposed
> IEEE 754 revision is not 'significance arithmetic', see:
>
> http://www2.hursley.ibm.com/decimal/decifaq4.html#signif
This paper makes the numerous rookie errors expected of someone who
hasn't ever actually worked in a significance environment. With two
years of live experience with a commercial time-sharing system, that
ONLY had significance arithmetic, that most of the customers used for
financial purposes, I can say that it works GREAT, that there are none
but trivial difficulties, but various methods are needed that your paper
didn't contemplate. The lack of this contemplation led to your erroneous
conclusion.
To illustrate, How do you count to 1000 in just 4 bits?! The answer is
that you take a 16/1000 chance of incrementing the 4 bits for each
count. On average, it will take 1000 such steps to overflow, with a
standard deviation error of sqr(16) * (16/1000) * 1000 = 64 steps. I
know, no one likes non-deterministic arithmetic!
Adding small quantities to inexact ones also works great if you have
enough guard digits - the limiting case being interval arithmetic - the
"other" form of significance arithmetic. From 50,000 feet, all
significance arithmetic systems are analogous.
These are just two solutions to a problem that the paper says has no
solution.
The really **BIG** thing that the paper missed is that obviously, I/O
must be modified a bit to work in a significance environment. When
"12.34" is read as input, is that +/-0.005, or is it exact/normalized?
At present, there is no distinguishing these in present (defective)
languages, and so you jumped to the confusion that the programmer could
not specify which was meant. With formatted I/O, separate format syntax
is sometimes needed to specify significance vs. exact entry.
The way we did this was to interpret the standard "F" format as
significance - which even when used with exact entries nonetheless
worked in the vast majority of applications, as most simple financial
computations do NOT do anything to lose significance. There was VERY
little use of the exact entry syntax, but it was there if anyone wanted it.
We also provided a library routine to in effect normalize its argument
and return it as the result. This was little used - mostly in successive
approximation procedures to normalize a poor initial guess, to then be
refined through iteration.
We also provided an optional mode to normalize all input and constants,
just in case the user didn't care about significance, his program had
some problem with it, and the user didn't want to change his program. To
my knowledge, this feature was never actually used! Why? Because there
usually were no problems to be fixed, in the rare case that there was a
problem, it was trivially fixed, and people really liked significance
because the computer stopped lying to them.
I REALLY like decimal arithmetic. However, there are some minor problems
mixing decimal and significance, because bits of significance are only
~.3 digits, so binary unnormalized methods aren't so "ragged" as decimal
ones that lose an entire digit at a time. However interval arithmetic
would work OK for this, as would unnormalized methods w/o the logic to
drop a bit under some conditions.
The conclusion the paper makes, that full significance must be
maintained, is obviously wrong once you see the alternative methods that
the paper obviously missed. I strongly recommend retiring the paper
until it is corrected.
Are you going to be at the computer show in Las Vegas on June 21-24? If
so, then we should sit down and work these various issues out. You'll
see that significance DOES work, once you see the methods needed to get
over some of its apparent "humps".
Steve Richfield
Steve Richfie1d
> Hrm, if you are doing multithreaded programming, there is *no* way to
> ensure that another thread will not step in between the access and the
> store of the incremented value. Unless you machine has either things
> like barrier instructions or an atomic read and increment. If you are
> not doing multithreaded programming there is no problem at all.
You missed the more common alternatives! Some computers inhibit
interrupts for one instruction following certain instructions, to
provide for such interlocking. Of course, if you are doing kernel
programming, you can just inhibit the interrupts yourself.
> > Hence, when a programmer redundantly specifies a particular order of
> > operations, a compiler can only optimize to the extent that it is
> > absolutely impossible for such unintended consequences to step in, which
> > pretty much means that the variables must all be non-static local
> > variables, with no embedded variable-modifying operators, etc.
>
> Wrong. Take a read of the C standard about sequence points.
I don't have a copy here. What does it say?
After the ANSI C committee FLATLY REFUSED when I and two others asked to
fix their order of operations problems before releasing the original
spec, I haven't trusted it for anything critical, and so program C only
when there aren't any reasonable alternatives. Hence, I don't have cause
to pay attention to lots of little details. I'm a
compiler/optimizer/vectorizer guy, not a big-time C user.
Note that commercial compilers often pay attention to details that are
NOT in the spec, as most specs are simply prescriptions for disaster
unless tempered with some intelligent caution.
> > While extremely few expressions may actually have unintended
> > consequences, the really BIG question is whether the optimizer-writer
> > fully understood this issue and went to the considerable trouble to
> > consider the impossibility of unintended consequences.
>
> Give an example of such unintended consequences, please.
Hypothetical situation: You are testing a variable that is really an
external I/O location. You write a statement that checks it twice and
loops until you get the same result twice in a row to avoid noise, etc.
To subvert the compiler optimizing your two tests into one test, you
surround the two tests with parentheses. If the compiler doesn't respect
your apparently redundant parentheses, then common subexpressions are
gathered and your fancy double test evaporates into a single test. Of
course this still "works", but is now sensitive to external noise.
>
> > Hence, I don't think that any broad accurate statement could be made
> > about optimizations, beyond that redundant parentheses could cause an
> > optimization problem for some compilers - a softer statement than my
> > original.
>
> Well, gcc seems to have no problems with it, but that is not a commercial
> compiler...
>
> Have you ever been concerned about such problems with the Fortran standard?
I have worked on two FORTRAN compilers. One was only slightly
optimizing, and didn't know enough to extract common subexpressions. The
other was probably the MOST optimizing ever written - the one CDC
developed for their supercomputers. However, they wrote their
operating-system level stuff in assembly language. Operating systems
written in vector code was REALLY wild. Since the computer was capable
of doing a complex associative memory lookup with a single instruction,
there really wasn't any reason for clever table construction to reduce
the number of locations to be examined, as such logic would actually
slow the processor down!
Steve Richfield
Steve Richfie1d
> I think you do not understand the IEEE standard well enough. You complained
> that in a NaN 23 bits are unused. This is false. The standard
> specifically allows to encode the reason of NaN generation in those
> bits (but that part is not standardised).
And, that part not being standardized IS this particular problem with
the IEEE standard, because there are LOTS of different NANs that need
DIFFERENT handling by the FP hardware, both in production, AND in HANDLING.
Leaving these bits undefined means that there is no special handling of
them during subsequent instructions, and so to the computer they ARE
unused, though of course you can store anything in them that might mean
something to you, though not the computer.
See, I DO understand the spec!
> Also, one bit is reserved
> to distinguish between a signalling NaN and a non-signalling NaN.
Of course, signaling should be under user control, because some
applications demand stopping at ALL exceptions, while others should be
left to run no matter what. Then, there are even a few applications that
want what the IEEE spec provides! This is a function of the environment
the application is being run in, and NOT of any requirement by the FP
hardware. We have the tail wagging the dog here.
> Another complaint from you: that IEEE only did have signed infinities.
> Take a look at the 8087. It had only one unsigned infinity. There
> is, mathematically, a distinction between a set of numbers with two
> signed infinities and one unsigned infinity. There were reasons
> (coming from numerical mathematics) to have the two signed version.
> I wonder in what case you want to generate an unsigned infinity.
You should have BOTH. Consider adding infinities. If they are of the
SAME sign, then you just get another infinity of the same sign that may
resolve if it finds it way into a future denominator. However, if they
are unsigned or of different signs, then you (should) get an indefinite
result that will poison future computation unless some future IF
statement skips it entirely. That makes the case for signed infinities.
However, some operations, like dividing by a completely underflowed/zero
denominator, cannot produce a sign to retain, so you also need unsigned
infinity to represent such result, to avoid representing a potentially
erroneous sign. This is unfortunately now missing from the standard. I
wonder, why did they discard it after the 8087? Any idea?
> And finally, you complained about the lack of indefinitives. (This
> definitively shows your CDC background ;-).) Indefinitives are
> renamed NaN's in IEEE.
Unfortunately, so is everything else but the kitchen sink, so I
reiterate my objection, specifically:
Indefinites ARE numbers that the particular implementation is unable to
either compute or represent, so NaN is really a misnomer. Maybe it
should be NaRN = Not a Representable Number.
However, the square root of a negative number or the arc sin of a number
greater than 1 is really NOT a number, and so would be a legitimate NaN.
It is REALLY important to distinguish between these two cases, because
you need to know whether you need to rearrange some expression, or chase
a genuine error in your logic.
> Significance arithmetic is a great tool in some fields of mathematics,
> but it has the same problem as interval arithmetic. Correlation
> between errors is not included. A really good numerical error analysis
> gets you much better results.
Have you ever actually tried this?! Most real-world problems are WAY too
complex for such complete error analysis. Further, real-world error
analysis in usually on variably significant numbers, so there is really
no choice but to track significance through computation. Retaining what
you can through computation, imperfect though it may be, is certainly a
LOT better than just normalizing everything and throwing all this
information away.
Steve Richfield
Jeff Kenton
...
> But, the fundamental maximum time limit of a projection is determined
> ONLY by the loss of significance, that can be VERY data-dependent.
I'd be at least as worried about incomplete and inaccurate software models.
If you have the wrong model, it doesn't matter how precise your answers are.
--
"Never argue with a fool."
Steve Richfie1d
>> But, the fundamental maximum time limit of a projection is determined
>> ONLY by the loss of significance, that can be VERY data-dependent.
>
>
> I'd be at least as worried about incomplete and inaccurate software
> models. If you have the wrong model, it doesn't matter how precise your
> answers are.
I don't think you meant "precise" in your reply, as all systems
propagate (often too much) precision very well. I suspect that you meant
to say something like "... If you have the wrong model, it doesn't
matter how well you propagate significance to your answers." Presuming
that is what you meant...
You're absolutely right, but... The problem is that without
significance, it's hard/impossible to tell whether a bad prediction is
due to a bad model, or due to a good model that has simply run out of
(unseen) significance. Hence, it can be pretty difficult to fix a wrong
model in the absence of significance, because you can't tell what is
wrong, or even if it is wrong.
This is one of those chicken-and-egg things - you need BOTH a good model
and significance, and you can't get a good model without significance.
However, you CAN get significance without a good model. Hence,
significance must come FIRST.
I suspect that there will be a LOT of work on people's models once they
have significance-propagating machines to show them what works, what
doesn't, and why.
Steve Richfield
Dik T. Winter
> > I agree .. when I investiated this a few years ago, the only place I
> > found significance arithmetic used regularly is among chemists.
>
> Financial and other predictive systems are fundamentally
> significance-limited, in that you can only predict the future until you
> lose all of your significance, after which your predictions are PURE
> garbage. In a sense, the significance is equally or perhaps even more
> important than the value being computed in these applications! However,
> people doing this do NOT currently use significance! Why? Because,
> unlike chemists, their computations are WAY too complex to do by hand,
> and IEEE-754 has deprived them of any opportunity to do this by machine.
> The results to our economy and ecology are all too tragic.
Tell that to the people that make weather forecasts. With significance
included, I do not think you can predict the weather beyond a few hours.
Dik T. Winter
> > Hrm, if you are doing multithreaded programming, there is *no* way to
> > ensure that another thread will not step in between the access and the
> > store of the incremented value. Unless you machine has either things
> > like barrier instructions or an atomic read and increment. If you are
> > not doing multithreaded programming there is no problem at all.
>
> You missed the more common alternatives! Some computers inhibit
> interrupts for one instruction following certain instructions, to
> provide for such interlocking. Of course, if you are doing kernel
> programming, you can just inhibit the interrupts yourself.
Yes, there are alternatives, of course, but it is the harware that needs
to help you.
> > > Hence, when a programmer redundantly specifies a particular order of
> > > operations, a compiler can only optimize to the extent that it is
> > > absolutely impossible for such unintended consequences to step in, which
> > > pretty much means that the variables must all be non-static local
> > > variables, with no embedded variable-modifying operators, etc.
> >
> > Wrong. Take a read of the C standard about sequence points.
>
> I don't have a copy here. What does it say?
From it you can find that the expression (a++ + a++) leads to undefined
behaviour. So you may put parenthesis around it as much as you would
like, it will not help. Side effects of operators can be done immediately
or delayed to the end of the expression, as far as the C standard is
concerned. (Actually that is where sequence points come in. Sequence
points are those places in an expression where all side effects are required
to be performed.)
> After the ANSI C committee FLATLY REFUSED when I and two others asked to
> fix their order of operations problems before releasing the original
> spec,
I still do not see *any* order of operation problems. You seem to think
that & is a logical operator. It is *not* in C. In C it is an arithmetic
operator (a bitwise and). So you need to write:
(a > b) & (c > d)
with parenthesis, but in that case you use an arithmetic operator, you
should use the logical one:
a > b && c > d
> I'm a
> compiler/optimizer/vectorizer guy, not a big-time C user.
But you critique a language without looking at the details.
> > > While extremely few expressions may actually have unintended
> > > consequences, the really BIG question is whether the optimizer-writer
> > > fully understood this issue and went to the considerable trouble to
> > > consider the impossibility of unintended consequences.
> >
> > Give an example of such unintended consequences, please.
>
> Hypothetical situation: You are testing a variable that is really an
> external I/O location. You write a statement that checks it twice and
> loops until you get the same result twice in a row to avoid noise, etc.
> To subvert the compiler optimizing your two tests into one test, you
> surround the two tests with parentheses.
I know of no language that helps when you surround by parenthesis. And C
indeed does not help. But C has a solution. Declare the variable as
"volatile" and the compiler is not allowed to collapse the two accesses.
In Fortran the situation was *much* worse. (If two function calls were
identical, the compiler was allowed to do only one call, regardless of
possible side effects.) Before you critique a language, please, check
what the language offers.
> If the compiler doesn't respect
> your apparently redundant parentheses, then common subexpressions are
> gathered and your fancy double test evaporates into a single test.
Not if the variable was declared "volatile".
> > Have you ever been concerned about such problems with the Fortran standard?
>
> I have worked on two FORTRAN compilers. One was only slightly
> optimizing, and didn't know enough to extract common subexpressions. The
> other was probably the MOST optimizing ever written - the one CDC
> developed for their supercomputers.
In my opinion Cray and NEC were better, but that is just an opinion.
> Operating systems
> written in vector code was REALLY wild. Since the computer was capable
> of doing a complex associative memory lookup with a single instruction,
> there really wasn't any reason for clever table construction to reduce
> the number of locations to be examined, as such logic would actually
> slow the processor down!
Yes, I know. We had one instruction for the 205 that could not be timed
because the timer overflowed a few times before the instruction finished.
But to get back to Fortran. I know of a library where some subroutines
needed to know whether two actual array arguments were in fact the same
array. There is *no* way to do that in standard Fortran 66 or Fortran 77.
Dik T. Winter
> > I think you do not understand the IEEE standard well enough. You complained
> > that in a NaN 23 bits are unused. This is false. The standard
> > specifically allows to encode the reason of NaN generation in those
> > bits (but that part is not standardised).
>
> And, that part not being standardized IS this particular problem with
> the IEEE standard, because there are LOTS of different NANs that need
> DIFFERENT handling by the FP hardware, both in production, AND in HANDLING.
It could not be standardised yet in that case because there was
insufficient knowledge.
> > Also, one bit is reserved
> > to distinguish between a signalling NaN and a non-signalling NaN.
>
> Of course, signaling should be under user control, because some
> applications demand stopping at ALL exceptions, while others should be
> left to run no matter what.
Whether a signalling or non-signalling NaN stops the program is under
user control... The distinction is mostly between generated NaN's and
propagated NaN's.
> However, some operations, like dividing by a completely underflowed/zero
> denominator, cannot produce a sign to retain, so you also need unsigned
> infinity to represent such result, to avoid representing a potentially
> erroneous sign.
But zero has a sign in IEEE. Just *because of underflow*.
> I
> wonder, why did they discard it after the 8087? Any idea?
They replaced it by signed infinities (which the 8087 did not have). There
are mathematically two realistic models, one with a single infinity and one
with two infinities.
> Indefinites ARE numbers that the particular implementation is unable to
> either compute or represent, so NaN is really a misnomer. Maybe it
> should be NaRN = Not a Representable Number.
Hrm. So +oo - +oo is not representable? Wrong. It is nonsense.
> However, the square root of a negative number or the arc sin of a number
> greater than 1 is really NOT a number, and so would be a legitimate NaN.
But the square root of -1 is i, and the arc sin of 2 is some other complex
number, so they are definitely numbers, but not representable.
I think you have this backwards.
> It is REALLY important to distinguish between these two cases, because
> you need to know whether you need to rearrange some expression, or chase
> a genuine error in your logic.
A good implementation of the arithmetic will provide you with the information.
Take a look at the Transputer, which will encode the source of the NaN.
> > Significance arithmetic is a great tool in some fields of mathematics,
> > but it has the same problem as interval arithmetic. Correlation
> > between errors is not included. A really good numerical error analysis
> > gets you much better results.
>
> Have you ever actually tried this?!
What do you think I have been doing all those years in the numerical
mathematics department of this institute?
Steve Richfie1d
> > > I think you do not understand the IEEE standard well enough. You complained
> > > that in a NaN 23 bits are unused. This is false. The standard
> > > specifically allows to encode the reason of NaN generation in those
> > > bits (but that part is not standardised).
> >
> > And, that part not being standardized IS this particular problem with
> > the IEEE standard, because there are LOTS of different NANs that need
> > DIFFERENT handling by the FP hardware, both in production, AND in HANDLING.
>
> It could not be standardised yet in that case because there was
> insufficient knowledge.
There is now! Wayne Albin developed this 30-years ago. I'm just working
from 30-year-old memories and common sense.
If you'd like, email me off-forum and I'll forward it to Wayne.
> > > Also, one bit is reserved
> > > to distinguish between a signalling NaN and a non-signalling NaN.
> >
> > Of course, signaling should be under user control, because some
> > applications demand stopping at ALL exceptions, while others should be
> > left to run no matter what.
>
> Whether a signalling or non-signalling NaN stops the program is under
> user control... The distinction is mostly between generated NaN's and
> propagated NaN's.
This subject is much deeper than the IEEE developers have yet realized.
There is a gradual demotion/promotion of special forms, where they can
get more or less vague through computations - sometimes even rising back
to real numbers, e.g. dividing by infinity. Whether a particular
demotion should interrupt the program is properly a function of the
application or subroutine that it is running in.
> > However, some operations, like dividing by a completely underflowed/zero
> > denominator, cannot produce a sign to retain, so you also need unsigned
> > infinity to represent such result, to avoid representing a potentially
> > erroneous sign.
>
> But zero has a sign in IEEE. Just *because of underflow*.
YES, and that often erroneous sign can get you into a LOT of needless
trouble further on.
I
> > wonder, why did they discard it after the 8087? Any idea?
>
> They replaced it by signed infinities (which the 8087 did not have). There
> are mathematically two realistic models, one with a single infinity and one
> with two infinities.
Still doesn't explain why, when both signed and unsigned are clearly
needed in a fully-functional system.
> > Indefinites ARE numbers that the particular implementation is unable to
> > either compute or represent, so NaN is really a misnomer. Maybe it
> > should be NaRN = Not a Representable Number.
>
> Hrm. So +oo - +oo is not representable? Wrong. It is nonsense.
With enough bits in the exponent, you wouldn't have any oo's. +oo is a
statement that your arithmetic system was unable to do what you asked it
to, but not because what you asked was reasonable, just beyond the
capacity of the exponent.
> > However, the square root of a negative number or the arc sin of a number
> > greater than 1 is really NOT a number, and so would be a legitimate NaN.
>
> But the square root of -1 is i, and the arc sin of 2 is some other complex
> number, so they are definitely numbers, but not representable.
>
> I think you have this backwards.
Come on now - you're trying to be perverse. We are talking about REAL
numbers here, unless of course you are programming in FORTRAN that has a
complex number form - but even these are handled as a pair of reals.
> > It is REALLY important to distinguish between these two cases, because
> > you need to know whether you need to rearrange some expression, or chase
> > a genuine error in your logic.
>
> A good implementation of the arithmetic will provide you with the information.
> Take a look at the Transputer, which will encode the source of the NaN.
Not throwing away the numeric/non-numeric nature of the NaN also helps.
The point you seem to be missing here is that the EASY part is putting
this information into the NaN like the Transputer does. Then comes the
interesting part - using this information in subsequent computations to
produce appropriate results through promotion/demotion.
> > > Significance arithmetic is a great tool in some fields of mathematics,
> > > but it has the same problem as interval arithmetic. Correlation
> > > between errors is not included. A really good numerical error analysis
> > > gets you much better results.
> >
> > Have you ever actually tried this?!
>
> What do you think I have been doing all those years in the numerical
> mathematics department of this institute?
1. Unfortunately, I don't know where you work or what you do. Please,
correct my ignorance and tell me a bit about yourself.
2. OK, by now you surely KNOW that manual error analysis is no trivial
task, if indeed it is even possible in complex cases, e.g. the presence
of many variable-significance inputs hitting non-linear functions. Why
on earth would you ever defend the IEEE FP standard that complicates
your work by throwing the significance information away? Employment
security?! What's the story here? Please explain.
Note, making significance work right REQUIRES that a lot of "loose ends"
in the IEEE FP standard be cleaned up, e.g. a real genuine guard digit
(not just a bit in the adder) is an absolute necessity. Hence, I'm
discussing these points, not to "throw stones" at the IEEE standard, but
to remove the subtle barriers to significance. There are some who claim
that significance cannot be made to work well, including the proposer of
the decimal standard (obviously absurd, since it has 2 years of
commercial service) citing a variety of needless IEEE-manufactured
impediments. Obviously, these impediments need removal for the standard
to truly advance.
Steve Richfield
Steve Richfie1d
> I still do not see *any* order of operation problems. You seem to think
> that & is a logical operator. It is *not* in C. In C it is an arithmetic
> operator (a bitwise and). So you need to write:
> (a > b) & (c > d)
> with parenthesis, but in that case you use an arithmetic operator, you
> should use the logical one:
> a > b && c > d
The problems are well-known, and even Denis Ritchie admits them. As I
recall, the most in-your-face case was mixing bitwise and boolean
operators. Obviously, the bitwise operators should be done first, but
the C standard specified, so the boolean operations would be performed
on the bitwise arguments, obvious nonsense, then bitwise operators would
be performed on the results of the boolean operations - even more
nonsense. This was all done with so much as a warning message.
> But you critique a language without looking at the details.
No, I was critiquing the best example I know of a COMPLETELY broken
standards process, where special interests all want to keep things just
as they are - bugs and all. I was offering some examples from memory of
15 years ago.
This same broken standards process appears to be hard at work with IEEE FP.
> Not if the variable was declared "volatile".
You're right.
> > I have worked on two FORTRAN compilers. One was only slightly
> > optimizing, and didn't know enough to extract common subexpressions. The
> > other was probably the MOST optimizing ever written - the one CDC
> > developed for their supercomputers.
>
> In my opinion Cray and NEC were better, but that is just an opinion.
Cray's optimizer was better, but CDC's vectorizer was better. This all
gets down to the issue that broke Cray away from CDC. Cray wanted better
scalar performance, while CDC wanted better vector performance. In the
end, they both got what they wanted - and both went down the tubes for
being too narrow in their views of things.
> Yes, I know. We had one instruction for the 205 that could not be timed
> because the timer overflowed a few times before the instruction finished.
Sounds like you too have been there and done that.
> But to get back to Fortran. I know of a library where some subroutines
> needed to know whether two actual array arguments were in fact the same
> array. There is *no* way to do that in standard Fortran 66 or Fortran 77.
Unless of course you are running on a Borroughs 5000/6000 system. There,
you didn't refer to a memory address, but an array descriptor and
subscripts as part of the way the hardware was wired. This produced MUCH
more reliable code than did any other machine I know of, as exceeding an
array bound caused an interrupt - without costing anything for the
subscript computation or tests.
If micros were only built this way, system crashes, worms, etc., would
be a thing of the past. Hey, Intel, would you like to make a RELIABLE
processor?!
Steve Richfield
Steve Richfie1d
> > Financial and other predictive systems are fundamentally
> > significance-limited, in that you can only predict the future until you
> > lose all of your significance, after which your predictions are PURE
> > garbage. In a sense, the significance is equally or perhaps even more
> > important than the value being computed in these applications! However,
> > people doing this do NOT currently use significance! Why? Because,
> > unlike chemists, their computations are WAY too complex to do by hand,
> > and IEEE-754 has deprived them of any opportunity to do this by machine.
> > The results to our economy and ecology are all too tragic.
>
> Tell that to the people that make weather forecasts. With significance
> included, I do not think you can predict the weather beyond a few hours.
Actually, I have some experience here, as weather forecasting used to be
done on CDC-205 computers while I was at CDC, and some of their problems
found their way onto my desk.
Weather forecasting is a gigantic simulation of a system that though not
entirely linear (as water vaporizes and condenses), it does NOT have the
really radical behavior of, say, financial systems that can "turn on a
dime". Hence, significance can be reasonably well estimated by simply
observing the accuracy of the system and expecting pretty much the same
in the future - which of course is exactly what they now do.
Climate prediction on the other hand is not so simple/linear, as climate
changes by their nature involve non-linear processes, like trying to
predict how much of the ocean will evaporate as our temperature rises,
how much will condense into glaciers, etc.
And, financial forecasting is the greatest challenge of all, because
money has little/no inertia, so huge amounts can move in an instant.
Here, significance appears to be the present barrier to advancing the
state of the art.
Steve Richfield
Dik T. Winter
> > > And, that part not being standardized IS this particular problem with
> > > the IEEE standard, because there are LOTS of different NANs that need
> > > DIFFERENT handling by the FP hardware, both in production, AND in HANDLING.
> >
> > It could not be standardised yet in that case because there was
> > insufficient knowledge.
>
> There is now! Wayne Albin developed this 30-years ago. I'm just working
> from 30-year-old memories and common sense.
Possibly, but at the time the standard emerged it was not available.
Kahan (the major one, but also one amongst the others) has given
compelling reasons for the IEEE standard.
> > Whether a signalling or non-signalling NaN stops the program is under
> > user control... The distinction is mostly between generated NaN's and
> > propagated NaN's.
>
> This subject is much deeper than the IEEE developers have yet realized.
> There is a gradual demotion/promotion of special forms, where they can
> get more or less vague through computations - sometimes even rising back
> to real numbers, e.g. dividing by infinity. Whether a particular
> demotion should interrupt the program is properly a function of the
> application or subroutine that it is running in.
Right so, so it is the application's or subroutine's responsiility, not
of the hardware. IEEE properly delegates this to that place.
> > > However, some operations, like dividing by a completely underflowed/zero
> > > denominator, cannot produce a sign to retain, so you also need unsigned
> > > infinity to represent such result, to avoid representing a potentially
> > > erroneous sign.
> >
> > But zero has a sign in IEEE. Just *because of underflow*.
>
> YES, and that often erroneous sign can get you into a LOT of needless
> trouble further on.
If your underflow gives an erroneous sign, there is a problem with the
application.
> > They replaced it by signed infinities (which the 8087 did not have). There
> > are mathematically two realistic models, one with a single infinity and one
> > with two infinities.
>
> Still doesn't explain why, when both signed and unsigned are clearly
> needed in a fully-functional system.
I still do not see that need. It is different. When I was involved with
Ada standardisation, standardising complex arithmetic had problems *because
of* the signed infinities.
> > Hrm. So +oo - +oo is not representable? Wrong. It is nonsense.
>
> With enough bits in the exponent, you wouldn't have any oo's. +oo is a
> statement that your arithmetic system was unable to do what you asked it
> to, but not because what you asked was reasonable, just beyond the
> capacity of the exponent.
You think so? How about division by an exact 0?
> > But the square root of -1 is i, and the arc sin of 2 is some other complex
> > number, so they are definitely numbers, but not representable.
> >
> > I think you have this backwards.
>
> Come on now - you're trying to be perverse. We are talking about REAL
> numbers here, unless of course you are programming in FORTRAN that has a
> complex number form - but even these are handled as a pair of reals.
Nope. I am (in the core) a mathematician. And when you talk about numbers,
there are other numbers than real numbers. Talking as a computer scientist,
REAL's are not the real numbers, they are only a subset of the rational
numbers.
> > A good implementation of the arithmetic will provide you with the information.
> > Take a look at the Transputer, which will encode the source of the NaN.
>
> Not throwing away the numeric/non-numeric nature of the NaN also helps.
>
> The point you seem to be missing here is that the EASY part is putting
> this information into the NaN like the Transputer does. Then comes the
> interesting part - using this information in subsequent computations to
> produce appropriate results through promotion/demotion.
That is *not* the responsibility of the hardware. It is the application's
responsibility whether a NaN coming from a square root of a negative
number should continue, whether the result should be replaced by 0, or
whatever.
> > > Have you ever actually tried this?!
> >
> > What do you think I have been doing all those years in the numerical
> > mathematics department of this institute?
>
> 1. Unfortunately, I don't know where you work or what you do. Please,
> correct my ignorance and tell me a bit about yourself.
For a long time involved in the development of libraries for numerical
mathematics in quite a few fields. Including error analysis.
> 2. OK, by now you surely KNOW that manual error analysis is no trivial
> task, if indeed it is even possible in complex cases, e.g. the presence
> of many variable-significance inputs hitting non-linear functions.
You know, even if they hit linear functions it may still be non-trivial.
Doing forward error analysis on linear systems of equations leads to the
conclusion that the results of 40 equations with 40 unknowns are bogus.
*Even when you assume the input is exact.* This has already been noted
by von Neumann. In the 50's, 60's, J.H. Wilkinson developed an approach
(backward error analysis) where it could be shown that the results where
the exact result for a question close to the original. And that is the
best you can have. Unless you are willing to go to the Karlsruhe method,
where they are able to provide solutions with guaranteed error bounds.
But that comes at a (huge) price.
> Note, making significance work right REQUIRES that a lot of "loose ends"
> in the IEEE FP standard be cleaned up, e.g. a real genuine guard digit
> (not just a bit in the adder) is an absolute necessity.
Have you ever considered what that would mean in hardware cost and speed?
Dik T. Winter
> > Tell that to the people that make weather forecasts. With significance
> > included, I do not think you can predict the weather beyond a few hours.
>
> entirely linear (as water vaporizes and condenses), it does NOT have the
> really radical behavior of, say, financial systems that can "turn on a
> dime".
Perhaps, but also weather forecasting is a bit chaotic.
> Hence, significance can be reasonably well estimated by simply
> observing the accuracy of the system and expecting pretty much the same
> in the future - which of course is exactly what they now do.
Rather they use error analysis, but I will ask my colleagues that are
working with large differential equations for such problems.
> And, financial forecasting is the greatest challenge of all, because
> money has little/no inertia, so huge amounts can move in an instant.
> Here, significance appears to be the present barrier to advancing the
> state of the art.
I do not think so. Most of finance prediction involves non-predictable
hype.
Dik T. Winter
> > I still do not see *any* order of operation problems. You seem to think
> > that & is a logical operator. It is *not* in C. In C it is an arithmetic
> > operator (a bitwise and). So you need to write:
> > (a > b) & (c > d)
> > with parenthesis, but in that case you use an arithmetic operator, you
> > should use the logical one:
> > a > b && c > d
>
> The problems are well-known, and even Denis Ritchie admits them.
Can you give a citation?
> As I
> recall, the most in-your-face case was mixing bitwise and boolean
> operators.
The most obvious is the absence of a boolean type. When that is not
present, the mixing is quite logical.
> Obviously, the bitwise operators should be done first, but
> the C standard specified, so the boolean operations would be performed
> on the bitwise arguments, obvious nonsense, then bitwise operators would
> be performed on the results of the boolean operations - even more
> nonsense. This was all done with so much as a warning message.
Eh? I cannot follow. The only logical operations in C are && and ||.
I see no problem with them.
> > In my opinion Cray and NEC were better, but that is just an opinion.
>
> Cray's optimizer was better, but CDC's vectorizer was better.
I doubt that. Both Cray and NEC performed pretty well, while on the 205
we had to use Q8 calls to get what we wanted. There was *no* way on the
205 to get the performance we obtained, except by using Q8 calls.
> > Yes, I know. We had one instruction for the 205 that could not be timed
> > because the timer overflowed a few times before the instruction finished.
>
> Sounds like you too have been there and done that.
Yes, been there, done that. On the 205, the Cray and the NEC.
> > But to get back to Fortran. I know of a library where some subroutines
> > needed to know whether two actual array arguments were in fact the same
> > array. There is *no* way to do that in standard Fortran 66 or Fortran 77.
>
> Unless of course you are running on a Borroughs 5000/6000 system. There,
> you didn't refer to a memory address, but an array descriptor and
> subscripts as part of the way the hardware was wired.
This is far aside from what I posted. How on the Burroughs 5000/6000 system
would you detect (in standard Fortran) whether two array arguments are
actually the same array? Honest. There is *no* way to do that.
> If micros were only built this way, system crashes, worms, etc., would
> be a thing of the past. Hey, Intel, would you like to make a RELIABLE
> processor?!
Only if it ran faster, I think. As a professor at Karlsruhe once said:
"The Cray is unsafe at any speed".
And talking about broken arithmetic. Consider the 205 where a == b did
*not* imply b == a (it is explicitly noted in the hardware manual, I
still have a copy here). Consider the Cray where with some f-p instructions
the result could be 0 rather than the more correct 1 (not documented in
manuals, the result of non-shifting before storing in some cases).
[ It was the floating point half precision multiply.] I have not done
enough with the NEC to know whether that machine had anomalies.
Steve Richfie1d
> > > > And, that part not being standardized IS this particular problem with
> > > > the IEEE standard, because there are LOTS of different NANs that need
> > > > DIFFERENT handling by the FP hardware, both in production, AND in HANDLING.
> > >
> > > It could not be standardised yet in that case because there was
> > > insufficient knowledge.
> >
> > There is now! Wayne Albin developed this 30-years ago. I'm just working
> > from 30-year-old memories and common sense.
>
> Possibly, but at the time the standard emerged it was not available.
> Kahan (the major one, but also one amongst the others) has given
> compelling reasons for the IEEE standard.
Well, the next time the standard is revised, then these should be fixed,
shouldn't they?
> > > Whether a signalling or non-signalling NaN stops the program is under
> > > user control... The distinction is mostly between generated NaN's and
> > > propagated NaN's.
> >
> > This subject is much deeper than the IEEE developers have yet realized.
> > There is a gradual demotion/promotion of special forms, where they can
> > get more or less vague through computations - sometimes even rising back
> > to real numbers, e.g. dividing by infinity. Whether a particular
> > demotion should interrupt the program is properly a function of the
> > application or subroutine that it is running in.
>
> Right so, so it is the application's or subroutine's responsiility, not
> of the hardware. IEEE properly delegates this to that place.
>
> > > > However, some operations, like dividing by a completely underflowed/zero
> > > > denominator, cannot produce a sign to retain, so you also need unsigned
> > > > infinity to represent such result, to avoid representing a potentially
> > > > erroneous sign.
> > >
> > > But zero has a sign in IEEE. Just *because of underflow*.
> >
> > YES, and that often erroneous sign can get you into a LOT of needless
> > trouble further on.
>
> If your underflow gives an erroneous sign, there is a problem with the
> application.
Consider the following code for which the IEEE standard (as I understand
it) produces the wrong answer. Of course, the constants shown would be
variables obtained elsewhere, with duplicate shown values being
different variables with the same value:
! This produces zero, but without knowing more about the two !
arguments, it is impossible to know the proper sign, if any.
! Of course, the present standard will arbitrarily invent a
! sign.
X = 5.0 - 5.0
! Same situation here, but with different arguments.
Y = 3.0 - 3.0
! This should produce an indefinite, and NOT a signed oo
! because there is no way to know if the signs of X and Y
! are the same.
Z = X + Y
! This should print a string indicating that the answer cannot
! be computed, but instead IEEE prints 4.
Print 4 + 13/Z
> > > They replaced it by signed infinities (which the 8087 did not have). There
> > > are mathematically two realistic models, one with a single infinity and one
> > > with two infinities.
> >
> > Still doesn't explain why, when both signed and unsigned are clearly
> > needed in a fully-functional system.
>
> I still do not see that need. It is different. When I was involved with
> Ada standardisation, standardising complex arithmetic had problems *because
> of* the signed infinities.
The above code also impinges on this problem. You need unsigned
infinities to deal with the reciprocals of unsigned zeros. You need
unsigned zeros for situations where the sign cannot be properly
determined, like the difference between identically valued arguments.
Sure, you can adopt a standard that doesn't do this, but then you can
end up getting wrong answers as in the above example.
I constructed the ADA linker for ROLM Corp's ADA compiler!
>
> > > Hrm. So +oo - +oo is not representable? Wrong. It is nonsense.
> >
> > With enough bits in the exponent, you wouldn't have any oo's. +oo is a
> > statement that your arithmetic system was unable to do what you asked it
> > to, but not because what you asked was reasonable, just beyond the
> > capacity of the exponent.
>
> You think so? How about division by an exact 0?
IEEE does not have an exact numerical representation, nor does it
specify mixed-mode arithmetic, e.g. 5.0 / 0 where 0 is an integer.
Note that Borroughs B5000/B6000 systems and that 30-year-old system I
keep talking about DID have mixed-mode representations - yet another
thing some would see as "missing" from the IEEE spec. There was a
special exponent that indicated that the fraction was an EXACT integer
vs. a number only known as precisely as there were bits to represent.
Division by integer zero on both of these systems was a special form all
its own that could only be gotten this way. To the 30-year old system,
this was functionally equivalent to an unsigned infinity in future
computation, so using it as a denominator could produce an unsigned zero
result.
> > > But the square root of -1 is i, and the arc sin of 2 is some other complex
> > > number, so they are definitely numbers, but not representable.
> > >
> > > I think you have this backwards.
> >
> > Come on now - you're trying to be perverse. We are talking about REAL
> > numbers here, unless of course you are programming in FORTRAN that has a
> > complex number form - but even these are handled as a pair of reals.
>
> Nope. I am (in the core) a mathematician. And when you talk about numbers,
> there are other numbers than real numbers. Talking as a computer scientist,
> REAL's are not the real numbers, they are only a subset of the rational
> numbers.
As a mathematician, IEEE FP must make you sick!
> > > A good implementation of the arithmetic will provide you with the information.
> > > Take a look at the Transputer, which will encode the source of the NaN.
> >
> > Not throwing away the numeric/non-numeric nature of the NaN also helps.
> >
> > The point you seem to be missing here is that the EASY part is putting
> > this information into the NaN like the Transputer does. Then comes the
> > interesting part - using this information in subsequent computations to
> > produce appropriate results through promotion/demotion.
>
> That is *not* the responsibility of the hardware. It is the application's
> responsibility whether a NaN coming from a square root of a negative
> number should continue, whether the result should be replaced by 0, or
> whatever.
But, how can it distinguish between the square root of a negative number
and other NaN producing operations? Not all NaNs are created equal,
except in IEEE FP. The result of a division by zero could be promoted by
using it in a denominator, whereas the square root of a negative number
could potentially be promoted by multiplying by zero. The whole reason
for distinguishing special forms is to provide for future promotion, and
either overspecification (like requiring that oo be signed) or
underspecification (like having no sign for oo) REALLY gets in the way
of proper future promotion.
Reading the IEEE spec, it seems obvious (to me) that promotion was never
even considered.
The theory behind all this goes as follows: Presume that the programmer
was an excellent mathematician but a less than stellar programmer, so
they have NOT re-arranged their expressions to avoid overflows, etc.
What could be then done with special forms to STILL make such code
produce correct answers whenever possible? Wayne worked this problem and
came up with the forms that I have been discussing.
Most real-world exceptions from good logic come from things like
dividing by the difference between identically-valued variables, and NOT
by manipulating things with astronomical exponents. There is no way that
special forms can recover 100% of these cases, as I could easily present
perverse examples that do NOT promote to numbers. However, a carefully
designed system of special forms promotes a large fraction of these
situations back to usable numbers. The quality of such systems is
measured by the fraction of such situations that can be promoted to
numbers by future computation. For IEEE, this fraction is not very good
at all. I think they only recover underflows and (n / oo).
> > > > Have you ever actually tried this?!
> > >
> > > What do you think I have been doing all those years in the numerical
> > > mathematics department of this institute?
> >
> > 1. Unfortunately, I don't know where you work or what you do. Please,
> > correct my ignorance and tell me a bit about yourself.
>
> For a long time involved in the development of libraries for numerical
> mathematics in quite a few fields. Including error analysis.
A kindred spirit.
> > 2. OK, by now you surely KNOW that manual error analysis is no trivial
> > task, if indeed it is even possible in complex cases, e.g. the presence
> > of many variable-significance inputs hitting non-linear functions.
>
> You know, even if they hit linear functions it may still be non-trivial.
> Doing forward error analysis on linear systems of equations leads to the
> conclusion that the results of 40 equations with 40 unknowns are bogus.
> *Even when you assume the input is exact.* This has already been noted
> by von Neumann. In the 50's, 60's, J.H. Wilkinson developed an approach
> (backward error analysis) where it could be shown that the results where
> the exact result for a question close to the original. And that is the
> best you can have. Unless you are willing to go to the Karlsruhe method,
> where they are able to provide solutions with guaranteed error bounds.
> But that comes at a (huge) price.
My point is that in the real world, almost no one has either the time
nor the education to do a proper error analysis on what they are doing,
even if it is technically possible, yet they often need some indication
of the significance of their result.
Now that you understand the problems with error analysis, I would think
that you would have already joined the significance bandwagon!
> > Note, making significance work right REQUIRES that a lot of "loose ends"
> > in the IEEE FP standard be cleaned up, e.g. a real genuine guard digit
> > (not just a bit in the adder) is an absolute necessity.
>
> Have you ever considered what that would mean in hardware cost and speed?
There isn't enough bits for a guard digit in 32-bit FP as it is
customarily used, but it is zero cost for 48-bit and longer FP. I think
the ONLY things that need to be done to implement a guard digit are:
1. Write it into the spec that it is there, so that people writing I/O
routines know not to display it on output.
2. Don't look at the guard digit when comparing arguments. Note that
subtracting and testing signs and for exactly zero doesn't work quite
right in the presence of a guard digit. However, considering a result to
be zero if all but the guard digit is zero works OK. Note that this
makes most tests for equality between FP numbers now work, whereas in
most implementations, e.g. IEEE this does NOT work right.
Steve Richfield
Steve Richfie1d
> > > I still do not see *any* order of operation problems. You seem to think
> > > that & is a logical operator. It is *not* in C. In C it is an arithmetic
> > > operator (a bitwise and). So you need to write:
> > > (a > b) & (c > d)
> > > with parenthesis, but in that case you use an arithmetic operator, you
> > > should use the logical one:
> > > a > b && c > d
> >
> > The problems are well-known, and even Denis Ritchie admits them.
>
> Can you give a citation?
Not really. As I recall, page 49 or the original K&R book listed the
precedences, which a little examination you'll see had && and || mixed
up - at least contrary to what would seem reasonable.
> > recall, the most in-your-face case was mixing bitwise and boolean
> > operators.
>
> The most obvious is the absence of a boolean type.
Agreed.
> When that is not
> present, the mixing is quite logical.
Not really given the particular precedence that C adopted.
>
> > Obviously, the bitwise operators should be done first, but
> > the C standard specified, so the boolean operations would be performed
> > on the bitwise arguments, obvious nonsense, then bitwise operators would
> > be performed on the results of the boolean operations - even more
> > nonsense. This was all done with so much as a warning message.
>
> Eh? I cannot follow. The only logical operations in C are && and ||.
> I see no problem with them.
Obviously you want to do the bitwise first, then the boolean, as bitwise
operation on a boolean result doesn't make much sense. However, C
specifies this the other way around.
>
> > > In my opinion Cray and NEC were better, but that is just an opinion.
> >
> > Cray's optimizer was better, but CDC's vectorizer was better.
>
> I doubt that. Both Cray and NEC performed pretty well, while on the 205
> we had to use Q8 calls to get what we wanted. There was *no* way on the
> 205 to get the performance we obtained, except by using Q8 calls.
According to an internal report that CDC prepared, Crays vectorized
about 50% of loops that weren't hand-modified for vectorization, whereas
CDCs vectorized about 70%. Both system used the common methods like
scalar promotion. Most of the difference was because CDCs had bit
vectors, so that most loops containing conditional statements could be
vectorized by first precomputing the results of all of the conditionals
into a bit vector that the compiler did quite well, then passing the bit
vector to following vector operations to suppress the storing of results
where the corresponding bit was a zero.
Of course, Crays didn't have to even bother to compute the results that
CDCs would suppress, so they gained a bunch there. The net results from
the internal benchmarks that I saw was that Crays won with scalars and
short vectors, whereas CDCs won with long vector applications.
Of course, most day to day applications were scalar and short vector,
but the REALLY big applications were mostly long vector. Hence,
universities were usually better off with Crays, while national
laboratories were usually better off with CDCs.
It's nice that we can now talk about these things, now that both
companies are no longer around to get upset with us over talking about
their innermost secrets.
> > > Yes, I know. We had one instruction for the 205 that could not be timed
> > > because the timer overflowed a few times before the instruction finished.
> >
> > Sounds like you too have been there and done that.
>
> Yes, been there, done that. On the 205, the Cray and the NEC.
I only have direct experience on the 205 at CDC.
> > > But to get back to Fortran. I know of a library where some subroutines
> > > needed to know whether two actual array arguments were in fact the same
> > > array. There is *no* way to do that in standard Fortran 66 or Fortran 77.
> >
> > Unless of course you are running on a Borroughs 5000/6000 system. There,
> > you didn't refer to a memory address, but an array descriptor and
> > subscripts as part of the way the hardware was wired.
>
> This is far aside from what I posted. How on the Burroughs 5000/6000 system
> would you detect (in standard Fortran) whether two array arguments are
> actually the same array? Honest. There is *no* way to do that.
First, I have no direct experience on the Burroughs computers - just a
bunch of long-past manual reading and discussing it with users. That
having been said, you should be able to examine the array descriptors
passed as arguments, though I don't think the language provided any
syntax for this. The 5000 series computers had "stream procedures", a
sort of in-line assembly language to do such brain-surgery. I don't know
how to approach this on the 6000. Maybe there is an old Burroughs guru
watching this thread who could set us both right?! WAKE UP you Burroughs
lurkers!
> > If micros were only built this way, system crashes, worms, etc., would
> > be a thing of the past. Hey, Intel, would you like to make a RELIABLE
> > processor?!
>
> Only if it ran faster, I think. As a professor at Karlsruhe once said:
> "The Cray is unsafe at any speed".
From what I can tell, array-descriptor/subscripting hardware is in ALL
ways superior to computing addresses and dealing with pointers as all
non-Burroughs implementations do. An in-between approach along this path
but not quite as good was as adopted in the IBM 360/370, where multiple
indexing in a single instruction didn't enforce subscript checking, but
nonetheless allowed for really efficient subscripting. The 360
architecture would have made MUCH better micros than Intel's current
structure, and the Burroughs approach would have been MUCH better still
than that, though the early units would have been a bit slower until
they got their transistor counts up high enough to do the extra work in
parallel.
The root problem of course is that the early micro people at Intel were
really lightweights in the computer department, who really weren't
looking very far into the future.
> And talking about broken arithmetic. Consider the 205 where a == b did
> *not* imply b == a (it is explicitly noted in the hardware manual, I
> still have a copy here). Consider the Cray where with some f-p instructions
> the result could be 0 rather than the more correct 1 (not documented in
> manuals, the result of non-shifting before storing in some cases).
> [ It was the floating point half precision multiply.] I have not done
> enough with the NEC to know whether that machine had anomalies.
... and when I talk about anomalies in IEEE FP everyone screams HERESY!
Sure, IEEE gives reasonable non-significance answers for straightforward
numerical computations, but the loss of significance is REALLY
horrendous for a small percentage of applications that are THE most
critical to our society.
Another topic we've barely touched is the absence of a 16-bit
logarithmic representation. With no standard for this, there has been no
merging of DSP and desktop technologies. Given all of the inherent
shortcoming of logarithmic arithmetic, you REALLY need to get your NaNs
right to make things work smoothly - which DSPs don't currently do. As
an aside, I know of methods to squeeze about 1,000 times the computing
power from a laptop with the SAME amount of silicon using memories with
embedded logarithmic processors. This can't happen unless/until the IEEE
FP spec is amended.
Note that when a system supports multiple FP formats, like single and
double precision, not to mention adding significance and/or logarithmic,
you get a new NaN problem - things that were NaNs in one representation
but may not have been in the other, e.g. where a single precision
computation overflowed to produce oo, and then was converted to double
precision where the result would have been representable. Now saying
that it is STILL oo with a much bigger exponent range misstates the
situation. I'm not sure what the best answer is here as I haven't (yet)
worked all of the cases out. Perhaps a "big" special form that works a
lot like oo, but prints out something like ">1E38". Any thoughts?
Steve Richfield
Steve Richfie1d
> > And, financial forecasting is the greatest challenge of all, because
> > money has little/no inertia, so huge amounts can move in an instant.
> > Here, significance appears to be the present barrier to advancing the
> > state of the art.
>
> I do not think so. Most of finance prediction involves non-predictable
> hype.
As I understand the situation (if indeed ANYONE does), most of the time
things are very predictable, but often they are not. Indeed, this is
where market volatility indicators came from. They can tell you that
there is going to be a big change, but can't even produce the sign of
that change.
In looking ahead, you often project yourself right into a volatile
situation, past which you really can't see very far at all.
The problem is that there are a LOT of things that can precipitate
market volatility, only some of which are incorporated into the
volatility indicators, so you can't really depend on such indicators,
which really only give you significance information to know when to
quit, and NOT the value information that you need to project further ahead.
Significance gives you a handle on the volatility factors that are NOT
part of market volatility indicators.
Of course, volatility is really a metastable condition with large
competing forces pulling in different directions. Metastable situations,
where the effect is the difference between large nearly-equal numbers,
is inherently a high-value, low-significance proposition, which tends to
poison further computation, at least if you are tracking significance.
Note that the whole concept of a market volatility indicator, where it
is the significance and NOT the value or sign that is being indicated,
sure shows the recognized value of significance information in financial
predictive applications.
The **BIG** point make in our paper about to be presented is that
tracking significance information will blunt market peaks and valleys by
normalizing valuations depending on the reliability (significance) of
the computations, and thereby (hopefully) stabilize world markets.
Steve Richfield
Tim Peters
...
> Consider the following code for which the IEEE standard (as I understand >
it) produces the wrong answer. Of course, the constants shown would be
> variables obtained elsewhere, with duplicate shown values being
> different variables with the same value:
>
> ! This produces zero, but without knowing more about the two
> ! arguments, it is impossible to know the proper sign, if any.
> ! Of course, the present standard will arbitrarily invent a
> ! sign.
> X = 5.0 - 5.0
The result is +0.0 under IEEE.
> ! Same situation here, but with different arguments.
> Y = 3.0 - 3.0
Ditto.
> ! This should produce an indefinite, and NOT a signed oo
Does "oo" mean infinity here?
> ! because there is no way to know if the signs of X and Y
> ! are the same.
> Z = X + Y
The result here is also +0.0 (+0.0 + +0.0 -> +0.0).
> ! This should print a string indicating that the answer cannot
> ! be computed, but instead IEEE prints 4.
> Print 4 + 13/Z
The 13/Z part signals the IEEE divide-by-0 condition. By default, that (a)
sets the divide-by-0 sticky status flag; and, (b) returns +oo (positive
infinity). 4 plus positive infinity is then unexceptional, and returns
positive infinity.
...
> IEEE does not have an exact numerical representation, nor does it
> specify mixed-mode arithmetic, e.g. 5.0 / 0 where 0 is an integer.
The letter and spirit of 754 take the representable numbers at face value,
as if exact. 754 operations treat 0.0 as exactly 0, 1.5 as exactly 3/2, and
so on. In particular, nothing in that standard views representable numbers
as approximations to "the real" results. That's a view you can layer on top
of it, but it's not the standard's view. It's true that, e.g., 1.0/10.0
can't be represented exactly in 754, and the standard then mandates that (a)
the closest representable number is the result; and, (b) the inexact
condition is signaled. But after that, the result is again taken at face
value, and, e.g., subtracting the result from itself does not signal
inexact.
Note that the standard requires support for directed rounding modes
primarily to aid efficient implementation of interval arithmetic. After 20
years, it's still hard to get at this stuff from common computer languages
in a portable way, so packages exploiting these features are finicky.
Here's a good interval package for C++, exploiting HW acceleration when it
can, but not visibly (not via the interface):
http://www.boost.org/libs/numeric/interval/doc/interval.htm
Develop software along those lines, and you can give users any arithmetic
model you want, with or without direct HW support.
Speaking of which, I've read all the msgs in this thread, and am afraid I
still don't know what you're proposing, just that it's going to save the
world somehow <wink>. Do you have a writeup available with a specific,
concrete, detailed proposal?
...
> According to an internal report that CDC prepared, Crays vectorized
> about 50% of loops that weren't hand-modified for vectorization, whereas
> CDCs vectorized about 70%.
Which universe of loops was considered? I worked on CFT and CFT77 from '79
to '88 at Cray. As the years went on, more vectorization benchmarks
appeared, with increasingly convoluted loop nests, and the percentage of
loops we vectorized on those decreased most of the time I was there. We
didn't care much at the time, because customers and potential customers told
us we were vectorizing more of the loops in their actual production code
than other vendors they evaluated. Since CFT was written in Cray assembler,
and a real bear to modify because of it, it's certainly true that Cray
encouraged rewriting loops for a long time, but in standard Fortran rather
than via calling Cray-specific libraries. Most customers were very happy
with that, because the suggested rewritings usually reduced clutter and
improved clarity, after which those loops often also ran faster on
non-vector machines with non-Cray compilers.
I haven't been to a $50K internal marketing party since the cold war ended
<wink>.
Jeff Kenton
> ... and when I talk about anomalies in IEEE FP everyone screams HERESY!
> Sure, IEEE gives reasonable non-significance answers for straightforward
> numerical computations, but the loss of significance is REALLY
> horrendous for a small percentage of applications that are THE most
> critical to our society.
Steve,
You keep talking about "applications that are THE most critical to our
society". For things that critical, if they really exist, I would expect the
few people involved (how many? 5?) would be willing to do the error analysis
to get the right answers.
You're talking about one software application in 100,000 or less, but you want
standards and hardware on every computer in the world changed to meet your
needs. It's not going to happen.
--
Toby Thain
Steve Richfie1d <St...@NOSPAM.smart-life.net> wrote:
> Dale,
>
> ...
> > I doubt this very much. Situations like the FDIV problem in Pentium are
> > seen by the public as "bugs" because they are anomalies. A bug, for most
> > people, is defined as a result that differs from the results on other kinds
> > of machines (calculators, other computers, etc.)
>
> ... or the correct answer?
>
> > For the problem with loss of significance, there's no expectation (for most
> > people) of anything different than the way a calculator works. If you were
> > to build a machine that reports result + significance, and it reports that a
> > particular calculation has lost all significance and hence there is no
> > answer of any value, most people would see THAT as a bug, and say "just give
> > me the darn answer".
>
> ... and THAT is why our economy and ecology is in the mess it is in.
Steve,
In one evening's reading, your jeremiad on the issue of significance
arithmetic has made you an instant hero of mine. I grok that simply "not
knowing" the significance of simulated results, and when they should be
ignored entirely, can be extremely dangerous.
I also appreciated your remarks on the shortcomings of IEEE 754 (e.g.,
undifferentiated NaN) and the C standards.
>
> ...
>
> I've tried to get interest in developing a new sort of "smart memory". I
> have a memory-process-compatible architecture for logarithmic vector
> processor cores, so that you could sprinkle dozens/hundreds throughout a
> large memory chip. This would provide somewhere around 3 orders of
> magnitude increase in the short-precision computing capability of a PC.
> Here's the kicker. For complex reasons I don't want to post, these would
> actually be as cheap or cheaper than making just the bare memory without
> the processors! As you have probably already guessed, there weren't any
> currently big applications needing that much crunching power, and there
> was no market demand for these non-existent devices, so no semi maker
> wanted to develop these. Hence, we are now stuck slogging with our
> clunky Pentiums rather than flying with these memories that would
> actually be cheaper to make.
I sense this is a very important idea! - meriting much more discussion
here and/or comp.arch?
Forget desktop PCs for the moment - would this not completely
revolutionise the HPC market? (Perhaps something Itanium architects
might care about!)
>
> ...
>
> Steve Richfield
Very glad to have encountered this thread, Steve, and I'll be reading
your future posts with interest.
Toby
Steve Richfie1d
Putting something in a spec does NOT make it right, just like
proclaiming that I am rich doesn't make it so. My objections below are
NOT with you or your interpretation of IEEE 754, but rather with
apparently bad design of the 754 spec.
The proper sign of 5.0 - 5.0 is NOT known because we don't know what low
bits have been discarded by prior operations. Saying that this has a
particular sign may be "standard", but it is just plain wrong and causes
problems further on.
Of course, IEEE is missing most of the machinery to promote abnormal
results back to numbers. Why bother fixing the engine when the wheels
are missing?
The place where abnormal results really need proper handling is in very
short (e.g. 16-bit logarithmic) representations that characteristically
produce so many more abnormal results. Most present non-standardized
implementations simply set overflows to the largest number, underflows
to zero, etc. Of course, this whole area is now missing from the IEEE
spec, yet it represents a rapidly growing application area.
There are SO many things wrong/missing with the IEEE FP spec, some of
which (like discarding significance) appear to be having SERIOUS effect
on our economy and ecology, yet no one (else) seems concerned about
these shortcomings. The present spec, if used in doing the computations
for a college freshman chemistry or physics lab, would get a failing
grade from any prof.
LETS FIX IT!
More detailed comments supporting this follow...
>>! This produces zero, but without knowing more about the two
>>! arguments, it is impossible to know the proper sign, if any.
>>! Of course, the present standard will arbitrarily invent a
>>! sign.
>>X = 5.0 - 5.0
>
> The result is +0.0 under IEEE.
THAT was what I was objecting to here. +0 indicates that the second
argument, if known to enough places (more than the implementation), is
NOT larger than the first argument. However, there is no way of knowing
this, so +0 overspecifies the result.
The presence of ANY such overstatement in a spec short-circuits a lot of
opportunities for promoting abnormal results.
> Does "oo" mean infinity here?
Yes.
>>! because there is no way to know if the signs of X and Y
>>! are the same.
>>Z = X + Y
>
>
> The result here is also +0.0 (+0.0 + +0.0 -> +0.0).
>
>
>>! This should print a string indicating that the answer cannot
>>! be computed, but instead IEEE prints 4.
>>Print 4 + 13/Z
>
>
> The 13/Z part signals the IEEE divide-by-0 condition.
You mean that 1/oo is treated as 1/0?!! This sounds like a BIG bug to
me, as it blocks the promotion of oo, so why even bother to have oo?!
Next to discarding underflows, this is probably the most basic promotion
there is.
> By default, that (a)
> sets the divide-by-0 sticky status flag; and, (b) returns +oo (positive
> infinity). 4 plus positive infinity is then unexceptional, and returns
> positive infinity.
Gee, a wrong answer that I hadn't contemplated. Even with existing IEEE
formats, it should have been demoted to a NaN because of the
non-computability of the sign, but then there was the substandard standard.
>>IEEE does not have an exact numerical representation, nor does it
>>specify mixed-mode arithmetic, e.g. 5.0 / 0 where 0 is an integer.
>
> The letter and spirit of 754 take the representable numbers at face value,
> as if exact. 754 operations treat 0.0 as exactly 0, 1.5 as exactly 3/2, and
> so on.
Obviously, this assumption is WRONG, as it ignores the probable presence
of lost bits below the least significant bit.
> In particular, nothing in that standard views representable numbers
> as approximations to "the real" results. That's a view you can layer on top
> of it, but it's not the standard's view. It's true that, e.g., 1.0/10.0
> can't be represented exactly in 754, and the standard then mandates that (a)
> the closest representable number is the result; and, (b) the inexact
> condition is signaled. But after that, the result is again taken at face
> value, and, e.g., subtracting the result from itself does not signal
> inexact.
However, in the real world, we all know that all floating point numbers
are +/- a half of a least significant bit - that anything could be out
there past the LSB. Knowing this affects some operations, e.g. 5.0 -
5.0, where we only know the most significant 23 bits of each operand,
could have either sign - or even be a true exact zero, so hence claiming
that the sign is anything but unknown is an ERROR.
> Note that the standard requires support for directed rounding modes
> primarily to aid efficient implementation of interval arithmetic. After 20
> years, it's still hard to get at this stuff from common computer languages
> in a portable way, so packages exploiting these features are finicky.
> Here's a good interval package for C++, exploiting HW acceleration when it
> can, but not visibly (not via the interface):
>
> http://www.boost.org/libs/numeric/interval/doc/interval.htm
Thanks for the link.
>
> Develop software along those lines, and you can give users any arithmetic
> model you want, with or without direct HW support.
>
> Speaking of which, I've read all the msgs in this thread, and am afraid I
> still don't know what you're proposing, just that it's going to save the
> world somehow <wink>. Do you have a writeup available with a specific,
> concrete, detailed proposal?
If you'll read the initial posting, I started this to flush any errors
out of a paper that has been accepted for the June AI meeting in Las Vegas.
For copyright reasons I can't post it, but I'll email it.
>>According to an internal report that CDC prepared, Crays vectorized
>>about 50% of loops that weren't hand-modified for vectorization, whereas
>>CDCs vectorized about 70%.
> Which universe of loops was considered? I worked on CFT and CFT77 from '79
> to '88 at Cray.
Mine was ~'82.
> As the years went on, more vectorization benchmarks
> appeared, with increasingly convoluted loop nests, and the percentage of
> loops we vectorized on those decreased most of the time I was there. We
> didn't care much at the time, because customers and potential customers told
> us we were vectorizing more of the loops in their actual production code
> than other vendors they evaluated. Since CFT was written in Cray assembler,
> and a real bear to modify because of it, it's certainly true that Cray
> encouraged rewriting loops for a long time, but in standard Fortran rather
> than via calling Cray-specific libraries. Most customers were very happy
> with that, because the suggested rewritings usually reduced clutter and
> improved clarity, after which those loops often also ran faster on
> non-vector machines with non-Cray compilers.
The study I say was with FORTRAN-only code. Yes, you're right, rewriting
cleaned up the code and often helped performance even on non-vector
machines.
> I haven't been to a $50K internal marketing party since the cold war ended
> <wink>.
In lots of ways, the good old days have come and gone.
Steve Richfield
Steve Richfie1d
> You keep talking about "applications that are THE most critical to our
> society". For things that critical, if they really exist, I would
> expect the few people involved (how many? 5?) would be willing to do the
> error analysis to get the right answers.
>
> You're talking about one software application in 100,000 or less, but
> you want standards and hardware on every computer in the world changed
> to meet your needs.
Supersetted would be more accurate.
It's not going to happen.
You're not making any good sense (to me) here. To clarify my apparent
logical break, I'll first restate back to you what you appear (to me) to
be saying:
. . . . . . . . . .
Sure, the IEEE FP standard has a long list of problems. If we fixed them
all, probably only 5% (my number) of users would see any useful benefit
at all, and only 1/100,000 users (your number, but sounds reasonable)
would achieve any really world-changing benefit from the improvements.
So, why do ANYTHING for just 5% or 1/100,000 users?
So what if those 1/100,000 users are exactly the SAME users whose
computational problems brought us global outsourcing and allowed special
interests to block any effective intervention into global warming?!
. . . . . . . . . .
Now, there is already consideration of a decimal enhancement of the IEEE
standard that doesn't really address any significant problems. The whole
idea of an evolving standard is to improve computers as we go, hopefully
in a way that doesn't hurt existing users.
Obviously, NO ONE is going to scrap IEEE-754, but what if we developed a
superset standard, capable of 754-compatible operation AND new operations.
For example, there are already bits available for the abnormal forms
I've been talking about. All the spec needs is to say is how to set them
and what to do with them in subsequent operations.
Significance probably works best in 48 bits. Logarithmic arithmetic is
probably best in 16 bits. Suppose we add new 16-bit and 48-bit modes of
operation, in addition to the 32 and 64 bit modes we already have,
increment the standard number, and release that for the next generation
of computers?
Alternatively, a "mode switching" operation could be provided whose
default was the present IEEE-754 spec. That wouldn't cause any problems,
would it?
Those who wanted could still put out 754-compliant hardware, while
others could meet the new spec that also ran 754-compliant code.
Hopefully, from these ramblings you can see our disconnect here and
correct it.
Thanks.
Steve Richfield
Steve Richfie1d
> In one evening's reading, your jeremiad on the issue of significance
> arithmetic has made you an instant hero of mine. I grok that simply "not
> knowing" the significance of simulated results, and when they should be
> ignored entirely, can be extremely dangerous.
>
> I also appreciated your remarks on the shortcomings of IEEE 754 (e.g.,
> undifferentiated NaN) and the C standards.
You have hereby officially earned the title "First Convert". Perhaps you
could explain why everyone else is showing so much resistance to these
concepts?
> I sense this is a very important idea! - meriting much more discussion
> here and/or comp.arch?
>
> Forget desktop PCs for the moment - would this not completely
> revolutionise the HPC market? (Perhaps something Itanium architects
> might care about!)
With these, you could plug in a bunch of newer cheaper memory SIMMS into
your computer and have enough computing power to simulate YOU, neuron by
neuron, synapse by synapse, in real time. If you'll dig up a copy of the
very first IJCNN, I made the closing presentation explaining most of how
to do this, complete with the numbers, clear back in 1987! (Careful, I
also made two other presentations there, so be sure to get the correct one).
There is a LOT of really neat technology just sitting on people's
shelves and hard-drives. Unfortunately, the entire field of technical
evaluation and standards adoption is **SERIOUSLY** broken, so most of
that technology is doomed to remain right where it now is.
This has been rejected by memory manufacturers for two primary reasons:
1. The numerical format is non-standard (logarithmic), that should be
addressed in a new IEEE FP format, but more importantly...
2. People chose to get into the memory fab business for one important
reason - memories are SIMPLE. They lack the mental capacity to dive into
the esoterics of computer science and architecture that entertain the
people on this forum. When they (try to) understand what I am proposing,
they look at each other with a sickly look, and say something like
"Don't call us, we'll call you.". Anyway, that was my take on my
interactions with Micron and Samsung.
This despite the drops in memory prices that now threaten the existence
of all memory manufacturers. What I need is some way to get past the
technology people to upper management, who hopefully would see the
astronomical value here and make it happen - over the revulsion of the
memory technology people.
Any idea on how to make this happen?
> Very glad to have encountered this thread, Steve, and I'll be reading
> your future posts with interest.
Thanks.
Steve Richfield
Mike Cowlishaw
>> For a discussion on why the decimal arithmetic in the proposed
>> IEEE 754 revision is not 'significance arithmetic', see:
>>
>> http://www2.hursley.ibm.com/decimal/decifaq4.html#signif
>
> This paper makes the numerous rookie errors expected of someone who
> hasn't ever actually worked in a significance environment.
The paper I referred to in that section of my FAQ was not 'my' paper;
it was written in 1958 (Computation with Approximate Numbers, Daniel
B. Delury, The Mathematics Teacher 51, pp521-530, November 1958).
Delury was no rookie.
Its conclusions are still valid; none of the papers I am familiar with (and
that
is quite a complete list; see the bibliography on my web site) disagree with
it. I certainly endorse its conclusion, as does Prof. Kahan of UCB who
introduced me to the paper some years ago.
If it is wrong, perhaps you should publish the refutation in some
peer-reviewed journal, instead of descending to ad hominem
attacks on writers quite likely long-since passed away?
Mike Cowlishaw
Mike Cowlishaw
> Dik,
>
>> > > > And, that part not being standardized IS this particular
>> problem with > > > the IEEE standard, because there are LOTS of
>> different NANs that need > > > DIFFERENT handling by the FP
>> hardware, both in production, AND in HANDLING. > >
>> > > It could not be standardised yet in that case because there was
>> > > insufficient knowledge.
>> >
>> > There is now! Wayne Albin developed this 30-years ago. I'm just
>> working > from 30-year-old memories and common sense.
>>
>> Possibly, but at the time the standard emerged it was not available.
>> Kahan (the major one, but also one amongst the others) has given
>> compelling reasons for the IEEE standard.
>
> Well, the next time the standard is revised, then these should be
> fixed, shouldn't they?
As it happens, the 754R committee has spent most its meetings
this year discussing exactly this issue. And it is clear that the
issues were not 'well understood', and there is no 'magic bullet'
soltion. A compromise solution is close, however.
mfc
Steve Richfie1d
>>>It's an interesting discussion, but I don't think it is very useful.
>>>For a discussion on why the decimal arithmetic in the proposed
>>>IEEE 754 revision is not 'significance arithmetic', see:
>>>
>>> http://www2.hursley.ibm.com/decimal/decifaq4.html#signif
>>
>>This paper makes the numerous rookie errors expected of someone who
>>hasn't ever actually worked in a significance environment.
>
>
> The paper I referred to in that section of my FAQ was not 'my' paper;
> it was written in 1958 (Computation with Approximate Numbers, Daniel
> B. Delury, The Mathematics Teacher 51, pp521-530, November 1958).
> Delury was no rookie.
>
> Its conclusions are still valid; none of the papers I am familiar with (and
> that
> is quite a complete list; see the bibliography on my web site) disagree with
> it. I certainly endorse its conclusion, as does Prof. Kahan of UCB who
> introduced me to the paper some years ago.
I don't believe in proof by authority. I do consider arguments and
experience. My 2 years of commercial experience with a significance
system tells me that it DOES work well.
I have heard NO counter argument regarding the astronomical value of the
loss of significance, that may be partially behind our present economic
and ecological woes because present predictive programs are unable to
tell just how far ahead they can predict.
> If it is wrong, perhaps you should publish the refutation in some
> peer-reviewed journal, instead of descending to ad hominem
> attacks on writers quite likely long-since passed away?
Have you seen the Ashenhurst and Metropolis paper of 1959 in the JACM?
That is the BEST discussion of significance methods that I have yet
seen. Unfortunately, it came just AFTER the paper you cited.
Everyone jumps on the one small problem that unnormalized significance
methods have in adding very small quantities to larger low-significance
quantities. In real life, this situation is EXTREMELY rare and trivially
handled. Certainly this non-problem isn't 1% of the cost of loss of
significance capability. In any case, this being an alternative mode,
users should be aware of the strengths and weaknesses of the tools they
are using.
We should engage in an open discussion, where you hold my feet to the
fire regarding the various issues connected with significance. There ARE
some additional language elements needed to handle all situations, e.g.
an alternate input format for precisely stated numbers, a library
routine to normalize a number, etc. Are you going to be at the June show
in Las Vegas?
Also, I am NOT proposing replacing the current methods with significance
methods, because I see way too many practical problems doing this. What
I propose are some alternate modes, akin to your decimal proposal,
including significance and logarithmic representation, as both are
GREATLY needed in their respective disciplines. There are some small
cross-format issues, e.g. handling NaNs with slightly different meanings
as discussed in some prior postings here.
Steve Richfield
==================
> Mike Cowlishaw
>
>
>
Jeff Kenton
> . . . . . . . . . .
>
> Sure, the IEEE FP standard has a long list of problems. If we fixed them
> all, probably only 5% (my number) of users would see any useful benefit
> at all, and only 1/100,000 users (your number, but sounds reasonable)
> would achieve any really world-changing benefit from the improvements.
>
> So, why do ANYTHING for just 5% or 1/100,000 users?
>
> So what if those 1/100,000 users are exactly the SAME users whose
> computational problems brought us global outsourcing and allowed special
> interests to block any effective intervention into global warming?!
> . . . . . . . . . .
The IEEE-754 Standard is causing global warming and outsourcing??? You can't
be serious.
Steve Richfie1d
Gee, my words of 30 years ago to Wayne Albin! Wayne then went on to
explain that if there was ANY overstatement on NaNs, e.g. forced from
having one missing, then you could not reliably promote NaNs as you
could/should. Ability to promote NaNs back to numbers is THE figure of
merit for NaN design. I argued a number of specific cases with Wayne,
but he won all of the arguments. Hence, I am **HIGHLY** skeptical of any
"compromise". You need them all, overflow, underflow, and indefinite,
and in each case with positive, negative, and unknown sign. Go ahead and
propose leaving any one out, and I'll gladly provide sample code that
won't work as well as it would with the form available.
In each case, this results in a sort of reductio ad absurdum discussion,
e.g. "I'm leaving out (say) unsigned infinity". "OK, then what are you
going to map those results to?". "Plus Infinity", "OK, then correct
indications of plus infinity are no longer to be believed. What does +oo
+ +oo become?" "+oo". Here, +oo can be incorrectly promoted to a number,
whereas the correct answer of unsigned indefinite cannot. Hence, in this
example, an entirely wrong numerical answer could be produced as a
result of not having unsigned infinity available. Other examples
available by request.
Also, how about loss of significance signals?! I know, everyone
considered it a boondoggle on the IBM-360, but there ARE a few
applications where it is important when using normalized methods. Of
course, this is no issue at all in significance arithmetic.
My opinion: The primary reason that normalized looks simpler than
unnormalized is because there are SO many subtle cases that IEEE 754
ignores or handles badly - many/most of which can't occur in a
significance system. The TOTAL complexity once all issues are addressed
is comparable, though I'd guess that significance might win by a hair.
Each system has some glaring shortfalls - and methods of addressing
those shortfalls. Without SNOBOL or FORMAC style statement of numbers as
arbitrarily long strings, there will ALWAYS be some "character" to all
systems - including even your decimal proposal. Again, I just LOVE
decimal arithmetic.
BTW, would you like a copy of the paper we'll be presenting? It's short
on significance details but long on the impacts of not having it and how
to mitigate some of the problems.
Steve Richfield
Steve Richfie1d
> The IEEE-754 Standard is causing global warming and outsourcing??? You
> can't be serious.
Close, but not exactly. The cause and effect chain goes as follows:
As predictive simulation programs run, the significance (if significance
is implemented) or accuracy (if significance is not implemented)
deteriorates some for each time step into the future. Because of
nonlinear and metastable conditions that will periodically occur in the
computations, this loss of significance is NOT ameniable to forward
error analysis. Hence, without significance, you model will lose
accuracy as it predicts further into the future, but there will be NO
indication whatever of just how much has been lost, or even if there is
any meaning at all left in the numerical output.
Hence, for program that want to pridict as far into the future as
possible, they have no idea just how far that is. Perhaps you've seen
various long-term climate models - where different models predict
RADICALLY different futures, but no one knows if any of them have any
accuracy at all (obviously, most must not have any accuracy)
Predictive simulations are by their very nature significance-limited.
This means that the significance information is arguably even MORE
important than the numeric result!
Seeing these uncontrolled models and lack of consensus, special
interests overcome reason in the government, and NOTHING is done about
problems whose effects cannot be consistently predicted.
Hence, while IEEE-754 problems are NOT directly responsible for these
problems, if IEEE-754 had a good implementation of significance, there's
a good chance that we wouldn't now have our present unaddressed global
warming and outsourcing problems as bad as they now are.
Obviously, there are plenty of people and lots of politics to share the
blame with in this complex cause and effect chain.
Yes, I AM SERIOUS. Eddie will be presenting our peer-reviewed paper at
the Las Vegas AI conference in late June explaining this and some ways
of stumbling along a little better w/o significance unless/until IEEE
gets signficance out to the world.
Perhaps you'd like to submit a paper arguing the opposite position?
BTW, email me off-forum and I'll email you a copy of the paper.
Steve Richfield
Tim Peters
> Putting something in a spec does NOT make it right, just like
> proclaiming that I am rich doesn't make it so. My objections below are
> NOT with you or your interpretation of IEEE 754, but rather with
> apparently bad design of the 754 spec.
I haven't expressed any opinion of the 754 design so far, I've only tried to
correct factual mistakes about what the spec says. If you want to convice
people to change it, I expect you need to demonstrate understanding of it.
> The proper sign of 5.0 - 5.0 is NOT known because we don't know what low
> bits have been discarded by prior operations. Saying that this has a
> particular sign may be "standard", but it is just plain wrong and causes
> problems further on.
Depending on the app, it may or may not be wrong. I don't know exactly what
you have in mind instead, but I'd be surprised if its usefulness were
independent of app.
...
> There are SO many things wrong/missing with the IEEE FP spec, some of
> which (like discarding significance) appear to be having SERIOUS effect
> on our economy and ecology,
Sorry, I don't know that that's so (while you've asserted it, I haven't seen
evidence to support it).
> yet no one (else) seems concerned about these shortcomings.
I grant the misuse of floating point is widespread. I can't grant that
there's a magic bullet that would kill the problems if only pig-headed
standards committees would listen to reason. In my numerical experience,
error analysis is difficult; the semantics of 754 arithmetic are an aid (but
don't make it easy); interval arithmetic is as good as I've seen for
maintaining error bounds automatically, but interval arithmetic is
essentially useless in the hands of a non-expert. As Dik Winter said
earlier, forward propagation of worst-case error bounds, without knowledge
of correlated errors, generally zooms out of control. The canonical trivial
example is x-x: no matter how fuzzy one's knowledge of x, x-x is exactly 0
in reality.
> The present spec, if used in doing the computations for a college
> freshman chemistry or physics lab, would get a failing grade from
> any prof.
>
> LETS FIX IT!
How, specifically? Interval arithmetic should do fine on a typical
20-operation freshman lab assignment, but may overstate the uncertainty a
bit.
> More detailed comments supporting this follow...
>
> >>! This produces zero, but without knowing more about the two
> >>! arguments, it is impossible to know the proper sign, if any.
> >>! Of course, the present standard will arbitrarily invent a
> >>! sign.
> >>X = 5.0 - 5.0
> >
> > The result is +0.0 under IEEE.
>
> THAT was what I was objecting to here. +0 indicates that the second
> argument, if known to enough places (more than the implementation), is
> NOT larger than the first argument.
The sign of a zero is mostly inconsequential in 754. While this operation
happens to produce +0, that compares equal to -0. The real purpose of
signed zeroes in 754 is to support crisp determination of branch cuts in
complex-valued functions; AFAIK, Kahan's paper "Branch Cuts for Complex
Elementary Functions, or Much Ado About Nothing's Sign Bit" is the only good
explanation available of 754's signed zero rationale. People who aren't
slinging complex functions generally ignore the sign of a 754 zero.
> However, there is no way of knowing this, so +0 overspecifies the
> result.
754 is certainly not any form of significance arithmetic (although, as
before, it provides directed rounding modes to *help* people implement
interval arithmetic packages).
> The presence of ANY such overstatement in a spec short-circuits a lot of >
opportunities for promoting abnormal results.
>> Does "oo" mean infinity here?
> Yes.
>>>! because there is no way to know if the signs of X and Y
>>>! are the same.
>>>Z = X + Y
>> The result here is also +0.0 (+0.0 + +0.0 -> +0.0).
Because of what you say later, note here that +0.0 is positive zero, not
positive infinity. I used decimal points because you earlier objected that
754 doesn't have mixed-type operations, so wanted to short-circuit that
complaint here.
>>>! This should print a string indicating that the answer cannot
>>>! be computed, but instead IEEE prints 4.
>>>Print 4 + 13/Z
>> The 13/Z part signals the IEEE divide-by-0 condition.
> You mean that 1/oo is treated as 1/0?!!
No, Z is zero, not infinity (see above). 1/+oo is +0 in 754, and 1/-oo
is -0 in 754. Both are unexceptional. The first time infinity appears in
your example is as a result of computing 13/Z. Nothing in your example
divides by an infinity.
...
>> By default, that (a) sets the divide-by-0 sticky status flag; and, (b)
>> returns +oo (positive infinity). 4 plus positive infinity is then
>> unexceptional, and returns positive infinity.
> Gee, a wrong answer that I hadn't contemplated.
Which one? You object to 13/zero returing infinity, to 4+infinity returning
infinity, or to both?
> Even with existing IEEE formats, it should have been demoted to a
> NaN because of the non-computability of the sign, but then there
> was the substandard standard.
I would be unhappy if 1/(x-x) returned a NaN.
>> The letter and spirit of 754 take the representable numbers at face
>> value, as if exact.
> Obviously, this assumption is WRONG, as it ignores the probable presence >
of lost bits below the least significant bit.
It may or may not be wrong, depending on the app. A simple example: on
32-bit machines, some apps use IEEE double for integer operations, because
they need more than 32 bits (but not more than 53), and some architectures
have lots of extra gimmicks to speed masses of fp operations. You can
inspect the state of the sticky inexact status flag after a bunch of these,
to determine whether any information was lost. There are many uses for
floating point. 754 provides partial support for 11 of them <wink>.
...
> However, in the real world, we all know that all floating point numbers
> are +/- a half of a least significant bit - that anything could be out
> there past the LSB.
Sorry, I don't know that, and it does depend on the app.
> Knowing this affects some operations, e.g. 5.0 - 5.0, where we only
> know the most significant 23 bits of each operand, could have either
> sign - or even be a true exact zero, so hence claiming that the sign
> is anything but unknown is an ERROR.
I've rarely had an app where it made any difference, apart from convergence
criteria in iterative algorithms. When I have, I've generally checked
results for "smallness", not just for "zeroness". Claiming to know the sign
of, say, 101.0 - 98.0 may be equally wrong, depending on the app (maybe
101.0 is known exactly, but 98.0 has an uncertainty of +/-5.0; then the true
result is "somewhere" in [-2, 8]). IOW, I don't see anything special about
5.0 - 5.0, and in many real apps the region of uncertainty is far greater
than 1/2 ULP.
...
>> Speaking of which, I've read all the msgs in this thread, and am
>> afraid I still don't know what you're proposing, just that it's going
>> to save the world somehow <wink>. Do you have a writeup available
>> with a specific, concrete, detailed proposal?
> If you'll read the initial posting, I started this to flush any errors
> out of a paper that has been accepted for the June AI meeting in Las
> Vegas.
>
> For copyright reasons I can't post it, but I'll email it.
That's generous and appreciated, but I'd prefer to mostly lurk in a public
place. I don't work in this field anymore, and can't afford the time a
serious review would require.
...
> In lots of ways, the good old days have come and gone.
Yes they have! I can't say that I miss 6.0/3.0 returning a result less than
2.0 (which it did on Cray Classic hardware), but the only exciting FP thing
that's happened since the world went to 754 was the introduction of fused
multipy-add hardware.
BTW, there's only one form of semi-painless error analysis I've seen that
real people could actually use: redo a computation from scratch at
different precision levels. Toward this end, Cray's CFT compiler had an
option to generate code to chop off the last N bits (N was your choice) of
each FP result. This was initially a marketing necessity, as people coming
from 32-bit boxes complained that 64-bit Crays "got the wrong answer".
There were some dramatic cases where plotting results against retained
precision showed essentially a random walk up to and somewhat beyond 24
retained bits, then settling down dramatically as it approached Cray's 48
significand bits.
Mike Cowlishaw's Decimal proposal also supports user-specified precision,
and the same kind of technique exploiting that also proved its practical
worth in REXX. It's not foolproof, but it's very easy to use, and if
results do "jump around" after changing precision a little you can be
certain you have a numeric problem (alas, you may have one even if they
don't jump). Of course people did a simple form of this in standard Fortran
for decades, switching between REAL and DOUBLE "just to see what happened".
Dik T. Winter
> > > > It could not be standardised yet in that case because there was
> > > > insufficient knowledge.
> > >
> > > There is now! Wayne Albin developed this 30-years ago. I'm just working
> > > from 30-year-old memories and common sense.
> >
> > Possibly, but at the time the standard emerged it was not available.
> > Kahan (the major one, but also one amongst the others) has given
> > compelling reasons for the IEEE standard.
>
> Well, the next time the standard is revised, then these should be fixed,
> shouldn't they?
Well, if you wish that, let us see a complete proposal. Not some ramblings.
> > If your underflow gives an erroneous sign, there is a problem with the
> > application.
>
> Consider the following code for which the IEEE standard (as I understand
> it) produces the wrong answer. Of course, the constants shown would be
> variables obtained elsewhere, with duplicate shown values being
> different variables with the same value:
>
> ! This produces zero, but without knowing more about the two !
> arguments, it is impossible to know the proper sign, if any.
> ! Of course, the present standard will arbitrarily invent a
> ! sign.
> X = 5.0 - 5.0
Yup, this gives +0.0 under IEEE. (BTW, I am not too enamoured with the
signed zeros and infinities either, I would much more prefer the affine
completion with only a single zero and infinity.)
> ! Same situation here, but with different arguments.
> Y = 3.0 - 3.0
Also +0.0 under IEEE.
> ! This should produce an indefinite, and NOT a signed oo
> ! because there is no way to know if the signs of X and Y
> ! are the same.
> Z = X + Y
Also +0.0 under IEEE.
> ! This should print a string indicating that the answer cannot
> ! be computed, but instead IEEE prints 4.
> Print 4 + 13/Z
No, Z = 0.0, so why do you think it prints 4?
> > I still do not see that need. It is different. When I was involved with
> > Ada standardisation, standardising complex arithmetic had problems *because
> > of* the signed infinities.
>
> The above code also impinges on this problem. You need unsigned
> infinities to deal with the reciprocals of unsigned zeros. You need
> unsigned zeros for situations where the sign cannot be properly
> determined, like the difference between identically valued arguments.
Yup, so I do not see the need for signed quantities (I know why it was
proposd by Kahan, but am not satisfied by his arguments).
> > > With enough bits in the exponent, you wouldn't have any oo's. +oo is a
> > > statement that your arithmetic system was unable to do what you asked it
> > > to, but not because what you asked was reasonable, just beyond the
> > > capacity of the exponent.
> >
> > You think so? How about division by an exact 0?
>
> IEEE does not have an exact numerical representation, nor does it
> specify mixed-mode arithmetic, e.g. 5.0 / 0 where 0 is an integer.
Then you misunderstand IEEE. The numbers are assumed to be exact, and the
results are the closest result based on that assumption. So mixed mode is
not needed, because converting an integer to a floating-point number
results in exactly the same floating-point number, when there is an
exact representation.
Note that this is similar to the 40 year old system on which I started.
Although it had both integers and floating point numbers, converting
an integer to a normalised floating point number meant just setting the
upper half of the floating point number to 0 bits. The normalisation
was Grau normalisation, look for an article by A.A. Grau in a fifties
or early sixties issue of CACM.
> > Nope. I am (in the core) a mathematician. And when you talk about
> > numbers, there are other numbers than real numbers. Talking as a
> > computer scientist, REAL's are not the real numbers, they are only a
> > subset of the rational numbers.
>
> As a mathematician, IEEE FP must make you sick!
Why? IEEE FP works pretty well within a special subring of the real
numbers. There is of course a problem with overflow and underflow, but
that is all.
> > That is *not* the responsibility of the hardware. It is the application's
> > responsibility whether a NaN coming from a square root of a negative
> > number should continue, whether the result should be replaced by 0, or
> > whatever.
>
> But, how can it distinguish between the square root of a negative number
> and other NaN producing operations? Not all NaNs are created equal,
> except in IEEE FP.
No, the standard leaves it open. It is a quality of implementation issue
whether the NaN's are distinguished or not.
> The result of a division by zero could be promoted by
> using it in a denominator,
Division by zero does not yield a NaN. And it *can* be promoted under IEEE.
> whereas the square root of a negative number
> could potentially be promoted by multiplying by zero. The whole reason
> for distinguishing special forms is to provide for future promotion, and
> either overspecification (like requiring that oo be signed) or
> underspecification (like having no sign for oo) REALLY gets in the way
> of proper future promotion.
However, as the generation of NaN's and handling of NaN's, infinities and
denormals are on most processors even not handled in hardware (they are
handled by interrupt routines in software), the point is a bit moot.
> Reading the IEEE spec, it seems obvious (to me) that promotion was never
> even considered.
Not for NaN's, not. Once a NaN, always a NaN.
> The theory behind all this goes as follows: Presume that the programmer
> was an excellent mathematician but a less than stellar programmer, so
> they have NOT re-arranged their expressions to avoid overflows, etc.
> What could be then done with special forms to STILL make such code
> produce correct answers whenever possible? Wayne worked this problem and
> came up with the forms that I have been discussing.
Possibly. Are those available somewhere, preferably online?
> Most real-world exceptions from good logic come from things like
> dividing by the difference between identically-valued variables, and NOT
> by manipulating things with astronomical exponents.
That is one of the reasons the 40 year old system I wrote about did not
have infinities, it just delivered the largest representable number. On
the other hand, the denormals found in IEEE come from that system, and
were in it for a very good reason.
> > For a long time involved in the development of libraries for numerical
> > mathematics in quite a few fields. Including error analysis.
>
> A kindred spirit.
It appears not.
> > You know, even if they hit linear functions it may still be non-trivial.
> > Doing forward error analysis on linear systems of equations leads to the
> > conclusion that the results of 40 equations with 40 unknowns are bogus.
> > *Even when you assume the input is exact.* This has already been noted
> > by von Neumann. In the 50's, 60's, J.H. Wilkinson developed an approach
> > (backward error analysis) where it could be shown that the results where
> > the exact result for a question close to the original. And that is the
> > best you can have. Unless you are willing to go to the Karlsruhe method,
> > where they are able to provide solutions with guaranteed error bounds.
> > But that comes at a (huge) price.
>
> My point is that in the real world, almost no one has either the time
> nor the education to do a proper error analysis on what they are doing,
> even if it is technically possible, yet they often need some indication
> of the significance of their result.
And as I said, if you do not understand what the results of a linear
system of 40 equations actually does mean... There is *no* good way
to get any indication of the significance of the result, except saying
that it is the exact result of a nearby system (in some measure).
> Now that you understand the problems with error analysis, I would think
> that you would have already joined the significance bandwagon!
Wrong. Significance is close to forward error analysis and that is close
to impossible for some problems.
> 2. Don't look at the guard digit when comparing arguments. Note that
> subtracting and testing signs and for exactly zero doesn't work quite
> right in the presence of a guard digit. However, considering a result to
> be zero if all but the guard digit is zero works OK.
I note that this would invalidate *all* of the error analysis I once did
for an arcsine routine I had written.
Dik T. Winter
> > Eh? I cannot follow. The only logical operations in C are && and ||.
> > I see no problem with them.
>
> Obviously you want to do the bitwise first, then the boolean, as bitwise
> operation on a boolean result doesn't make much sense. However, C
> specifies this the other way around.
Eh? Consider:
2 & 2 && 1 & 1, which delivers 1, or true
vs.
2 & (2 && 1) & 1, which delivers 0, or false.
Clearly the bitwise operations go before the logical operations. (Yes,
every non-zero value is considered true in the logical operations.)
> > I doubt that. Both Cray and NEC performed pretty well, while on the 205
> > we had to use Q8 calls to get what we wanted. There was *no* way on the
> > 205 to get the performance we obtained, except by using Q8 calls.
>
> According to an internal report that CDC prepared, Crays vectorized
> about 50% of loops that weren't hand-modified for vectorization, whereas
> CDCs vectorized about 70%.
We had to hand modify code to get good vectorisation on the Cray and the
NEC, but doing that was beneficial to both systems. We also had to hand
modify for the 205, but that involved Q8 calls (which was just syntactic
sugar for assembly).
> Most of the difference was because CDCs had bit
> vectors, so that most loops containing conditional statements could be
> vectorized by first precomputing the results of all of the conditionals
> into a bit vector that the compiler did quite well, then passing the bit
> vector to following vector operations to suppress the storing of results
> where the corresponding bit was a zero.
Note what was happening. The results were computed, but not stored. So
they took time. Other machines used contract, calculate and expand, which
has the same effect but is better when the calculations are large.
> Of course, Crays didn't have to even bother to compute the results that
> CDCs would suppress, so they gained a bunch there. The net results from
> the internal benchmarks that I saw was that Crays won with scalars and
> short vectors, whereas CDCs won with long vector applications.
The reason for that is simply that Cray had 64 word vector registers and
CDC had a continuous flow to memory with a maximum of 65536 (or somesuch)
words. If I remember right, CDC's compilers where flabbergasted if the
vector length exceeded that number. Cray had no problem with vectors
larger than 64 words. Also the relative location of the vectors
involved played an important role on CDC, much less so on Cray. I have
done intensive studies on that (and they have been published somewhere,
although I disremember where). On the CDC it could mean a factor of two
in performance if the source vectors had a wrong displacement. The best
displacement would be something like (actually these figures are completely
out of my hat, but they were similar): 2nd source vector at 64n + 19 words
from the 1st source, destination vector at 64n + 13 words from the 1st
source. (I have done similar tests on the IBM 3090 and the Alliant FX/2,
they were similar to CDC's.)
> Of course, most day to day applications were scalar and short vector,
> but the REALLY big applications were mostly long vector. Hence,
> universities were usually better off with Crays, while national
> laboratories were usually better off with CDCs.
The institution that bought the first supercomputer in the Netherlands
asked me (amongst others) to check both. So I have run various tests
on both and my conclusion was that Cray was better (it is a national
laboratory in your sense). But CDC was chosen.
> It's nice that we can now talk about these things, now that both
> companies are no longer around to get upset with us over talking about
> their innermost secrets.
I have been talking about it all the time. And I do not talk about
innermost secrets, I am talking about things that can be found out by
a ordinary user.
> > This is far aside from what I posted. How on the Burroughs 5000/6000
> > system would you detect (in standard Fortran) whether two array arguments
> > are actually the same array? Honest. There is *no* way to do that.
>
> First, I have no direct experience on the Burroughs computers - just a
> bunch of long-past manual reading and discussing it with users. That
> having been said, you should be able to examine the array descriptors
> passed as arguments, though I don't think the language provided any
> syntax for this.
Pray, please. I am talking about standard Fortran, yes?
> The 5000 series computers had "stream procedures", a
> sort of in-line assembly language to do such brain-surgery. I don't know
> how to approach this on the 6000.
I have manuals for the 5700 and some other Burroughs machine, but there is
no relation to the discussion.
> > And talking about broken arithmetic. Consider the 205 where a == b did
> > *not* imply b == a (it is explicitly noted in the hardware manual, I
> > still have a copy here). Consider the Cray where with some f-p instructions
> > the result could be 0 rather than the more correct 1 (not documented in
> > manuals, the result of non-shifting before storing in some cases).
> > [ It was the floating point half precision multiply.] I have not done
> > enough with the NEC to know whether that machine had anomalies.
>
> ... and when I talk about anomalies in IEEE FP everyone screams HERESY!
So you consider the problems with IEEE just as serious than a system where
a == b does not imply b == a, or a system were an instruction could deliver
0 rather than 1? Oh, well.
> Another topic we've barely touched is the absence of a 16-bit
> logarithmic representation. With no standard for this, there has been no
> merging of DSP and desktop technologies.
Write a proposal, including how to do addition and subtraction. As far as
I know, it has been proposed once (I remember seeing an article in the
CACM), but never flew.
Dik T. Winter
> Dik,
> > > And, financial forecasting is the greatest challenge of all, because
> > > money has little/no inertia, so huge amounts can move in an instant.
> > > Here, significance appears to be the present barrier to advancing the
> > > state of the art.
> >
> > I do not think so. Most of finance prediction involves non-predictable
> > hype.
>
> As I understand the situation (if indeed ANYONE does), most of the time
> things are very predictable, but often they are not. Indeed, this is
> where market volatility indicators came from. They can tell you that
> there is going to be a big change, but can't even produce the sign of
> that change.
Oh, fine, so they can tell me there will be a big change. How do they
predict that the books of some company are fraudulent? My prediction is
that on the stock market there will be a big change in the next month.
Do I win a price?
Steve Richfie1d
Lots of good points. My comments follow about a few...
> I haven't expressed any opinion of the 754 design so far, I've only tried to
> correct factual mistakes about what the spec says.
I need all the help I can get.
> If you want to convice
> people to change it, I expect you need to demonstrate understanding of it.
Yes, and no. I've been through a LOT to determine how to make a
significance system work, and to work out a system of NaNs that maximize
the prospects for promotion. It takes about 5 minutes of looking at the
754 spec to see that little if any of this is present. It takes a
**LOT** more work to pose precise examples where 754 produces
predictable wrong answers, even though it is entirely obvious (to me)
that when you leave something critical out, that there WILL be
consequences. Hence, please don't condemn me because I miss a detail
here and there.
>>The proper sign of 5.0 - 5.0 is NOT known because we don't know what low
>>bits have been discarded by prior operations. Saying that this has a
>>particular sign may be "standard", but it is just plain wrong and causes
>>problems further on.
> Depending on the app, it may or may not be wrong. I don't know exactly what
> you have in mind instead, but I'd be surprised if its usefulness were
> independent of app.
I don't see how this has anything at all to do with the app. Subtracting
two different variables, each of which may have discarded unknown low
bits but having the same surviving value, is of unknown sign in ANY
application where you don't have any a priori knowledge to the contrary.
>>There are SO many things wrong/missing with the IEEE FP spec, some of
>>which (like discarding significance) appear to be having SERIOUS effect
>>on our economy and ecology,
>
>
> Sorry, I don't know that that's so (while you've asserted it, I haven't seen
> evidence to support it).
OK, let's keep working on the details...
>>yet no one (else) seems concerned about these shortcomings.
>
> I grant the misuse of floating point is widespread. I can't grant that
> there's a magic bullet that would kill the problems if only pig-headed
> standards committees would listen to reason.
You're right, that things like significance won't do much good at all
UNLESS people "use" them, i.e. make the little changes in their
significance-blind programs so that significance can work.
> In my numerical experience,
> error analysis is difficult; the semantics of 754 arithmetic are an aid (but
> don't make it easy); interval arithmetic is as good as I've seen for
> maintaining error bounds automatically, but interval arithmetic is
> essentially useless in the hands of a non-expert.
All good points. However, in most applications, effects like variable
error correlation between arguments causes enough problems that the best
and worst significance systems produce about equally good/bad results.
> As Dik Winter said
> earlier, forward propagation of worst-case error bounds, without knowledge
> of correlated errors, generally zooms out of control. The canonical trivial
> example is x-x: no matter how fuzzy one's knowledge of x, x-x is exactly 0
> in reality.
The systems I've worked with have assumed a distribution, not worst case.
>>The present spec, if used in doing the computations for a college
>>freshman chemistry or physics lab, would get a failing grade from
>>any prof.
>>
>>LETS FIX IT!
>
>
> How, specifically? Interval arithmetic should do fine on a typical
> 20-operation freshman lab assignment, but may overstate the uncertainty a
> bit.
OK, then let's put interval arithmetic into the IEEE FP spec! Let's put
SOMETHING that tracks significance into the spec. I suggested
unnormalized methods because they don't require much more storage, and
so require less program modification, but I wouldn't drag even one toe
if someone were to push for including interval arithmetic.
>>More detailed comments supporting this follow...
>>>>! This produces zero, but without knowing more about the two
>>>>! arguments, it is impossible to know the proper sign, if any.
>>>>! Of course, the present standard will arbitrarily invent a
>>>>! sign.
>>>>X = 5.0 - 5.0
>>>
>>>The result is +0.0 under IEEE.
>>
>>THAT was what I was objecting to here. +0 indicates that the second
>>argument, if known to enough places (more than the implementation), is
>>NOT larger than the first argument.
> The sign of a zero is mostly inconsequential in 754. While this operation
> happens to produce +0, that compares equal to -0.
This is true UNTIL you try dividing by it. Then, depending on the sigh,
you get different signs of oo. Still OK, until you start subtracting oo
from oo, whereupon the signs determine the type of NaN that you SHOULD
get. Some can later be promoted to numeric, and some can't.
Unfortunately, IEEE spec ends up making erroneous numeric results when
in fact no such results can be computed.
> The real purpose of
> signed zeroes in 754 is to support crisp determination of branch cuts in
> complex-valued functions; AFAIK, Kahan's paper "Branch Cuts for Complex
> Elementary Functions, or Much Ado About Nothing's Sign Bit" is the only good
> explanation available of 754's signed zero rationale. People who aren't
> slinging complex functions generally ignore the sign of a 754 zero.
WRONG. You came up with one example of something that works with things
as they are, and advanced that as a reason to let other things not work.
Let's get EVERYTHING to work, including BOTH the branch cuts AND the
proper demotion/promotion of NaN forms. Certainly, sticking an arbitrary
and often wrong sign onto a zero isn't going to HELP anything at all.
>>However, there is no way of knowing this, so +0 overspecifies the
>>result.
> 754 is certainly not any form of significance arithmetic (although, as
> before, it provides directed rounding modes to *help* people implement
> interval arithmetic packages).
I'm not sure what you're saying here.
The following stuff is pretty scrambled from too many comments and too
much editing.
>>>>! because there is no way to know if the signs of X and Y
>>>>! are the same.
>>>>Z = X + Y
>
>
>>>The result here is also +0.0 (+0.0 + +0.0 -> +0.0).
The point I was trying to make is that in the earlier (deleted)
statements where two equal variables were subtracted, you can't know the
correct sign that IEEE makes + regardless what what it should properly
be. Taking their reciprocal produced infinities of unknown sign, though
again IEEE throws this information away and makes them +.
Then, adding infinities of unknown signs produces an indefinite about
which you can't say much of anything - is it overflow, underflow,
numeric, positive, negative - it is COMPLETELY unknowable. However, IEEE
simply says that this is a +oo. Now, continuing with the example with
Z=indefinite (my claim) or +oo (IEEE)...
>
>
> Because of what you say later, note here that +0.0 is positive zero, not
> positive infinity. I used decimal points because you earlier objected that
> 754 doesn't have mixed-type operations, so wanted to short-circuit that
> complaint here.
>
>
>>>>! This should print a string indicating that the answer cannot
>>>>! be computed, but instead IEEE prints 4.
>>>>Print 4 + 13/Z
Won't IEEE indeed print 4 here, when in fact the proper value is a
completely incalculable indefinite?
I think your following comments come from the scrambling of the example...
>>>The 13/Z part signals the IEEE divide-by-0 condition.
>
>
>>You mean that 1/oo is treated as 1/0?!!
>
>
> No, Z is zero, not infinity (see above). 1/+oo is +0 in 754, and 1/-oo
> is -0 in 754. Both are unexceptional. The first time infinity appears in
> your example is as a result of computing 13/Z. Nothing in your example
> divides by an infinity.
>
> ...
>
>
>>>By default, that (a) sets the divide-by-0 sticky status flag; and, (b)
>>>returns +oo (positive infinity). 4 plus positive infinity is then
>>>unexceptional, and returns positive infinity.
>
>
>>Gee, a wrong answer that I hadn't contemplated.
>
>
> Which one? You object to 13/zero returing infinity, to 4+infinity returning
> infinity, or to both?
As stated above, it should be indefinite.
>>Even with existing IEEE formats, it should have been demoted to a
>>NaN because of the non-computability of the sign, but then there
>>was the substandard standard.
>
>
> I would be unhappy if 1/(x-x) returned a NaN.
Hmmm, an interesting question. Of course, most compilers would see X-X
and substitute zero, then try to compute 1/0 and produce a compile-time
error.
>>>The letter and spirit of 754 take the representable numbers at face
>>>value, as if exact.
>
>
>>Obviously, this assumption is WRONG, as it ignores the probable presence >
>
> of lost bits below the least significant bit.
>
> It may or may not be wrong, depending on the app. A simple example: on
> 32-bit machines, some apps use IEEE double for integer operations, because
> they need more than 32 bits (but not more than 53), and some architectures
> have lots of extra gimmicks to speed masses of fp operations. You can
> inspect the state of the sticky inexact status flag after a bunch of these,
> to determine whether any information was lost. There are many uses for
> floating point. 754 provides partial support for 11 of them <wink>.
Your example isn't app-wide, just local to a loop or particular
subroutine. Except in perverse cases (all people who are unable to
produce perverse cases may now leave this discussion - well I see that
no one has left) I stand by my point. It's one thing to produce perverse
cases to shoot something down, but quite another to produce a perverse
case to try to resurrect something that should DIE DIE DIE, like the
presumption that FP values are exact. As I understand things, this
difference in presumption has no affect on numerical results, only on
the NaNs that should be produced. Hence, shifting the presumption would
have no harmful effect on the example you advanced.
Note also, along the line of semi-perverse examples, IEEE does NOT
provide for a Fused Multiply Add instruction as on a Power PC or
Itanium. Fusing these instructions provides different/better results
than doing them as separate operations, as you have all of the low bits
left over from the multiply to use in the subsequent addition.
>
>>However, in the real world, we all know that all floating point numbers
>>are +/- a half of a least significant bit - that anything could be out
>>there past the LSB.
>
>
> Sorry, I don't know that, and it does depend on the app.
You seem to believe that being able to construct a very perverse example
of something to make it work means that it is good for everyone,
everywhere. However, as I pointed out above, changing this assumption I
think only affects the production of NaNs.
>>Knowing this affects some operations, e.g. 5.0 - 5.0, where we only
>>know the most significant 23 bits of each operand, could have either
>>sign - or even be a true exact zero, so hence claiming that the sign
>>is anything but unknown is an ERROR.
>
>
> I've rarely had an app where it made any difference, apart from convergence
> criteria in iterative algorithms. When I have, I've generally checked
> results for "smallness", not just for "zeroness". Claiming to know the sign
> of, say, 101.0 - 98.0 may be equally wrong, depending on the app (maybe
> 101.0 is known exactly, but 98.0 has an uncertainty of +/-5.0; then the true
> result is "somewhere" in [-2, 8]).
This all works itself out in significance implementations, where the
guard digit is ignored for zero testing.
> IOW, I don't see anything special about
> 5.0 - 5.0, and in many real apps the region of uncertainty is far greater
> than 1/2 ULP.
Right, that is why you can't assign a sign.
The ULTIMATE way to evaluate significance is to intentionally put random
bits below the ones you know. Run your program several times this way,
and observe the spread of results. Not only does this convey
significance, but it conveys the DISTRIBUTIONS of results given the
computations being done.
> Mike Cowlishaw's Decimal proposal also supports user-specified precision,
> and the same kind of technique exploiting that also proved its practical
> worth in REXX. It's not foolproof, but it's very easy to use, and if
> results do "jump around" after changing precision a little you can be
> certain you have a numeric problem (alas, you may have one even if they
> don't jump). Of course people did a simple form of this in standard Fortran
> for decades, switching between REAL and DOUBLE "just to see what happened".
I'd like to get EVERYTHING that IEEE FP needs into the next rev. Mike
would work decimal. I'd turn NaNs over to Wayne Albin (if he would take
them), as Wayne knows more about this than everyone else I know
combined. I'd turn unnormalized significance over to Curtis Lacy,
because he is a PhD mathematician who has already implemented a
commercial system of this that worked VERY well. Me, I'd work on the
"loose ends" like logarithmic representation, etc.
I'd put all of this behind a "mode/format selection" interface that
provided for smooth limitless expansion.
This whole job can be done WELL if everyone just stops arguing about
doing it and sets down to making it so.
Who cares if significance works for everyone?! I'll be first to admit
that I like to use floating point instructions for wild bit
manipulations. The languages/system will come up in as-is 754 unless
they change it, so nothing can possibly be broken by what I am suggesting.
Do you see any problems with this approach?
Steve Richfield
Steve Richfie1d
> > > > > It could not be standardised yet in that case because there was
> > > > > insufficient knowledge.
> > > >
> > > > There is now! Wayne Albin developed this 30-years ago. I'm just working
> > > > from 30-year-old memories and common sense.
> > >
> > > Possibly, but at the time the standard emerged it was not available.
> > > Kahan (the major one, but also one amongst the others) has given
> > > compelling reasons for the IEEE standard.
> >
> > Well, the next time the standard is revised, then these should be fixed,
> > shouldn't they?
>
> Well, if you wish that, let us see a complete proposal. Not some ramblings.
I agree, but...
That would make me the only unpaid member of the committee, if indeed I
were even a member of the committee. Constructing these things, taking
care of ALL of the cases is no easy task, though a design job with about
as many problems as the present spec might not be all that much trouble.
Personally, I would prefer to do a cleaner job.
HEY all you lurkers and participants. Who among you is interested in
participating in this effort? Together we could do a LOT better job than
just me alone. Anyone interested? How about you Dik?
> > > If your underflow gives an erroneous sign, there is a problem with the
> > > application.
> >
> > Consider the following code for which the IEEE standard (as I understand
> > it) produces the wrong answer. Of course, the constants shown would be
> > variables obtained elsewhere, with duplicate shown values being
> > different variables with the same value:
> >
> > ! This produces zero, but without knowing more about the two !
> > arguments, it is impossible to know the proper sign, if any.
> > ! Of course, the present standard will arbitrarily invent a
> > ! sign.
> > X = 5.0 - 5.0
>
> Yup, this gives +0.0 under IEEE. (BTW, I am not too enamoured with the
> signed zeros and infinities either, I would much more prefer the affine
> completion with only a single zero and infinity.)
A single zero and infinity reduces the opportunities for promotion after
doing additional computations with them. Examples upon request.
> > ! Same situation here, but with different arguments.
> > Y = 3.0 - 3.0
>
> Also +0.0 under IEEE.
>
> > ! This should produce an indefinite, and NOT a signed oo
> > ! because there is no way to know if the signs of X and Y
> > ! are the same.
> > Z = X + Y
Oops, probably my fault. Sorry about that. Should be:
Z = 1/X + 1/Y
>
> Also +0.0 under IEEE.
>
> > ! This should print a string indicating that the answer cannot
> > ! be computed, but instead IEEE prints 4.
> > Print 4 + 13/Z
>
> No, Z = 0.0, so why do you think it prints 4?
>
> > > I still do not see that need. It is different. When I was involved with
> > > Ada standardisation, standardising complex arithmetic had problems *because
> > > of* the signed infinities.
> >
> > The above code also impinges on this problem. You need unsigned
> > infinities to deal with the reciprocals of unsigned zeros. You need
> > unsigned zeros for situations where the sign cannot be properly
> > determined, like the difference between identically valued arguments.
>
> Yup, so I do not see the need for signed quantities (I know why it was
> proposd by Kahan, but am not satisfied by his arguments).
Only unsigned is better than only signed as far as producing NaNs goes,
but both is better because you can add some infinities and get
infinities with signs, whereas they must all be demoted to indefinites
without signs.
> > > > With enough bits in the exponent, you wouldn't have any oo's. +oo is a
> > > > statement that your arithmetic system was unable to do what you asked it
> > > > to, but not because what you asked was reasonable, just beyond the
> > > > capacity of the exponent.
> > >
> > > You think so? How about division by an exact 0?
> >
> > IEEE does not have an exact numerical representation, nor does it
> > specify mixed-mode arithmetic, e.g. 5.0 / 0 where 0 is an integer.
>
> Then you misunderstand IEEE. The numbers are assumed to be exact,
Even though they obviously are NOT usually exact.
> and the
> results are the closest result based on that assumption.
Which produces the SAME numerical results, though different NaNs.
> So mixed mode is
> not needed, because converting an integer to a floating-point number
> results in exactly the same floating-point number, when there is an
> exact representation.
>
> Note that this is similar to the 40 year old system on which I started.
> Although it had both integers and floating point numbers, converting
> an integer to a normalised floating point number meant just setting the
> upper half of the floating point number to 0 bits.
I like systems where the exponent bias is such that integers ARE the
same in floating point. To normalize, just add 0.0. This sure gets rid
of float operations.
> The normalisation
> was Grau normalisation, look for an article by A.A. Grau in a fifties
> or early sixties issue of CACM.
>
> > > Nope. I am (in the core) a mathematician. And when you talk about
> > > numbers, there are other numbers than real numbers. Talking as a
> > > computer scientist, REAL's are not the real numbers, they are only a
> > > subset of the rational numbers.
> >
> > As a mathematician, IEEE FP must make you sick!
>
> Why? IEEE FP works pretty well within a special subring of the real
> numbers. There is of course a problem with overflow and underflow, but
> that is all.
Back to value, significance, and dimensionality. Compilers can track
dimensionality. IEEE FP tracks value, but who tracks significance? The
missing third of every variable should bother you.
> > > That is *not* the responsibility of the hardware. It is the application's
> > > responsibility whether a NaN coming from a square root of a negative
> > > number should continue, whether the result should be replaced by 0, or
> > > whatever.
> >
> > But, how can it distinguish between the square root of a negative number
> > and other NaN producing operations? Not all NaNs are created equal,
> > except in IEEE FP.
>
> No, the standard leaves it open. It is a quality of implementation issue
> whether the NaN's are distinguished or not.
What good is a nonstandard standard? Some results are standardized and
some are not. Ugh.
>
> > The result of a division by zero could be promoted by
> > using it in a denominator,
>
> Division by zero does not yield a NaN. And it *can* be promoted under IEEE.
>
> > whereas the square root of a negative number
> > could potentially be promoted by multiplying by zero. The whole reason
> > for distinguishing special forms is to provide for future promotion, and
> > either overspecification (like requiring that oo be signed) or
> > underspecification (like having no sign for oo) REALLY gets in the way
> > of proper future promotion.
>
> However, as the generation of NaN's and handling of NaN's, infinities and
> denormals are on most processors even not handled in hardware (they are
> handled by interrupt routines in software), the point is a bit moot.
IEEE is leaving the desing of a part of the system that even they cannot
design up to users? I see no hope for reasonable operation doing this.
>
> > Reading the IEEE spec, it seems obvious (to me) that promotion was never
> > even considered.
>
> Not for NaN's, not. Once a NaN, always a NaN.
When in fact there are very few NaNs that are completely beyond any hope
of promotion, even the square root of a negative number.
>
> > The theory behind all this goes as follows: Presume that the programmer
> > was an excellent mathematician but a less than stellar programmer, so
> > they have NOT re-arranged their expressions to avoid overflows, etc.
> > What could be then done with special forms to STILL make such code
> > produce correct answers whenever possible? Wayne worked this problem and
> > came up with the forms that I have been discussing.
>
> Possibly. Are those available somewhere, preferably online?
All part of a 30-year old design. I'll call around and see if anything
survives on paper.
>
> > Most real-world exceptions from good logic come from things like
> > dividing by the difference between identically-valued variables, and NOT
> > by manipulating things with astronomical exponents.
>
> That is one of the reasons the 40 year old system I wrote about did not
> have infinities, it just delivered the largest representable number.
I've seen this on a number of systems. Works pretty well in most
real-world cases, and better than some NaN systems.
Agreed, but those problems are in the minority. Let's fix what we can.
> > 2. Don't look at the guard digit when comparing arguments. Note that
> > subtracting and testing signs and for exactly zero doesn't work quite
> > right in the presence of a guard digit. However, considering a result to
> > be zero if all but the guard digit is zero works OK.
>
> I note that this would invalidate *all* of the error analysis I once did
> for an arcsine routine I had written.
Yes, I agree. Library routines are a special case, because they estimate
value at the sacrifice of both significance and dimensionality. Our uses
of our normalization procedure were mostly in support of the
trancendental library.
There, it is often easier just to take another term in the Taylor series
(or whatever is being used) than to go to the effort to compute how many
terms are needed, unless of course there was a lot of variability in the
number of terms that were needed.
Most of our trancendental routines separately computed significance,
then tacked it on at the end. This was related to the slope of the
function at the point of interest.
Steve Richfield
Steve Richfie1d
> Write a proposal, including how to do addition and subtraction. As far as
> I know, it has been proposed once (I remember seeing an article in the
> CACM), but never flew.
There's the excellent 1959 Ashenhurst & Metropolis paper that guided our
30-year-old efforts. Did you possibly see something else more recent?
The situation in 1959 was that all of the computer manufacturers had
just announced their next generation of machines, so it would be another
decade or so before this could be considered. Further, the article was
VERY good and VERY complete, even by today's standards, so it wasn't
followed by other articles that built on it. By the time that the
manufacturers were ready to design their next generation, pretty much
everyone had forgotten the article.
However, our efforts didn't go unnoticed at HP, as we had implemented
our significance system on an **HP** 2116 computer. In 1973 I
interviewed with HP to design a similar hardware significance system for
their future 3000 series computers, and was accepted. However, during
the week between interview and starting work, HP put an employment
freeze into place that lasted until the 3000 design was completed, and
THAT is why there was no significance system on HP 3000 computer
systems. If not for the unfortunate timing of that hiring freeze, I'd
guess that IEEE-754 would now already incorporate significance, as it
would have then been common knowledge when 754 was being designed.
Anyway, an interesting tidbit of history.
Your experience with forward error analysis obviously exceeds mine.
Would you be interested in helping produce a proposal? No system can
work perfectly for everything. I'd just like to get a significance
system working as well as practical. Any interest?
Steve Richfield
Steve Richfie1d
> In article <aad5472df7d60539...@news.teranews.com> St...@NOSPAM.smart-life.net writes:
> > Dik,
> > > > And, financial forecasting is the greatest challenge of all, because
> > > > money has little/no inertia, so huge amounts can move in an instant.
> > > > Here, significance appears to be the present barrier to advancing the
> > > > state of the art.
> > >
> > > I do not think so. Most of finance prediction involves non-predictable
> > > hype.
> >
> > As I understand the situation (if indeed ANYONE does), most of the time
> > things are very predictable, but often they are not. Indeed, this is
> > where market volatility indicators came from. They can tell you that
> > there is going to be a big change, but can't even produce the sign of
> > that change.
>
> Oh, fine, so they can tell me there will be a big change. How do they
> predict that the books of some company are fraudulent? My prediction is
> that on the stock market there will be a big change in the next month.
> Do I win a price?
Of course, market volatility indicators are really indicating the
expected significance of estimates during that period, though even the
people who prepare these indicators don'r realize that this is what they
are doing. High volatility = low significance.
You prize: Knowing this, you'll reduce your investment in the market to
protect yourself from a possible crash. Hence, when things go unstable,
you get to survive financially, while some fraction of those around you
don't.
Of course, there are always the folks why say "Good, now I can make some
REAL money" and they invest heavily. They often do win a time or two of
this, but eventually they lose everything, get fired, and move into
another line of work.
Steve Richfield
Dik T. Winter
...
> > Well, if you wish that, let us see a complete proposal. Not some ramblings.
>
> I agree, but...
>
> That would make me the only unpaid member of the committee, if indeed I
> were even a member of the committee.
Do you think the committee members are paid by IEEE? I think you are
wrong. I think at most the committee's secretary gets some compensation.
> HEY all you lurkers and participants. Who among you is interested in
> participating in this effort? Together we could do a LOT better job than
> just me alone. Anyone interested? How about you Dik?
Not really, it is no longer my field.
> > > IEEE does not have an exact numerical representation, nor does it
> > > specify mixed-mode arithmetic, e.g. 5.0 / 0 where 0 is an integer.
> >
> > Then you misunderstand IEEE. The numbers are assumed to be exact,
>
> Even though they obviously are NOT usually exact.
But they can be. If I understand significance arithmetic well enough, in
that *no* numbers are exact.
> > Why? IEEE FP works pretty well within a special subring of the real
> > numbers. There is of course a problem with overflow and underflow, but
> > that is all.
>
> Back to value, significance, and dimensionality. Compilers can track
> dimensionality. IEEE FP tracks value, but who tracks significance? The
> missing third of every variable should bother you.
Significance is not a mathematical term, so I do not know much about that.
And I have absolutely *no* idea what you mean by "dimensionality".
> > > But, how can it distinguish between the square root of a negative number
> > > and other NaN producing operations? Not all NaNs are created equal,
> > > except in IEEE FP.
> >
> > No, the standard leaves it open. It is a quality of implementation issue
> > whether the NaN's are distinguished or not.
>
> What good is a nonstandard standard? Some results are standardized and
> some are not. Ugh.
What *was* important at that time was that the arithmetic finally was right.
Considering the number of machines with extremely sloppy arithmetic that
was a good step.
> > However, as the generation of NaN's and handling of NaN's, infinities and
> > denormals are on most processors even not handled in hardware (they are
> > handled by interrupt routines in software), the point is a bit moot.
>
> IEEE is leaving the desing of a part of the system that even they cannot
> design up to users? I see no hope for reasonable operation doing this.
IEEE can not require something to be done in hardware. There are even
systems without hardware floating point that nevertheless implement IEEE.
And they conform to the standard. Almost all current processors do the
standard cases in hardware, leaving the remainder to interrupt routines.
> > > Reading the IEEE spec, it seems obvious (to me) that promotion was never
> > > even considered.
> >
> > Not for NaN's, not. Once a NaN, always a NaN.
>
> When in fact there are very few NaNs that are completely beyond any hope
> of promotion, even the square root of a negative number.
I would argue that when your program tries to extract the square root of
a negative number, there is something wrong in the logic of the program.
> > That is one of the reasons the 40 year old system I wrote about did not
> > have infinities, it just delivered the largest representable number.
>
> I've seen this on a number of systems. Works pretty well in most
> real-world cases, and better than some NaN systems.
Eh? They remove the infinities...
> > > Now that you understand the problems with error analysis, I would think
> > > that you would have already joined the significance bandwagon!
> >
> > Wrong. Significance is close to forward error analysis and that is close
> > to impossible for some problems.
>
> Agreed, but those problems are in the minority. Let's fix what we can.
Eh? Solving linear systems is a *huge* part of most programs.
> > I note that this would invalidate *all* of the error analysis I once did
> > for an arcsine routine I had written.
...
> There, it is often easier just to take another term in the Taylor series
> (or whatever is being used) than to go to the effort to compute how many
> terms are needed, unless of course there was a lot of variability in the
> number of terms that were needed.
You obviously have no idea how a well-designed library routine for the
mathematical functions *is* designed. Moreover, most of the error
analysis effort in the arcsine was in the argument modification part,
not in the part that finally calculates some actual function value.
Jeff Kenton
Steve,
I have read the paper you emailed, and I have a concrete suggestion for your
significance requirements. I don't think anyone is going to change the single
or double precision FP spec to your satisfaction. However, the extended FP
that most computers implement might be an area to work on.
First, the high bit is explicit, making it possible to deal with unnormalized
numbers.
Second, there are enough extra bits, so you could have the 4 guard bits you want.
Third, not many people use extended FP explicitly (even though intermediate
results are kept extended internally), so you would break fewer existing programs.
OTOH, you still have to specify exactly how significance arithmetic works in
the new environment, how the arithmetic modes would be controlled by the
hardware, and how to integrate all that into programming languages. Plus,
whatever it is you want to do with NaNs and infinities and zeroes, which are
all a separate issue.
jeff
Herman Rubin
> > > It could not be standardised yet in that case because there was
> > > insufficient knowledge.
> > There is now! Wayne Albin developed this 30-years ago. I'm just working
> > from 30-year-old memories and common sense.
>Possibly, but at the time the standard emerged it was not available.
>Kahan (the major one, but also one amongst the others) has given
>compelling reasons for the IEEE standard.
I have never seen any remotely decent reason for the flaws
which were pointed out AT THE TIME in the standard. It is
easy, and not that costly, to put in modifications which do
not really affect the hardware at the time of engineering
it, but it is extremely costly to handle it with software
later. Not all instruction bits need to be decoded when
the instruction is issued.
In my own work, I definitely do not agree with Steve about
significance. In fact, it used to be a standard exercise
BC (before computers) to ask students to compute means and
variances using significance arithmetic, with examples
where the computed variance becomes negative. I have many
other examples where the significance kept at computing
time should be considerably more than in the data, even
when one may be combining items of apparently different
significance.
My objection is to the forced normalization and hidden
bit. Try representing multiple precision floating numbers
intelligently with that restriction. Another objection is
separate integer and floating registers, and I even have
numerous situations where I want to use Boolean operations
with floats.
> > > Whether a signalling or non-signalling NaN stops the program is under
> > > user control... The distinction is mostly between generated NaN's and
> > > propagated NaN's.
> > This subject is much deeper than the IEEE developers have yet realized.
> > There is a gradual demotion/promotion of special forms, where they can
> > get more or less vague through computations - sometimes even rising back
> > to real numbers, e.g. dividing by infinity. Whether a particular
> > demotion should interrupt the program is properly a function of the
> > application or subroutine that it is running in.
>Right so, so it is the application's or subroutine's responsiility, not
>of the hardware. IEEE properly delegates this to that place.
Often at a very great cost. There has been a posting about this
causing a factor of almost 1000 in running time. I have seen one
machine's hardware with a substantial range in exponents of NaN's;
this would enable rapid recovery in many cases. That was a very
CISCy machine, which however could almost issue instructions on
a RISC-type basis.
...................
There has been a rapid decline in the ability of computers
to do good precise mathematics quickly. It is partly due
to "simplifications" in engineering, and partly due to the
fact that our programming languages are designed to keep
the user from doing what those who understand what goes on
in computing can show is simple from the actual bit point
of view.
--
This address is for information only. I do not claim that these views
are those of the Statistics Department or of Purdue University.
Herman Rubin, Department of Statistics, Purdue University
hru...@stat.purdue.edu Phone: (765)494-6054 FAX: (765)494-0558
Herman Rubin
>In article <f449052e77689c3f...@news.teranews.com> St...@NOSPAM.smart-life.net writes:
> > > Eh? I cannot follow. The only logical operations in C are && and ||.
> > > I see no problem with them.
> > Obviously you want to do the bitwise first, then the boolean, as bitwise
> > operation on a boolean result doesn't make much sense. However, C
> > specifies this the other way around.
>Eh? Consider:
> 2 & 2 && 1 & 1, which delivers 1, or true
>vs.
> 2 & (2 && 1) & 1, which delivers 0, or false.
>Clearly the bitwise operations go before the logical operations. (Yes,
>every non-zero value is considered true in the logical operations.)
> > > I doubt that. Both Cray and NEC performed pretty well, while on the 205
> > > we had to use Q8 calls to get what we wanted. There was *no* way on the
> > > 205 to get the performance we obtained, except by using Q8 calls.
> > According to an internal report that CDC prepared, Crays vectorized
> > about 50% of loops that weren't hand-modified for vectorization, whereas
> > CDCs vectorized about 70%.
>We had to hand modify code to get good vectorisation on the Cray and the
>NEC, but doing that was beneficial to both systems. We also had to hand
>modify for the 205, but that involved Q8 calls (which was just syntactic
>sugar for assembly).
What was bad about that? The 205 optimized instructions
after compiling, so assembler instructions were included
in this. Frankly, I would not mind if there was a decent
assembler language, and this can be done, with user-defined
macros in an easy-to-write version, etc. ALL compilers
need to allow more use of the hardware's operations that
the compiler writers did not consider, and interactive user
participation.
> > Most of the difference was because CDCs had bit
> > vectors, so that most loops containing conditional statements could be
> > vectorized by first precomputing the results of all of the conditionals
> > into a bit vector that the compiler did quite well, then passing the bit
> > vector to following vector operations to suppress the storing of results
> > where the corresponding bit was a zero.
>Note what was happening. The results were computed, but not stored. So
>they took time. Other machines used contract, calculate and expand, which
>has the same effect but is better when the calculations are large.
The bit vectors were stored. And the CDCs, if programmed
intelligently, did it that way.
> > Of course, Crays didn't have to even bother to compute the results that
> > CDCs would suppress, so they gained a bunch there. The net results from
> > the internal benchmarks that I saw was that Crays won with scalars and
> > short vectors, whereas CDCs won with long vector applications.
See the above. An intelligent programmer will almost automatically
outperform a highly limited compiler. Compiler writers are more
concerned with what idiots might do than how to do things well
which are not in their limited ken.
>The reason for that is simply that Cray had 64 word vector registers and
>CDC had a continuous flow to memory with a maximum of 65536 (or somesuch)
>words. If I remember right, CDC's compilers where flabbergasted if the
>vector length exceeded that number. Cray had no problem with vectors
>larger than 64 words.
See my above comments on compiler writers. They did not foresee
that there were uses with vector longer than that; I managed
those vectors myself, with no trouble. Try to anticipate anything
which can be done, and you get into a straitjacket.
..................
>The institution that bought the first supercomputer in the Netherlands
>asked me (amongst others) to check both. So I have run various tests
>on both and my conclusion was that Cray was better (it is a national
>laboratory in your sense). But CDC was chosen.
I was asked at Purdue. My conclusion, based on the type of things
I was doing and considering, was quite otherwise. One of the
considerations was multiple precision, which was quite reasonable
on the CDC, but extremely difficult on the Cray. The treatment
of vectors which needed different operations on different elements
was next to impossible on the early Crays.
Fred J. Tydeman
>
> > That would make me the only unpaid member of the committee, if indeed I
> > were even a member of the committee.
>
> Do you think the committee members are paid by IEEE? I think you are
> wrong. I think at most the committee's secretary gets some compensation.
Based upon the one IEEE-754R meeting I have been to, it is the same as
C.
We pay our own way to the meetings.
On the other hand, IEEE-754R does have people call in to
teleconference.
And, most of the work is done via email (no paper mailings, and no
separate
subgroup meetings), so it is possible to "attend" on the cheap.
---
Fred J. Tydeman Tydeman Consulting
tyd...@tybor.com Programming, testing, numerics
+1 (775) 287-5904 Vice-chair of J11 (ANSI "C")
Sample C99+FPCE tests: ftp://jump.net/pub/tybor/
Savers sleep well, investors eat well, spenders work forever.
Steve Richfie1d
> Do you think the committee members are paid by IEEE? I think you are
> wrong. I think at most the committee's secretary gets some compensation.
No, they all work for the respective special interests that they
represent, who pay them. That IEEE committee members are unpaid by the
IEEE is one of the BIG reasons that so many specs have so many problems.
I'd rather pay a little more in IEEE dues and get better standards. At
minimum, specs should be accepted by their DETRACTORS. For example, if
YOU accepted the details of a significance proposal, then it would
probably be a LOT better than if it was prepared by some special
interest, and then accepted by other who didn't care.
> > HEY all you lurkers and participants. Who among you is interested in
> > participating in this effort? Together we could do a LOT better job than
> > just me alone. Anyone interested? How about you Dik?
>
> Not really, it is no longer my field.
Now, it is not ANYONE's field. For me, this stuff is 30 years old.
Someone should stand up and do this.
>
> > > > IEEE does not have an exact numerical representation, nor does it
> > > > specify mixed-mode arithmetic, e.g. 5.0 / 0 where 0 is an integer.
> > >
> > > Then you misunderstand IEEE. The numbers are assumed to be exact,
> >
> > Even though they obviously are NOT usually exact.
>
> But they can be. If I understand significance arithmetic well enough, in
> that *no* numbers are exact.
Though it depends on the implementation, the only difference between
significance and normalized arithmetic is that some mechanism tracks the
significance, beyond which you shouldn't trust the result. However,
significance methods only work for ~99.9% of the cases. For the
remainder, normalization is required. We had a normalizing library
routine for these rare cases but only needed it a few times. We also had
an exact representation - basically, an integer in floating point form
with an exact flag.
>
> > > Why? IEEE FP works pretty well within a special subring of the real
> > > numbers. There is of course a problem with overflow and underflow, but
> > > that is all.
> >
> > Back to value, significance, and dimensionality. Compilers can track
> > dimensionality. IEEE FP tracks value, but who tracks significance? The
> > missing third of every variable should bother you.
>
> Significance is not a mathematical term, so I do not know much about that.
> And I have absolutely *no* idea what you mean by "dimensionality".
12.3 gallons. Value = 12.3, significance = +/-0.05, dimensionality =
gallons, or in the MKS system, M^^3 or K depending on whether the
gallons are by volume or weight. These are the basic properties of ALL
measures, that in a perfect world would be carried through ALL computations.
> > > > But, how can it distinguish between the square root of a negative number
> > > > and other NaN producing operations? Not all NaNs are created equal,
> > > > except in IEEE FP.
> > >
> > > No, the standard leaves it open. It is a quality of implementation issue
> > > whether the NaN's are distinguished or not.
> >
> > What good is a nonstandard standard? Some results are standardized and
> > some are not. Ugh.
>
> What *was* important at that time was that the arithmetic finally was right.
> Considering the number of machines with extremely sloppy arithmetic that
> was a good step.
OK, time to move on and fix the stuff that was left for later.
> > > However, as the generation of NaN's and handling of NaN's, infinities and
> > > denormals are on most processors even not handled in hardware (they are
> > > handled by interrupt routines in software), the point is a bit moot.
> >
> > IEEE is leaving the desing of a part of the system that even they cannot
> > design up to users? I see no hope for reasonable operation doing this.
>
> IEEE can not require something to be done in hardware. There are even
> systems without hardware floating point that nevertheless implement IEEE.
> And they conform to the standard. Almost all current processors do the
> standard cases in hardware, leaving the remainder to interrupt routines.
I said nothing about putting things into hardware, just providing a
standard for hardware or software that works - so every user doesn't
have to hire guys like us to finish the design just for them.
>
> > > > Reading the IEEE spec, it seems obvious (to me) that promotion was never
> > > > even considered.
> > >
> > > Not for NaN's, not. Once a NaN, always a NaN.
> >
> > When in fact there are very few NaNs that are completely beyond any hope
> > of promotion, even the square root of a negative number.
>
> I would argue that when your program tries to extract the square root of
> a negative number, there is something wrong in the logic of the program.
Agreed.
> > > That is one of the reasons the 40 year old system I wrote about did not
> > > have infinities, it just delivered the largest representable number.
> >
> > I've seen this on a number of systems. Works pretty well in most
> > real-world cases, and better than some NaN systems.
>
> Eh? They remove the infinities...
You lose something (infinities, reliability of large results, etc.) and
gain something (more correct numerical results where you would have had
a NaN for output). As a user at that time, I wanted good numerical
answers, do I liked the force-to-maximum method.
The ultimate NaN system should do about as well.
> > > > Now that you understand the problems with error analysis, I would think
> > > > that you would have already joined the significance bandwagon!
> > >
> > > Wrong. Significance is close to forward error analysis and that is close
> > > to impossible for some problems.
> >
> > Agreed, but those problems are in the minority. Let's fix what we can.
>
> Eh? Solving linear systems is a *huge* part of most programs.
Hey, the rest of you lurkers. How many of your programs are solving
linear systems? Dik says most, whereas I suspect that the number might
be a few percent. How about a quick poll?
> > > I note that this would invalidate *all* of the error analysis I once did
> > > for an arcsine routine I had written.
> ...
> > There, it is often easier just to take another term in the Taylor series
> > (or whatever is being used) than to go to the effort to compute how many
> > terms are needed, unless of course there was a lot of variability in the
> > number of terms that were needed.
>
> You obviously have no idea how a well-designed library routine for the
> mathematical functions *is* designed. Moreover, most of the error
> analysis effort in the arcsine was in the argument modification part,
> not in the part that finally calculates some actual function value.
OK, guilty as charged. I've only written one arcsin function, and I just
used the formulas from the Handbook of Mathematical Functions that
worked great.
Steve Richfield
Steve Richfie1d
EXCELLENT input.
The following comments follows (my understanding of) the X.25 concept
for standards to work across versions, so that cross-version
communication is possible even between systems using different versions
of a standard.
My imperfectly formed thought was that somewhere there needs to be
separate version, format, and length information as to what the software
expects from the hardware, none of which was incorporated in the current
FP spec. This would be kept outside of the numbers themselves, and set
by a subroutine library call or special instruction. Once these become a
part of the spec, then the dangers of fixes and enhancements to the
standard are reduced by a couple of orders of magnitude. Then,
applications can specify which version of which type of FP to put into
their 32-bit (or whatever) words. With this, FP could move smoothly into
the future.
X.25 discusses at length how to deal with unimplemented combinations,
incompatibilities, etc., much of which could be adapted and brought into
this context. Here, I suspect that if, say, an FADD instruction is
performed but the hardware doesn't support the particular version,
format, and length combination that was specified, then this should trap
into a vector of subroutine calls to routines to handle these cases.
Once this critical superstructure is in place, then new proposals like
Mike's and mine can easily be written and debugged in software, and then
even used with existing proprietary programming systems.
Manufacturers can then compete with each other on the basis of which
formats their processors respect - as it should be for technology to
proceed in this area.
There are a couple of other details that must be examined to make sure
things work smoothly with all known alternative numerical technologies,
such as potentially ignoring guard bits for comparisons as significance
methods need. I'm not sure whether there is actually anything else to do
here, but this should be carefully examined.
In short, there needs to be a superstructure where there is none now,
with current IEEE-754 being the default "Version 0.0" if nothing is done
to select other options.
What legitimate reason might there be to resist such a move?
Again, thanks for your comments.
Steve Richfield
================
Steve Richfie1d
> I have never seen any remotely decent reason for the flaws
> which were pointed out AT THE TIME in the standard. It is
> easy, and not that costly, to put in modifications which do
> not really affect the hardware at the time of engineering
> it, but it is extremely costly to handle it with software
> later. Not all instruction bits need to be decoded when
> the instruction is issued.
YES, especially when there are important operations in other arithmetic
systems that were not included here.
> In my own work, I definitely do not agree with Steve about
> significance. In fact, it used to be a standard exercise
> BC (before computers) to ask students to compute means and
> variances using significance arithmetic, with examples
> where the computed variance becomes negative. I have many
> other examples where the significance kept at computing
> time should be considerably more than in the data, even
> when one may be combining items of apparently different
> significance.
A little too much significance is certainly better than a LOT too much
significance from normalization! However...
Methods of periodically reducing significance have been proposed in
various articles. This is discussed in detail in the 1959 paper by
Ashenhurst and Metropolis in the JACM. This solves the problem I think
you are describing, but is a bit counterintuitive in some applications.
At Remote Time-Sharing we didn't automatically reduce significance, but
provided the tools for users to do this themselves in their code under a
little more control. I never heard any objection to this approach. To
adjust the significance, a user might use code like:
! This produces a zero with the significance of X.
Adjuster = X - X
! Now, limit a variables significance to that of X.
TOTAL = TOTAL + X
> My objection is to the forced normalization and hidden
> bit. Try representing multiple precision floating numbers
> intelligently with that restriction. Another objection is
> separate integer and floating registers, and I even have
> numerous situations where I want to use Boolean operations
> with floats.
More good points.
>>>>Whether a signalling or non-signalling NaN stops the program is under
>>>>user control... The distinction is mostly between generated NaN's and
>>>>propagated NaN's.
>
>
>>>This subject is much deeper than the IEEE developers have yet realized.
>>>There is a gradual demotion/promotion of special forms, where they can
>>>get more or less vague through computations - sometimes even rising back
>>>to real numbers, e.g. dividing by infinity. Whether a particular
>>>demotion should interrupt the program is properly a function of the
>>>application or subroutine that it is running in.
>
>
>>Right so, so it is the application's or subroutine's responsiility, not
>>of the hardware. IEEE properly delegates this to that place.
>
>
> Often at a very great cost. There has been a posting about this
> causing a factor of almost 1000 in running time. I have seen one
> machine's hardware with a substantial range in exponents of NaN's;
> this would enable rapid recovery in many cases. That was a very
> CISCy machine, which however could almost issue instructions on
> a RISC-type basis.
NaN promotion/demotion should be done automatically in the hardware,
with control thrown to software only when the user requests it.
Steve Richfield
=================
Steve Richfie1d
I tried to contact them. All I could get from the IEEE was a name and
phone number. I left a message, but no one called back.
Do you have some working contact information?
Steve Richfield
==================
Boudewijn Dijkstra
news:d2ed3690ccb8b932...@news.teranews.com...
> > Eh? Solving linear systems is a *huge* part of most programs.
>
> Hey, the rest of you lurkers. How many of your programs are solving
> linear systems? Dik says most, whereas I suspect that the number might
> be a few percent. How about a quick poll?
Most of my programs were created for my own personal enjoyment, none of them
are financial and most of the interesting ones are physical simulations.
None of my programs /solve/ any linear system. If any linear system is
involved, the programs just calculate the result of the solved system,
assuming that the environment in which the program runs, behaves acceptably
linear.
Just my $0:01111001:01000111101011100001010
Dik T. Winter
> > Significance is not a mathematical term, so I do not know much about that.
> > And I have absolutely *no* idea what you mean by "dimensionality".
>
> 12.3 gallons. Value = 12.3, significance = +/-0.05, dimensionality =
> gallons, or in the MKS system, M^^3 or K depending on whether the
> gallons are by volume or weight. These are the basic properties of ALL
> measures, that in a perfect world would be carried through ALL computations.
Interesting, and explains why I never met the term. The numbers I have
been working with were dimenionless. (Calculating special functions,
like sine, acsine, etc., doing linear systems, calculating eigenvalues
and singular values, integrating functions, integrating differential
equations, etc.) However, it is *not* the task of the processor to
keep abreast of those dimensions. Unless you give a good indication
about what dimenstions it should carry through. (Personally I prefer
SI.)
> > > IEEE is leaving the desing of a part of the system that even they cannot
> > > design up to users? I see no hope for reasonable operation doing this.
> >
> > IEEE can not require something to be done in hardware. There are even
> > systems without hardware floating point that nevertheless implement IEEE.
> > And they conform to the standard. Almost all current processors do the
> > standard cases in hardware, leaving the remainder to interrupt routines.
>
> I said nothing about putting things into hardware, just providing a
> standard for hardware or software that works - so every user doesn't
> have to hire guys like us to finish the design just for them.
In another article you say something else, I will come back to that.
> > > When in fact there are very few NaNs that are completely beyond any hope
> > > of promotion, even the square root of a negative number.
> >
> > I would argue that when your program tries to extract the square root of
> > a negative number, there is something wrong in the logic of the program.
>
> Agreed.
So why would something that is the result of an error in programming logic
be promoted?
> > You obviously have no idea how a well-designed library routine for the
> > mathematical functions *is* designed. Moreover, most of the error
> > analysis effort in the arcsine was in the argument modification part,
> > not in the part that finally calculates some actual function value.
>
> OK, guilty as charged. I've only written one arcsin function, and I just
> used the formulas from the Handbook of Mathematical Functions that
> worked great.
Worked great. Granted. But did not give you the best quality.
Dik T. Winter
> NaN promotion/demotion should be done automatically in the hardware,
> with control thrown to software only when the user requests it.
In an answer to me you were saying that you were not talking about the
difference in hardware and software, here you require something done
in hardware. For that, please, talk to the hardware producers, not the
standard producers. (There is a good reason it is not done in hardware
on almost all processors.)
Steve Richfie1d
> > > I would argue that when your program tries to extract the square root of
> > > a negative number, there is something wrong in the logic of the program.
> >
> > Agreed.
>
> So why would something that is the result of an error in programming logic
> be promoted?
Promoting something requires doing just the right thing, which is
different depending on the particular thing that might be promoted.
Further, if you look at NaNs and what it might take to promote them, it
is often exactly what you would expect the next operation to be in a
real-world program that just "stepped over the edge", you'd expect that
overflows would find their way into a denominator.
Now, suppose that you are taking the square roots of computed numbers
near zero, but then they go a bit negative. You could analyze why this
might happen, and there might be some reasonable explanation. However,
the test of promotability is whether an operation is done which could
promote a particular NaN. In the case of imaginary numbers, there isn't
much you can do to promote them, EXCEPT multiplying them by zero. Not
likely, but if that were indeed the next thing done with the imaginary,
just maybe the algorithm was good for the result.
OK, you're right in the sense that while the above argument may be
reasonable, it probably isn't worth one single transistor to make it work.
> > > You obviously have no idea how a well-designed library routine for the
> > > mathematical functions *is* designed. Moreover, most of the error
> > > analysis effort in the arcsine was in the argument modification part,
> > > not in the part that finally calculates some actual function value.
> >
> > OK, guilty as charged. I've only written one arcsin function, and I just
> > used the formulas from the Handbook of Mathematical Functions that
> > worked great.
>
> Worked great. Granted. But did not give you the best quality.
... unless reasonable-but-technically-wrong numbers were worth more to
you than NaNs.
Steve Richfield
Steve Richfie1d
> In an answer to me you were saying that you were not talking about the
> difference in hardware and software, here you require something done
> in hardware. For that, please, talk to the hardware producers, not the
> standard producers. (There is a good reason it is not done in hardware
> on almost all processors.)
Please explain. What is the "good reason"? Some array computations
produce a predominance of abnormal results, so that forcing these into
software effectively slows the hardware down to software speeds.
I do agree with you that this is outside of the standard. I was just
wondering why (apparently to me, needlessly) push it into software.
Wherever it is done, the functionality should be fully specified in the
spec.
Steve Richfield
Dik T. Winter
> > So why would something that is the result of an error in programming logic
> > be promoted?
>
> Promoting something requires doing just the right thing, which is
> different depending on the particular thing that might be promoted.
> Further, if you look at NaNs and what it might take to promote them, it
> is often exactly what you would expect the next operation to be in a
> real-world program that just "stepped over the edge", you'd expect that
> overflows would find their way into a denominator.
>
> Now, suppose that you are taking the square roots of computed numbers
> near zero, but then they go a bit negative. You could analyze why this
> might happen, and there might be some reasonable explanation.
Yup, there might be, and if that is the case the best way to go is to change
them to 0, without requiring any multiplication. The upshot is that *when*
a NaN is generated you have to analise the reason and you should *not*
rely on some harware solution that might give you very wrong results.
When standardising the Ada numerics library there was some discussion
about this, e.g. what should the sqrt function do if the argument was
only slightly negative. The general opinion was: raise an exception,
or whatever. It is the task of the user to force that the argument is
positive (or zero for you non-Bourkaki-ists).
> However,
> the test of promotability is whether an operation is done which could
> promote a particular NaN. In the case of imaginary numbers, there isn't
> much you can do to promote them, EXCEPT multiplying them by zero. Not
> likely, but if that were indeed the next thing done with the imaginary,
> just maybe the algorithm was good for the result.
Or maybe it was just more wrong than it was originally. When a square
root is requested for a negative number that either indicates a precision
problem (in which case flushing to zero would be the best) or a problem
in the programming logic.
> OK, you're right in the sense that while the above argument may be
> reasonable, it probably isn't worth one single transistor to make it work.
I do not even think it is reasonable.
> > > OK, guilty as charged. I've only written one arcsin function, and I just
> > > used the formulas from the Handbook of Mathematical Functions that
> > > worked great.
> >
> > Worked great. Granted. But did not give you the best quality.
>
> ... unless reasonable-but-technically-wrong numbers were worth more to
> you than NaNs.
Let's say: speed. A reasonably designed algorithm will generally outperform
Abramowitz and Stegun's formulas by a factor 2 or more. A well-designed
algorithm may even perform better. And all giving you exactly (or nearly
exactly) the same results, or better. But my perspective is black-box
design, obviously.
Dik T. Winter
> Dik T. Winter wrote:
> > In an answer to me you were saying that you were not talking about the
> > difference in hardware and software, here you require something done
> > in hardware. For that, please, talk to the hardware producers, not the
> > standard producers. (There is a good reason it is not done in hardware
> > on almost all processors.)
>
> Please explain. What is the "good reason"?
Speed. If you have a fully pipelined floating point processor, where you
can issue one operation every clock-cycle, you need a fixed length of
the pipe (i.e. a fixed number of cycles) to complete the operation. A
denormal is something that requires special processing at the hardware
level, so doing that in a pipelined processor is at least problematical.
(It can be done, but you need quite a bit of additional hardware to allow
stalling the processor.) On many current FP processors floating point
multiply, add and subtract are pipelined. BTW, the first processor I
have used where a floating point multiply took essentially the same cycles
as an add or subtract was a CDC Cyber 750.
> Some array computations
> produce a predominance of abnormal results, so that forcing these into
> software effectively slows the hardware down to software speeds.
Been there, seen that. In some cases adding 1 to each element of a
matrix reduced the time for the solution of a linear system by a
considerable amount. Without that addition there were quite a lot of
underflows resulting in denormals.
> I do agree with you that this is outside of the standard. I was just
> wondering why (apparently to me, needlessly) push it into software.
>
> Wherever it is done, the functionality should be fully specified in the
> spec.
IEEE did quite well (I think) at that point of time. Although Herman
Rubin disagrees (yes, I know his arguments). One problem with the
CDC Cyber 750 was that because post normalisation was not performed
on addition and subtraction (which would have cost at most one cycle
in latency) made those operations actually more expensive than
multiplication (in well coded software by a factor of two). And indeed,
when I designed the routine for calculations on the zeros of the Riemann
zeta functions, add and subtract were the bottleneck.
Steve Richfie1d
> > > In an answer to me you were saying that you were not talking about the
> > > difference in hardware and software, here you require something done
> > > in hardware. For that, please, talk to the hardware producers, not the
> > > standard producers. (There is a good reason it is not done in hardware
> > > on almost all processors.)
> >
> > Please explain. What is the "good reason"?
>
> Speed. If you have a fully pipelined floating point processor, where you
> can issue one operation every clock-cycle, you need a fixed length of
> the pipe (i.e. a fixed number of cycles) to complete the operation. A
> denormal is something that requires special processing at the hardware
> level, so doing that in a pipelined processor is at least problematical.
> (It can be done, but you need quite a bit of additional hardware to allow
> stalling the processor.)
Certainly, stalling the processor is cheaper than handling the exception
in software. Am I missing something here?
> On many current FP processors floating point
> multiply, add and subtract are pipelined. BTW, the first processor I
> have used where a floating point multiply took essentially the same cycles
> as an add or subtract was a CDC Cyber 750.
On logarithmic processors, multiply and divide are the easy operations,
while add and subtract take a bunch of pipeline stages. Such is the
world of DSPs.
> > Some array computations
> > produce a predominance of abnormal results, so that forcing these into
> > software effectively slows the hardware down to software speeds.
>
> Been there, seen that. In some cases adding 1 to each element of a
> matrix reduced the time for the solution of a linear system by a
> considerable amount. Without that addition there were quite a lot of
> underflows resulting in denormals.
>
> > I do agree with you that this is outside of the standard. I was just
> > wondering why (apparently to me, needlessly) push it into software.
> >
> > Wherever it is done, the functionality should be fully specified in the
> > spec.
>
> IEEE did quite well (I think) at that point of time. Although Herman
> Rubin disagrees (yes, I know his arguments).
Is there any reason not to now finish the job?
> One problem with the
> CDC Cyber 750 was that because post normalisation was not performed
> on addition and subtraction (which would have cost at most one cycle
> in latency) made those operations actually more expensive than
> multiplication (in well coded software by a factor of two).
Why? Was normalization needed here? Many computers had both a normalized
and an unnormalized floating add so you could take your choice. I don't
have any experience with the 750. I guess it didn't?
I know that if you throw two denormalized numbers into a multiplier, the
result is denormalized by the sum of the two denormalizations. Is THAT
the problem here? If so - bad design by CDC.
> And indeed,
> when I designed the routine for calculations on the zeros of the Riemann
> zeta functions, add and subtract were the bottleneck.
To force normalization? Obviously, there is an education here that I
have not (yet) had. If I haven't seen it, I presume that lots of the
rest of the readers here haven't seen it. If it wouldn't wear your
keyboard out, please explain.
Steve Richfield
Dik T. Winter
> > > > In an answer to me you were saying that you were not talking about the
> > > > difference in hardware and software, here you require something done
> > > > in hardware. For that, please, talk to the hardware producers, not the
> > > > standard producers. (There is a good reason it is not done in hardware
> > > > on almost all processors.)
> > >
> > > Please explain. What is the "good reason"?
> >
> > Speed. If you have a fully pipelined floating point processor, where you
> > can issue one operation every clock-cycle, you need a fixed length of
> > the pipe (i.e. a fixed number of cycles) to complete the operation. A
> > denormal is something that requires special processing at the hardware
> > level, so doing that in a pipelined processor is at least problematical.
> > (It can be done, but you need quite a bit of additional hardware to allow
> > stalling the processor.)
>
> Certainly, stalling the processor is cheaper than handling the exception
> in software. Am I missing something here?
It would speed up the case when there was something special, but it would
decrease the speed when there was nothing special. Moreover, quite a bit
of extra hardware would be needed to make the stalling possible.
> > One problem with the
> > CDC Cyber 750 was that because post normalisation was not performed
> > on addition and subtraction (which would have cost at most one cycle
> > in latency) made those operations actually more expensive than
> > multiplication (in well coded software by a factor of two).
>
> Why? Was normalization needed here? Many computers had both a normalized
> and an unnormalized floating add so you could take your choice. I don't
> have any experience with the 750. I guess it didn't?
Not it did not, it had only unnormalised add.
> I know that if you throw two denormalized numbers into a multiplier, the
> result is denormalized by the sum of the two denormalizations. Is THAT
> the problem here? If so - bad design by CDC.
Yup. But why bad design? Or do you think they should have supplied
normalised add only? Providing both was not possible considering the
instruction coding. All 64 possible op-codes were filled, with no spare
bits in the instructions. (6 bit opcode, 3 bit register designator for
each of the two source and the destination register.)
> > And indeed,
> > when I designed the routine for calculations on the zeros of the Riemann
> > zeta functions, add and subtract were the bottleneck.
>
> To force normalization?
Yup.
Thomas R. Truscott
People routinely code expressions in which the intended
and actual order of operations differ,
and the typical fix is another pair of parentheses:
x = i<<3 + j; /* oops, + binds tighter than << */
if (p != NULL
&& p[0] == 1 || p[0] == 3) /* oops, && binds tighter than ||
There are plenty of mistakes where booleans and non-booleans are mixed:
if (! flags&TASK_ACTIVE) /* oops, ! binds tighter than & */
if (flags&MASK == SUBSET) /* oops, == binds tighter than & */
Of course the order of these is defined by the C language,
but I have seen such mistakes too often by too many excellent programmers
to buy the 'victim had it coming' argument.
A compiler warning message for such hazards can avoid much grief.
Tom Truscott
Herman Rubin
>In article <1bd6b8968473ff86...@news.teranews.com> St...@NOSPAM.smart-life.net writes:
> > Dik T. Winter wrote:
> > > In an answer to me you were saying that you were not talking about the
> > > difference in hardware and software, here you require something done
> > > in hardware. For that, please, talk to the hardware producers, not the
> > > standard producers. (There is a good reason it is not done in hardware
> > > on almost all processors.)
> > Please explain. What is the "good reason"?
>Speed. If you have a fully pipelined floating point processor, where you
>can issue one operation every clock-cycle, you need a fixed length of
>the pipe (i.e. a fixed number of cycles) to complete the operation. A
>denormal is something that requires special processing at the hardware
>level, so doing that in a pipelined processor is at least problematical.
>(It can be done, but you need quite a bit of additional hardware to allow
>stalling the processor.) On many current FP processors floating point
>multiply, add and subtract are pipelined. BTW, the first processor I
>have used where a floating point multiply took essentially the same cycles
>as an add or subtract was a CDC Cyber 750.
This is floating point arithmetic. It is difficult to do
now, but the reason that a floating add or subtract takes
so long is that it is doing four operations BEFORE the
addition or subtractions take place; there is overlap,
but not much. It also has some problems after the addition
or subtraction occurs. On the other hand, for floating
point multiply, those operations go in parallel.
> > Some array computations
> > produce a predominance of abnormal results, so that forcing these into
> > software effectively slows the hardware down to software speeds.
>Been there, seen that. In some cases adding 1 to each element of a
>matrix reduced the time for the solution of a linear system by a
>considerable amount. Without that addition there were quite a lot of
>underflows resulting in denormals.
Do you not have the option to treat denormals as 0?
How much error is introduced in that type of procedure?
This also becomes less important if there is no hidden
bit, and denormals become 0 if they get too small, unless
the user requests an interrupt. Otherwise, they just get
used "as is". I can see many uses for this, especially
if combined with an "add to exponent" operation.
> > I do agree with you that this is outside of the standard. I was just
> > wondering why (apparently to me, needlessly) push it into software.
> > Wherever it is done, the functionality should be fully specified in the
> > spec.
>IEEE did quite well (I think) at that point of time. Although Herman
>Rubin disagrees (yes, I know his arguments). One problem with the
>CDC Cyber 750 was that because post normalisation was not performed
>on addition and subtraction (which would have cost at most one cycle
>in latency) made those operations actually more expensive than
>multiplication (in well coded software by a factor of two).
See the previous comments.
And indeed,
>when I designed the routine for calculations on the zeros of the Riemann
>zeta functions, add and subtract were the bottleneck.
These usually require multiple precision arithmetic, for
which IEEE already greatly increases the time.