I was mostly interested in +inf + +inf. (Positive infinity + Positive
infinity)
However perhaps that situation is now unlikely in my algorithm so my
interest in this special situation is dwindling/decreasing ;)
But maybe this situation might still pop-up and then my interest in it might
go back up ;) :)
So don't expect me to look to deeply into it for now ;) (If my interest in
it changes, I'll let you know, could be within a few days, otherwise years,
or maybe never...)
Bye,
Skybuck.
Here are some interesting properties of the IEEE floating point
standard, which is the underlying technology behind most home computers.
The behaviour of high-level programming languages tend to mirror it
closely. Let x be some positive, regular, floating point number. The
basic mathematical model is the affinely extended reals, but the
standard defines a bit more (see division by zero).
Zeros:
* There are two zeros, +0 and -0.
* -0 = -(+0)
* Floating point comparison is defined so that +0 = -0 is true.
Not-A-Number (NaN):
* Produced as an error value, for example with +0 / +0.
* Multiplying, adding, subtracting or dividing with a NaN produces a
NaN. The NaN is said to be propagated.
* All comparisons (<, >, =, !=, <=, >=) with a NaN yield false.
Infinities:
* There is infinity (oo) and minus infinity (-oo).
* -oo = -(oo)
* oo + oo = oo
* -oo + -oo = -oo
* oo + -oo = NaN
* +-x + oo = oo
* +-x + -oo = -oo
* Subtraction similarly...
Division by zero:
* +-x / +0 = +-oo
* +-x / -0 = -+oo
* +-0 / +-0 = NaN
Multiplication by infinities:
* +-x * oo = +-oo
* +-x * -oo = -+oo
* +0 * +-oo = +-0
* -0 * +-oo = -+0
Division by infinities:
* +-x / oo = +-0
* +-x / -oo = -+0
> Multiplication by infinities:
>
> * +-x * oo = +-oo
> * +-x * -oo = -+oo
> * +0 * +-oo = +-0
> * -0 * +-oo = -+0
The 0 times infinity case does not agree with IEEE754.
And note that not all comparisons with NaN yield "false": x != NaN and
NaN != x yields "true".
The IEEE 754 model is incompatible with most language's arithmetic
models, so it isn't possible (even in theory) to answer that question
for the vast majority of languages. If you really want to know the
details, you will have to start by studying various language standards
and specifications - i.e. the definitions of the languages, not
someone's summary of them.
>> > Let x be some positive, regular, floating point number. The
>> >basic mathematical model is the affinely extended reals, but the
>> >standard defines a bit more (see division by zero).
>>
>> Its signed zero handling is incompatible with that model in several
>> respects where traditional floating-point isn't.
>
>I disagree. Mathematics has one zero, the IEEE floating point has two.
>However, they negative and positive zero can not be differentiated by
>their action, expect when dividing by zero. But dividing by zero is not
>defined in the affinely extended reals. Thus, unless I am missing
>something, the IEEE floating point is a fine model for the affinely
>extended reals.
You are. IEEE 754 also has a sign test function, and that means
that -(x-y) is distinguishable from y-x, whereas -(x-y) = y-x for
in that form of mathematics.
Regards,
Nick Maclaren.
Thank you for introducing me to the affinely extended reals.
I struggled vainly to come to grips with many odd objects in theoretical
physics, until I finally grasped that, for someone with a pedestrian
mind like mine, these discussions make sense only if you are willing to
dig into the implied limiting processes in detail.
Appealing to arbitrary standards is well past silly. If you can't
define the limiting processes you are assuming in pedestrian terms, you
are either a genius or you don't know what you are talking about.
Robert.
Yes.
>I've got a copy of that standard, and am somewhat familiar with it (I
>wouldn't claim to be an expert, by any means). I do consider myself a
>fair expert on the C99 standard, and I haven't noticed any
>incompatibilities of the very fundamental nature that you're implying.
>Your citation from the Fortran standard didn't do anything to clarify
>the issue for me.
>
>I'm not saying you're wrong. As it stands, I have no idea what you're
>talking about, so I can't judge whether or not you're right. If you have
>any inclination to be less cryptic, please let me know.
Sorry - things have been a little stressful.
IEEE 754 is based on a serial assembler model, where arithmetic
is a sequence of basic operations. And both aspects are critical.
Fortran is based on an evaluation of mathematical expressions,
and is not in terms of basic operations, nor even necessarily serial.
There just isn't any match between the models.
Most languages are like Fortran, though usually less explicit, and
with less implied freedom for the implementor. If I recall from
my time on WG2, Pascal is one such. Python definitely is. I can't
remember about Ada, but I vaguely recall it being, too. As are
most of the other hundreds of languages I have looked at in my time.
The original intent of WG14 was similar (I was involved - remember
the unary plus proposal?) and the standards was deliberately left
ambiguous (NOT my choice). Later revisionists claimed that C had
always specified the execution order implied by one reading of the
syntax, but that is completely false, and is contradicted by other
wording, anyway (try 6.5#3, for a start). As I said, I am NOT going
to go into the complete mess that is FLT_EVAL_METHOD, the pragmas,
Annex F and all that - I tried to get some sanity injected during
C99's standardisation, and failed, which is the main reason I voted
against it.
C++ is similar, but different. I raised this there and WG21 kicked
it into the long grass. As I understand it, the intent is that the
order is unconstrained, but serial execution is assumed.
For Java, see Kahan's paper. Its value semantics match; its flag
ones don't.
Regards,
Nick Maclaren.
The IEE 754 defines divide by zero to be an exception. It does not
differentiate between integer and floating point.
Mathematically the result is undefined.
> For integers this seems to be the case but for floating point it could
> be different.
>
> When one thinks about it: dividing a number by a very small number gives
> a very big number.
Except when that "very small: number is zero, then the result is
undefined, by definition .
>
> So dividing a number by zero would lead to infinity, so in a way it
> makes sense to define the result of divizion by zero for floating point
> numbers to either
> be negative infinity or positive infinity this would save the need to
> use branches to catch zero cases and assign infinity.
>
No.
The result is undefined.
To be a defined result the inverse operation must generate the original
values and be unique.
so, if :
1.0/0.0 = inf, then inf * 0.0 = 1.0
and
2.0/0.0 = inf, then inf*0.0 = 2.0
That is why division by zero and calculations in the reals with inf are
undefined.
> Example:
>
> Nasty situation:
>
> if Delta <> 0 then
> begin
> Result := Number / Delta;
> end else
> begin
> if Number < 0 then
> begin
> Result := NegativeInfinity;
> end else
> begin
> Result := PositiveInfinity;
> end;
> end;
>
> Nice situation:
>
> Result := Number / Delta;
>
> ^ It would be nice if the processor would automatically fill result with
> the appriorate infinity so the possibly expensive branches up there can
> be avoided.
>
> I have at least seen one programmer claim that current X86/X64
> processors can already do this ? I am not sure and have not tried yet,
> seems a bit risky to leave these branches out.
>
> It could also be as you say and it could be compiler specific.
>
Hardware does not understand numbers, software as produced by a compiler
does. A number or a letter is no more than a series of bits in a
register to hardware that is described by the software.
I have used languages where, if you are careless, you can add the ascii
value of letters.
\Larry
So in what respect is that incompatible with IEEE 754? C specifies
not only that in (a+b)+c, a+b happens first, but that in a+b+c,
a+b happens first, so it's more prescriptive than Fortran.
It says things are otherwise unsequenced, but how can IEEE 754
require more than that? If IEEE 754 required a total order on all
floating-point operations in a program it would have been
unimplementable on any SIMD computer from ILLIAC IV onwards.
C _allows_ implementations that are incompatible with IEEE 754
(e.g. ones that use a different format, or flush denormals to zero)
but that's a long way from _being_ incompatible with IEEE 754.
>
> No, it didn't, BECAUSE IT'S NOT PORTABLE.
You scare me, Nick.
That's only fair, because, to the extent that you pay attention to
anything I say, I probably scare you.
What you are really saying is that, in a code where mysterious and
unreliable things are happening, you can't blindly transfer the code
from one ad hoc hardware and software standard to another. Listen
very carefully:
THAT'S A GOOD THING.
It may be the last thing standing between us and careerists who don't
have a clue "validating" their results by transferring a meaningless
calculation from one machine to another with no clue that something
totally crazy is going on.
The idea that the universe would be a safer place if only people would
listen to you doesn't even pass the laugh test.
Robert.
Nick does have a good point though, notwithstanding the last couple of
posts in which the authors asserted that C (at least) now *does* specify
a total order of operations within expressions (even ones that appear
ambiguous because of expectations of associativity). I didn't realize
that, and I'm still not sure that I believe them.
Nick's point (I believe) is that floating point arithmetic is fraught,
because it is fairly fundamentally unlike the mathematics that expression-
based languages appear to be offering. In maths and exact computer
arithmetic (eg integer), addition is associative and a bunch of other
nice rules for "simplification" apply. In floating point they don't,
because every floating point multiply (a * b), for example, is really
something like round(_g_current_FPU_rounding_mode, multiply(a,b)), and
every addition or subtraction is even more complicated, with a bunch of
normalizations thrown in for good measure. Round is, naturally, a lossy
operation. Throw in exceptions that can be raised in a bunch of ways and
you wind up with something that can only really be reasoned about as a
discrete sequence of assembly-language-like operations that both produce
results and mutate and depend on the "system state" in obscure and
platform-dependent ways. I haven't even mentioned an IEEE standard here:
it ought to be possible to define a stateless floating point method,
where rounding modes were explicit in the instructions and exceptions
were either defined away as in-band values or made of precice exceptions,
but I've never heard of it being done.
Clearly you can still have parallelism between independent operations,
but you can't manipulate floating point expressions that look like maths
as though they were maths, at least not without giving up being able to
reason about the answer.
Which I think just emphasises your point, too: people doing floating
point need to have a clue...
Cheers,
--
Andrew
Actually, no. It's tricky, but not THAT tricky. My real point is
that the C99 and other modern 'improvements' make it appear to the
naive as if it is more portable and reliable, but actually introduce
ten times as many portability and reliability problems as they solve.
Almost everyone who has actually tried it has discovered that, but
most people code for one system, run a couple of sets of well-behaved
data, and then claim that their program is portable and robust and
any failures must be the fault of the compiler!
>Clearly you can still have parallelism between independent operations,
>but you can't manipulate floating point expressions that look like maths
>as though they were maths, at least not without giving up being able to
>reason about the answer.
Yes, you can. Look at most of the classic numerical analysis books.
You have to do it rather differently, and very, very few modern
'computer scientists' would know how to start. Wilkinson and Reinsch
"The Algebraic Eigenvalue Problem" is one example of such reasoning
that I have used in the past.
>Which I think just emphasises your point, too: people doing floating
>point need to have a clue...
Oh, THERE, I am in 100% agreement. When I teach it, I tell people
that the key is to think floating-point, not mathematical real,
but that 50+ years of Fortran experience shows that it's not as
hard as all that. Inter alia, we used to write code that was both
reliable and portable across a range of arithmetics that most
people nowadays cannot imagine :-)
There ARE some books on how to write robust, portable floating-point
code, but all are very old-fashioned. And there were a large number
of experiments showing both that the exact details of the arithmetic
didn't matter much (though directed versus nearest rounding did),
and that relying on the the exact details led to the code being LESS
robust, rather than more.
Of courses, that's now all regarded as heresy ....
Regards,
Nick Maclaren.
Start thinking about IEEE floating point number as a probable number -
number You can't measure exactly, there is always some immeasurable error.
In this case -0 means: number so small so we can assume it is zero, but
we certain it is negative. +0 is same, but positive.
So You never can say it is zero, because You never can be certain it is.
You can only compare to zero up to some precision.
--
Arivald
That was what I was saying, with the further provision that
a + b + (c + d) is equivalent to ((a+b) + (c+d)). I.e. in C,
parentheses are a lexical feature and don't act as a further
constraint on the parse. The compiler has no more leeway
to reassociate a+b+c than it does to reassociate (a+b)+c.
The point is that there is no difference in C between a+b+c
and (a+b)+c. Any order implied by the latter is also implied
by the former. The point is that operator precedence in C
uniquely defines the "evaluation tree". I don't believe anyone
has asserted that there is a total order of all computations
within the tree, i.e. between the LHS and RHS of a binary
operator.
> Nick's point (I believe) is that floating point arithmetic is fraught
We all know it that floating-point arithmetic is not the same as
real arithmetic. Nick appears to be making an abstruse point,
about the relationship between ISO C and IEEE 754, without
being able to explain what that point is.
I'm not sure exactly what you mean by the statement that they "don't act
as a further constraint on the parse". To clarify, do you agree that
each of the following expressions is parsed into a different evaluation
tree?
a + b + c + d
a + (b + c) + d
a + b + (c + d)
a + (b + (c + d)
In particular, the conditions that make each of those expressions
produce an overflow may be quite different.
> ... The compiler has no more leeway
> to reassociate a+b+c than it does to reassociate (a+b)+c.
I'll agree with that
so, if :
1.0/0.0 = inf, then inf * 0.0 = 1.0
and
2.0/0.0 = inf, then inf*0.0 = 2.0
That is why division by zero and calculations in the reals with inf are
undefined.
"
Feels as if math has a little short coming here.
Here is an idea:
How about:
Infinity * 0.0 = Anything
Bye,
Skybuck.
Yes. Also, the third one is the only "balanced" tree where
two of the adds can happen at the same time.
> In particular, the conditions that make each of those expressions
> produce an overflow may be quite different.
And in floating-point the result may be numerically different...
<snip>
> I haven't even mentioned an IEEE standard here:
> it ought to be possible to define a stateless floating point method,
> where rounding modes were explicit in the instructions and exceptions
> were either defined away as in-band values or made of precice exceptions,
> but I've never heard of it being done.
>
<snip>
> Which I think just emphasises your point, too: people doing floating
> point need to have a clue...
>
You make some good points that are worth remembering.
I'm not sure, though, that a stateless and platform-independent
definition of floating point arithmetic is either necessary or
desirable or even possible within the bounds of practical utility.
If the floating point details that people spend so much time on here
make all that much difference, there is probably something flaky about
either the algorithm or the code or both. Given that at least some of
the people who spend so much time talking about this issue here
probably already understand the claim I just made, I don't understand
why this alchemist's pursuit of turning lead into gold continues.
Given the choice between hammering it into people that floating point
arithmetic is not for the naive or even necessarily replicable from
one situation to another, and inventing an arbitrary standard that
makes it at least make the same mistake every time, I'd prefer the
current chaos to artificial predictability.
I could and would and do make the same objection about the
correspondence to mathematics that you make for floating point
arithmetic to every code that purports to represent mathematics where
a differential operator appears in the calculation. I know how to
make that correspondence precise and completely unarbitrary, but I am
repeatedly told that doing so is simply too expensive.
I conclude, as I believe I have said before, that this continuing
discussion is an instance of Nazrudin's lost key: looking where there
is light. Given that many if not most codes that use floating point
arithmetic rely on absurd mathematics (repeatedly differentiating
derivatives that are piecewise continuous even only in theory), I just
can't understand all the fuss about the last bit (rounding), which
idealized properties floating point arithmetic does or does not
possess, or what arbitrary thing the hardware does when the algorithm
leads to a nonsensical or ambiguous result.
Robert.