# Numerical accuracy/precision - this is a bug or a feature?

175 views

### slawek

Jul 4, 2011, 6:47:53 AM7/4/11
to
Let Mathematica (6.x, 7.x) compute quite a simple product

In:= N[N[2.0, 20] * N[Sqrt , 20], 20]

Out= 2.82843

This is a bug.

Why?

Now we analyze it in details:

1. N[2.0,20] should give 2 with accuracy/precision/whatsever about 20
decimal digits, i.e. 2.00000000000000000000

2. Sqrt should give... Sqrt

3. N[Sqrt] should give 1.4142135623730950488 (this is copy-paste from
Mathematica output to N[Sqrt] )

4. The product 2.00000000000000000000 * 1.4142135623730950488 is
2.8284271247461900976 (again copy-paste)

5. BUT THE RESULT OF N[2.0, 20] * N[Sqrt , 20] "truncated to 20 digits"
is Out= 2.82843

Where are missing digits?!

slawek

### Kevin J. McCann

Jul 4, 2011, 7:14:46 AM7/4/11
to
The answer to this puzzle is that the N[2.0,20] is 2.0, not
2.00000000... Try N[2,20] and all is well. I think that when you put 2.0
in you have already limited yourself to machine precision, and N[2.0,20]
is then just machine accuracy.

Kevin

### Kevin J. McCann

Jul 4, 2011, 7:18:14 AM7/4/11
to
BTW, you can skip the outer N[]:

N[2,20]N[Sqrt,20]

produces:

2.8284271247461900976

Kevin

On 7/4/2011 6:47 AM, slawek wrote:

### slawek

Jul 5, 2011, 5:17:49 AM7/5/11
to

Uďż˝ytkownik "Kevin J. McCann" <k...@KevinMcCann.com> napisaďż˝ w wiadomoďż˝ci grup
dyskusyjnych:ius7b6\$30t\$1...@smc.vnet.net...

> The answer to this puzzle is that the N[2.0,20] is 2.0, not
> 2.00000000... Try N[2,20] and all is well. I think that when you put 2.0
> in you have already limited yourself to machine precision, and N[2.0,20]
> is then just machine accuracy.

It is still a-bug-and-a-feature.
And this bug make Mathematica nearly useless in numerical computations. "MS
Windows Calculator" is much more reliable!

The number of written digits IS NEITHER the precision NOR the accuracy.
Mathematica treat 2.0 as a 2.0+-0.1, but it is not the proper way to handle
numbers.

I know, that it is common mistake to treat 2.0 as "not an integer number"
and/or "exact" number, but 2.0 is an integer number AND also it is a
rational number AND also a real number AND also a complex number. And 2.0 is
simply 1+1+ 0/10 . Therefore, as you see, there is no "roudning", "limited
precision", "error" or "uncertinaity". It is only a matter of a notation of
decimal fractions. And decimal fractions are exact. Any "tolerance" is not
indicated in any way by this notation. Thus it is a bug. Nasty, big, fat bug
in the core of Mathematica.

Even from "CS view" 2.0 is translated to IEEE representation with 56-bits of
the mantisa. Nobody declare float x = 2.0000000000 to iniject the float
point two into a code.

slawek

### Richard Fateman

Jul 5, 2011, 5:15:47 AM7/5/11
to
In brief: yes to both.

On 7/4/2011 4:18 AM, Kevin J. McCann wrote:
...

>>
>> 1. N[2.0,20] should give 2 with accuracy/precision/whatsever about 20
>> decimal digits, i.e. 2.00000000000000000000
>

WRI will defend this as a feature.

You thought that the semantics of N[] were the same as SetPrecision
e.g.

N[SetPrecision[2.0, 20]*N[Sqrt, 20], 20]

works as you expected.

So from your perspective, and from the perspective of anyone else who
thinks along the same lines, it is a bug. I would prefer to call it
a design error.

2.0 (indeed, any floating point number) is a perfectly respectable way
of denoting a number that can be expressed in higher precision by adding
binary zeros. WRI doesn't agree.

RJF

### slawek

Jul 6, 2011, 5:34:31 AM7/6/11
to

U=BFytkownik "Richard Fateman" <fat...@cs.berkeley.edu> napisa=B3 w wiadomo=B6ci
grup dyskusyjnych:iuuko3\$eqi\$1...@smc.vnet.net...

> 2.0 (indeed, any floating point number) is a perfectly respectable way
> of denoting a number that can be expressed in higher precision by adding
> binary zeros. WRI doesn't agree.

Mathematica is very simple to use. Of course, it is the entire programming
language, advanced concepts, etc. However it seems that in order to
calculate 2 Sqrt is need no special knowledge nor special care.

Nevertheless, it appears that although the language itself does not enforce
this, calculations require very formal definitions of all variables.

slawek

### Kevin J. McCann

Jul 6, 2011, 5:35:02 AM7/6/11
to
Well, I certainly disagree with your assessment. The rules regarding
exact and machine precision numbers are straightforward and stated in
the documentation. But if you feel that the MS Windows Calculator is a
better option, by all means use that. It is free and easy to use.

2.0 is not an integer nor a rational, it is machine precision. On the
other hand 2 is an integer and exact.

As far as whether or not 2.0 should be considered as 2.0+-0.1 or as
2.000..., 2.0 is a machine precision number, which means
2.000000000000000 or as close as you can get in about 16 places. Telling
Mathematica that you want 20 places is somewhat like asking Mathematica to give more
accuracy *after* the calculation is finished. At any rate my friend
David Park reminded me that you can specify your 2.0 to 20 places with
the following:

2.0`20
Precision[%]

20.

Kevin

On 7/5/2011 5:17 AM, slawek wrote:
> U�ytkownik "Kevin J. McCann"<k...@KevinMcCann.com> napisa� w wiadomo�ci grup

### Oleksandr Rasputinov

Jul 6, 2011, 5:39:07 AM7/6/11
to
On Tue, 05 Jul 2011 10:17:49 +0100, slawek <sla...@host.pl> wrote:

> U=BFytkownik "Kevin J. McCann" <k...@KevinMcCann.com> napisa=B3 w wiadomo=B6ci

> grup dyskusyjnych:ius7b6\$30t\$1...@smc.vnet.net...

>> The answer to this puzzle is that the N[2.0,20] is 2.0, not
>> 2.00000000... Try N[2,20] and all is well. I think that when you put 2.0
>> in you have already limited yourself to machine precision, and N[2.0,20]
>> is then just machine accuracy.
>

> It is still a-bug-and-a-feature.
> And this bug make Mathematica nearly useless in numerical computations..

> "MS Windows Calculator" is much more reliable!
>

> The number of written digits IS NEITHER the precision NOR the accuracy..

> Mathematica treat 2.0 as a 2.0+-0.1, but it is not the proper way to
> handle numbers.
>

In fact:

In :=
Interval[2.0] // InputForm

Out//InputForm =
Interval[{1.9999999999999998, 2.0000000000000004}]

While:

In :=
Interval[2.0``1] // InputForm

Out//InputForm =
Interval[{1.875`1.2730012720637347, 2.125`1.327358934386329}]

Precision (which, as defined by Mathematica, means relative uncertainty)
and accuracy (absolute uncertainty) are expressed as annotations after the
number. In the special case of a number with a decimal point and no
annotation, the number is taken to be a machine precision real. I agree
that these definitions and the notational convention chosen by Mathematica
are strange. However, there is nothing "improper" about it as a choice of
formalism--at least, this is no worse a design choice than for Mathematica
to have standardized on decimal notation for input and presentation of
numerical values rather than binary as it uses internally.

The purpose of this approach is as a crude but often adequate
approximation to interval arithmetic, whereby these (approximations of)
errors are carried through arithmetic operations using first-order
algebraic methods. When functions (such as N) that pay attention to
Precision and Accuracy (by Mathematica's definitions) see them decreasing,
they increase the working precision so as to avoid numerical instability
being expressed in the final result. This is by no means intended to be
rigorous; it is merely a heuristic, but one that comes at little cost and
works in many cases. Of course, if a user's own code treats this
approximation as somehow sacrosanct and ignores the precision adjustments
necessary during the calculation while taking the final answer as correct,
it is more likely that the approximation will have fallen apart somewhere
down the line.

If you don't like significance arithmetic, you have (at least) two other
options at hand: either work in fixed precision (\$MinPrecision
\$MaxPrecision = prec) or use interval arithmetic. These have their own
drawbacks, of course (most notably that Mathematica tacitly assumes all
intervals are uncorrelated), but your hand isn't forced either way and you
may even use all three methods simultaneously if you wish. Alternatively
you may program your own, more accurate algebraic or Monte Carlo error
propagation methods if you prefer.

### DrMajorBob

Jul 6, 2011, 5:41:10 AM7/6/11
to
"Mathematica treat 2.0 as a 2.0+-0.1"

No, it treats it as 2 with machine precision -- almost 16 digits at my
machine -- not two digits, as you're describing.

Bobby

On Tue, 05 Jul 2011 04:12:05 -0500, slawek <sla...@host.pl> wrote:

>
> U=BFytkownik "Kevin J. McCann" <k...@KevinMcCann.com> napisa=B3 w wiadomo=B6ci
> grup
> dyskusyjnych:ius7b6\$30t\$1...@smc.vnet.net...
>> The answer to this puzzle is that the N[2.0,20] is 2.0, not
>> 2.00000000... Try N[2,20] and all is well. I think that when you put 2.0
>> in you have already limited yourself to machine precision, and N[2.0,20]
>> is then just machine accuracy.
>
> It is still a-bug-and-a-feature.
> And this bug make Mathematica nearly useless in numerical computations.

> "MS
> Windows Calculator" is much more reliable!
>
>
> The number of written digits IS NEITHER the precision NOR the accuracy.

> Mathematica treat 2.0 as a 2.0+-0.1, but it is not the proper way to
> handle
> numbers.
>

> I know, that it is common mistake to treat 2.0 as "not an integer number"
> and/or "exact" number, but 2.0 is an integer number AND also it is a
> rational number AND also a real number AND also a complex number. And
> 2.0 is
> simply 1+1+ 0/10 . Therefore, as you see, there is no "roudning",
> "limited
> precision", "error" or "uncertinaity". It is only a matter of a notation
> of
> decimal fractions. And decimal fractions are exact. Any "tolerance" is
> not
> indicated in any way by this notation. Thus it is a bug. Nasty, big, fat
> bug
> in the core of Mathematica.
>
> Even from "CS view" 2.0 is translated to IEEE representation with
> 56-bits of

> the mantisa. Nobody declare float x == 2.0000000000 to iniject the float

> point two into a code.
>
> slawek
>
>

### Richard Fateman

Jul 7, 2011, 7:33:49 AM7/7/11
to
>....

>

This is an excellent summary of Mathematica's approach to arithmetic on
numbers. Unfortunately many people come to use Mathematica with their
own notions of numbers, accuracy, precision, and equality. These words
are redefined in a non-standard way in Mathematica, leading to
unfortunate situations sometimes. "unexplainable" behavior. confusion.
Or worse, erroneous results silently delivered and accepted as true by a
user, who "knows" about precision, accuracy, floating-point arithmetic, etc.

WRI argues that this is a winning proposition. Perhaps Wolfram still
believes that someday all the world will use Mathematica for all
programming purposes and everyone will accept his definition of terms
like Precision and Accuracy, and that (see separate thread on how to
write a mathematical paper) it will all be natural and consistent.
(or that people who want to hold to the standard usage will be forced to
use something like SetGlobalPrecision[prec_]:=
\$MaxPrecision=MinPrecision=prec.
I believe this is routinely used by people who find Mathematica's
purportedly "user-friendly" amateurish error control to be hazardous.
)

.........

'When I use a word,' Humpty Dumpty said, in rather a scornful tone, 'it
means just what I choose it to mean =97 neither more nor less.'

'The question is,' said Alice, 'whether you can make words mean so many
different things.'

'The question is,' said Humpty Dumpty, 'which is to be master =97 that's
all.'

### Daniel Lichtblau

Jul 7, 2011, 7:34:50 AM7/7/11
to
On Jul 5, 4:17 am, "slawek" <sla...@host.pl> wrote:
> U ytkownik "Kevin J. McCann" <k...@KevinMcCann.com> napisa w wiadomo ci grup
> dyskusyjnych:ius7b6\$30...@smc.vnet.net...

>
> > The answer to this puzzle is that the N[2.0,20] is 2.0, not
> > 2.00000000... Try N[2,20] and all is well. I think that when you put 2.0
> > in you have already limited yourself to machine precision, and N[2.0,20]
> > is then just machine accuracy.
>
> It is still a-bug-and-a-feature.
> And this bug make Mathematica nearly useless in numerical computations. "MS
> Windows Calculator" is much more reliable!

I have used Mathematica for numerical computation for nearly 20 years
and will say that on most days I do not find it to be useless. I am
not familiar with MS Windows Calculator so cannot comment on any
comparison between the two.

> The number of written digits IS NEITHER the precision NOR the accuracy.
> Mathematica treat 2.0 as a 2.0+-0.1, but it is not the proper way to handle
> numbers.

As written this is not correct. In Mathematica if you input 2.0 then
it is treated as a machine double, which on all platforms today has
around 16 decimal places. In any case it is 2. +-\$MachineEpsilon.

The formatting of machine floats attempts (I believe always
successfully) to show the decimal approximating to the underlying
binary value (that is, within half an error-in-the-last place) that
has the least number of digits. So for example 2.1 will be printed as
2.1, even though the underlying binary representation is not
identically equal to 21/10.

> I know, that it is common mistake to treat 2.0 as "not an integer number"
> and/or "exact" number, but 2.0 is an integer number AND also it is a
> rational number AND also a real number AND also a complex number.

This is a matter to be determined by the language semantics (for any
language, not just Mathematica). A "design decision", if you will (and
as another response states). You may not like the design, and you may
have reasons for not liking it, or for preferring other programs. That
does not make this design intrinsically wrong.

> And 2.0 is
> simply 1+1+ 0/10 . Therefore, as you see, there is no "roudning", "limited
> precision", "error" or "uncertinaity". It is only a matter of a notation of
> decimal fractions. And decimal fractions are exact.

This is only true if the underlying representation is decimal. In some
programs that might be the case. In Mathematica underlying
representations are binary based. So a decimal such as 2.1 will not be
2 + 1/10 because that cannot be supported by finite binary
representation.

A different tack would be to translate, at parse time say, decimal
input such as 2.1 to exact input such as 21/10. I will spare anyone
who has read this far from all the reasons why that would be an
incredibly bad idea. Suffice it to say that if one wants to work with
exactly 21/10 then one should use 21/10, and not an approximate input
variant thereof.

> Any "tolerance" is not
> indicated in any way by this notation. Thus it is a bug. Nasty, big, fat bug
> in the core of Mathematica.
>
> Even from "CS view" 2.0 is translated to IEEE representation with 56-bits of
> the mantisa. Nobody declare float x = 2.0000000000 to iniject the float
> point two into a code.
>
> slawek

Daniel Lichtblau
Wolfram Research

### Daniel Lichtblau

Jul 7, 2011, 7:35:52 AM7/7/11
to

I don't think this holds up for all floats in the sense you seem to
indicate. Take 2.1 for example. We can (and will) use SetPrecision to
pad with binary zeroes, just as you use it above on 2.0.

In:= N[SetPrecision[2.1, 20]*N[Sqrt, 20], 20]

Out= 2.9698484809834997281

But this is not an accurate representation of 21/10 * Sqrt to 20
places.

In:= N[21/10*Sqrt, 20]
Out= 2.9698484809834996025

They are off by arould a machine epsilon. No surprise here, I think.

In:= % - %%
Out= -1.256*10^-16

I also think that this sort of numeric discrepancy is inevitable* in
any program that uses decimal input and binary representation.

Daniel Lichtblau
Wolfram Research

*Perhaps more inevitable than these threads. Now that's pretty
inevitable.

### slawek

Jul 7, 2011, 7:40:29 AM7/7/11
to

U=BFytkownik "Kevin J. McCann" <Kevin....@umbc.edu> napisa=B3 w wiadomo=B6ci
grup dyskusyjnych:iv1a86\$sji\$1...@smc.vnet.net...

> 2.0 is not an integer nor a rational, it is machine precision. On the
> other hand 2 is an integer and exact.

The decimal fraction 2.0 is in mathematics (the science) exactly the same as
the sum 2 + 0/10

The convention that 2.0 is less accurate than 2.00 is applied ONLY in
Mathematica (the computer program).

### Noqsi

Jul 8, 2011, 4:51:53 AM7/8/11
to
On Jul 7, 5:40 am, "slawek" <sla...@host.pl> wrote:

> The convention that 2.0 is less accurate than 2.00 is applied ONLY in
> Mathematica (the computer program).

Not true. This is a long-standing convention in experimental science.

### Richard Fateman

Jul 8, 2011, 4:53:25 AM7/8/11
to
On 7/7/2011 4:35 AM, Daniel Lichtblau wrote:
> On Jul 5, 4:15 am, Richard Fateman<fate...@cs.berkeley.edu> wrote:
>> In brief: yes to both.
>>
>> On 7/4/2011 4:18 AM, Kevin J. McCann wrote:
>> ...
>>
>>
>>
>>>> 1. N[2.0,20] should give 2 with accuracy/precision/whatsever about 20
>>>> decimal digits, i.e. 2.00000000000000000000
>>
>> WRI will defend this as a feature.
>>
>> You thought that the semantics of N[] were the same as SetPrecision
>> e.g.
>>
>> N[SetPrecision[2.0, 20]*N[Sqrt, 20], 20]
>>
>> works as you expected.
>>
>> So from your perspective, and from the perspective of anyone else who
>> thinks along the same lines, it is a bug. I would prefer to call it
>> a design error.
>>
>> 2.0 (indeed, any floating point number) is a perfectly respectable way
>> of denoting a number that can be expressed in higher precision by adding
>> binary zeros. WRI doesn't agree.
>>
>> RJF
>
> I don't think this holds up for all floats in the sense you seem to
> indicate. Take 2.1 for example.

My contention is that 2.1, written that way, is a perfectly respectable

way of denoting a number that can be expressed in higher precision by

adding binary zeros. 2.1 is a double-float in most systems today.
What is that number? In decimal, to higher precision, it looks like
2.10000000000000008881784197001....

A program that decodes a float into its component fraction, exponent,
and sign gives this information for the fraction:
4728779608739021 or in binary
10000110011001100110011001100110011001100110011001101

That is the number to which we add binary zeros.

Let us call this number P.

>We can (and will) use SetPrecision to
> pad with binary zeroes, just as you use it above on 2.0.
>
> In:= N[SetPrecision[2.1, 20]*N[Sqrt, 20], 20]
>
> Out= 2.9698484809834997281
>
> But this is not an accurate representation of 21/10 * Sqrt to 20
> places.

Just because you are doing arithmetic on numbers represented to 20
digits does not mean that the result of the arithmetic will be right to
20 digits, however, in the single operation of multiplication, you
should not lose any digits. What's going on here?

You simply computed two different quantities. Let S= sqrt(2) to 20
decimal places. In line 606 you multiplied P*S and rounded it to 20
digits [ actually you did something to the binary reps, but it doesn't
matter here].
In line 607, below, you multiplied 21/10 by S. Since P is not equal
to 21/10, why should the results be the same? Oh, Mathematica thinks
that P ==21/10, but that is merely the terribly broken notion of
equality that Mathematica uses. We used to call that "close enough for
government work". But it is terribly broken because == fails to
satisfy the axioms of "equivalence relations" such as a==b and b==c
implies a==c.

>
> In:= N[21/10*Sqrt, 20]
> Out= 2.9698484809834996025
>
> They are off by arould a machine epsilon. No surprise here, I think.
>
> In:= % - %%
> Out= -1.256*10^-16
>
> I also think that this sort of numeric discrepancy is inevitable* in
> any program that uses decimal input and binary representation.

Absent any interior calculations, I would tend to agree. But there is
something else going on. I think that Mathematica has embedded in it
several design decisions that elevate the (typical)
one-half-unit-in-last-place error in binary-to-decimal and
decimal-to-binary conversion that happens in input or output, into a
global slush bucket that infects and propagates into all interior
computations with "software floats".

>
> Daniel Lichtblau
> Wolfram Research
>
> *Perhaps more inevitable than these threads. Now that's pretty
> inevitable.
>

maybe so.

### Oleksandr Rasputinov

Jul 8, 2011, 5:04:08 AM7/8/11
to

Apart from the curious definitions given to Precision and Accuracy (one
imagines ApproximateRelativeError and ApproximateAbsoluteError were
considered too verbose), I do not think Mathematica's way of doing things
is particularly arbitrary or confusing in the broader context of
multiprecision arithmetic. Mathematically, finite-precision numbers
represent distributions over finite fields, and they therefore possess
quantized upper and lower bounds, as well as quantized expectation values.
Strictly, then, any two such distributions cannot be said to be equal if
they represent numbers of different precision: they are then distributions
over two entirely different fields, irrespective of whether their
expectation values may be equal.

However, this definition is not very useful numerically and we are usually
satisfied in practice that two finite-precision numbers equal if the
expectation values are equal within quantization error. Note that the
question of true equality for numbers of different precisions, i.e. the
means of the distributions being equal, is impossible to resolve in
general given that the means (which represent the exact values) are not
available to us. Heuristically, the mean should be close, in a relative
sense, to the expectation value, hence the tolerance employed by Equal;
the exact magnitude of this tolerance may perhaps be a matter for debate
but either way it is set using Internal`\$EqualTolerance, which takes a
machine real value indicating the number of decimal digits' tolerance that
should be applied, i.e. Log/Log times the number of least
significant bits one wishes to ignore. This setting has been discussed in
this forum at least once in the past: see e.g.
<http://forums.wolfram.com/mathgroup/archive/2009/Dec/msg00013.html>.

Note that if one wishes to be more rigorous when determining equality,
SameQ operates in a similar manner to Equal for numeric comparands, except
that its tolerance is 1 (binary) ulp. This is also adjustable, via
Internal`\$SameQTolerance.

In regard to erroneous results: undoubtedly it is a possibility. However,
one would expect that an approximate first order method for dealing with
error propagation should at least be better in the majority of cases than
a zeroth-order method such as working in fixed precision. As stated
previously, if one desires more accurate approximations then one is in any
case free to implement them, although given the above it should be clear
that all that is generally possible within the domain of finite-precision
numbers is a reasonable approximation unless other information is
available from which to make stronger deductions. I will also note that
none of the example "problems" in this present topic are anything directly
to do with significance arithmetic; they instead represent a combination
of confusion due to Mathematica's (admittedly confusing) choice of
notation, combined with an apparent misunderstanding of concepts related
to multiprecision arithmetic in general.

### Bill Rowe

Jul 8, 2011, 5:06:40 AM7/8/11
to
On 7/7/11 at 7:31 AM, sla...@host.pl (slawek) wrote:

>U=BFytkownik "Kevin J. McCann" <Kevin....@umbc.edu> napisa=B3 w

>>2.0 is not an integer nor a rational, it is machine precision. On
>>the other hand 2 is an integer and exact.

>The decimal fraction 2.0 is in mathematics (the science) exactly the
>same as the sum 2 + 0/10

True enough.

>The convention that 2.0 is less accurate than 2.00 is applied ONLY
>in Mathematica (the computer program).

Not true.

First

In:= Precision[2.0] === Precision[2.00]

Out= True

In:= Accuracy[2.0] === Accuracy[2.00]

Out= True

But it is true in Mathematica 2 + 0/10 is different than 2.0.
But the fact these are different is not at all unique to
Mathematica. Any application that understands floating point
values and integer values that is compliant with IEEE standards
will report 2 + 0/10 is different than 2.0

### DrMajorBob

Jul 8, 2011, 5:09:44 AM7/8/11
to
This happens in EVERY language that converts decimal (real) input to
binary storage.

Binary storage is not mathematically required, but the result is MUCH
faster code. Perhaps, at some point, we'll abandon this trade-off when
computers are so fast that the gain in speed isn't worth the confusion.
But that day isn't here, yet.

(Well. It is for ME, but not for a lot of other, very impatient people. If
they can get a result in a second with decimal arithmetic, they'd rather
have it in a microsecond with binary arithmetic, no matter WHAT the cost
in confusion factors.)

Bobby

On Thu, 07 Jul 2011 06:31:49 -0500, slawek <sla...@host.pl> wrote:

>
> U=BFytkownik "Kevin J. McCann" <Kevin....@umbc.edu> napisa=B3 w
> grup dyskusyjnych:iv1a86\$sji\$1...@smc.vnet.net...
>> 2.0 is not an integer nor a rational, it is machine precision. On the
>> other hand 2 is an integer and exact.
>
> The decimal fraction 2.0 is in mathematics (the science) exactly the
> same as
> the sum 2 + 0/10
>

> The convention that 2.0 is less accurate than 2.00 is applied ONLY in
> Mathematica (the computer program).
>
>
>
>

### Richard Fateman

Jul 9, 2011, 7:34:48 AM7/9/11
to
On 7/8/2011 2:04 AM, Oleksandr Rasputinov wrote:

.. snip...

>
> Apart from the curious definitions given to Precision and Accuracy (one
> imagines ApproximateRelativeError and ApproximateAbsoluteError were
> considered too verbose), I do not think Mathematica's way of doing things
> is particularly arbitrary or confusing in the broader context of
> multiprecision arithmetic. Mathematically, finite-precision numbers
> represent distributions over finite fields, and they therefore possess
> quantized upper and lower bounds, as well as quantized expectation values.

This is one view, but one that is not especially useful computationally.
A much more useful view is that a finite-precision number is simply a
single value. That's all. The number 3 has no fuzz around it. The
double-float number 0.1d0 has no fuzz around it either. It is not
exactly 1/10, but

7205759403792794 X 2 ^(-56).

exactly. It is a finite precision number because the fraction part, here
7205759403792794, is limited to a fixed number of bits. There is, I
repeat, nothing that makes it a distribution.
Another way of writing it is

3602879701896397/36028797018963968

The advantages of this view:
(1) arithmetic is well-defined and executable on a computer
(2) should you choose to implement some kind of distribution arithmetic
-- intervals, Gaussian bell curves, significance arithmetic,
fuzzy-set-theoretic whatever, you can do so knowing that the underlying
implementation of arithmetic is supportive of any model that can be
reduced to ordinary mathematics.

> Strictly, then, any two such distributions cannot be said to be equal if
> they represent numbers of different precision: they are then distributions
> over two entirely different fields, irrespective of whether their
> expectation values may be equal.

This is something you are free to implement. I personally object to an
arithmetic system and a notion of equality that does not support the
fundamental properties of equivalence relations. I much prefer that if
a==b, then a-b==0. This of course fails, in Mathematica.

>
> However, this definition is not very useful numerically and we are usually
> satisfied in practice that two finite-precision numbers equal if the
> expectation values are equal within quantization error.

No, I disagree. It used to be that programmers were taught that one
should never (or almost never) compare floating point numbers for
equality. You can easily ask if two numbers are relatively or
absolutely close. That is not the same as being equal.

Note that the
> question of true equality for numbers of different precisions, i.e. the
> means of the distributions being equal, is impossible to resolve in
> general given that the means (which represent the exact values) are not
> available to us.

The fault here is that you view numbers as distributions. I have no
problem telling if a number 0.1d0 is equal to
3602879701896397/36028797018963968 .

It is. That is its value. It is equal to all other objects with the
same exact value. That's what equal means.

Is it equal to 0.099999999? No, but it is relatively close by some measure.

Heuristically, the mean should be close, in a relative
> sense, to the expectation value, hence the tolerance employed by Equal;

You may choose to believe this, but it is mathematical nonsense.

> the exact magnitude of this tolerance may perhaps be a matter for debate
> but either way it is set using Internal`\$EqualTolerance, which takes a
> machine real value indicating the number of decimal digits' tolerance that
> should be applied, i.e. Log/Log times the number of least
> significant bits one wishes to ignore. This setting has been discussed in
> this forum at least once in the past: see e.g.
> <http://forums.wolfram.com/mathgroup/archive/2009/Dec/msg00013.html>.

I suspect there are more insightful discussions in the archives.

>
> Note that if one wishes to be more rigorous when determining equality,
> SameQ operates in a similar manner to Equal for numeric comparands, except
> that its tolerance is 1 (binary) ulp. This is also adjustable, via
> Internal`\$SameQTolerance.

It is nice that one can try to scrape out all the garbage of the
arithmetic by setting internal flags, but the way it is set up in
Mathematica, the ordinary user with default system settings is exposed
to a really defective arithmetic system.

>
> In regard to erroneous results: undoubtedly it is a possibility. However,
> one would expect that an approximate first order method for dealing with
> error propagation should at least be better in the majority of cases than
> a zeroth-order method such as working in fixed precision.

You might think that, but the choice is not so clear.
If you set \$MinPrecision=\$MaxPrecision
then
(i = 1.100000000000000000; Do[(i = 2*i - i; Print[i]), {4}])
gives Overflow[] 4 times.

In the default setting, \$MinPRecision is 0.
then
(i = 1.100000000000000000; Do[(i = 2*i - i; Print[i]), {50}]

sets i to decreasingly precise values, ending in 0., 0., 0X10^1, ... 0X10^5

(try it.. it is only one line!)

What is the naive user to do?

If you think this example is a good result, imagine what would happen to
a naive user (most are) who has a more elaborate program (most are)
which internally produces an answer " 0. " which is entirely bogus?

As stated
> previously, if one desires more accurate approximations then one is in any
> case free to implement them, although given the above it should be clear
> that all that is generally possible within the domain of finite-precision
> numbers is a reasonable approximation unless other information is
> available from which to make stronger deductions.

Quite the opposite. All numbers in a computer calculation should be
considered exact unless other information is available from which to
make WEAKER deductions. Then the kind of deductions (based, perhaps, on
knowledge of physical measurements or uncertainties) can be incorporated
in the calculation.

I will also note that
> none of the example "problems" in this present topic are anything directly
> to do with significance arithmetic; they instead represent a combination
> of confusion due to Mathematica's (admittedly confusing) choice of
> notation,

I think that is right.

combined with an apparent misunderstanding of concepts related
> to multiprecision arithmetic in general.

I think that the concepts of multiprecision arithmetic as implemented
in Mathematica are different from other implementations in general.
Bringing such outside knowledge to bear on an utterance in Mathematica
is hazardous.

### Oleksandr Rasputinov

Jul 9, 2011, 7:37:23 AM7/9/11
to
On Fri, 08 Jul 2011 09:51:53 +0100, Noqsi <noqsiae...@gmail.com> wrote:

> On Jul 7, 5:40 am, "slawek" <sla...@host.pl> wrote:
>

>> The convention that 2.0 is less accurate than 2.00 is applied ONLY in
>> Mathematica (the computer program).
>

> Not true. This is a long-standing convention in experimental science.
>

Unfortunately so, given that it is severely erroneous: see e.g.
<http://www.av8n.com/physics/uncertainty.htm>. However, Mathematica's
approximation of how these uncertainties propagate is first-order, not
zeroth-order. This does not make it completely reliable, of course, but
certainly it is not almost always wrong as is the significant digits
convention. Within the bounds of its own applicability, Mathematica's
approximation is reasonable, although it would still be a mistake to apply
it to experimental uncertainty analysis given the much broader scope of
the latter.

### FranD

Jul 10, 2011, 5:08:00 AM7/10/11
to
Is using "`" an acceptable work-around?

a = 2.`30; b = Sqrt; c = a b;
check[x_] := {x, Precision[x]}

check[a] {2.00000000000000000000000000000, 30.}
check[b] {Sqrt, \[Infinity]}
check[c] {2.82842712474619009760337744842, 30.}

-FranD

Slawek wrote:

> Let Mathematica (6.x, 7.x) compute quite a simple
> product
>
> In:= N[N[2.0, 20] * N[Sqrt , 20], 20]
>
> Out= 2.82843
>
> This is a bug.
>
> Why?
>
> Now we analyze it in details:
>

> 1. N[2.0,20] should give 2 with
> accuracy/precision/whatsever about 20
> decimal digits, i.e. 2.00000000000000000000
>

### slawek

Jul 12, 2011, 7:01:05 AM7/12/11
to

Użytkownik "DrMajorBob" <btr...@austin.rr.com> napisał w wiadomości grup
dyskusyjnych:iv6hgo\$sch\$1...@smc.vnet.net...

> Binary storage is not mathematically required, but the result is MUCH
> faster code. Perhaps, at some point, we'll abandon this trade-off when
> computers are so fast that the gain in speed isn't worth the confusion.
> But that day isn't here, yet.

BCD - Binary Coded Decimals.

### slawek

Jul 12, 2011, 7:02:07 AM7/12/11
to

Użytkownik "Oleksandr Rasputinov" <oleksandr_...@hmamail.com> napisał
w wiadomości grup dyskusyjnych:iv6h68\$s97\$1...@smc.vnet.net...

> considered too verbose), I do not think Mathematica's way of doing things
> is particularly arbitrary or confusing in the broader context of

If 1.4 is not the same as 14/10, then Mathematica should evaluate 1.4 ==
14/10 as False.

### slawek

Jul 12, 2011, 7:01:36 AM7/12/11
to

Użytkownik "Noqsi" <noqsiae...@gmail.com> napisał w wiadomości grup
dyskusyjnych:iv6gf9\$ruo\$1...@smc.vnet.net...

There is no such convention. There is always possible that 2.00 +- 0.53 .

Nobody should believe that 2.00 is more exact than 2.0 or even 2 . (If so,
then 2 Pi have got 10% std. dev. ;)

### Noqsi

Jul 13, 2011, 3:11:53 AM7/13/11
to
On Jul 9, 4:37 am, "Oleksandr Rasputinov"
<oleksandr_rasputi...@hmamail.com> wrote:

> On Fri, 08 Jul 2011 09:51:53 +0100, Noqsi <noqsiaerosp...@gmail.com> wrote:
> > On Jul 7, 5:40 am, "slawek" <sla...@host.pl> wrote:
>
> >> The convention that 2.0 is less accurate than 2.00 is applied ONLY in
> >> Mathematica (the computer program).
>
> > Not true. This is a long-standing convention in experimental science.
>
> Unfortunately so, given that it is severely erroneous:

Such an unqualified statement is unjustified. Whether this produces
troublesome error or not depends on what you're doing.

I have some sympathy with this point of view. But I recall a rather
vehement dispute between two colleagues in a tame case where recording
only significant digits was fine. That dispute poisoned human
relationships unnecessarily. The real folly is to be ideological about
this, instead of understanding what you're doing *from the perspective
of the problem domain*. One size does not fit all.

> However, Mathematica's
> approximation of how these uncertainties propagate is first-order, not
> zeroth-order. This does not make it completely reliable, of course, but
> certainly it is not almost always wrong as is the significant digits
> convention. Within the bounds of its own applicability, Mathematica's
> approximation is reasonable, although it would still be a mistake to apply
> it to experimental uncertainty analysis given the much broader scope of
> the latter.

Experimental uncertainty analysis almost always involves
approximation. Avoiding shortcuts in the statistical formulation can
lead to extremely large calculations even when the data set is very
small. See, for example, Loredo and Lamb 1989, bibcode 1989NYASA.
571..601L. This example still required numerical approximation. In
this case, I think the enormous human and computational effort was
justified, but the appropriateness of any particular approach depends
on the problem to be solved.

### Bill Rowe

Jul 13, 2011, 3:13:56 AM7/13/11
to
On 7/12/11 at 6:59 AM, sla...@host.pl (slawek) wrote:

>U=C2=BFytkownik "Oleksandr Rasputinov" <oleksandr_...@hmamail.com>
>napisa=C2=B3 w wiadomo=C2=B6ci grup dyskusyjnych:iv6h68\$s97\$1...@smc.vnet.net...

>>considered too verbose), I do not think Mathematica's way of doing
>>things is particularly arbitrary or confusing in the broader
>>context of

>If 1.4 is not the same as 14/10, then Mathematica should evaluate
>1.4 == 14/10 as False.

The documentation for Equal (==) specifically states:

Approximate numbers with machine precision or higher are
considered equal if they differ in at most their last seven
binary digits (roughly their last two decimal digits).

Since the exact value 14/10 differs from less than the last
seven bits of the binary representation of 1.4 14/10 == 1.4
returns true.

Note, by default whenever both machine precision values and
exact values are in the same expression, Mathematica evaluates
the expression as if everything was machine precision. And in
general this is a good thing.

The other choices would seem to be either leave 1.4 == 14/10
unevaluated or to return False. Both seem undesirable as they
would likely cause far more problem than returning True as
Mathematica presently does. Either of the other choices would
certainly make expressions containing both exact and approximate
values much more problematic to evaluate.

Ultimately, since the developers are unlikely to change such a
fundamental aspect of Mathematica, the only sensible thing is to
understand how Mathematica does things if you want to
effectively use Mathematica. The alternative would be to find a
system that operates more to your liking.

### Richard Fateman

Jul 13, 2011, 3:17:00 AM7/13/11
to
On 7/12/2011 4:01 AM, slawek wrote:
> U=BFytkownik "DrMajorBob"<btr...@austin.rr.com> napisa=B3 w wiadomo=B6=

ci grup
> dyskusyjnych:iv6hgo\$sch\$1...@smc.vnet.net...
>> Binary storage is not mathematically required, but the result is MUCH
>> faster code. Perhaps, at some point, we'll abandon this trade-off when
>> computers are so fast that the gain in speed isn't worth the confusion=

.
>> But that day isn't here, yet.
>
> BCD - Binary Coded Decimals.
>
>

IEEE 754-2008 is an update to the floating-point standard that not only
includes binary, but decimal arithmetic (and radix-independent). This
is presumably implemented in hardware, or intended to be implemented in
hardware.

At least one computer algebra system does software arbitrary precision
floats in what looks like decimal. Though I think it is base 1000 coded
in 1024 bits or some such thing.
For more reading on IEEE use google or just check out
http://en.wikipedia.org/wiki/IEEE_754-2008

### Noqsi

Jul 13, 2011, 3:12:24 AM7/13/11
to
On Jul 12, 4:01 am, "slawek" <sla...@host.pl> wrote:
> U=BFytkownik "Noqsi" <noqsiaerosp...@gmail.com> napisa=B3 w wiadomo=B6ci grup
> dyskusyjnych:iv6gf9\$ru...@smc.vnet.net...

>
> > On Jul 7, 5:40 am, "slawek" <sla...@host.pl> wrote:
>
> >> The convention that 2.0 is less accurate than 2.00 is applied ONLY in
> >> Mathematica (the computer program).
>
> > Not true. This is a long-standing convention in experimental science.
>
> There is no such convention.

You mean you're not familiar with it. But it exists.

> There is always possible that 2.00 +- 0.53 .

That's a different convention. It also exists.

> Nobody should believe that 2.00 is more exact than 2.0 or even 2 . (If so,
> then 2 Pi have got 10% std. dev. ;)

There are different traditions here. Those who use the "2.00" to
indicate that further digits are unknown do generally not find "2 Pi"
confusing. The clash of conventions in engineering, though, is more
troublesome, where the units change by factors of 1000 and fractions
are often avoided. So, does "200 mm" mean "0.2 m" or "0.200 m"? Still,
this rarely causes serious confusion.

### Richard Fateman

Jul 14, 2011, 5:22:10 AM7/14/11
to
On 7/13/2011 12:13 AM, Bill Rowe wrote:
> On 7/12/11 at 6:59 AM, sla...@host.pl (slawek) wrote:
>
>> U=C2=BFytkownik "Oleksandr Rasputinov"<oleksandr_...@hmamail.com>
>> napisa=C2=B3 w wiadomo=C2=B6ci grup dyskusyjnych:iv6h68\$s97\$1...@smc.vnet.net...
>>> considered too verbose), I do not think Mathematica's way of doing
>>> things is particularly arbitrary or confusing in the broader
>>> context of
>
>> If 1.4 is not the same as 14/10, then Mathematica should evaluate
>> 1.4 == 14/10 as False.
>
> The documentation for Equal (==) specifically states:
>
> Approximate numbers with machine precision or higher are
> considered equal if they differ in at most their last seven
> binary digits (roughly their last two decimal digits).

You mean "considered Equal[] by Mathematica.

>
> Since the exact value 14/10 differs from less than the last
> seven bits of the binary representation of 1.4 14/10 == 1.4
> returns true.
>
> Note, by default whenever both machine precision values and
> exact values are in the same expression, Mathematica evaluates
> the expression as if everything was machine precision. And in
> general this is a good thing.
>
> The other choices would seem to be either leave 1.4 == 14/10
> unevaluated or to return False.

Clearly Rasputinov thinks that if they are not equal they should not be
Equal. Thus the answer is False.

Both seem undesirable as they
> would likely cause far more problem than returning True as
> Mathematica presently does.

Actually, you are assuming users are routinely comparing floats and
exact numbers for equality which they should not be doing anyway.
Programmers in FORTRAN are told to not test for equality involving floats.

Either of the other choices would
> certainly make expressions containing both exact and approximate
> values much more problematic to evaluate.

Or less, depending on what you expect for numbers.

>
> Ultimately, since the developers are unlikely to change such a
> fundamental aspect of Mathematica, the only sensible thing is to
> understand how Mathematica does things if you want to
> effectively use Mathematica. The alternative would be to find a
> system that operates more to your liking.

It might be fun to test to see if any of your code broke if you did this:

Unprotect[Equal]
Equal[a_Real,b_]:= Equal[Rationalize[SetAccuracy[a,Infinity]],b]
Equal[a_,b_Real]:= Equal[a,Rationalize[SetAccuracy[b,Infinity]]]

For example,
0.1d0 is exactly p= 3602879701896397/36028797018963968

so my new and improved Equal thinks that
0.1d0 and 1/10 are NOT equal, (indeed, they differ by 1/180143985094819840)

but

0.1d0 and p ARE equal.

So the question is: would any of YOUR code break if you used this patch
on Equal? Really?

RJF

>
>

### Richard Fateman

Jul 14, 2011, 5:23:11 AM7/14/11
to
On 7/13/2011 12:11 AM, Noqsi wrote:
..

..
Learning mathematics from a physicist is hazardous.
Learning computer science from a physicist is hazardous too.
Numbers in a computer are different from experimental measurements.

nevertheless, I like this article. It says, among other things,

The technique of propagating the uncertainty from step to step
throughout the calculation is a very bad technique. It might sometimes
work for super-simple “textbook” problems but it is unlikely to work for
real-world problems.

### Christoph Lhotka

Jul 14, 2011, 9:18:56 PM7/14/11
to
Dear group!

The number of posts has already become too large to read every of them, so
I appologize if I repeat something which is already said but I think some
people would like to see Equal to behave like SameQ.

Some now might think it is too much to write

In:= 1.4===14/10
Out= False

In:= 1.4==14/10
Out= True

so one way would be to overwrite Equal itself:

In:= Unprotect[Equal]
Out= {Equal}

In:= Equal[a_,b__]:=SameQ[a,b]

In:= Protect[Equal]
Out= {Equal}

In:= 1.4==14/10
Out= False

My question is: how dangerous is it to overwrite such a fundamental
function as Equal?

Best,

Christoph

### Andrzej Kozlowski

Jul 14, 2011, 9:19:27 PM7/14/11
to

On 14 Jul 2011, at 11:21, Richard Fateman wrote:

> On 7/13/2011 12:11 AM, Noqsi wrote:
> ..
>

> ..
> Learning mathematics from a physicist is hazardous.
> Learning computer science from a physicist is hazardous too.
> Numbers in a computer are different from experimental measurements.
>
>
> nevertheless, I like this article. It says, among other things,
>
> The technique of propagating the uncertainty from step to step
> throughout the calculation is a very bad technique. It might sometimes
> work for super-simple textbook problems but it is unlikely to work for
> real-world problems.
>

Well, here is a quote from a very well known book on numerical analysis by a mathematician (Henrici, "Elements of Numerical Analysis").

> It is plain that, on a given machine and for a given problem, the local
> rounding errors are not, in fact, random variables. If the same problem
> is run on the same machine a number of times, there will result always the
> same local rounding errors, and therefore also the same accumulated
> error. We may, however, adopt a stochastic model of the propagation of
> rounding error, where the local errors are treated as if they were random
> variables. This stochastic model has been applied in the literature to a
> number of different numerical problems and has produced results that are
> in complete agreement with experimentally observed results in several
> important cases.

The book then describes the statistical method of error propagation of which Mathematica's approach can be regarded as a first order approximation (as pointed out by Oleksandr Rasputinov, who should not be confused with the OP of this thread so:

> Clearly Rasputinov thinks that if they are not equal they should not be
> Equal. Thus the answer is False.

is *clearly* False. In fact Oleksander expressed something closer to the opposite view.)

And if one quote is not enough, here is another, from another text on numerical analysis. (Conte, de Boor, ""Elementary Numerical Analysis).
It describes 4 approaches to error analysis, interval arithmetic, significant-digit arithmetic, the "statistical approach" and backward error analysis. Here is what it says about the second and the third one:

> A third approach is significant-digit arithmetic. As pointed out earlier, whenever two nearly equal machine numbers are subtracted, there is a danger that some significant digits will be lost. In significant-digit arithmetic an attempt is made to keep track of digits so lost. In one version
> only the significant digits in any number are retained, all others being discarded. At the end of a computation we will thus be assured that all digits retained are significant. The main objection to this method is that some information is lost whenever digits are discarded, and that the results obtained are likely to be much too conservative. Experimentation with this technique is still going on, although the experience to date is not too promising.
>
> A fourth method which gives considerable promise of providing an adequate mathematical theory of round-off-error propagation is based on a statistical approach. It begins with the assumption that round-off errors are independent. This assumption is, of course, not valid, because if the same problem is run on the same machine several times, the answers will always be the same. We can, however, adopt a stochastic model of the propagation of round-off errors in which the local errors are treated as if they were random variables. Thus we can assume that the local round-off errors are either uniformly or normally distributed between their extreme values. Using statistical methods, we can then obtain the standard devia- tion, the variance of distribution, and estimates of the accumulated round- off error. The statistical approach is considered in some detail by Ham- ming  and Henrici . The method does involve substantial analysis and additional computer time, but in the experiments conducted to date it has obtained error estimates which are in remarkable agreement with experimentally available evidence.

The fundamental paper of Mathematica's error propagation is "Precise numerical computation" by Mark Sofroniou and Giulia Spaletta in The Journal of Logic and Algebraic Programming 64 (2005) 113=96134. This paper describes Mathematica's "significance arithmetic" as a first order approximation to Interval Arithmetic. It makes no mention of distributions. Oleksandr Rasputionov, in an earlier post here, interpreted "significance arithmetic" as a first order approximation to the fourth method above. I have not considered this very carefully, but it seems pretty clear that he is right, and that the two "first order" approximations are in fact isomorphic. The first order approach is, of course, justified on grounds of performance. It is perfectly "rigorous" in the same sense as any "first order" approach is (i.e. taking a linear approximation to the Taylor series of a non-linear function). It works fine under certain conditions and will produce nonsense when these conditions do not hold.

The fact that significance arithmetic is "useful" needs no justification other than the fact that it is used successfully by NSolve and Reduce in achieving validated symbolic results by numerical methods which are vastly faster than purely symbolic ones.

It is also useful, for users such as myself, who sometimes need fast first order error anlysis.

I have lots of posts by Richard on this topic (or perhaps it was the same post lots of time, it's so hard to tell), but I have never understood what his main point is. It seems to me that is because he himself has not yet decided this, although he has been posting on this topic for over 20 years (I think).

Sometimes he seems to be disparaging significance arithmetic itself. When Daniel points out how effective it is in his implementation of numerical Groebner basis, or in Adam Strzebonski's work on Reduce, he either ignores this altogether or claims that Groebner bases, etc. are themselves not "useful".

On other occasions he takes on the role of the defender of the interest of the "naive user" (presumably like the OP, who however would be better described as "intentionally naive") who is going to be confused by the "quirky" nature of significance arithmetic (at low precision). In doing so he conveniently ignores that fact of the existence of thousands of "naive users" who never become confused (sometimes because they always work with machine precision numbers and only use significance arrhythmic unknowingly, e.g. when using Reduce). And moreover, for those who do find a need for some sort of error analysis he offers no alternative, except perhaps to learn backward error analysis. Except, of course, that should they do so they would no longer be "naive" and thus outside of Richard's area of concern. And in any case, anyone who needs and understands backward error analysis can use it now, and can't imagine that even Richard would claim that reducing users' options is a good thing.

Finally, perhaps all that Richard is so upset about is simply Mathematica's habit of defining numbers as "fuzz balls" or "distributions". In other words, if Mathematica used a more usual "definition" of number, and used significance arithmetic for "error" propagation, that would be done by applying some separate function or turning on an option, than everything would be fine? If that is all, than it seems to me that Richard has for years been making mountains of molehills.

Andrzej Kozlowski

### Noqsi

Jul 14, 2011, 9:23:04 PM7/14/11
to
On Jul 14, 2:23 am, Richard Fateman <fate...@cs.berkeley.edu> wrote:
> On 7/13/2011 12:11 AM, Noqsi wrote:
> ..
>
> >> see e.g.
> >> <http://www.av8n.com/physics/uncertainty.htm>.
>
> ..
> Learning mathematics from a physicist is hazardous.

In some ways. But if you want to relate mathematics to reality, you
might do well to consider the viewpoints of physicists.

> Learning computer science from a physicist is hazardous too.

Learning computer science from computer scientists is in some ways
worse, unless your interest is pointless bit pushing. Matthew 23:24 is
very relevant (some things never change).

In extracting useful results from computers in the real world
ignorance of the application domain is more debilitating than
ignorance of computer science. One can muddle through the latter, but
not the former.

Consider the howler you yourself committed a little while ago,
discussing the role of computers in the Apollo moon landings:

> you've got to wonder how the Russians, probably WITHOUT much in the way
> of computers, put up an artificial satellite.

This illustrates a profound ignorance of the problem. If you don't
care what orbit you go into, the computations are readily performed by
slide rule ahead of time. But Apollo had to perform a series of
precise maneuvers using imprecise rockets, with repeated re-
computation of the trajectories. For the early missions, they didn't
even know the lunar gravity very well, so actual orbits diverged
rapidly from predictions even when the initial conditions were known.
Apollo's indirect "lunar orbital rendezvous" approach was thus a
triumph of computation. Even the Saturn V was not big enough to
support the less computationally intensive direct approach.

Perhaps the "programmer" I learned the most from in a long career was
a physicist whose code was littered with silly (from a CS point of
view) constructions like:

TA=(COS(EA)-EC)/(1.-EC*COS(EA))
IF(ABS(TA).GT.1.) TA=SIGN(.99999,TA)
TA=ACOS(TA)

What was so great about his code? It's that every program he wrote was
an illuminating exercise in extracting important knowledge from
measurable information. The sloppy technique didn't matter so much. He
put rigor into the place it really counted: serving the needs of his
research. There were several much better technical programmers in that
research group, but they were not as good at conceptualizing how to
actually *use* the computer, rather than simply programming it.

> Numbers in a computer are different from experimental measurements.

But the experimental measurements relate much better to reality.

>
> nevertheless, I like this article. It says, among other things,
>

> The technique of propagating the uncertainty from step to=

step
> throughout the calculation is a very bad technique. It might sometimes

> work for super-simple =93textbook=94 problems but it is unlikely to work =
for
> real-world problems.

Except that error propagation techniques are used successfully in many
fields. Your cell phone works because engineers found a good balance
of power consumption and radio sensitivity from error propagation
methods, rather than the impractical method of tracking each electron
through the circuits. Getting back to orbits, one extremely useful
application of error propagation is to use it "backwards" to determine
which observations would best improve knowledge of an orbit.

There is no universal method for tracking uncertainty that is accurate
and practical. Your own ideologically favored method, interval
arithmetic, yields unrealistically large estimates of error in many
cases, and that can be a very bad thing. Or it can be useful to have
an upper bound. What's good depends on what the *application* needs,
not some ivory tower ideology.

I am *really* tired of your smug, patronizing attitude. You're a blind
man attempting to explain a rainbow. Why not, instead of whining all
the time that Mathematica doesn't conform to your profoundly narrow
notions of what computation is, spend some time with it actually
computing something of relevance to the real world? If you were
actually interested in applications, you would *rejoice* in the fact
that there are a variety of approaches available. But instead, you
obviously see Mathematica as a threat to your narrow ideology.

### W. Craig Carter

Jul 14, 2011, 9:25:07 PM7/14/11
to

On Jul 14, 2011, at 11:21 AM, Richard Fateman <fat...@cs.berkeley.edu> wrote:
> Learning mathematics from a physicist is hazardous.
> Learning computer science from a physicist is hazardous too.
> Numbers in a computer are different from experimental measurements.

I believe that I won't be the only one who objects to this hyperbole and limited world view. I've learned poor quality math from poor quality instructors---but this is not "hazardous.". I've learned what I believe to be high quality math from excellent physics and mathematical physics instructors.

Every experimental measurement of a continuously varying quantity will have some imprecision. Entering that number into a computer may or may not add to the imprecision. Keeping track of the sources of imprecision is what good experimentalist does.

Of course, an experiment is different from a number; but the experimentally d
etermined number and its sources of imprecision in a computer is just another number on the computer. There is no difference in meaning unless the precision of the experiment exceeds that of the measurement.

WCC

### Richard Fateman

Jul 14, 2011, 9:26:08 PM7/14/11
to
On 7/14/2011 7:12 AM, W. Craig Carter wrote:
> On Jul 14, 2011, at 11:21 AM, Richard Fateman<fat...@cs.berkeley.edu> wrote:
>> Learning mathematics from a physicist is hazardous.
>> Learning computer science from a physicist is hazardous too.
>> Numbers in a computer are different from experimental measurements.
> I believe that I won't be the only one who objects to this hyperbole and limited world view. I've learned poor quality math from poor quality instructors---but this is not "hazardous.". I've learned what I believe to be high quality math from excellent physics and mathematical physics instructors.
Generally hazardous. You may however be lucky. I too have had some
excellent physics instructors, but it is generally hazardous, judging
from papers I've read, and programs I've looked at written by
physicists. By the way, I was a physics major as an undergraduate.

> Every experimental measurement of a continuously varying quantity will have some imprecision. Entering that number into a computer may or may not add to the imprecision.
At this point the notion of precision is philosophically different.
Assume we are talking about floating-point representation. Next assume
that the number is scaled so that it is within range of some
(floating-point) representation. The external number can then be stored
in the computer with at most one rounding error, good to half a unit in
the last place (ULP). The precision of this number is related to the
floating-point data format, and varies depending upon
single/double/extended/ software-arbitrary-precision etc choices.
If you, additionally, wish to represent the uncertainty [[NOT
PRECISION]] of the number in the computer, you can do so by storing
more data in the computer.

> Keeping track of the sources of imprecision is what good experimentalist does.
OK. I would add that estimating or bounding the error in a
computational result is often a goal of a numerical computational
scientist. Estimating the computed error would depend on the uncertainty
of the input and the computational steps.
> Of course, an experiment is different from a number; but the experimentally determined number and its sources of imprecision in a computer is just another number on the computer.
Yes, though I would again not use "imprecision" but "uncertainty". I
would also distinguish between the uncertainty of the input and the
potential for subsequent computational errors.

> There is no difference in meaning unless the precision of the experiment exceeds that of the measurement.
You lost me here. Or maybe you mean something like taking 10
measurements, and computing an average, to get an extra
decimal digit??? I don't see how that is related to numerical
computation (as on a computer).

Fundamentally, taking the rules of thumb that you learn in an
or a thermometer, etc, sort of work if you are only doing a few
arithmetic operations on them and you know personally that the numbers
are not correlated. If you put a few thousand numbers in a computer and
do a few billion arithmetic operations on them, the same rules don't
work so well.

RJF

### Richard Fateman

Jul 14, 2011, 9:26:38 PM7/14/11
to
On 7/14/2011 6:22 AM, Christoph Lhotka wrote:
> Dear group!
>
....

> My question is: how dangerous is it to overwrite such a fundamental
> function as Equal?
>
> Best,
>
> Christoph
>

An excellent question. Anyone who runs Mathematica in serious
applications could try it
and report back.

Unfortunately, SameQ doesn't operate the way I suspect you think it does.

Consider a = 1.100000000000000000000000000; b=1.1

SameQ[a,b] is True.

Yet a and b are different. You can see this if you do
SetPrecision[a,100] and SetPrecision[b,100].

1.10000000000000000000000000000000000000117549435082228750796873653722\
2245677818665556772087521508752

1.10000000000000008881784197001252323389053344726562500000000000000000\
0000000000000000000000000000000

RJF

### Andrzej Kozlowski

Jul 15, 2011, 4:12:25 AM7/15/11
to
You are clearly confusing significant digit arithmetic, which is not
what Mathematica uses, with significance arithmetic, which is a first
order approximation to interval arithmetic or distribution based
approach. Obviously you don't read the posts you reply to and confuse
both the posters and the contents of what they post. Here is a quote
from Oleksandr Rasputionov that makes this completely clear:

Unfortunately so, given that it is severely erroneous: see e.g.
<http://www.av8n.com/physics/uncertainty.htm>. However, Mathematica's

approximation of how these uncertainties propagate is first-order, not
zeroth-order. This does not make it completely reliable, of course, but
certainly it is not almost always wrong as is the significant digits
convention. Within the bounds of its own applicability, Mathematica's
approximation is reasonable, although it would still be a mistake to apply
it to experimental uncertainty analysis given the much broader scope of
the latter.

Note the "first-order not zeroth-order". Also, do take a look at Sofroniou and Spaletta and then you may perhaps understand what "order" means and how "significance arithmetic" differs from the "significant digits" convention. Good grief, did you ever learn about the Taylor series? It must have been a long time ago, I take.

Andrzej Kozlowski

On 14 Jul 2011, at 19:55, Richard Fateman wrote:

> On 7/14/2011 6:27 AM, Andrzej Kozlowski wrote:

>> On 14 Jul 2011, at 11:21, Richard Fateman wrote:
>>
>>> On 7/13/2011 12:11 AM, Noqsi wrote:
>>> ..
>>>

>>> ..

>>> Learning mathematics from a physicist is hazardous.
>>> Learning computer science from a physicist is hazardous too.
>>> Numbers in a computer are different from experimental measurements.
>>>
>>>

>>> nevertheless, I like this article. It says, among other things,
>>>

>>> The technique of propagating the uncertainty from step to step

>>> throughout the calculation is a very bad technique. It might sometimes

>>> work for super-simple textbook problems but it is unlikely to work for
>>> real-world problems.
>>>
>> Well, here is a quote from a very well known book on numerical
analysis by a mathematician (Henrici, "Elements of Numerical Analysis").
>>
>>
>>> It is plain that, on a given machine and for a given problem, the local
>>> rounding errors are not, in fact, random variables. If the same problem
>>> is run on the same machine a number of times, there will result always the
>>> same local rounding errors, and therefore also the same accumulated
>>> error. We may, however, adopt a stochastic model of the propagation of
>>> rounding error, where the local errors are treated as if they were random
>>> variables. This stochastic model has been applied in the literature to a
>>> number of different numerical problems and has produced results that are
>>> in complete agreement with experimentally observed results in several
>>> important cases.
>>
>> The book then describes the statistical method of error propagation of which Mathematica's approach can be regarded as a first order approximation (as pointed out by Oleksandr Rasputinov, who should not be confused with the OP of this thread so:
>

> So are we to conclude that Henrici recommends this as a general numerical computational method?
>
> I don't see that here. I think what he is saying is that if you do some mathematics (see below), then
> you will get results consistent with what you will get if you actually run the experiment on the computer.
> This is not surprising. It is a result that says that "theoretical" numerical analysis agrees with
> "computer experiments" in arithmetic. It doesn't say Henrici recommends running a computation this way.
>
> When Henrici says "adopt a stochastic model ...." he doesn't mean to write a program. He means to
> think about each operation like this.. (I show for multiplication of numbers P and Q with errors a b resp.)
>
> P*(1+a) times Q*(1+b) P*Q *(1+a)*(1+b)* (1+c) where c is a new "error" bounded by roundoff, e.g. half unit in last place.
>
> For each operation in the calculation, make up another error letter... a,b,c,d,e,f,g...
> assume they are uncorrelated.
>
> The fact that this theoretical approach and numerically running "several important cases" is a statement
> about correlation of roundoff in these cases, not a statement of advisability of whatever for a model of
> how to write a computer system.
>
> By the way, I think that Henrici was an extremely fine theoretical numerical analyst, and a fine writer too.

>
>
>
>>> Clearly Rasputinov thinks that if they are not equal they should not be
>>> Equal. Thus the answer is False.
>> is *clearly* False. In fact Oleksander expressed something closer to the opposite view.)

> This thread is too long. I don't know at this point if you are agreeing that it is false or contradicting that "is False" is false.

>> And if one quote is not enough, here is another, from another text on numerical analysis. (Conte, de Boor, ""Elementary Numerical Analysis).
>> It describes 4 approaches to error analysis, interval arithmetic, significant-digit arithmetic, the "statistical approach" and backward error analysis. Here is what it says about the second and the third one:

> Huh, if we are talking about the second and third one, why does he say third and fourth?
> Are you using 0-based indexing and deBoor is using 1-based indexing????

>
>
>>> A third approach is significant-digit arithmetic. As pointed out earlier, whenever two nearly equal machine numbers are subtracted, there is a danger that some significant digits will be lost. In significant-digit arithmetic an attempt is made to keep track of digits so lost. In one version
>>> only the significant digits in any number are retained, all others being discarded. At the end of a computation we will thus be assured that all digits retained are significant. The main objection to this method is that some information is lost whenever digits are discarded, and that the results obtained are likely to be much too conservative. Experimentation with this technique is still going on, although the experience to date is not too promising.
>>>

> OK, so deBoor (who is retired and therefore not likely to revise the "experience to date") says this method "is not too promising".
> This sounds to me like he is not endorsing what Mathematica does.

>
>>> A fourth method which gives considerable promise of providing an adequate mathematical theory of round-off-error propagation is based on a statistical approach. It begins with the assumption that round-off errors are independent. This assumption is, of course, not valid, because if the same problem is run on the same machine several times, the answers will always be the same. We can, however, adopt a stochastic model of the propagation of round-off errors in which the local errors are treated as if they were random variables. Thus we can assume that the local round-off errors are either uniformly or normally distributed between their extreme values. Using statistical methods, we can then obtain the standard devia- tion, the variance of distribution, and estimates of the accumulated round- off error. The statistical approach is considered in some detail by Ham- ming  and Henrici . The method does involve substantial analysis and additional computer time, but in the experiments conducted to date it has obtained error estimates which are in remarkable agreement with experimentally available evidence.

> deBoor is essentially quoting Henrici, and this statistical approach is to say that all those error terms I mentioned above, a,b,c,d,e,f,...
> can be chosen from some distribution (the way I've written it, a ,....,z .... would essentially be chosen from {-u,u} where u = 2^(-W) where the fraction part of the floating-point number is W bits. ) and you can compute the final expression as ANSWER+<somehorrendousfunctionof>(a,b,c,....).
>
> What deBoor says is that this (theoretical numerical analysis) "method" promises to provide
> a theory of round-off error propagation. He is not saying this is a practical method for scientific computing. When he uses the work "method"
> he means a mathematical method for analyzing roundoff.
>
> In any case, Mathematica does not do this. I would further argue that Mathematica makes it hard to carry out the experiments that might be done to demonstrate that this theory applied in any particular sample computation.

>>
>> The fundamental paper of Mathematica's error propagation is "Precise numerical computation" by Mark Sofroniou and Giulia Spaletta in The Journal of Logic and Algebraic Programming 64 (2005) 113=96134. This paper describes Mathematica's "significance arithmetic" as a first order approximation to Interval Arithmetic. It makes no mention of distributions.

> I thought I read this paper in some Mathematica documentation or conference proceedings.

>> Oleksandr Rasputionov, in an earlier post here, interpreted "significance arithmetic" as a first order approximation to the fourth method above.

> Huh? First of all, the original poster was slawek. Rasputinov seems to think that Mathematica numbers are like Intervals (basically a good intuition until you think about equality.) and refers to them as distributions. This is not deBoor's 4th "method" of theoretically analyzing round-off.
> In fact it is deBoor's 1st method, interval arithmetic. This has the advantage of being maybe 4 to 8 times slower than regular arithmetic, and also has a huge literature (see "Reliable Computation") describing variations, advantages, disadvantages, etc.

>
>> I have not considered this very carefully, but it seems pretty clear that he is right, and that the two "first order" approximations are in fact isomorphic.

> Uh, this is unclear, unless you mean that Mathematica's number system is essentially interval arithmetic, but with a confusing front end.

>> The first order approach is, of course, justified on grounds of performance. It is perfectly "rigorous" in the same sense as any "first order" approach is (i.e. taking a linear approximation to the Taylor series of a non-linear function). It works fine under certain conditions and will produce nonsense when these conditions do not hold.
>>
>> The fact that significance arithmetic is "useful" needs no justification other than the fact that it is used successfully by NSolve and Reduce in achieving validated symbolic results by numerical methods which are vastly faster than purely symbolic ones.

> Really?
>
> 1. This does not mean that other methods, e.g. validated methods for accurately evaluating polynomials (etc) NOT based on significance arithmetic would not be faster and better. DanL claims it is used and useful there, so that's nice. BUT..
>
> 2. This does not mean that this arithmetic should be used by default by user arithmetic.

>> It is also useful, for users such as myself, who sometimes need fast first order error anlysis.

> OK, If you find it useful yourself, fine. I daresay you are not a typical user.

>
>> I have lots of posts by Richard on this topic (or perhaps it was the same post lots of time, it's so hard to tell), but I have never understood what his main point is.
>

> Main points: Mathematica's number system is non-standard, peculiar, hard to explain, capable of returning mysterious non-answers without indications of error to innocent users, a bad foundation for building higher-level functionality.
>
> I have other criticisms of Mathematica, but I think that the sentence above is enough for you to process today.

>
>> It seems to me that is because he himself has not yet decided this, although he has been posting on this topic for over 20 years (I think).
>
>> Sometimes he seems to be disparaging significance arithmetic itself.

> I think deBoor did that.

>
>> When Daniel points out how effective it is in his implementation of numerical Groebner basis, or in Adam Strzebonski's work on Reduce, he either ignores this altogether or claims that Groebner bases, etc. are themselves not "useful".

> I think Reduce is a very nice program when it works. If I am not mistaken, all work on numerical Groebner basis should be reducible to the evaluation of polynomials, for which there are faster and more accurate methods available not using significance arithmetic. On the other hand, I might be mischaracterizing DanL work, since I admit to not having studied it.

>
>
>> On other occasions he takes on the role of the defender of the interest of the "naive user" (presumably like the OP, who however would be better described as "intentionally naive") who is going to be confused by the "quirky" nature of significance arithmetic (at low precision).

> No, I don't think that slawek was a "troll". I think he was genuinely confused, as might anyone be who has some prior computer arithmetic exposure and for the first time encounters a system with exact rational arithmetic.

>> In doing so he conveniently ignores that fact of the existence of thousands of "naive users" who never become confused (sometimes because they always work with machine precision numbers and only use significance arrhythmic unknowingly, e.g. when using Reduce).

> Most naive users don't tread near the dangerous spots. Some naive users never notice that their answers are nonsense.
> Every so often we get a naive user who DOES notice, and he/she sends email to this newsgroup.

>> And moreover, for those who do find a need for some sort of error analysis he offers no alternative, except perhaps to learn backward error analysis.

> Actually, there's a whole bunch of libraries of methods with error estimates for common computations, where backward error analysis or some other method was used to provide extra information.

>
>> Except, of course, that should they do so they would no longer be "naive" and thus outside of Richard's area of concern.

> They would probably not be writing in Mathematica.

>> And in any case, anyone who needs and understands backward error analysis can use it now, and can't imagine that even Richard would claim that reducing users' options is a good thing.

> I have no problem with Mathematica providing as an option, some other arithmetic. It is WRI that has made it rather hard for the user to
> figure out how to "use the arithmetic of the rest of the world".

>
>> Finally, perhaps all that Richard is so upset about is simply Mathematica's habit of defining numbers as "fuzz balls" or "distributions".

> I'm not sure that "upset" is the right term. After all, I don't have to use Mathematica for numerical computation. And I rarely do.

>> In other words, if Mathematica used a more usual "definition" of number, and used significance arithmetic for "error" propagation, that would be done by applying some separate function or turning on an option, than everything would be fine?

> Actually, that's pretty close to correct.

>> If that is all, than it seems to me that Richard has for years been making mountains of molehills.

> So why are you (and WRI) so opposed to this notion? Note that it would also have to affect other parts of the system including of course,
> Equal and friends.
>
>
>>
>> Andrzej Kozlowski
>>
>>
>>
>>
>
>

### James Stein

Jul 15, 2011, 4:09:20 AM7/15/11
to
Richard, thanks for pointing this out.
>From what you say, it seems that the documentation for SameQ is incorrect.

On Thu, Jul 14, 2011 at 6:21 PM, Richard Fateman
<fat...@eecs.berkeley.edu>wrote:

### Richard Fateman

Jul 15, 2011, 4:10:53 AM7/15/11
to
On 7/14/2011 6:23 PM, Noqsi wrote:
> On Jul 14, 2:23 am, Richard Fateman<fate...@cs.berkeley.edu> wrote:
>> On 7/13/2011 12:11 AM, Noqsi wrote:
>> ..
>>
>>>> see e.g.
>>>> <http://www.av8n.com/physics/uncertainty.htm>.
>>
>> ..
>> Learning mathematics from a physicist is hazardous.
>
> In some ways. But if you want to relate mathematics to reality, you
> might do well to consider the viewpoints of physicists.
>
>> Learning computer science from a physicist is hazardous too.
>
> Learning computer science from computer scientists is in some ways
> worse, unless your interest is pointless bit pushing. Matthew 23:24 is
> very relevant (some things never change).

I suppose it depends on what you mean by computer science.

>
> In extracting useful results from computers in the real world
> ignorance of the application domain is more debilitating than
> ignorance of computer science. One can muddle through the latter, but
> not the former.

I think you may be confusing "computer science" with "applications of
computers to whatever [e.g. physics, engineering, business]" which is
usually taught in the department of 'whatever'.

>
> Consider the howler you yourself committed a little while ago,
> discussing the role of computers in the Apollo moon landings:
>
>> you've got to wonder how the Russians, probably WITHOUT much in the way
>> of computers, put up an artificial satellite.
>
> This illustrates a profound ignorance of the problem. If you don't
> care what orbit you go into, the computations are readily performed by
> slide rule ahead of time. But Apollo had to perform a series of
> precise maneuvers using imprecise rockets, with repeated re-
> computation of the trajectories. For the early missions, they didn't
> even know the lunar gravity very well, so actual orbits diverged
> rapidly from predictions even when the initial conditions were known.
> Apollo's indirect "lunar orbital rendezvous" approach was thus a
> triumph of computation. Even the Saturn V was not big enough to
> support the less computationally intensive direct approach.

Do you know for a fact that the Russians didn't care what orbit the
first manned satellite had?

>
> Perhaps the "programmer" I learned the most from in a long career was
> a physicist whose code was littered with silly (from a CS point of
> view) constructions like:
>
> TA=(COS(EA)-EC)/(1.-EC*COS(EA))
> IF(ABS(TA).GT.1.) TA=SIGN(.99999,TA)
> TA=ACOS(TA)
>
> What was so great about his code? It's that every program he wrote was
> an illuminating exercise in extracting important knowledge from
> measurable information. The sloppy technique didn't matter so much. He
> put rigor into the place it really counted: serving the needs of his
> research. There were several much better technical programmers in that
> research group, but they were not as good at conceptualizing how to
> actually *use* the computer, rather than simply programming it.

I think you are confusing application knowledge with computer science.
It is fairly clear that computer scientists cannot be held responsible
for the content of all programs.

>
>> Numbers in a computer are different from experimental measurements.
>
> But the experimental measurements relate much better to reality.

The computation deals with representations in the computer. Mapping
those representations to the external world is a separate matter that
deals with sensors, actuators (robots, displays, computer-controlled
instruments, sound boards, digital cameras, scanners, microphones...)
These are usually parts of some other engineering discipline.

>
>>
>> nevertheless, I like this article. It says, among other things,
>>
>> The technique of propagating the uncertainty from step to=
> step
>> throughout the calculation is a very bad technique. It might sometimes
>> work for super-simple =93textbook=94 problems but it is unlikely to work =
> for
>> real-world problems.
>
> Except that error propagation techniques are used successfully in many
> fields.

A simple problem perhaps.

Your cell phone works because engineers found a good balance
> of power consumption and radio sensitivity from error propagation
> methods, rather than the impractical method of tracking each electron
> through the circuits.

Of course there are many methods that can be programmed. You are
assuming that analysis of signals is done by some kind of particle
tracking? I assume that programs are designed by persons familiar with
differential equations and electromagnetic radiation, as well as more
seat-of-the-pants stuff like antenna design and sun spots.

Also I assume
that cell phones use signal strength and feedback, and do not need great
accuracy. Though maybe GPS stuff is tricky if you have few
triangulation points. Using 10 points, maybe not so tricky. Not
something I've cared to look at.

Anyway, after a few billion computations, significance arithmetic tends
to lose.

Getting back to orbits, one extremely useful
> application of error propagation is to use it "backwards" to determine
> which observations would best improve knowledge of an orbit.
>
> There is no universal method for tracking uncertainty that is accurate
> and practical.

Ah, so you are saying that Mathematica is not accurate and practical??

Your own ideologically favored method, interval
> arithmetic, yields unrealistically large estimates of error in many
> cases, and that can be a very bad thing. Or it can be useful to have
> an upper bound. What's good depends on what the *application* needs,
> not some ivory tower ideology.

It's not my favorite. I point it out as a method that has been widely
studied. It does not provide estimates of error. It provides bounds on
error, and those bounds may be very pessimistic.

>
> I am *really* tired of your smug, patronizing attitude. You're a blind
> man attempting to explain a rainbow. Why not, instead of whining all
> the time that Mathematica doesn't conform to your profoundly narrow
> notions of what computation is, spend some time with it actually
> computing something of relevance to the real world?

My concern is that someone attempting to compute something of relevance
will fall into a real-world hole.

If you were
> actually interested in applications, you would *rejoice* in the fact
> that there are a variety of approaches available. But instead, you
> obviously see Mathematica as a threat to your narrow ideology.

I am primarily interested in building systems appropriate for a range of
applications. (That's more computer science). From that perspective I
think that Mathematica falls short. I don't see that as a threat, but I
am inclined to object to statements that claim (in my view incorrectly)
that Mathematica (arithmetically speaking) is the best, or even the only
way to do floating-point calculations. In some ways the Mathematica
system is just fine, if you use its library routines for special
functions for arbitrary precision, and you are within the appropriate
ranges where they actually deliver what is promised. (I have mixed
experiences near singular points...)
But I think I've stated my perspective pretty clearly, even if
patronizingly.

RJF

>
>

### Christoph Lhotka

Jul 15, 2011, 5:07:40 AM7/15/11
to
yes, indeed!

On 15/07/2011 09:34, Andrzej Kozlowski wrote:
> Well, here is one example of what would happen:
>
> In:= Unprotect[Equal]
>
> Out= {Equal}
>
> In:= Equal[a_, b__] := SameQ[a, b]
>
> In:= Protect[Equal]
>
> Out= {Equal}
>
> In:= Solve[3 x == 1, x]
>
> Out= {}
>
> Doesn't look like a great idea to me.
>
> Andrzej Kozlowski

>
>
>
>
> On 15 Jul 2011, at 03:18, Christoph Lhotka wrote:
>
>> Dear group!
>>

>> The number of posts has already become too large to read every of them, so
>> I appologize if I repeat something which is already said but I think some
>> people would like to see Equal to behave like SameQ.
>>
>> Some now might think it is too much to write
>>
>> In:= 1.4===14/10
>> Out= False
>>
>>
>> In:= 1.4==14/10
>> Out= True
>>
>> so one way would be to overwrite Equal itself:
>>
>> In:= Unprotect[Equal]
>> Out= {Equal}
>>
>> In:= Equal[a_,b__]:=SameQ[a,b]
>>
>> In:= Protect[Equal]
>> Out= {Equal}
>>
>> In:= 1.4==14/10
>> Out= False
>>

>> My question is: how dangerous is it to overwrite such a fundamental
>> function as Equal?
>>
>> Best,
>>
>> Christoph
>>
>>

>>> Clearly Rasputinov thinks that if they are not equal they should not be
>>> Equal. Thus the answer is False.
>>>

### Andrzej Kozlowski

Jul 15, 2011, 5:07:09 AM7/15/11
to

### Bill Rowe

Jul 16, 2011, 5:43:56 AM7/16/11
to
On 7/14/11 at 5:20 AM, fat...@cs.berkeley.edu (Richard Fateman)
wrote:

>On 7/13/2011 12:13 AM, Bill Rowe wrote:
>>On 7/12/11 at 6:59 AM, sla...@host.pl (slawek) wrote:

>>The documentation for Equal (==) specifically states:

>>Approximate numbers with machine precision or higher are considered
>>equal if they differ in at most their last seven binary digits
>>(roughly their last two decimal digits).

>You mean "considered Equal[] by Mathematica.

No. I meant exactly what I wrote. The excerpt was copy/paste
from the documentation for Equal. Now if you want to interpret
the documentation fine. But this is what is in the
documentation. And if your comment is to the effect something
other than Mathematica might treat 1.4 as unequal to 14/10, so
what? The existence of other ways to treating this has no impact
on what is in the Mathematica documentation or how Mathematica works.

>>Ultimately, since the developers are unlikely to change such a
>>fundamental aspect of Mathematica, the only sensible thing is to
>>understand how Mathematica does things if you want to effectively
>>use Mathematica. The alternative would be to find a system that
>>operates more to your liking.

>It might be fun to test to see if any of your code broke if you did
>this:

>Unprotect[Equal] Equal[a_Real,b_]:=
>Equal[Rationalize[SetAccuracy[a,Infinity]],b] Equal[a_,b_Real]:=
>Equal[a,Rationalize[SetAccuracy[b,Infinity]]]

Fun?? Modifying built-in Mathematica commands is risky. To much
of Mathematica uses other Mathematica functions. I could do then
this spend lots of time testing and getting results that lead me
to believe there is no impact only to get burned much later and
likely have a very difficult time determining why something
didn't work as expected. This isn't my idea of fun. I have far
better things to do with my time.

>For example, 0.1d0 is exactly p= 3602879701896397/36028797018963968

>so my new and improved Equal thinks that 0.1d0 and 1/10 are NOT
>equal, (indeed, they differ by 1/180143985094819840)

>but

>0.1d0 and p ARE equal.

>So the question is: would any of YOUR code break if you used this
>patch on Equal? Really?

I don't know the answer. And since I assume you don't have
access to Mathematica's source code, you cannot know the answer
either. I have no intention of attempting to determine the
answer for the reasons stated above. So, a discussion of whether
this patch would or would not break some code is pointless to me.

### Andrzej Kozlowski

Jul 16, 2011, 5:45:28 AM7/16/11
to

On 15 Jul 2011, at 16:49, Richard Fateman wrote:

> On 7/14/2011 11:55 PM, Andrzej Kozlowski wrote:
>
> Gee, Andrzej, all I can think of is the childhood playground chant (I don't know where you might have been a child, so this may not
> bring back memories...)
>
> "I'm rubber, you're glue; everything You say sticks to YOU!"

Yes, I can also think of a few playground chants that would apply nicely, but unfortunately you would not understand them. So let's have something English instead, like "Sticks and stones...".

Now, I really have no time or patience to go over all your misrepresentations in detail. But let me point out (or remind you perhaps) that the already mentioned paper of Sofroniou and Spaletta makes the following remark:

> The choice of significance arithmetic as the default in Mathematica has not been universally favorable (see for example ) although some of these criticisms relate to early deficiencies in the implementation.

Reference  is, of course:

 R.J. Fateman, A review of Mathematica, J. Symbolic Comput. 13 (1992) 545=96579.

In fact the "early" version was much more like "significant digits convention", while the current version is quite a lot more sophisticated, as described in the article. Somehow you don't seem to have noticed it since that is the "early version" that you always seem to be describing. (Significant digits is something that used to be taught in primary school, if I recall correctly).

As for difference between theoretical analysis of the propagation of round-off error and practical scientific computations, I am well aware of it, but was it not you who gleefully quoted the statement:

The technique of propagating the uncertainty from step to step

throughout the calculation is a very bad technique. It might sometimes

work for super-simple textbook problems but it is unlikely to work for
real-world problems.

So if agree with this (as you seem to), are you implying that the entire chapter 16 of Henrici's book is concerned with something that only applies to "super-simple text book problems"?

The point is, of course, that you never make it clearly what exactly you are talking about probably because you find it convenient for rhetorical purposes (90% of these discussions are really just rhetorics).

I should add that I myself long ago and more then once wrote on this forum that I do not think Mathematica's "approximate numbers" are a good model for empirical errors, but they are very good tools for solving certain purely mathematical problems. Often it can be accomplished in 100% provably correct way (barring bugs of course).

As for things like numerical Groebner basis etc, you keep arguing that it probably can be done in some other way (without significance arithmetic) and in a certain, rather trivial, sense you undoubtedly right. But the fact is, as Daniel has pointed out, that, in practice, nobody seems to have done it in another way, and not for want of trying. So again here we have the distinction between "theory" and "practical application" but except that this time the shoe is on the other foot.

Andrzej

>> You are clearly confusing significant digit arithmetic, which is not what Mathematica uses, with significance arithmetic, which is a first order approximation to interval arithmetic or distribution based approach.
>

> I view significant digit arithmetic, if by digit you allow binary digit or "bit", to be a crude version of interval arithmetic where the endpoints of the interval are not numbers of (essentially) arbitrary precision, but where the endpoints must be represented by the central point plus or minus some power of two. It is an approximation to interval arithmetic. I think it is a reasonable idea to restrict the =
number of digits carried in the value of the interval endpoints (making =
them "cruder" than you might really know) in the interests of efficiency =
in time and space. Thus using properly rounded machine floats is =
appropriate, at least when the range is suitable and the difference =
between upper and lower bounds is large. Using just one bit makes the =
width of the interval grow faster than using more bits...
>
> Since you do not understand the context of the paragraphs you have =
quoted, nor do you seem to acknowledge the difference between =
theoretical analysis of the propagation of round-off error in =
well-chosen standard examples, and somehow running interval arithmetic =
all the time, it is hard to know how to educate you. How comforting is =
your proud quote, below? Mathematica's arithmetic is "not ... =
completely reliable". Rasputinov says,
> further, "it is not almost always wrong". I'm not sure why he says =
the significant digit convention is almost always wrong unless he means =
almost always too pessimistic. I'm also not aware of why Rasputinov's =
opinion should sway the discussion especially (sorry Oleksandr).
>
>
> My point remains:
> WRI could have a much more direct notion of number, and equality, that =
would make it easy to implement my choice of arithmetic or theirs or =
yours.
> They didn't. The default is not completely reliable. People write to =
this newsgroup, periodically, saying Huh? what's going on? I found a =
bug!
>
> As for the experimental and computational notions of (im)precision, I =
think there is an issue of using the same words with different meanings. =
Similar but unfortunately different. Precision of a floating point =
number F is simply the number of bits in the fraction. If you impute =
some uncertainty to F, you can store that in another piece of data D in =
the computer. If you want to assert that F and D together represent a =
distribution of a certain kind you can also compute with that.
>
> RJF
>
>> Obviously you don't read the posts you reply to and confuse both the =
posters and the contents of what they post. Here is a quote from =

Oleksandr Rasputionov that makes this completely clear:
>>
>>

>> Unfortunately so, given that it is severely erroneous: see e.g.
>> <http://www.av8n.com/physics/uncertainty.htm>. However, Mathematica's
>> approximation of how these uncertainties propagate is first-order, =
not
>> zeroth-order. This does not make it completely reliable, of course, =

but
>> certainly it is not almost always wrong as is the significant digits
>> convention. Within the bounds of its own applicability, Mathematica's

>> approximation is reasonable, although it would still be a mistake to =
apply
>> it to experimental uncertainty analysis given the much broader scope =
of
>> the latter.
>>
>> Note the "first-order not zeroth-order". Also, do take a look at =
Sofroniou and Spaletta and then you may perhaps understand what "order" =
means and how "significance arithmetic" differs from the "significant =
digits" convention. Good grief, did you ever learn about the Taylor =

series? It must have been a long time ago, I take.
>>
>>
>> Andrzej Kozlowski
>>
>>
>> On 14 Jul 2011, at 19:55, Richard Fateman wrote:
>>
>>> On 7/14/2011 6:27 AM, Andrzej Kozlowski wrote:

>>>> On 14 Jul 2011, at 11:21, Richard Fateman wrote:
>>>>
>>>>> On 7/13/2011 12:11 AM, Noqsi wrote:
>>>>> ..
>>>>>

>>>>> ..
>>>>> Learning mathematics from a physicist is hazardous.

>>>>> Learning computer science from a physicist is hazardous too.

>>>>> Numbers in a computer are different from experimental =
measurements.

>>>>>
>>>>>
>>>>> nevertheless, I like this article. It says, among other things,
>>>>>

>>>>> The technique of propagating the uncertainty from step to step
>>>>> throughout the calculation is a very bad technique. It might =
sometimes
>>>>> work for super-simple =84textbook=89 problems but it is unlikely =

to work for
>>>>> real-world problems.
>>>>>

>>>> Well, here is a quote from a very well known book on numerical =

analysis by a mathematician (Henrici, "Elements of Numerical Analysis").
>>>>
>>>>

>>>>> It is plain that, on a given machine and for a given problem, the =
local
>>>>> rounding errors are not, in fact, random variables. If the same =
problem
>>>>> is run on the same machine a number of times, there will result =
always the
>>>>> same local rounding errors, and therefore also the same =
accumulated
>>>>> error. We may, however, adopt a stochastic model of the =
propagation of
>>>>> rounding error, where the local errors are treated as if they were =
random
>>>>> variables. This stochastic model has been applied in the =
literature to a
>>>>> number of different numerical problems and has produced results =
that are
>>>>> in complete agreement with experimentally observed results in =
several
>>>>> important cases.
>>>> The book then describes the statistical method of error propagation =
of which Mathematica's approach can be regarded as a first order =
approximation (as pointed out by Oleksandr Rasputinov, who should not be =

confused with the OP of this thread so:

>>> So are we to conclude that Henrici recommends this as a general =
numerical computational method?
>>>
>>> I don't see that here. I think what he is saying is that if you do =

some mathematics (see below), then

>>> you will get results consistent with what you will get if you =

actually run the experiment on the computer.

>>> This is not surprising. It is a result that says that "theoretical" =
numerical analysis agrees with
>>> "computer experiments" in arithmetic. It doesn't say Henrici =

recommends running a computation this way.
>>>

>>> When Henrici says "adopt a stochastic model ...." he doesn't mean =

to write a program. He means to

>>> think about each operation like this.. (I show for multiplication =

of numbers P and Q with errors a b resp.)
>>>

>>> P*(1+a) times Q*(1+b) = P*Q *(1+a)*(1+b)* (1+c) where c is a =

new "error" bounded by roundoff, e.g. half unit in last place.
>>>

>>> For each operation in the calculation, make up another error =

letter... a,b,c,d,e,f,g...
>>> assume they are uncorrelated.
>>>

>>> The fact that this theoretical approach and numerically running =

"several important cases" is a statement

>>> about correlation of roundoff in these cases, not a statement of =

advisability of whatever for a model of
>>> how to write a computer system.
>>>

>>> By the way, I think that Henrici was an extremely fine theoretical =

numerical analyst, and a fine writer too.
>>>
>>>
>>>

>>>>> Clearly Rasputinov thinks that if they are not equal they should =

not be
>>>>> Equal. Thus the answer is False.

>>>> is *clearly* False. In fact Oleksander expressed something closer =
to the opposite view.)
>>> This thread is too long. I don't know at this point if you are =

agreeing that it is false or contradicting that "is False" is false.

>>>> And if one quote is not enough, here is another, from another text =
on numerical analysis. (Conte, de Boor, ""Elementary Numerical =
Analysis).
>>>> It describes 4 approaches to error analysis, interval arithmetic, =
significant-digit arithmetic, the "statistical approach" and backward =

error analysis. Here is what it says about the second and the third one:

>>> Huh, if we are talking about the second and third one, why does he =
say third and fourth?
>>> Are you using 0-based indexing and deBoor is using 1-based =
indexing????
>>>
>>>
>>>>> A third approach is significant-digit arithmetic. As pointed out =
earlier, whenever two nearly equal machine numbers are subtracted, there =
is a danger that some significant digits will be lost. In =
significant-digit arithmetic an attempt is made to keep track of digits =

so lost. In one version

>>>>> only the significant digits in any number are retained, all others =
being discarded. At the end of a computation we will thus be assured =
that all digits retained are significant. The main objection to this =
method is that some information is lost whenever digits are discarded, =
and that the results obtained are likely to be much too conservative. =
Experimentation with this technique is still going on, although the =

experience to date is not too promising.
>>>>>

>>> OK, so deBoor (who is retired and therefore not likely to revise the =

"experience to date") says this method "is not too promising".
>>> This sounds to me like he is not endorsing what Mathematica does.
>>>

>>>>> A fourth method which gives considerable promise of providing an =
adequate mathematical theory of round-off-error propagation is based on =
a statistical approach. It begins with the assumption that round-off =
errors are independent. This assumption is, of course, not valid, =
because if the same problem is run on the same machine several times, =
the answers will always be the same. We can, however, adopt a stochastic =
model of the propagation of round-off errors in which the local errors =
are treated as if they were random variables. Thus we can assume that =
the local round-off errors are either uniformly or normally distributed =
between their extreme values. Using statistical methods, we can then =
obtain the standard devia- tion, the variance of distribution, and =
estimates of the accumulated round- off error. The statistical approach =
is considered in some detail by Ham- ming  and Henrici . The =
method does involve substantial analysis and additional computer time, =
but in the experiments conducted to date it has obtained error estimates =
which are in remarkable agreement with experimentally available =
evidence.
>>> deBoor is essentially quoting Henrici, and this statistical approach =

is to say that all those error terms I mentioned above, a,b,c,d,e,f,...

>>> can be chosen from some distribution (the way I've written it, a =
,....,z .... would essentially be chosen from {-u,u} where u = =
2^(-W) where the fraction part of the floating-point number is W bits. =
) and you can compute the final expression as =
>>>
>>> What deBoor says is that this (theoretical numerical analysis) =
"method" promises to provide
>>> a theory of round-off error propagation. He is not saying this is =
a practical method for scientific computing. When he uses the work =

"method"
>>> he means a mathematical method for analyzing roundoff.
>>>

>>> In any case, Mathematica does not do this. I would further argue =
that Mathematica makes it hard to carry out the experiments that might =
be done to demonstrate that this theory applied in any particular =
sample computation.
>>>> The fundamental paper of Mathematica's error propagation is =
"Precise numerical computation" by Mark Sofroniou and Giulia Spaletta =
in The Journal of Logic and Algebraic Programming 64 (2005) 113=96134. =
This paper describes Mathematica's "significance arithmetic" as a first =
order approximation to Interval Arithmetic. It makes no mention of =
distributions.
>>> I thought I read this paper in some Mathematica documentation or =
conference proceedings.
>>>> Oleksandr Rasputionov, in an earlier post here, interpreted =
"significance arithmetic" as a first order approximation to the fourth =
method above.
>>> Huh? First of all, the original poster was slawek. Rasputinov seems =
to think that Mathematica numbers are like Intervals (basically a good =
intuition until you think about equality.) and refers to them as =
distributions. This is not deBoor's 4th "method" of theoretically =
analyzing round-off.
>>> In fact it is deBoor's 1st method, interval arithmetic. This has =
the advantage of being maybe 4 to 8 times slower than regular =
arithmetic, and also has a huge literature (see "Reliable Computation") =

>>>

>>>> I have not considered this very carefully, but it seems pretty =
clear that he is right, and that the two "first order" approximations =
are in fact isomorphic.
>>> Uh, this is unclear, unless you mean that Mathematica's number =
system is essentially interval arithmetic, but with a confusing front =
end.
>>>> The first order approach is, of course, justified on grounds of =
performance. It is perfectly "rigorous" in the same sense as any "first =
order" approach is (i.e. taking a linear approximation to the Taylor =
series of a non-linear function). It works fine under certain conditions =

and will produce nonsense when these conditions do not hold.
>>>>

>>>> The fact that significance arithmetic is "useful" needs no =
justification other than the fact that it is used successfully by NSolve =
and Reduce in achieving validated symbolic results by numerical methods =

which are vastly faster than purely symbolic ones.
>>> Really?
>>>

>>> 1. This does not mean that other methods, e.g. validated methods for =
accurately evaluating polynomials (etc) NOT based on significance =
arithmetic would not be faster and better. DanL claims it is used and =

useful there, so that's nice. BUT..
>>>

>>> 2. This does not mean that this arithmetic should be used by default =
by user arithmetic.
>>>> It is also useful, for users such as myself, who sometimes need =

fast first order error anlysis.

>>> OK, If you find it useful yourself, fine. I daresay you are not a =
typical user.
>>>
>>>> I have lots of posts by Richard on this topic (or perhaps it was =
the same post lots of time, it's so hard to tell), but I have never =

understood what his main point is.

>>> Main points: Mathematica's number system is non-standard, peculiar, =
hard to explain, capable of returning mysterious non-answers without =
indications of error to innocent users, a bad foundation for building =
higher-level functionality.
>>>
>>> I have other criticisms of Mathematica, but I think that the =

sentence above is enough for you to process today.
>>>

>>>> It seems to me that is because he himself has not yet decided =
this, although he has been posting on this topic for over 20 years (I =
think).
>>>> Sometimes he seems to be disparaging significance arithmetic =

itself.
>>> I think deBoor did that.
>>>

>>>> When Daniel points out how effective it is in his implementation =
of numerical Groebner basis, or in Adam Strzebonski's work on Reduce, he =
either ignores this altogether or claims that Groebner bases, etc. are =
themselves not "useful".
>>> I think Reduce is a very nice program when it works. If I am not =
mistaken, all work on numerical Groebner basis should be reducible to =
the evaluation of polynomials, for which there are faster and more =
accurate methods available not using significance arithmetic. On the =
other hand, I might be mischaracterizing DanL work, since I admit to not =
having studied it.
>>>
>>>
>>>> On other occasions he takes on the role of the defender of the =
interest of the "naive user" (presumably like the OP, who however would =
be better described as "intentionally naive") who is going to be =
confused by the "quirky" nature of significance arithmetic (at low =
precision).
>>> No, I don't think that slawek was a "troll". I think he was =
genuinely confused, as might anyone be who has some prior computer =
arithmetic exposure and for the first time encounters a system with =
exact rational arithmetic.
>>>> In doing so he conveniently ignores that fact of the existence of =
thousands of "naive users" who never become confused (sometimes because =
they always work with machine precision numbers and only use =

significance arrhythmic unknowingly, e.g. when using Reduce).

>>> Most naive users don't tread near the dangerous spots. Some naive =

users never notice that their answers are nonsense.

>>> Every so often we get a naive user who DOES notice, and he/she sends =
email to this newsgroup.
>>>> And moreover, for those who do find a need for some sort of error =
analysis he offers no alternative, except perhaps to learn backward =
error analysis.
>>> Actually, there's a whole bunch of libraries of methods with error =
estimates for common computations, where backward error analysis or some =

other method was used to provide extra information.
>>>

>>>> Except, of course, that should they do so they would no longer be =

"naive" and thus outside of Richard's area of concern.
>>> They would probably not be writing in Mathematica.

>>>> And in any case, anyone who needs and understands backward error =
analysis can use it now, and can't imagine that even Richard would claim =

that reducing users' options is a good thing.

>>> I have no problem with Mathematica providing as an option, some =
other arithmetic. It is WRI that has made it rather hard for the user =

to
>>> figure out how to "use the arithmetic of the rest of the world".
>>>

>>>> Finally, perhaps all that Richard is so upset about is simply =
Mathematica's habit of defining numbers as "fuzz balls" or =
"distributions".
>>> I'm not sure that "upset" is the right term. After all, I don't =

have to use Mathematica for numerical computation. And I rarely do.

>>>> In other words, if Mathematica used a more usual "definition" of =
number, and used significance arithmetic for "error" propagation, that =
would be done by applying some separate function or turning on an =

option, than everything would be fine?
>>> Actually, that's pretty close to correct.

>>>> If that is all, than it seems to me that Richard has for years been =
making mountains of molehills.
>>> So why are you (and WRI) so opposed to this notion? Note that it =

### Richard Fateman

Jul 16, 2011, 5:46:30 AM7/16/11
to
On 7/15/2011 11:54 AM, Andrzej Kozlowski wrote:
>
> On 15 Jul 2011, at 16:49, Richard Fateman wrote:
>
>> On 7/14/2011 11:55 PM, Andrzej Kozlowski wrote:
>>
>> Gee, Andrzej, all I can think of is the childhood playground chant
(I don't know where you might have been a child, so this may not
>> bring back memories...)
>>
>> "I'm rubber, you're glue; everything You say sticks to YOU!"
>
> Yes, I can also think of a few playground chants that would apply nicely,
but unfortunately you would not understand them. So let's have something English instead, like "Sticks and stones...".
>
> Now, I really have no time or patience to go over all your misrepresentations in detail. But let me point out (or remind you perhaps) that the already mentioned paper of Sofroniou and Spaletta makes the following remark:
>
>> The choice of significance arithmetic as the default in Mathematica has not been universally favorable (see for example ) although some of these criticisms relate to early deficiencies in the implementation.
>
> Reference  is, of course:
>
>  R.J. Fateman, A review of Mathematica, J. Symbolic Comput. 13 (1992) 545=E2=80=93579.

>
> In fact the "early" version was much more like "significant digits convention", while the current version is quite a lot more sophisticated, as described in the article. Somehow you don't seem to have noticed it since that is the "early version" that you always seem to be describing. (Significant digits is something that used to be taught in primary school, if I recall correctly).

(x = 1.00000000000000000000; While[x != 0, x = 2*x - x; Print[x]])

terminates when Equal[x,0] is True.

And[x == 1, x == 0, x == -1]

returns True.

says it all.

> As for difference between theoretical analysis of the propagation of round-off error and practical scientific computations, I am well aware of it, but was it not you who gleefully quoted the statement:
>
> The technique of propagating the uncertainty from step to step
> throughout the calculation is a very bad technique. It might sometimes
> work for super-simple textbook problems but it is unlikely to work for
> real-world problems.

While you claim to understand the difference, you now show you don't.
The technique of replacing x*y by (x*(1+a)) * (y*(1+b))-->
z*(1+c) for
suitable a,b,c, for theoretical analysis of propagation of roundoff
as a function of a, b, is different from doing the actual
calculations with fuzz-balls.

> So if agree with this (as you seem to), are you implying that the entire chapter 16 of Henrici's book is concerned with something that only applies to "super-simple text book problems"?

I don't have a copy of his book handy. It is true that the
theoretical analysis becomes pretty hairy, e.g. for programs with
branches.

> The point is, of course, that you never make it clearly what exactly you are talking about probably because you find it convenient for rhetorical purposes (90% of these discussions are really just rhetorics).
>
> I should add that I myself long ago and more then once wrote on this forum that I do not think Mathematica's "approximate numbers" are a good model for empirical errors, but they are very good tools for solving certain purely mathematical problems. Often it can be accomplished in 100% provably correct way (barring bugs of course).
>
> As for things like numerical Groebner basis etc, you keep arguing that it probably can be done in some other way (without significance arithmetic) and in a certain, rather trivial, sense you undoubtedly right. But the fact is, as Daniel has pointed out, that, in practice, nobody seems to have done it in another way, and not for want of trying. So again here we have the distinction between "theory" and "practical application" but except that this time the shoe is on the other foot.

It seems to me that numerical computation of GB has a certain history.
E.g.
(see references)

and several PhD dissertations.

Given that most people messing with floating-point GB find themselves
facing stability problems,
the introduction of arbitrary precision numbers seems like a way out
of some difficulties. Most
people seem to concentrate on other ways though, because double-floats
are so much faster
than software floats.
It is nice that Dan has done some software-float implementation and
put it in Mathematica. I am not aware of
others who have tried this tactic and failed. Even you seem to agree
that Mathematica's particular kind of
bigfloat arithmetic is not essential to this. So I don't see your point.

<big snip>

RJF

### Daniel Lichtblau

Jul 16, 2011, 5:47:31 AM7/16/11
to
On 07/08/2011 03:50 AM, Richard Fateman wrote:
> On 7/7/2011 4:35 AM, Daniel Lichtblau wrote:
>> On Jul 5, 4:15 am, Richard Fateman<fate...@cs.berkeley.edu> wrote:
>>> In brief: yes to both.
>>>
>>> On 7/4/2011 4:18 AM, Kevin J. McCann wrote:
>>> ...

>>>
>>>
>>>
>>>>> 1. N[2.0,20] should give 2 with accuracy/precision/whatsever about 20
>>>>> decimal digits, i.e. 2.00000000000000000000
>>>
>>> WRI will defend this as a feature.
>>>
>>> You thought that the semantics of N[] were the same as SetPrecision
>>> e.g.
>>>
>>> N[SetPrecision[2.0, 20]*N[Sqrt, 20], 20]
>>>
>>> works as you expected.
>>>
>>> So from your perspective, and from the perspective of anyone else who
>>> thinks along the same lines, it is a bug. I would prefer to call it
>>> a design error.
>>>
>>> 2.0 (indeed, any floating point number) is a perfectly respectable way
>>> of denoting a number that can be expressed in higher precision by adding
>>> binary zeros. WRI doesn't agree.
>>>
>>> RJF
>>
>> I don't think this holds up for all floats in the sense you seem to
>> indicate. Take 2.1 for example.
>
> My contention is that 2.1, written that way, is a perfectly respectable
> way of denoting a number that can be expressed in higher precision by
> adding binary zeros. 2.1 is a double-float in most systems today.
> What is that number? In decimal, to higher precision, it looks like
> 2.10000000000000008881784197001....
>
> A program that decodes a float into its component fraction, exponent,
> and sign gives this information for the fraction:
> 4728779608739021 or in binary
> 10000110011001100110011001100110011001100110011001101
>
> That is the number to which we add binary zeros.
>
> Let us call this number P.

This is what SetPrecision will do. N will not do that.

If I recall correctly (unlikely, by now) the original point of
contention was that N[] did not also do this. That is by design. N[]
will not invent new digits, binary or otherwise. For that one requires
the more arcane SetPrecision.

>> We can (and will) use SetPrecision to
>> pad with binary zeroes, just as you use it above on 2.0.
>>
>> In:= N[SetPrecision[2.1, 20]*N[Sqrt, 20], 20]
>>
>> Out= 2.9698484809834997281
>>
>> But this is not an accurate representation of 21/10 * Sqrt to 20
>> places.
>
>
> Just because you are doing arithmetic on numbers represented to 20
> digits does not mean that the result of the arithmetic will be right to
> 20 digits, however, in the single operation of multiplication, you
> should not lose any digits. What's going on here?
>
> You simply computed two different quantities. Let S= sqrt(2) to 20
> decimal places. In line 606 you multiplied P*S and rounded it to 20
> digits [ actually you did something to the binary reps, but it doesn't
> matter here].

I was (and remain) fairly sure that your notion, applied to 2.1 rather
than 2.0, would be to do exactly this. I never claimed it was a good idea...

> In line 607, below, you multiplied 21/10 by S. Since P is not equal
> to 21/10, why should the results be the same? Oh, Mathematica thinks
> that P ==21/10, but that is merely the terribly broken notion of
> equality that Mathematica uses. We used to call that "close enough for
> government work". But it is terribly broken because == fails to
> satisfy the axioms of "equivalence relations" such as a==b and b==c
> implies a==c.

It satisfies trichotomy. That is to say, it plays nice with inequalities.

In any implementation of equality and inequality predicates that
involves approximate numbers, something from exact arithmetic must be
jettisoned. The choice we took might not be the one you would have
preferred. But it has its justifications.

I think we've discussed this before. (Recursively, I probably said this
the last time around.)

>> In:= N[21/10*Sqrt, 20]
>> Out= 2.9698484809834996025
>>
>> They are off by arould a machine epsilon. No surprise here, I think.
>>
>> In:= % - %%
>> Out= -1.256*10^-16
>>
>> I also think that this sort of numeric discrepancy is inevitable* in
>> any program that uses decimal input and binary representation.
>
> Absent any interior calculations, I would tend to agree. But there is
> something else going on. I think that Mathematica has embedded in it
> several design decisions that elevate the (typical)
> one-half-unit-in-last-place error in binary-to-decimal and
> decimal-to-binary conversion that happens in input or output, into a
> global slush bucket that infects and propagates into all interior
> computations with "software floats".

For input, I don't see that this slush bucket is any different than the
underpinnings of other approximate floating point arithmetic. For output
purposes it seems irrelevant. Maybe I'm misunderstanding the claim you
are making.

Daniel Lichtblau
Wolfram Research

### Richard Fateman

Jul 16, 2011, 5:42:54 AM7/16/11
to
(in a different thread, where replacing Equal by SameQ was proposed)

On 7/15/2011 2:07 AM, Andrzej Kozlowski wrote:
> Well, here is one example of what would happen:
>
> In:= Unprotect[Equal]
>
> Out= {Equal}
>
> In:= Equal[a_, b__] := SameQ[a, b]
>
> In:= Protect[Equal]
>
> Out= {Equal}
>
> In:= Solve[3 x == 1, x]
>
> Out= {}
>
> Doesn't look like a great idea to me.
>
> Andrzej Kozlowski
>

That's because 3*x==1 is immediately changed to False, and
Solve[False, x] is indeed not a great idea.

Equal[a_?NumberQ, b_?NumberQ] := SameQ[a, b];

works better. That Solve example works just fine.

Indeed, if one does this..

Equal[a_?NumberQ, b_?NumberQ] := (Print[test[a, b]]; SameQ[a, b]);

one can detect when, if ever, the Solve program calls Equal on two
numbers. It will print test[...] on such occasions.

When does Solve call Equal on 2 numbers? I poked around a little and
found -- never. I tried Integrate, Factor, Do, Product, Plot.. --never.

note that f[x_] := If [x == 0, 1, f[x - 1]*x]; f computes 5! and
calls Equal 6 times... but Factorial does not.

Perhaps the internal support routines do not call Equal?
or calls to it are compiled away and not in reach of redefinition by the
user? Or do they call SameQ? (no, they don't. I tried it).

Cheers

RJF

### Andrzej Kozlowski

Jul 16, 2011, 5:40:49 AM7/16/11
to
I should have added that actually Mathematica's significance arithmetic
is considerably more sophisticated than the description as a "first
order" approximation suggests. This is also explained by Sofroniou and
Spaletta:

Numerical algorithms for computing elementary functions can be written
in terms of addition and multiplication at some level. However, relying
on the error propagation rules for these operations would often give
very pessimistic error bounds in significance arithmetic. Much tighter
bounds can be obtained by directly imposing error estimates based on
properties of functions during their numerical computation.

In other words, Mathematica uses special estimates for a variety of
functions, based on properties of these functions and thus obtains bounds
are considerably tighter than a simple minded approach would produce.

Andrzej Kozlowski

On 15 Jul 2011, at 10:09, Andrzej Kozlowski wrote:

> You are clearly confusing significant digit arithmetic, which is not
> what Mathematica uses, with significance arithmetic, which is a first
> order approximation to interval arithmetic or distribution based

> approach. Obviously you don't read the posts you reply to and confuse
> both the posters and the contents of what they post. Here is a quote
> from Oleksandr Rasputionov that makes this completely clear:
>
>
> Unfortunately so, given that it is severely erroneous: see e.g.
> <http://www.av8n.com/physics/uncertainty.htm>. However, Mathematica's

> approximation of how these uncertainties propagate is first-order, not
> zeroth-order. This does not make it completely reliable, of course, but
> certainly it is not almost always wrong as is the significant digits
> convention. Within the bounds of its own applicability, Mathematica's
> approximation is reasonable, although it would still be a mistake to apply
> it to experimental uncertainty analysis given the much broader scope of
> the latter.
>

> Note the "first-order not zeroth-order". Also, do take a look at Sofroniou and Spaletta and then you may perhaps understand what "order" means and how "significance arithmetic" differs from the "significant digits" convention. Good grief, did you ever learn about the Taylor series? It must have been a long time ago, I take.

>
>
> Andrzej Kozlowski
>
>
> On 14 Jul 2011, at 19:55, Richard Fateman wrote:
>
>> On 7/14/2011 6:27 AM, Andrzej Kozlowski wrote:
>>> On 14 Jul 2011, at 11:21, Richard Fateman wrote:
>>>
>>>> On 7/13/2011 12:11 AM, Noqsi wrote:
>>>> ..
>>>>

>>>>>> see e.g.
>>>>>> <http://www.av8n.com/physics/uncertainty.htm>.
>>>>>

>>>> ..
>>>> Learning mathematics from a physicist is hazardous.
>>>> Learning computer science from a physicist is hazardous too.

>>>> Numbers in a computer are different from experimental measurements.

>>>>
>>>>
>>>> nevertheless, I like this article. It says, among other things,
>>>>

>>>> The technique of propagating the uncertainty from step to step
>>>> throughout the calculation is a very bad technique. It might sometimes
>>>> work for super-simple textbook problems but it is unlikely to work for
>>>> real-world problems.
>>>>

>>> Well, here is a quote from a very well known book on numerical

> analysis by a mathematician (Henrici, "Elements of Numerical Analysis").
>>>
>>>

>>>> It is plain that, on a given machine and for a given problem, the local
>>>> rounding errors are not, in fact, random variables. If the same problem
>>>> is run on the same machine a number of times, there will result always the
>>>> same local rounding errors, and therefore also the same accumulated
>>>> error. We may, however, adopt a stochastic model of the propagation of
>>>> rounding error, where the local errors are treated as if they were random
>>>> variables. This stochastic model has been applied in the literature to a
>>>> number of different numerical problems and has produced results that are
>>>> in complete agreement with experimentally observed results in several
>>>> important cases.
>>>
>>> The book then describes the statistical method of error propagation of which Mathematica's approach can be regarded as a first order approximation (as pointed out by Oleksandr Rasputinov, who should not be confused with the OP of this thread so:
>>
>> So are we to conclude that Henrici recommends this as a general numerical computational method?
>>
>> I don't see that here. I think what he is saying is that if you do some mathematics (see below), then
>> you will get results consistent with what you will get if you actually run the experiment on the computer.
>> This is not surprising. It is a result that says that "theoretical" numerical analysis agrees with
>> "computer experiments" in arithmetic. It doesn't say Henrici recommends running a computation this way.
>>
>> When Henrici says "adopt a stochastic model ...." he doesn't mean to write a program. He means to
>> think about each operation like this.. (I show for multiplication of numbers P and Q with errors a b resp.)
>>
>> P*(1+a) times Q*(1+b) P*Q *(1+a)*(1+b)* (1+c) where c is a new "error" bounded by roundoff, e.g. half unit in last place.
>>
>> For each operation in the calculation, make up another error letter... a,b,c,d,e,f,g...
>> assume they are uncorrelated.
>>
>> The fact that this theoretical approach and numerically running "several important cases" is a statement
>> about correlation of roundoff in these cases, not a statement of advisability of whatever for a model of

>> how to write a computer system.
>>

>> By the way, I think that Henrici was an extremely fine theoretical numerical analyst, and a fine writer too.
>>
>>
>>
>>>> Clearly Rasputinov thinks that if they are not equal they should not be

>>>> Equal. Thus the answer is False.

>>> is *clearly* False. In fact Oleksander expressed something closer to the opposite view.)
>> This thread is too long. I don't know at this point if you are agreeing that it is false or contradicting that "is False" is false.
>>> And if one quote is not enough, here is another, from another text on numerical analysis. (Conte, de Boor, ""Elementary Numerical Analysis).
>>> It describes 4 approaches to error analysis, interval arithmetic, significant-digit arithmetic, the "statistical approach" and backward error analysis. Here is what it says about the second and the third one:
>> Huh, if we are talking about the second and third one, why does he say third and fourth?
>> Are you using 0-based indexing and deBoor is using 1-based indexing????
>>
>>
>>>> A third approach is significant-digit arithmetic. As pointed out earlier, whenever two nearly equal machine numbers are subtracted, there is a danger that some significant digits will be lost. In significant-digit arithmetic an attempt is made to keep track of digits so lost. In one version
>>>> only the significant digits in any number are retained, all others being discarded. At the end of a computation we will thus be assured that all digits retained are significant. The main objection to this method is that some information is lost whenever digits are discarded, and that the results obtained are likely to be much too conservative. Experimentation with this technique is still going on, although the experience to date is not too promising.
>>>>
>> OK, so deBoor (who is retired and therefore not likely to revise the "experience to date") says this method "is not too promising".

>> This sounds to me like he is not endorsing what Mathematica does.
>>

>>>> A fourth method which gives considerable promise of providing an adequate mathematical theory of round-off-error propagation is based on a statistical approach. It begins with the assumption that round-off errors are independent. This assumption is, of course, not valid, because if the same problem is run on the same machine several times, the answers will always be the same. We can, however, adopt a stochastic model of the propagation of round-off errors in which the local errors are treated as if they were random variables. Thus we can assume that the local round-off errors are either uniformly or normally distributed between their extreme values. Using statistical methods, we can then obtain the standard devia- tion, the variance of distribution, and estimates of the accumulated round- off error. The statistical approach is considered in some detail by Ham- ming  and Henrici . The method does involve substantial analysis and additional computer time, but in the ex!
> periments conducted to date it has obtained error estimates which are in remarkable agreement with experimentally available evidence.
>> deBoor is essentially quoting Henrici, and this statistical approach is to say that all those error terms I mentioned above, a,b,c,d,e,f,...
>> can be chosen from some distribution (the way I've written it, a ,....,z .... would essentially be chosen from {-u,u} where u == 2^(-W) where the fraction part of the floating-point number is W bits. ) and you can compute the final expression as ANSWER+<somehorrendousfunctionof>(a,b,c,....).
>>
>> What deBoor says is that this (theoretical numerical analysis) "method" promises to provide
>> a theory of round-off error propagation. He is not saying this is a practical method for scientific computing. When he uses the work "method"

>> he means a mathematical method for analyzing roundoff.
>>

>> In any case, Mathematica does not do this. I would further argue that Mathematica makes it hard to carry out the experiments that might be done to demonstrate that this theory applied in any particular sample computation.
>>>
>>> The fundamental paper of Mathematica's error propagation is "Precise numerical computation" by Mark Sofroniou and Giulia Spaletta in The Journal of Logic and Algebraic Programming 64 (2005) 1139=6134. This paper describes Mathematica's "significance arithmetic" as a first order approximation to Interval Arithmetic. It makes no mention of distributions.
>> I thought I read this paper in some Mathematica documentation or conference proceedings.
>>> Oleksandr Rasputionov, in an earlier post here, interpreted "significance arithmetic" as a first order approximation to the fourth method above.
>> Huh? First of all, the original poster was slawek. Rasputinov seems to think that Mathematica numbers are like Intervals (basically a good intuition until you think about equality.) and refers to them as distributions. This is not deBoor's 4th "method" of theoretically analyzing round-off.
>> In fact it is deBoor's 1st method, interval arithmetic. This has the advantage of being maybe 4 to 8 times slower than regular arithmetic, and also has a huge literature (see "Reliable Computation") describing variations, advantages, disadvantages, etc.
>>
>>> I have not considered this very carefully, but it seems pretty clear that he is right, and that the two "first order" approximations are in fact isomorphic.
>> Uh, this is unclear, unless you mean that Mathematica's number system is essentially interval arithmetic, but with a confusing front end.
>>> The first order approach is, of course, justified on grounds of performance. It is perfectly "rigorous" in the same sense as any "first order" approach is (i.e. taking a linear approximation to the Taylor series of a non-linear function). It works fine under certain conditions and will produce nonsense when these conditions do not hold.
>>>
>>> The fact that significance arithmetic is "useful" needs no justification other than the fact that it is used successfully by NSolve and Reduce in achieving validated symbolic results by numerical methods which are vastly faster than purely symbolic ones.
>> Really?
>>
>> 1. This does not mean that other methods, e.g. validated methods for accurately evaluating polynomials (etc) NOT based on significance arithmetic would not be faster and better. DanL claims it is used and useful there, so that's nice. BUT..
>>
>> 2. This does not mean that this arithmetic should be used by default by user arithmetic.
>>> It is also useful, for users such as myself, who sometimes need fast first order error anlysis.
>> OK, If you find it useful yourself, fine. I daresay you are not a typical user.
>>
>>> I have lots of posts by Richard on this topic (or perhaps it was the same post lots of time, it's so hard to tell), but I have never understood what his main point is.
>>
>> Main points: Mathematica's number system is non-standard, peculiar, hard to explain, capable of returning mysterious non-answers without indications of error to innocent users, a bad foundation for building higher-level functionality.
>>
>> I have other criticisms of Mathematica, but I think that the sentence above is enough for you to process today.
>>
>>> It seems to me that is because he himself has not yet decided this, although he has been posting on this topic for over 20 years (I think).
>>
>>> Sometimes he seems to be disparaging significance arithmetic itself.

>> I think deBoor did that.
>>

>>> When Daniel points out how effective it is in his implementation of numerical Groebner basis, or in Adam Strzebonski's work on Reduce, he either ignores this altogether or claims that Groebner bases, etc. are themselves not "useful".
>> I think Reduce is a very nice program when it works. If I am not mistaken, all work on numerical Groebner basis should be reducible to the evaluation of polynomials, for which there are faster and more accurate methods available not using significance arithmetic. On the other hand, I might be mischaracterizing DanL work, since I admit to not having studied it.
>>
>>
>>> On other occasions he takes on the role of the defender of the interest of the "naive user" (presumably like the OP, who however would be better described as "intentionally naive") who is going to be confused by the "quirky" nature of significance arithmetic (at low =

>> I have no problem with Mathematica providing as an option, some other =
arithmetic. It is WRI that has made it rather hard for the user to

>> figure out how to "use the arithmetic of the rest of the world".
>>
>>> Finally, perhaps all that Richard is so upset about is simply =
Mathematica's habit of defining numbers as "fuzz balls" or =
"distributions".

>> I'm not sure that "upset" is the right term. After all, I don't have =

### Noqsi

Jul 16, 2011, 5:41:52 AM7/16/11
to
On Jul 15, 1:10 am, Richard Fateman <fate...@cs.berkeley.edu> wrote:

> > In extracting useful results from computers in the real world
> > ignorance of the application domain is more debilitating than
> > ignorance of computer science. One can muddle through the latter, but
> > not the former.
>
> I think you may be confusing "computer science" with "applications of
> computers to whatever [e.g. physics, engineering, business]" which is
> usually taught in the department of 'whatever'.

So, in your view, the substance is all in the departments of
'whatever'? Because without applications, what's left? Computers are
not natural objects: they are artificial, so without applications the
whole field is just a made-up story. A true science needs the
discipline of relating to *something* in the real world. You're
evading this discipline. Your view makes computer science
indistinguishable from religious scholasticism.

> > Consider the howler you yourself committed a little while ago,
> > discussing the role of computers in the Apollo moon landings:
>

> >> you've got to wonder how the Russians, probably WITHOUT much in the wa=

y
> >> of computers, put up an artificial satellite.
>
> > This illustrates a profound ignorance of the problem. If you don't
> > care what orbit you go into, the computations are readily performed by
> > slide rule ahead of time. But Apollo had to perform a series of
> > precise maneuvers using imprecise rockets, with repeated re-
> > computation of the trajectories. For the early missions, they didn't
> > even know the lunar gravity very well, so actual orbits diverged
> > rapidly from predictions even when the initial conditions were known.
> > Apollo's indirect "lunar orbital rendezvous" approach was thus a
> > triumph of computation. Even the Saturn V was not big enough to
> > support the less computationally intensive direct approach.
>
> Do you know for a fact that the Russians didn't care what orbit the
> first manned satellite had?

I know the consequences of getting it wrong. Basically, they knew if
they got the perigee high enough to go around once, nothing really bad
could result from the other orbital parameters. The rocket wasn't
powerful enough to do something silly like put Gagarin into an escape
trajectory. The limitations of the rocket combined with very basic
orbital mechanics guaranteed that about an hour and a half after
launch, the spacecraft would return to a point near overhead to where
the launch site had been. Consider the rotation of the Earth, and then
all they had to do was fire the retrorocket in roughly the right
direction at roughly the right time and Gagarin was guaranteed to come
down about 23 degrees west of where he was launched. The Soviet Union
was a huge place: they didn't need to do this accurately at all.

Satisfying a single inequality is enormously easier than rendezvous in
orbit around a body with poorly known gravity, where you must satisfy
six equations to high precision.

> I think you are confusing application knowledge with computer science.

Without applications, computer science is vacuous.

> Of course there are many methods that can be programmed. You are
> assuming that analysis of signals is done by some kind of particle
> tracking? I assume that programs are designed by persons familiar with
> differential equations and electromagnetic radiation, as well as more
> seat-of-the-pants stuff like antenna design and sun spots.
>
> Also I assume
> that cell phones use signal strength and feedback, and do not need great
> accuracy.

How do you determine how large to make the transistors when
manufacturing the cell phone? This requires calculation. How would you
do that calculation?

> Though maybe GPS stuff is tricky if you have few
> triangulation points. Using 10 points, maybe not so tricky. Not
> something I've cared to look at.

If von Neumann was alive, he'd understand the issues completely before
you'd even finished describing the problem. He was a *real* computer
scientist.

> > There is no universal method for tracking uncertainty that is accurate
> > and practical.
>
> Ah, so you are saying that Mathematica is not accurate and practical??

Not universally. However, given a specific problem, it is often the
tool of choice.

> I am primarily interested in building systems appropriate for a range of
> applications. (That's more computer science).

Without knowledge of applications, you have no foundation to stand on
here. And that's Wolfram's advantage: he and his people *do*
understand a very wide range of applications.

> From that perspective I
> think that Mathematica falls short.

Since many of us find Mathematica a very effective tool in real
applications, this judgement is obviously based on nothing but
ideology. You have repeatedly demonstrated your complete lack of any
useful perspective here, and your unwillingness to do the necessary
studying to gain that perspective. Instead, you carefully define
"computer science" in a way that excuses you from studying anything
you don't wish to study.

> I don't see that as a threat, but I
> am inclined to object to statements that claim (in my view incorrectly)
> that Mathematica (arithmetically speaking) is the best, or even the only
> way to do floating-point calculations.

There is no best. It depends on what you're doing. Mathematica is very
effective over a wide range of applications. It is not the right tool
for every application. But you need the application knowledge to
understand this.