4 views

Skip to first unread message

Aug 4, 2010, 6:49:47 AM8/4/10

to sage-devel

Although there is not a fully functioning 64-bit SPARC build of Sage, there is a

sufficiently functioning build to run doctests. One failure observed by John

Palmieri

sufficiently functioning build to run doctests. One failure observed by John

Palmieri

File

"/home/palmieri/fulvia/sage-4.5.2.rc0/devel/sage-main/sage/symbolic/expression.pyx",

line 508\

8:

sage: maxima('asinh(1.0)')

Expected:

0.881373587019543

Got:

.8813735870195429

Clearly there is some numerical noise issues, which need to be investigated, but

are probably trivial. But what is odd is that the "Got" value lacks the leading zero

See

http://trac.sagemath.org/sage_trac/ticket/9099#comment:5

Has anyone seen this sort of problem before, where a leading zero is missing?

Any ideas of a likely cause?

Dave

Aug 5, 2010, 11:41:07 AM8/5/10

to Dr. David Kirkby

Hello, David

You wrote 4 августа 2010 г., 14:49:47:

> 8:

> sage: maxima('asinh(1.0)')

> Expected:

> 0.881373587019543

> Got:

> .8813735870195429

> Clearly there is some numerical noise issues, which need to be

> investigated, but are probably trivial.

I think that it is more than trivial issues. You can't require maxima

or any other numerical program to return same results up to 16th digit

with every combination of CPU, compiler, optimization settings.

You can get 0,.......543 when you compile with -O1, and 0,.......999

when you compile with -O2. Well, you can try to make results same up

to the last bit, but it is hard to achieve even with optimization

turned off.

The right move is to compare results using only 10-14 leading digits

(depending on problem and its stability properties). However I don't

know how to do it using doctest framework. Python tries to output

numbers with full precision, and there is no way to tell doctest

framework to compare decimal fractions using only N leading digits.

--

With best regards,

Sergey mailto:sergey.b...@alglib.net

Aug 5, 2010, 12:17:11 PM8/5/10

to sage-...@googlegroups.com

On 8/5/10 8:41 AM, Sergey Bochkanov wrote:

> Hello, David

>

> Hello, David

>

> You wrote 4 О©╫О©╫О©╫О©╫О©╫О©╫О©╫ 2010 О©╫., 14:49:47:

>> 8:

>> sage: maxima('asinh(1.0)')

>> Expected:

>> 0.881373587019543

>> Got:

>> .8813735870195429

>> Clearly there is some numerical noise issues, which need to be

>> investigated, but are probably trivial.

>

> I think that it is more than trivial issues. You can't require maxima

> or any other numerical program to return same results up to 16th digit

> with every combination of CPU, compiler, optimization settings.

>

> You can get 0,.......543 when you compile with -O1, and 0,.......999

> when you compile with -O2. Well, you can try to make results same up

> to the last bit, but it is hard to achieve even with optimization

> turned off.

>

> The right move is to compare results using only 10-14 leading digits

> (depending on problem and its stability properties). However I don't

> know how to do it using doctest framework. Python tries to output

> numbers with full precision,and there is no way to tell doctest>> 8:

>> sage: maxima('asinh(1.0)')

>> Expected:

>> 0.881373587019543

>> Got:

>> .8813735870195429

>> Clearly there is some numerical noise issues, which need to be

>> investigated, but are probably trivial.

>

> I think that it is more than trivial issues. You can't require maxima

> or any other numerical program to return same results up to 16th digit

> with every combination of CPU, compiler, optimization settings.

>

> You can get 0,.......543 when you compile with -O1, and 0,.......999

> when you compile with -O2. Well, you can try to make results same up

> to the last bit, but it is hard to achieve even with optimization

> turned off.

>

> The right move is to compare results using only 10-14 leading digits

> (depending on problem and its stability properties). However I don't

> know how to do it using doctest framework. Python tries to output

> framework to compare decimal fractions using only N leading digits.

>

>

Yes, there is. The doctest below will compare only the digits listed:

sage: maxima('asinh(1.0)')

0.88137...

If you look through the Sage library, you'll see lots of places where

we use the triple-dot "wildcard" to ignore the last few digits of a

numerical answer.

Thanks,

Jason

Aug 5, 2010, 12:39:46 PM8/5/10

to Jason Grout

Hello, Jason.

You wrote 5 августа 2010 г., 20:17:11:

>> (depending on problem and its stability properties). However I don't

>> know how to do it using doctest framework. Python tries to output

>> numbers with full precision,and there is no way to tell doctest

>> framework to compare decimal fractions using only N leading digits.

> Yes, there is. The doctest below will compare only the digits listed:

> sage: maxima('asinh(1.0)')

> 0.88137...

Thanks! I didn't noticed this ELLIPSIS option while reading doctest's

documentation. Everything becomes a bit simpler now :)

Aug 5, 2010, 12:44:53 PM8/5/10

to sage-...@googlegroups.com

2010/8/5 Sergey Bochkanov <sergey.b...@alglib.net>:

Yes, that's what I mean by trivial. But the missing leading zero is

less trivial.

With all the numerical noise issues I've seen in Sage, the three dots

solves its. So if we expect

1.00000000000000

but get

1.00000000000001

we can change that to

1.0000000000000...

and the test will pass.

However, what if we get

0.99999999999999

That's very close to 1, but not a single digit is the same. I'm not

sure how one would handle that case.

Dave

Aug 5, 2010, 12:57:47 PM8/5/10

to sage-devel

On Aug 5, 6:44 pm, David Kirkby <david.kir...@onetel.net> wrote:

> 2010/8/5 Sergey Bochkanov <sergey.bochka...@alglib.net>:

doing the comparison in this case

(alternatively, one might add certain \epssilon>0 to shift the thing

above 1.0)

Dima

>

> Dave

> 2010/8/5 Sergey Bochkanov <sergey.bochka...@alglib.net>:

> > Hello, Jason.

>

> > You wrote 5 августа 2010 г., 20:17:11:

> >>> (depending on problem and its stability properties). However I don't

> >>> know how to do it using doctest framework. Python tries to output

> >>> numbers with full precision,and there is no way to tell doctest

> >>> framework to compare decimal fractions using only N leading digits.

>

> >> Yes, there is. The doctest below will compare only the digits listed:

> >> sage: maxima('asinh(1.0)')

> >> 0.88137...

>

> > Thanks! I didn't noticed this ELLIPSIS option while reading doctest's

> > documentation. Everything becomes a bit simpler now :)

>

> > --

> > With best regards,

> > Sergey mailto:sergey.bochka...@alglib.net
>

> > You wrote 5 августа 2010 г., 20:17:11:

> >>> (depending on problem and its stability properties). However I don't

> >>> know how to do it using doctest framework. Python tries to output

> >>> numbers with full precision,and there is no way to tell doctest

> >>> framework to compare decimal fractions using only N leading digits.

>

> >> Yes, there is. The doctest below will compare only the digits listed:

> >> sage: maxima('asinh(1.0)')

> >> 0.88137...

>

> > Thanks! I didn't noticed this ELLIPSIS option while reading doctest's

> > documentation. Everything becomes a bit simpler now :)

>

> > --

> > With best regards,

>

> Yes, that's what I mean by trivial. But the missing leading zero is

> less trivial.

>

> With all the numerical noise issues I've seen in Sage, the three dots

> solves its. So if we expect

>

> 1.00000000000000

> but get

> 1.00000000000001

> we can change that to

> 1.0000000000000...

> and the test will pass.

>

> However, what if we get

>

> 0.99999999999999

>

> That's very close to 1, but not a single digit is the same. I'm not

> sure how one would handle that case.

well, one needs to call a function to do a proper rounding before
> Yes, that's what I mean by trivial. But the missing leading zero is

> less trivial.

>

> With all the numerical noise issues I've seen in Sage, the three dots

> solves its. So if we expect

>

> 1.00000000000000

> but get

> 1.00000000000001

> we can change that to

> 1.0000000000000...

> and the test will pass.

>

> However, what if we get

>

> 0.99999999999999

>

> That's very close to 1, but not a single digit is the same. I'm not

> sure how one would handle that case.

doing the comparison in this case

(alternatively, one might add certain \epssilon>0 to shift the thing

above 1.0)

Dima

>

> Dave

Aug 5, 2010, 1:01:13 PM8/5/10

to David Kirkby

Hello, David.

You wrote 5 августа 2010 г., 20:44:53:

> With all the numerical noise issues I've seen in Sage, the three dots

> solves its. So if we expect

> 1.00000000000000

> but get

> 1.00000000000001

> we can change that to

> 1.0000000000000...

> and the test will pass.

> However, what if we get

> 0.99999999999999

> That's very close to 1, but not a single digit is the same. I'm not

> sure how one would handle that case.

Hmmm... Didn't thought about this situation yet. Definitely we can't

solve this problem with any kind of regular expressions. One possible

solution is to round data before printing. So both 1.00000000000001

and 0.99999999999999 will become 1.000000.

As for me (testing interaction between ALGLIB and Sage), I can write

function which prints arrays/matrices with rounding up to specified

number of digits. Such doctests will look like

> sage: a = sqrt(2)

> sage: my_own_print(a,4)

> 1.414

But it will be more test than doc (less human readable). Don't know

whether it is worth using beyond Sage wrapper for ALGLIB.

Aug 5, 2010, 1:17:42 PM8/5/10

to Sergey Bochkanov

Hello,

> Hmmm... Didn't thought about this situation yet. Definitely we can't

> solve this problem with any kind of regular expressions. One

> possible solution is to round data before printing. So both

> 1.00000000000001 and 0.99999999999999 will become 1.000000.

...however, we still can have problems when rounding X=0.000499999 up

to three digits. With original X we will have 0.000. But with

perturbation as small as 0.000000002 we will round to 0.001.

My automated Python-ALGLIB tests just calculate difference between

desired and actual answers, but they are not doctests; they are just

automatically generated Python scripts. We can't calculate differences

with doctests.

Aug 5, 2010, 1:23:05 PM8/5/10

to sage-...@googlegroups.com

Thank you. That makes perfect sense.

BTW, do you have any ideas why the second failure at #9099 might occur

sage -t -long devel/sage/sage/symbolic/expression.pyx

**********************************************************************

File "/home/palmieri/fulvia/sage-4.5.2.rc0/devel/sage-main/sage/symbolic/expression.pyx",

line 498\

3:

sage: maxima('sinh(1.0)')

Expected:

1.175201193643801

Got:

1.175201193643802

**********************************************************************

File "/home/palmieri/fulvia/sage-4.5.2.rc0/devel/sage-main/sage/symbolic/expression.pyx",

line 508\

8:

sage: maxima('asinh(1.0)')

Expected:

0.881373587019543

Got:

.8813735870195429

**********************************************************************

Why should the zero be missing in the case observed on Solaris x86?i

In other words, why do we see .8813735870195429 instead of

0.8813735870195429 ? Te exact value of the last digit is unimportant,

and one clearly can't expect to get all digits spot on, but one would

hope that a zero should be printed when its needed.

Dave

Aug 5, 2010, 1:45:53 PM8/5/10

to sage-...@googlegroups.com

On Thu, Aug 5, 2010 at 9:44 AM, David Kirkby <david....@onetel.net> wrote:

> With all the numerical noise issues I've seen in Sage, the three dots

> solves its. So if we expect

>

> 1.00000000000000

> but get

> 1.00000000000001

> we can change that to

> 1.0000000000000...

> and the test will pass.

>

> However, what if we get

>

> 0.99999999999999

>

> That's very close to 1, but not a single digit is the same. I'm not

> sure how one would handle that case.

> With all the numerical noise issues I've seen in Sage, the three dots

> solves its. So if we expect

>

> 1.00000000000000

> but get

> 1.00000000000001

> we can change that to

> 1.0000000000000...

> and the test will pass.

>

> However, what if we get

>

> 0.99999999999999

>

> That's very close to 1, but not a single digit is the same. I'm not

> sure how one would handle that case.

I've seen a couple of approaches used in this case.

First, simply change the doctest. For example, if you're testing a

numerical root-finder on the polynomial x^2-x, change the polynomial

to x^2-x-1.

Second, include the test code in the doctest. Change:

sage: foo()

1.0000000000000

to:

sage: abs(foo() - 1) < 1e-12

True

This is less preferred, because while it keeps the doctest useful as a

test, it can reduce its value as documentation (especially if the test

code is more complicated than the above).

Carl

Aug 5, 2010, 1:55:23 PM8/5/10

to sage-...@googlegroups.com

Well, it seems likely that the issue is within maxima (either in

maxima itself or in the Common Lisp implementation). To verify that

the problem has nothing to do with Sage, you can use "sage -maxima"

and try the test case on both machines:

cwitty@red-spider:~/sage$ ./sage -maxima

;;; Loading #P"/home/cwitty/sage/local/lib/ecl/defsystem.fas"

;;; Loading #P"/home/cwitty/sage/local/lib/ecl/cmp.fas"

;;; Loading #P"/home/cwitty/sage/local/lib/ecl/sysfun.lsp"

Maxima 5.20.1 http://maxima.sourceforge.net

using Lisp ECL 10.2.1

Distributed under the GNU Public License. See the file COPYING.

Dedicated to the memory of William Schelter.

The function bug_report() provides bug reporting information.

(%i1) asinh(1.0);

(%o1) 0.881373587019543

(Don't forget the semicolon on the "asinh(1.0);" line; without that,

maxima will wait forever for more input.)

If the results do differ with "sage -maxima", I guess the next step

would be to report the problem to the maxima mailing list or the bug

tracker (bearing in mind that the problem may actually be in the

Common Lisp implementation we use).

Carl

Aug 5, 2010, 5:59:25 PM8/5/10

to sage-...@googlegroups.com

On 08/ 5/10 06:17 PM, Sergey Bochkanov wrote:

> Hello,

>

>> Hmmm... Didn't thought about this situation yet. Definitely we can't

>> solve this problem with any kind of regular expressions. One

>> possible solution is to round data before printing. So both

>> 1.00000000000001 and 0.99999999999999 will become 1.000000.

>

> ...however, we still can have problems when rounding X=0.000499999 up

> to three digits. With original X we will have 0.000. But with

> perturbation as small as 0.000000002 we will round to 0.001.

> Hello,

>

>> Hmmm... Didn't thought about this situation yet. Definitely we can't

>> solve this problem with any kind of regular expressions. One

>> possible solution is to round data before printing. So both

>> 1.00000000000001 and 0.99999999999999 will become 1.000000.

>

> ...however, we still can have problems when rounding X=0.000499999 up

> to three digits. With original X we will have 0.000. But with

> perturbation as small as 0.000000002 we will round to 0.001.

I've never done this, but the most logical thing to me seems to be to look at

the absolute magnitude of the relative error, for all cases except when the

expected value is 0. But I guess on cases where the numbers are not close to 1,

or other special cases one could think up, the current system is probably the

simplest.

Dave

Aug 5, 2010, 8:07:05 PM8/5/10

to sage-...@googlegroups.com, sage-s...@googlegroups.com

Carl, thank you for your input.

I tried what you suggested, and see the result you show above on the Solaris 10

SPARC system, and a similar result, but lacking the leading zero, on the Solaris

10 x86 system.

I've stuck the full outputs at

http://trac.sagemath.org/sage_trac/ticket/9693

and have emailed both the Maxima and ECL mailing lists.

Unfortunately, we are not running the latest versions of either ECL or Maxima,

so I expect I'm likely to be asked "do you see this is the latest version?"

IIRC, there were some problems when we tried to update ECL and Maxima recently,

so its not as simple as one might like to determine this.

Dave

Aug 5, 2010, 10:13:51 PM8/5/10

to sage-...@googlegroups.com

That would be nice. But it's a bit tricky to write, since the doctest

check happens solely on the basis of string comparison; to do

something clever like looking at relative error would require some

sort of marking in the expected output to turn on the relative-error

check, and some sort of parser to find the numbers in the

expected/found outputs, and possibly some sort of marking on a

per-number basis in the expected output to say what relative error is

allowed on that number. (If the expected output is 1.23456*x^10000,

with a relative error allowed of 1e-3, you probably don't want to

allow 1.23456*x^10001 !)

I would be in favor of something like this, if we could decide on a

set of markings in the expected output that didn't interfere too much

with the documentation aspect; but nobody has written it yet.

The three-dots notation is much, much stupider than this... it is just

a wildcard that will match an arbitrary string of characters (the

equivalent of .* as a regex, or * as shell globbing). And we didn't

even write it; it's a standard part of the Python doctest system.

Carl

Aug 6, 2010, 12:02:46 AM8/6/10

to sage-...@googlegroups.com

And the three dots is much prettier to read too.

Personally, I just choose doctests whose results are not subject to

the 1.0000000000 vs. 0.99999999999 kind of noise, unless I want to

make a point that the function nails the result right on (in which

case 1.000000000 is a good doctests for that).

- Robert

Reply all

Reply to author

Forward

0 new messages

Search

Clear search

Close search

Google apps

Main menu