doctest failure with leading zero not printing

4 views
Skip to first unread message

Dr. David Kirkby

unread,
Aug 4, 2010, 6:49:47 AM8/4/10
to sage-devel
Although there is not a fully functioning 64-bit SPARC build of Sage, there is a
sufficiently functioning build to run doctests. One failure observed by John
Palmieri


File
"/home/palmieri/fulvia/sage-4.5.2.rc0/devel/sage-main/sage/symbolic/expression.pyx",
line 508\
8:
sage: maxima('asinh(1.0)')
Expected:
0.881373587019543
Got:
.8813735870195429


Clearly there is some numerical noise issues, which need to be investigated, but
are probably trivial. But what is odd is that the "Got" value lacks the leading zero

See
http://trac.sagemath.org/sage_trac/ticket/9099#comment:5

Has anyone seen this sort of problem before, where a leading zero is missing?
Any ideas of a likely cause?

Dave

Sergey Bochkanov

unread,
Aug 5, 2010, 11:41:07 AM8/5/10
to Dr. David Kirkby
Hello, David

You wrote 4 августа 2010 г., 14:49:47:
> 8:
> sage: maxima('asinh(1.0)')
> Expected:
> 0.881373587019543
> Got:
> .8813735870195429
> Clearly there is some numerical noise issues, which need to be
> investigated, but are probably trivial.

I think that it is more than trivial issues. You can't require maxima
or any other numerical program to return same results up to 16th digit
with every combination of CPU, compiler, optimization settings.

You can get 0,.......543 when you compile with -O1, and 0,.......999
when you compile with -O2. Well, you can try to make results same up
to the last bit, but it is hard to achieve even with optimization
turned off.

The right move is to compare results using only 10-14 leading digits
(depending on problem and its stability properties). However I don't
know how to do it using doctest framework. Python tries to output
numbers with full precision, and there is no way to tell doctest
framework to compare decimal fractions using only N leading digits.


--
With best regards,
Sergey mailto:sergey.b...@alglib.net

Jason Grout

unread,
Aug 5, 2010, 12:17:11 PM8/5/10
to sage-...@googlegroups.com
On 8/5/10 8:41 AM, Sergey Bochkanov wrote:
> Hello, David
>
> You wrote 4 О©╫О©╫О©╫О©╫О©╫О©╫О©╫ 2010 О©╫., 14:49:47:
>> 8:
>> sage: maxima('asinh(1.0)')
>> Expected:
>> 0.881373587019543
>> Got:
>> .8813735870195429
>> Clearly there is some numerical noise issues, which need to be
>> investigated, but are probably trivial.
>
> I think that it is more than trivial issues. You can't require maxima
> or any other numerical program to return same results up to 16th digit
> with every combination of CPU, compiler, optimization settings.
>
> You can get 0,.......543 when you compile with -O1, and 0,.......999
> when you compile with -O2. Well, you can try to make results same up
> to the last bit, but it is hard to achieve even with optimization
> turned off.
>
> The right move is to compare results using only 10-14 leading digits
> (depending on problem and its stability properties). However I don't
> know how to do it using doctest framework. Python tries to output
> numbers with full precision,and there is no way to tell doctest

> framework to compare decimal fractions using only N leading digits.
>
>

Yes, there is. The doctest below will compare only the digits listed:

sage: maxima('asinh(1.0)')
0.88137...

If you look through the Sage library, you'll see lots of places where
we use the triple-dot "wildcard" to ignore the last few digits of a
numerical answer.

Thanks,

Jason

Sergey Bochkanov

unread,
Aug 5, 2010, 12:39:46 PM8/5/10
to Jason Grout
Hello, Jason.

You wrote 5 августа 2010 г., 20:17:11:
>> (depending on problem and its stability properties). However I don't
>> know how to do it using doctest framework. Python tries to output
>> numbers with full precision,and there is no way to tell doctest
>> framework to compare decimal fractions using only N leading digits.

> Yes, there is. The doctest below will compare only the digits listed:
> sage: maxima('asinh(1.0)')
> 0.88137...

Thanks! I didn't noticed this ELLIPSIS option while reading doctest's
documentation. Everything becomes a bit simpler now :)

David Kirkby

unread,
Aug 5, 2010, 12:44:53 PM8/5/10
to sage-...@googlegroups.com
2010/8/5 Sergey Bochkanov <sergey.b...@alglib.net>:

Yes, that's what I mean by trivial. But the missing leading zero is
less trivial.

With all the numerical noise issues I've seen in Sage, the three dots
solves its. So if we expect

1.00000000000000
but get
1.00000000000001
we can change that to
1.0000000000000...
and the test will pass.

However, what if we get

0.99999999999999

That's very close to 1, but not a single digit is the same. I'm not
sure how one would handle that case.

Dave

Dima Pasechnik

unread,
Aug 5, 2010, 12:57:47 PM8/5/10
to sage-devel
On Aug 5, 6:44 pm, David Kirkby <david.kir...@onetel.net> wrote:
> 2010/8/5 Sergey Bochkanov <sergey.bochka...@alglib.net>:
> > Hello, Jason.
>
> > You wrote 5 августа 2010 г., 20:17:11:
> >>> (depending  on  problem and its stability properties). However I don't
> >>> know  how  to  do  it  using doctest framework. Python tries to output
> >>> numbers  with  full  precision,and  there  is no way to tell doctest
> >>> framework to compare decimal fractions using only N leading digits.
>
> >> Yes, there is.  The doctest below will compare only the digits listed:
> >> sage: maxima('asinh(1.0)')
> >> 0.88137...
>
> > Thanks!  I didn't noticed this ELLIPSIS option while reading doctest's
> > documentation. Everything becomes a bit simpler now :)
>
> > --
> > With best regards,
> >  Sergey                          mailto:sergey.bochka...@alglib.net
>
> Yes, that's what I mean by trivial. But the missing leading zero is
> less trivial.
>
> With all the numerical noise issues I've seen in Sage, the three dots
> solves its. So if we expect
>
> 1.00000000000000
> but get
> 1.00000000000001
> we can change that to
> 1.0000000000000...
> and the test will pass.
>
> However, what if we get
>
> 0.99999999999999
>
> That's very close to 1, but not a single digit is the same. I'm not
> sure how one would handle that case.

well, one needs to call a function to do a proper rounding before
doing the comparison in this case
(alternatively, one might add certain \epssilon>0 to shift the thing
above 1.0)

Dima

>
> Dave

Sergey Bochkanov

unread,
Aug 5, 2010, 1:01:13 PM8/5/10
to David Kirkby
Hello, David.

You wrote 5 августа 2010 г., 20:44:53:
> With all the numerical noise issues I've seen in Sage, the three dots
> solves its. So if we expect
> 1.00000000000000
> but get
> 1.00000000000001
> we can change that to
> 1.0000000000000...
> and the test will pass.
> However, what if we get
> 0.99999999999999
> That's very close to 1, but not a single digit is the same. I'm not
> sure how one would handle that case.

Hmmm... Didn't thought about this situation yet. Definitely we can't
solve this problem with any kind of regular expressions. One possible
solution is to round data before printing. So both 1.00000000000001
and 0.99999999999999 will become 1.000000.

As for me (testing interaction between ALGLIB and Sage), I can write
function which prints arrays/matrices with rounding up to specified
number of digits. Such doctests will look like

> sage: a = sqrt(2)
> sage: my_own_print(a,4)
> 1.414

But it will be more test than doc (less human readable). Don't know
whether it is worth using beyond Sage wrapper for ALGLIB.

Sergey Bochkanov

unread,
Aug 5, 2010, 1:17:42 PM8/5/10
to Sergey Bochkanov
Hello,

> Hmmm... Didn't thought about this situation yet. Definitely we can't
> solve this problem with any kind of regular expressions. One
> possible solution is to round data before printing. So both
> 1.00000000000001 and 0.99999999999999 will become 1.000000.

...however, we still can have problems when rounding X=0.000499999 up
to three digits. With original X we will have 0.000. But with
perturbation as small as 0.000000002 we will round to 0.001.

My automated Python-ALGLIB tests just calculate difference between
desired and actual answers, but they are not doctests; they are just
automatically generated Python scripts. We can't calculate differences
with doctests.

David Kirkby

unread,
Aug 5, 2010, 1:23:05 PM8/5/10
to sage-...@googlegroups.com

Thank you. That makes perfect sense.

BTW, do you have any ideas why the second failure at #9099 might occur

sage -t -long devel/sage/sage/symbolic/expression.pyx
**********************************************************************
File "/home/palmieri/fulvia/sage-4.5.2.rc0/devel/sage-main/sage/symbolic/expression.pyx",
line 498\
3:
sage: maxima('sinh(1.0)')
Expected:
1.175201193643801
Got:
1.175201193643802
**********************************************************************


File "/home/palmieri/fulvia/sage-4.5.2.rc0/devel/sage-main/sage/symbolic/expression.pyx",
line 508\

8:
sage: maxima('asinh(1.0)')
Expected:
0.881373587019543
Got:
.8813735870195429

**********************************************************************


Why should the zero be missing in the case observed on Solaris x86?i
In other words, why do we see .8813735870195429 instead of
0.8813735870195429 ? Te exact value of the last digit is unimportant,
and one clearly can't expect to get all digits spot on, but one would
hope that a zero should be printed when its needed.

Dave

Carl Witty

unread,
Aug 5, 2010, 1:45:53 PM8/5/10
to sage-...@googlegroups.com
On Thu, Aug 5, 2010 at 9:44 AM, David Kirkby <david....@onetel.net> wrote:
> With all the numerical noise issues I've seen in Sage, the three dots
> solves its. So if we expect
>
> 1.00000000000000
> but get
> 1.00000000000001
> we can change that to
> 1.0000000000000...
> and the test will pass.
>
> However, what if we get
>
> 0.99999999999999
>
> That's very close to 1, but not a single digit is the same. I'm not
> sure how one would handle that case.

I've seen a couple of approaches used in this case.

First, simply change the doctest. For example, if you're testing a
numerical root-finder on the polynomial x^2-x, change the polynomial
to x^2-x-1.

Second, include the test code in the doctest. Change:

sage: foo()
1.0000000000000

to:

sage: abs(foo() - 1) < 1e-12
True

This is less preferred, because while it keeps the doctest useful as a
test, it can reduce its value as documentation (especially if the test
code is more complicated than the above).

Carl

Carl Witty

unread,
Aug 5, 2010, 1:55:23 PM8/5/10
to sage-...@googlegroups.com

Well, it seems likely that the issue is within maxima (either in
maxima itself or in the Common Lisp implementation). To verify that
the problem has nothing to do with Sage, you can use "sage -maxima"
and try the test case on both machines:

cwitty@red-spider:~/sage$ ./sage -maxima
;;; Loading #P"/home/cwitty/sage/local/lib/ecl/defsystem.fas"
;;; Loading #P"/home/cwitty/sage/local/lib/ecl/cmp.fas"
;;; Loading #P"/home/cwitty/sage/local/lib/ecl/sysfun.lsp"
Maxima 5.20.1 http://maxima.sourceforge.net
using Lisp ECL 10.2.1
Distributed under the GNU Public License. See the file COPYING.
Dedicated to the memory of William Schelter.
The function bug_report() provides bug reporting information.
(%i1) asinh(1.0);
(%o1) 0.881373587019543

(Don't forget the semicolon on the "asinh(1.0);" line; without that,
maxima will wait forever for more input.)

If the results do differ with "sage -maxima", I guess the next step
would be to report the problem to the maxima mailing list or the bug
tracker (bearing in mind that the problem may actually be in the
Common Lisp implementation we use).

Carl

Dr. David Kirkby

unread,
Aug 5, 2010, 5:59:25 PM8/5/10
to sage-...@googlegroups.com
On 08/ 5/10 06:17 PM, Sergey Bochkanov wrote:
> Hello,
>
>> Hmmm... Didn't thought about this situation yet. Definitely we can't
>> solve this problem with any kind of regular expressions. One
>> possible solution is to round data before printing. So both
>> 1.00000000000001 and 0.99999999999999 will become 1.000000.
>
> ...however, we still can have problems when rounding X=0.000499999 up
> to three digits. With original X we will have 0.000. But with
> perturbation as small as 0.000000002 we will round to 0.001.

I've never done this, but the most logical thing to me seems to be to look at
the absolute magnitude of the relative error, for all cases except when the
expected value is 0. But I guess on cases where the numbers are not close to 1,
or other special cases one could think up, the current system is probably the
simplest.

Dave

Dr. David Kirkby

unread,
Aug 5, 2010, 8:07:05 PM8/5/10
to sage-...@googlegroups.com, sage-s...@googlegroups.com

Carl, thank you for your input.

I tried what you suggested, and see the result you show above on the Solaris 10
SPARC system, and a similar result, but lacking the leading zero, on the Solaris
10 x86 system.

I've stuck the full outputs at

http://trac.sagemath.org/sage_trac/ticket/9693

and have emailed both the Maxima and ECL mailing lists.

Unfortunately, we are not running the latest versions of either ECL or Maxima,
so I expect I'm likely to be asked "do you see this is the latest version?"

IIRC, there were some problems when we tried to update ECL and Maxima recently,
so its not as simple as one might like to determine this.

Dave

Carl Witty

unread,
Aug 5, 2010, 10:13:51 PM8/5/10
to sage-...@googlegroups.com

That would be nice. But it's a bit tricky to write, since the doctest
check happens solely on the basis of string comparison; to do
something clever like looking at relative error would require some
sort of marking in the expected output to turn on the relative-error
check, and some sort of parser to find the numbers in the
expected/found outputs, and possibly some sort of marking on a
per-number basis in the expected output to say what relative error is
allowed on that number. (If the expected output is 1.23456*x^10000,
with a relative error allowed of 1e-3, you probably don't want to
allow 1.23456*x^10001 !)

I would be in favor of something like this, if we could decide on a
set of markings in the expected output that didn't interfere too much
with the documentation aspect; but nobody has written it yet.

The three-dots notation is much, much stupider than this... it is just
a wildcard that will match an arbitrary string of characters (the
equivalent of .* as a regex, or * as shell globbing). And we didn't
even write it; it's a standard part of the Python doctest system.

Carl

Robert Bradshaw

unread,
Aug 6, 2010, 12:02:46 AM8/6/10
to sage-...@googlegroups.com

And the three dots is much prettier to read too.

Personally, I just choose doctests whose results are not subject to
the 1.0000000000 vs. 0.99999999999 kind of noise, unless I want to
make a point that the function nails the result right on (in which
case 1.000000000 is a good doctests for that).

- Robert

Reply all
Reply to author
Forward
0 new messages