Whats the plan for random doctests with floating point numbers?

[Volker Braun]

There are doctests of the form 

    sage: x = random_value()

    sage: abs(floating_point_computation(x) - exact_value(x)) < tolerance

    True 

but every floating point computation has SOME values where it is ill-conditioned. I'm finding a steady trickle of test failures due to the (now) random seeds for each run. Whats the plan for that?

A) mark them all as # random

B) identify all such tests and pick a fixed seed for those

C) identify all such tests and do the error analysis to find a tolerance(x) that takes the numerical instabilities into account

D) ignore issue and pick a fixed seed for the buildbot, will still randomly fail for users

PS: Here is one to illustrate the issue:

sage -t --long --warn-long 50.7 --random-seed=269711721119345330713403621242624346446 src/sage/functions/exp_integral.py
**********************************************************************
File "src/sage/functions/exp_integral.py", line 1493, in sage.functions.exp_integral.exponential_integral_1
Failed example:
    for prec in [20..128]:  # long time (15s on sage.math, 2013)
        R = RealField(prec)
        S = RealField(prec+64)
        a = R.random_element(-15,10).exp()
        n = 2^ZZ.random_element(14)
        x = exponential_integral_1(a, n)
        y = exponential_integral_1(S(a), n)
        c = RDF(4 * max(1.0, y[0]))
        for i in range(n):
            e = float(abs(S(x[i]) - y[i]) << prec)
            if e >= c:
                print("exponential_integral_1(%s, %s)[%s] with precision %s has error of %s >= %s"%(a, n, i, prec, e, c))
Expected nothing
Got:
    exponential_integral_1(57.77854062056109797473161078505723378, 1)[0] with precision 127 has error of 4.75102130393081 >= 4.0
**********************************************************************
1 item had failures:
   1 of  11 in sage.functions.exp_integral.exponential_integral_1
    [275 tests, 1 failure, 7.23 s]
----------------------------------------------------------------------
sage -t --long --warn-long 50.7 --random-seed=269711721119345330713403621242624346446 src/sage/functions/exp_integral.py  # 1 doctest failed
----------------------------------------------------------------------

[Michael Orlitzky]

On Wed, 2021-12-29 at 13:35 -0800, Volker Braun wrote:
> There are doctests of the form
>
> sage: x = random_value()
> sage: abs(floating_point_computation(x) - exact_value(x)) < tolerance
> True
>
> but every floating point computation has SOME values where it is
> ill-conditioned. I'm finding a steady trickle of test failures due to the
> (now) random seeds for each run. Whats the plan for that?
>
>

I think we have to consider what the point of these tests is. If
someone picked a tolerance that turned out to be too naive, then either
the code or the test is wrong, and we should fix whichever one it is.

If the existing tolerance was just someone's pulled-it-out-of-my-butt
guess, then we can fix it by adding an order of magnitude to the
tolerance without much thought. But if someone carefully picked a
tolerance that is actually violated (as in your example), someone
should probably take a look at it and do the analysis to pick a better
value.

[dmo...@deductivepress.ca]

In the long run, we want to fix the doctests so they always pass (as described in the previous comment).

However, in the short run, I think it is fine to add

set_random_seed(0)  # failed doctest - see  trac ticket nnnnn

where the description of ticket nnnnn says it is adding `set_random_seed` to fix a bunch of failed doctests (and should also have comments that specify the `sage -t` calls that produce the failures).  That way, we still get doctest coverage, and a search of the source code for `set_random_seed` will quickly reveal the problems that were found.  I don't think that opening a ticket for each of these failures needs to be the job of the person who finds the errors.

PS I'm sorry to hear that the randomization has been causing problems for you!

[Kwankyu Lee]

On Thursday, December 30, 2021 at 8:40:35 AM UTC+9 dmo...@deductivepress.ca wrote:

In the long run, we want to fix the doctests so they always pass (as described in the previous comment).

However, in the short run, I think it is fine to add

set_random_seed(0)  # failed doctest - see  trac ticket nnnnn

+1. So this is (B). The ticket nnnnn is cc'ed to the writer of the failed doctests.

[Tobia...@gmx.de]

It would also be an idea to replace these absolute tolerance tests, i.e. abs(x - y) < something, by a relative tests abs(x-y)/x < something. In general, this should be more stable. Such an approach is used by pytest, see https://docs.pytest.org/en/6.2.x/reference.html#pytest-approx. Maybe this helper function could be directly re-used, which would also simplify comparing lists etc.

[David Roe]

We already have a relative tolerance check available:

sage: print("0.9999")    # rel tol 1e-4
1.0
sage: print("1.00001")   # abs tol 1e-5
1.0

David

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/8717e8ca-820a-460f-aff6-299dfe97dca1n%40googlegroups.com.

[jonatha...@googlemail.com]

Sorry for being late to the party.

To save Volker the headache, I would propose setting the environment variable SAGE_DOCTEST_RANDOM_SEED to zero for buildbots (see https://trac.sagemath.org/ticket/32761).

It's not the job of buildbots to discover random test failures (patchbots on the other hand should check on this).

+1 to the approach of Kwankyu Lee (mark failures, open ticket, cc the person that wrote the offending test).

I think, doctests that fail for some values should be removed or fixed. E.g. claiming that matrix inversion can be done with fixed absolute error is just wrong.

Relative error will probably work for many purposes.

I had some instances, were I used maximum of some absolute error and some relative error, e.g.:

sage: x = random_value()

sage: abs(x) * abs(floating_point_computation(x) - exact_value(x)) < tolerance * max(1, abs(x), abs(x)^2)

True

This might need to be altered, depending on where the calculation is not stable.

In general, I would wrap this in a loop and experiment how large the error can get and if it stabilizes.

Sorry this creates work, but on the other hand as mentioned above, our doctests shouldn't claim some stability that just cannot be achieved.

Jonathan