Sage Integral Test Suite

[Tim Lahey]

Hi,

I've added a quite a few integrals to the test suites now.

There are two problems currently:
a) Maxima on occasion takes input which raises an exception.
b) Some of the integrals have multiple solutions and I haven't
found a good way to include and compare these. However, since I
can't integrate these through the Sage-Maxima interface
sometimes because of a), this isn't a big problem.

I've deleted the script from my Dropbox since it's quite out of
date now. I now have 6 sets of tests (matching Tim Daly's CATS)
and I have much better reporting on errors.

So, once again, the latest is at:
http://github.com/tjl/sage_int_testing/tree

I've also added issue tracking at Lighthouse for it. The link is
in the README (or on the above web page).

I need to figure out a robust way of
1) Handling multiple solutions.
2) Supporting multiple comparison strategies since sometimes it
may be necessary to follow a specific procedure to compare the
results. This may be a sequence of various Sage calls.

The current system is quite crude, but my first priority is to
get the complete test set put together.

Cheers,

Tim.

---
Tim Lahey
PhD Candidate, Systems Design Engineering
University of Waterloo

[Bill Page]

Tim,

What version of sage is required to run this? I get the following error:

wspage@debian:~$ ./sage*/sage
----------------------------------------------------------------------
| SAGE Version 3.1.4, Release Date: 2008-10-20 |
| Type notebook() for the GUI, and license() for information. |
----------------------------------------------------------------------

sage: exit
Exiting SAGE (CPU time 0m0.02s, Wall time 0m1.94s).

wspage@debian:~$ ./sage*/sage integral_test1.sage
Test 1 : Test Passed.
Maxima Time:
Traceback (most recent call last):
File "integral_test1.py", line 96, in <module>
time_Maxima_friCAS(integrand)
File "integral_test1.py", line 81, in time_Maxima_friCAS
print "Maxima Time:", mx_time.stats[_sage_const_3 ],
mx_time.stats[_sage_const_4 ]
AttributeError: 'str' object has no attribute 'stats'

Regards,
Bill Page.

On Tue, Nov 25, 2008 at 6:48 PM, Tim Lahey wrote:
>
> Hi,
>
> I've added a quite a few integrals to the test suites now.

>...

[Mike Hansen]

On Tue, Nov 25, 2008 at 4:15 PM, Bill Page <bill...@newsynthesis.org> wrote:
>
> Tim,
>
> What version of sage is required to run this? I get the following error:

It requires Sage 3.2.

--Mike

[Tim Lahey]

On Nov 25, 2008, at 7:15 PM, Bill Page wrote:

>
> Tim,
>
> What version of sage is required to run this? I get the following
> error:
>
> wspage@debian:~$ ./sage*/sage
> ----------------------------------------------------------------------
> | SAGE Version 3.1.4, Release Date: 2008-10-20 |
> | Type notebook() for the GUI, and license() for information. |
> ----------------------------------------------------------------------
>
> sage: exit
> Exiting SAGE (CPU time 0m0.02s, Wall time 0m1.94s).
>
> wspage@debian:~$ ./sage*/sage integral_test1.sage
> Test 1 : Test Passed.
> Maxima Time:
> Traceback (most recent call last):
> File "integral_test1.py", line 96, in <module>
> time_Maxima_friCAS(integrand)
> File "integral_test1.py", line 81, in time_Maxima_friCAS
> print "Maxima Time:", mx_time.stats[_sage_const_3 ],
> mx_time.stats[_sage_const_4 ]
> AttributeError: 'str' object has no attribute 'stats'
>
> Regards,
> Bill Page.

As Mike said, it requires Sage 3.2. Also, you should just be able to run
integral_test1.sage directly if sage is in your PATH and SAGE_ROOT is
defined. It also requires FriCAS to run those tests. If FriCAS isn't
installed, there are only a few lines to comment out.

Cheers,

Tim.

[Bill Page]

On Tue, Nov 25, 2008 at 7:58 PM, Tim Lahey wrote:
>
> As Mike said, it requires Sage 3.2.

After an apparently sucessful 'sage -upgrade':

wspage@debian:~$ sage
----------------------------------------------------------------------
| Sage Version 3.2, Release Date: 2008-11-20 |

| Type notebook() for the GUI, and license() for information. |
----------------------------------------------------------------------

sage: exit
Exiting SAGE (CPU time 0m0.03s, Wall time 0m1.88s).

> Also, you should just be able to run integral_test1.sage
> directly if sage is in your PATH and SAGE_ROOT
> is defined.

Done.

> It also requires FriCAS to run those tests. If FriCAS isn't
> installed, there are only a few lines to comment out.
>

FriCAS is installed.

But I still get:

wspage@debian:~$ ./integral_test1.sage

Test 1 : Test Passed.
Maxima Time:
Traceback (most recent call last):

File "./integral_test1.py", line 96, in <module>
time_Maxima_friCAS(integrand)
File "./integral_test1.py", line 81, in time_Maxima_friCAS

print "Maxima Time:", mx_time.stats[_sage_const_3 ],
mx_time.stats[_sage_const_4 ]
AttributeError: 'str' object has no attribute 'stats'

----

The script runs ok if I comment out the timing:

else:
test_eval(test,test_int,desired_result)
#time_Maxima_friCAS(integrand)

Any ideas?

Regards,
Bill Page.

[Tim Lahey]

On Nov 25, 2008, at 10:16 PM, Bill Page wrote:

>
> wspage@debian:~$ ./integral_test1.sage
> Test 1 : Test Passed.
> Maxima Time:
> Traceback (most recent call last):
> File "./integral_test1.py", line 96, in <module>
> time_Maxima_friCAS(integrand)
> File "./integral_test1.py", line 81, in time_Maxima_friCAS
> print "Maxima Time:", mx_time.stats[_sage_const_3 ],
> mx_time.stats[_sage_const_4 ]
> AttributeError: 'str' object has no attribute 'stats'
>
> ----
>
> The script runs ok if I comment out the timing:
>
> else:
> test_eval(test,test_int,desired_result)
> #time_Maxima_friCAS(integrand)
>
>

Strange,

The timeit code is in sage/misc/sage_timeit.py and sage/misc/
sage_timeit_class.py
so I'd check to see if the code there has the stats attribute. If it
does, I'd
assume that somehow Sage didn't pick up the change.

Cheers,

Tim.

[Bill Page]

On Tue, Nov 25, 2008 at 10:32 PM, Tim Lahey wrote:
>
> Strange,
>
> The timeit code is in sage/misc/sage_timeit.py and

> sage/misc/sage_timeit_class.py so I'd check to see if the

> code there has the stats attribute. If it does, I'd assume that
> somehow Sage didn't pick up the change.
>

I do not have any file named 'sage/misc/sage_timeit_class.py', instead
there is 'sage/misc/sage_timeit_class.c' and
'sage/misc/sage_timeit_class.pyx'

wspage@debian:~/sage-3.1.4$ find . -name sage_timeit.py
./devel/sage-main/build/lib.linux-i686-2.5/sage/misc/sage_timeit.py
./devel/sage-main/build/sage/misc/sage_timeit.py
./devel/sage-main/sage/misc/sage_timeit.py

wspage@debian:~/sage-3.1.4$ ls -l ./devel/sage-main/sage/misc/sage_timeit*
-rw-r--r-- 1 wspage wspage 50391 2008-11-25 20:07
./devel/sage-main/sage/misc/sage_timeit_class.c
-rw-r--r-- 1 wspage wspage 3316 2008-11-25 19:41
./devel/sage-main/sage/misc/sage_timeit_class.pyx
-rw-r--r-- 1 wspage wspage 3381 2008-10-16 03:52
./devel/sage-main/sage/misc/sage_timeit.py

wspage@debian:~/sage-3.1.4$ grep stats ./devel/sage-main/sage/misc/sage_timeit*
wspage@debian:~/sage-3.1.4$

---

It seems that none of these files has anything named 'stats':

It appears that 'sage_timeit.py' was not updated by '-upgrade'.

Note: Directory sage-3.1.4 is where I previously built sage from
source. I did not change this name when I did '-upgrade'.

Regards.
Bill Page.

[Mike Hansen]

Hi Bill,

On Tue, Nov 25, 2008 at 8:00 PM, Bill Page <bill...@newsynthesis.org> wrote:
> It seems that none of these files has anything named 'stats':
>
> It appears that 'sage_timeit.py' was not updated by '-upgrade'.
>
> Note: Directory sage-3.1.4 is where I previously built sage from
> source. I did not change this name when I did '-upgrade'.

What is the output of running "hg heads" in ~/sage-3.1.4/devel/sage-main?

My guess is that there is more than one head (due to you making
changes in the repository). To fix it, run "hg merge" and resolve any
conflicts if there are any.

--Mike

[Tim Lahey]

Checking again, sage_timeit_class.py doesn't exist. Sorry about that.
However, it sage_timeit.py there should be the class,

class SageTimeitResult():
r"""
I represent the statistics of a timeit() command. I print as a
string so
that I can be easily returned to a user.
"""
def __init__(self, stats):
self.stats = stats

def __repr__(self):
return "%d loops, best of %d: %.*g %s per loop" % self.stat

That's where I'm getting the stats from. I agree with Mike Hansen that
you
probably have two heads that need to be merged.

Sorry about this.

[Bill Page]

On Tue, Nov 25, 2008 at 11:05 PM, Mike Hansen wrote:

>
>
> On Tue, Nov 25, 2008 at 8:00 PM, Bill Page wrote:
>> It seems that none of these files has anything named 'stats':
>>
>> It appears that 'sage_timeit.py' was not updated by '-upgrade'.
>>
>> Note: Directory sage-3.1.4 is where I previously built sage from
>> source. I did not change this name when I did '-upgrade'.
>
> What is the output of running "hg heads" in ~/sage-3.1.4/devel/sage-main?
>

wspage@debian:~/sage-3.1.4/devel/sage-main$ hg heads
changeset: 10984:8be12623525e
tag: tip
parent: 10737:3859ace86968
user: wspage@debian
date: Tue Nov 25 19:41:35 2008 -0500
summary: merge

changeset: 10983:4952e4331644
user: William Stein <wst...@gmail.com>
date: Thu Nov 20 13:00:39 2008 -0800
summary: 3.2

> My guess is that there is more than one head (due to you making
> changes in the repository).

Maybe you are right. Very confusing.

> To fix it, run "hg merge" and resolve any conflicts if there are any.
>

wspage@debian:~/sage-3.1.4/devel/sage-main$ hg merge
merging sage/interfaces/expect.py
merging sage/interfaces/sage0.py
merging sage/rings/integer_mod.pyx
merging sage/rings/integer_mod_ring.py
merging sage/rings/real_mpfr.pyx
233 files updated, 5 files merged, 59 files removed, 0 files unresolved
(branch merge, don't forget to commit)

-----

wspage@debian:~/sage-3.1.4/devel/sage-main$ cd ~
wspage@debian:~$ export PATH=/home/wspage/sage-3.1.4:$PATH
wspage@debian:~$ export SAGE_ROOT=/home/wspage/sage-3.1.4

wspage@debian:~$ sage
----------------------------------------------------------------------
| Sage Version 3.2, Release Date: 2008-11-20 |
| Type notebook() for the GUI, and license() for information. |
----------------------------------------------------------------------

sage: exit
Exiting SAGE (CPU time 0m0.02s, Wall time 0m2.52s).

wspage@debian:~$ ./integral_test1.sage
Test 1 : Test Passed.
Maxima Time:
Traceback (most recent call last):
File "./integral_test1.py", line 96, in <module>
time_Maxima_friCAS(integrand)
File "./integral_test1.py", line 81, in time_Maxima_friCAS
print "Maxima Time:", mx_time.stats[_sage_const_3 ],
mx_time.stats[_sage_const_4 ]
AttributeError: 'str' object has no attribute 'stats'

---

Still unhappy. Did it really mean 'don't forget to commit'? Where?

Regards,
Bill Page.

[Mike Hansen]

On Tue, Nov 25, 2008 at 8:38 PM, Bill Page <bill...@newsynthesis.org> wrote:
>
> Still unhappy. Did it really mean 'don't forget to commit'? Where?
>
> Regards,
> Bill Page.

You just run "hg ci" to commit the merge. Then to make the changes
active, you have to start sage with "sage -br".

--Mike

[Bill Page]

On Tue, Nov 25, 2008 at 11:10 PM, Tim Lahey wrote:
> ...

> However, it sage_timeit.py there should be the class,
>
> class SageTimeitResult():
> r"""
> I represent the statistics of a timeit() command. I print as a
> string so
> that I can be easily returned to a user.
> """
> def __init__(self, stats):
> self.stats = stats
>
> def __repr__(self):
> return "%d loops, best of %d: %.*g %s per loop" % self.stat
>
> That's where I'm getting the stats from. I agree with Mike Hansen
> that you probably have two heads that need to be merged.
>

After the merge the dates on the 'sage_timeit.py' file did not change
and there is no 'class SageTimeitResult' in that file.

> Sorry about this.
>

No problem. I'll try to get this right. If all else fails I guess I
can re-build from source and re-apply the changes (actually just
Mike's patches from the trac item for axiom).

Regards,
Bill Page.

[William Stein]

On Tue, Nov 25, 2008 at 8:44 PM, Bill Page <bill...@newsynthesis.org> wrote:
>
> On Tue, Nov 25, 2008 at 11:10 PM, Tim Lahey wrote:
>> ...
>> However, it sage_timeit.py there should be the class,
>>
>> class SageTimeitResult():
>> r"""
>> I represent the statistics of a timeit() command. I print as a
>> string so
>> that I can be easily returned to a user.
>> """
>> def __init__(self, stats):
>> self.stats = stats
>>
>> def __repr__(self):
>> return "%d loops, best of %d: %.*g %s per loop" % self.stat
>>
>> That's where I'm getting the stats from. I agree with Mike Hansen
>> that you probably have two heads that need to be merged.
>>
>
> After the merge the dates on the 'sage_timeit.py' file did not change
> and there is no 'class SageTimeitResult' in that file.

You definitely haven't upgraded to sage-3.2 then. Try this from sage:

sage: hg_sage.ci()
sage: hg_sage.pull()
sage: hg_sage.merge()
sage: hg_sage.update()
sage: quit

sage -br

Then try the integration stuff.

>
>> Sorry about this.
>>
>
> No problem. I'll try to get this right. If all else fails I guess I
> can re-build from source and re-apply the changes (actually just
> Mike's patches from the trac item for axiom).
>
> Regards,
> Bill Page.
>
> >
>

--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

[Bill Page]

On Tue, Nov 25, 2008 at 11:42 PM, Mike Hansen wrote:

>
> On Tue, Nov 25, 2008 at 8:38 PM, Bill Page wrote:
>>
>> Still unhappy. Did it really mean 'don't forget to commit'? Where?
>>

> You just run "hg ci" to commit the merge. Then to make the
> changes active, you have to start sage with "sage -br".
>

That magic worked. 'integral_test1.sage' runs as expected. Now I'm happy. :-)

Thanks!

Regards,
Bill Page.

[Tim Lahey]

On Nov 26, 2008, at 12:25 AM, Bill Page wrote:

>
> That magic worked. 'integral_test1.sage' runs as expected. Now I'm
> happy. :-)
>
> Thanks!
>

Great. If you have any suggestions (or code), I'd be happy to hear them.

[Bill Page]

On Wed, Nov 26, 2008 at 12:14 AM, William Stein wrote:
> Bill Page wrote:
>> After the merge the dates on the 'sage_timeit.py' file did not
>> change and there is no 'class SageTimeitResult' in that file.
>
> You definitely haven't upgraded to sage-3.2 then. Try this from
> sage:
>
> sage: hg_sage.ci()
> sage: hg_sage.pull()
> sage: hg_sage.merge()
> sage: hg_sage.update()
> sage: quit
>
> sage -br
>
> Then try the integration stuff.
>

Is there any good reason to want to issue the hg commands from inside
Sage rather than 'sage -upgrade' and using the 'hg merge + ci' that
Mike recommended? My version of Sage now runs Tim's integration tests
without a trace back. Is there any reason to think I might still be
missing something?

Regards,
Bill Page.

[William Stein]

I only posted the above since you had mentioned that what Mike suggested
didn't work. You subsequently posted that it *did* work, but that was
after I posted the above. So please never mind regarding the above,
and simply move on to having fun testing integration and helping
Tim build that awesome test suite.

-- William

[Bill Page]

On Wed, Nov 26, 2008 at 12:28 AM, Tim Lahey wrote:
>
> On Nov 26, 2008, at 12:25 AM, Bill Page wrote:
>>
>> That magic worked. 'integral_test1.sage' runs as expected. Now
>> I'm happy. :-)
>>
>> Thanks!
>
> Great. If you have any suggestions (or code), I'd be happy to hear
> them.
>

One thing that I wanted to do with your code is to include comparisons
of the answers produced by both Maxima and FriCAS as well as the
comparison to the Schaum's tabulated value. Here is a small patch to
'integral_test1.sage' (changes to other files are identical):

wspage@debian:~/integration$ git diff
diff --git a/integral_test1.sage b/integral_test1.sage
index 0984a27..14ea330 100755
--- a/integral_test1.sage
+++ b/integral_test1.sage
@@ -73,13 +73,18 @@ def test_eval(test, test_int, desired_result):
print "Comparison Integral: ", desired_result

# Time integration of Maxima and FriCAS for integral.
-def time_Maxima_friCAS(integrand):
+def time_Maxima_friCAS(test_int,integrand):
mx_time = timeit.eval('integrand.integrate(x)')
- fCAS_time= timeit.eval('axiom.integrate(integrand,x)')
+ fCAS_time = timeit.eval('axiom.integrate(integrand,x)')
+ fricas_int = axiom.integrate(integrand,x).sage()
+ fricas_cmp =
(test_int.simplify_full()-fricas_int.simplify_full()).simplify_full()
+ if (fricas_cmp == 0):
+ print "FriCAS agrees with Maxima."
+ else:
+ print "FriCAS begs to differ!"
print "Maxima Time:", mx_time.stats[3], mx_time.stats[4]
print "FriCAS Time:", fCAS_time.stats[3], fCAS_time.stats[4]

-
# Loop over tests
for test in int_table.keys():
test_set = int_table[test]
@@ -90,5 +95,6 @@ for test in int_table.keys():
except:
print "Test", test,": Test failed due to exception."
else:
- test_eval(test,test_int,desired_result)
- time_Maxima_friCAS(integrand)
+ test_eval(test,test_int,desired_result)
+ time_Maxima_friCAS(test_int,integrand)
+

-----

Notice that the comparison of the FriCAS result:

fricas_int = axiom.integrate(integrand,x).sage()

is done using Sage "native" form (internally it's actually Maxima, I think).

In order to run these tests, it is also necessary to make some simple
additions to the 'axiom.py' interface:

wspage@debian:~/sage-3.1.4/devel/sage-main/sage/interfaces$ hg diff
diff -r ed3f78f99d2a sage/interfaces/axiom.py
--- a/sage/interfaces/axiom.py Tue Nov 25 23:45:43 2008 -0500
+++ b/sage/interfaces/axiom.py Wed Nov 26 19:43:59 2008 -0500
@@ -729,7 +729,10 @@
s = P.eval('unparse(%s::InputForm)'%self._name)
if 'translation error' in s or 'Cannot convert' in s:
raise NotImplementedError
- s = multiple_replace({'\r\n':'', # fix stupid Fortran-ish
+ s = multiple_replace({'\r\n':'', # fix stupid Fortran-ish
+ 'DLOG(':'log(',
+ 'DEXP(':'exp(',
+ '::(':'', ',Symbol)':'',
'DSIN(':'sin(',
'DCOS(':'cos(',
'DTAN(':'tan(',

----

These integrals produce some additional "Fortran-style" names and a
coercion that have to be translated before the input form can be
processed by Sage.

With these changes the test script shows that FriCAS and Maxima
produce the same results on at least the first 25 integrals.

wspage@debian:~/integration$ sage integral_test1.sage

Test 1 : Test Passed.
FriCAS agrees with Maxima.
Maxima Time: 10.3870391846 ms
FriCAS Time: 8.20741653442 ms
Test 2 : Test Passed.
FriCAS agrees with Maxima.
Maxima Time: 12.8763580322 ms
FriCAS Time: 10.127363205 ms
Test 3 Difference in Results:
2
3 b
- ----
3
2 a
Correct within a constant of integration.
Test Passed.
FriCAS agrees with Maxima.
Maxima Time: 14.6490764618 ms
FriCAS Time: 12.3098373413 ms
Test 4 Difference in Results:
3
11 b
-----
4
6 a
Correct within a constant of integration.
Test Passed.
FriCAS agrees with Maxima.
Maxima Time: 16.8963623047 ms
FriCAS Time: 15.8182048798 ms

...

Finally, one more thing that I found rather interesting was that the
total run time for FriCAS across all 120 integrals was approximately
30% greater than the total run time for Maxima. I was a bit surprised
since I had come to believe that FriCAS as actually a bit faster based
on a much less representative comparison. Both Maxima and FriCAS are
built using Clisp for these tests, so that does not explain the
difference.

Regards,
Bill Page.

[mabshoff]

On Nov 26, 4:54 pm, "Bill Page" <bill.p...@newsynthesis.org> wrote:
> On Wed, Nov 26, 2008 at 12:28 AM, Tim Lahey wrote:

<SNIP>

Hi Bill,

> In order to run these tests, it is also necessary to make some simple
> additions to the 'axiom.py' interface:
>
> wspage@debian:~/sage-3.1.4/devel/sage-main/sage/interfaces$ hg diff
> diff -r ed3f78f99d2a sage/interfaces/axiom.py
> --- a/sage/interfaces/axiom.py  Tue Nov 25 23:45:43 2008 -0500
> +++ b/sage/interfaces/axiom.py  Wed Nov 26 19:43:59 2008 -0500
> @@ -729,7 +729,10 @@
>          s = P.eval('unparse(%s::InputForm)'%self._name)
>          if 'translation error' in s or 'Cannot convert' in s:
>              raise NotImplementedError
> -        s = multiple_replace({'\r\n':'', # fix stupid Fortran-ish
> +        s = multiple_replace({'\r\n':'', # fix stupid Fortran-ish
> +                              'DLOG(':'log(',
> +                              'DEXP(':'exp(',
> +                              '::(':'', ',Symbol)':'',
>                                'DSIN(':'sin(',
>                                'DCOS(':'cos(',
>                                'DTAN(':'tan(',
>
> ----
>
> These integrals produce some additional "Fortran-style" names and a
> coercion that have to be translated before the input form can be
> processed by Sage.

<SNIP>

Please open a ticket and attach a patch so this can get merged in
3.2.1.

There is also #4036 still open which has various improvements for the
panAxiom interface. Only a little review is required, so let's also
get this in.

> Regards,
> Bill Page.

Cheers,

Michael

[Tim Lahey]

On Nov 26, 2008, at 7:54 PM, Bill Page wrote:

> One thing that I wanted to do with your code is to include comparisons
> of the answers produced by both Maxima and FriCAS as well as the
> comparison to the Schaum's tabulated value. Here is a small patch to
> 'integral_test1.sage' (changes to other files are identical):
>

Thanks for these additions. What I think I'll do is maintain a branch
with these changes until the FriCAS interface changes make it into
Sage. Do you want to submit a Trac ticket with interface changes? I
don't have a Trac account as of yet.

I thought that as well, but I certainly noticed that in many cases
Maxima
is faster and sometimes is a lot faster. Sometimes, SymPy is faster than
both Maxima and FriCAS.

[mabshoff]

On Nov 26, 5:13 pm, Tim Lahey <tim.la...@gmail.com> wrote:
> On Nov 26, 2008, at 7:54 PM, Bill Page wrote:

Hi Tim,

> > One thing that I wanted to do with your code is to include comparisons
> > of the answers produced by both Maxima and FriCAS as well as the
> > comparison to the Schaum's tabulated value. Here is a small patch to
> > 'integral_test1.sage' (changes to other files are identical):
>
> Thanks for these additions. What I think I'll do is maintain a branch
> with these changes until the FriCAS interface changes make it into
> Sage. Do you want to submit a Trac ticket with interface changes?

I asked Bill to do so, but so far nothing has happened.

> I don't have a Trac account as of yet.

Send me an email off list after reading the bit about account names
and password at http://wiki.sagemath.org/TracGuidelines

Cheers,

Michael