Sage 3.4.2.rc0 release!

3 views
Skip to first unread message

mabshoff

unread,
Apr 30, 2009, 8:23:09 AM4/30/09
to sage-devel
Hello folks,

here goes 3.4.2.rc0 - a little later than planned, but it seems like
we fixed all the issues (and more) that needed to be fixed. We finally
merged the symbolic logic code written well over *18* months ago, but
"no doctests, no merge" is something we take seriously. On top of that
we are at

Overall weighted coverage score: 71.1%
Total number of functions: 22348
We need 881 more function to get to 75% coverage.
We need 1999 more function to get to 80% coverage.
We need 3116 more function to get to 85% coverage.

Note that this is also due to splitting DSage into its own spkg. We
will discuss what to do with the DSage codebase at SD15 I assume.

The source tarball, upgrade bits as well as a sage.math binary are in
the usual place at

http://sage.math.washington.edu/home/mabshoff/release-cycles-3.4.2/

I have done some preliminary build testing and it seems to at least
build. From here on we will only merge truly critical bugs that fix
doctests and grave issues, otherwise better luck in Sage 4.0 :).

Please build, doctest and report any issues.

Cheers,

Michael

Merged, but needs review:

#5849: Michael Abshoff: Update MPIR to 1.1.1

Merged in Sage 3.4.2.rc0:

#545: Chris Gorecki: polish the new symbolic logic code [Reviewed by
Minh Van Nguyen, Georg Weber, Wilfried Huss, Michael Abshoff]
#2740: Robert Bradshaw: Downloading and uploading folders of
worksheets [Reviewed by William Stein]
#4435: John Palmieri: %latex -- don't use \usepackage{fullpage}
[Reviewed by William Stein]
#5479: Alex Ghitza: schemes/generic/spec.py: Spec.__call__ is
basically not implemented [Reviewed by David Roe]
#5588: John Palmieri: developer's guide: more information about
docstrings [Reviewed by John Cremona]
#5765: Alex Ghitza: improve doctest coverage for schemes/generic/
algebraic_scheme.py [Reviewed by John Cremona]
#5791: John Palmieri: Allow custom packages to be injected or %latex
and the Sage latex mode [Reviewed by Rob Beezer]
#5804: William Stein, Bill Hart, Michael Abshoff: Solaris 10/Sparc:
number_of_partitions(100000) is broken [Reviewed by Mike Hansen]
#5820: Alex Ghitza: implement comparison of ring coercion morphisms
[Reviewed by David Roe]
#5824: William Stein: Move DSage to its own spkg [Reviewed by Michael
Abshoff]
#5846: Chris Wuthrich: small bug in caching the precision for p-adic L-
series [Reviewed by Georg Weber]
#5853: Chris Wuthrich: Restify and include more documentation on
elliptic curves [Reviewed by John Cremona]
#5855: Robert Miller: implement squarefree_divisors function [Reviewed
by William Stein]
#5856: Alex Ghitza: elliptic curves over Z/pZ are treated totally
differently than elliptic curves over GF(p) [Reviewed by John Cremona]
#5860: William Stein: delete sage/catalogue [Reviewed by David Harvey]
#5877: Franco Saliola: calling a group character is broken [Reviewed
by David Harvey]
#5880: William Stein, Rob Beezer: notebook -- greatly reduce the
number of actions that trigger taking a snapshot [Reviewed by Rob
Beezer, William Stein]
#5881: Mike Hansen: __cmp__ is random-ish in root_system/type_dual.py
also (analog to #5811) [Reviewed by Michael Abshoff]
#5890: Alex Ghitza: clean up schemes/elliptic_curves/ell_generic.py
[Reviewed by John Cremona]
#5896: Craig Ctiro: Documentation fix for lcalc interface [Reviewed by
John Cremona]
#5898: Karl-Dieter Crisman: Plot Field doc [Reviewed by Minh Van
Nguyen]
#5904: Alex Ghitza: improve doctest coverage for schemes/jacobians/
abstract_jacobian.py [Reviewed by John Cremona]
#5905: John Cremona: minor fix to ReST markup in ell_rational_field.py
[Reviewed by John Palmieri]
#5912: William Stein: %fortran mode is broken in the notebook
[Reviewed by Chris Swierczewski]
#5914: Robert Miller: default edge color is *invisible* [Reviewed by
William Stein]
#5918: Franco Saliola: bring doctest coverage for posets to 100%
[Reviewed by Dan Drake]
#5919: John Cremona: bug in conversion of polys over GF(2,e) from NTL
to singular [Reviewed by Martin Albrecht]
#5921: Alex Ghitza: factoring integer polynomials does not factor the
content [Reviewed by John Cremona]
#5926: William Stein: solaris top sometimes fails due to race
condition [Reviewed by Michael Abshoff]
#5928: Alex Ghitza: binary operations on factorisations should coerce
factors into a common universe [Reviewed by John Cremona]
#5933: Karl-Dieter Crisman: Bring primes.py to 100% [Reviewed by John
Cremona]
#5946: David Roe: bug in content for p-adic polynomials [Reviewed by
Alex Ghitza]

Jaap Spies

unread,
Apr 30, 2009, 1:38:10 PM4/30/09
to sage-...@googlegroups.com
mabshoff wrote:
> Hello folks,
>
> here goes 3.4.2.rc0 - a little later than planned, but it seems like
> we fixed all the issues (and more) that needed to be fixed. We finally
> merged the symbolic logic code written well over *18* months ago, but
> "no doctests, no merge" is something we take seriously. On top of that
> we are at
>
> Overall weighted coverage score: 71.1%
> Total number of functions: 22348
> We need 881 more function to get to 75% coverage.
> We need 1999 more function to get to 80% coverage.
> We need 3116 more function to get to 85% coverage.
>
> Note that this is also due to splitting DSage into its own spkg. We
> will discuss what to do with the DSage codebase at SD15 I assume.
>
> The source tarball, upgrade bits as well as a sage.math binary are in
> the usual place at
>
> http://sage.math.washington.edu/home/mabshoff/release-cycles-3.4.2/
>
> I have done some preliminary build testing and it seems to at least
> build. From here on we will only merge truly critical bugs that fix
> doctests and grave issues, otherwise better luck in Sage 4.0 :).
>
> Please build, doctest and report any issues.
>

After an upgrade from alpha0 on Fedora 9, 32 bit one failure:

[jaap@paix sage-3.4.2.alpha0]$ ./sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
**********************************************************************
File "/home/jaap/downloads/sage-3.4.2.alpha0/devel/sage/sage/matrix/matrix_symbolic_dense.pyx", line 413:
sage: M.determinant()
Expected:
4*x - 6
Got:
determinant(sage513)
**********************************************************************
1 items had failures:
1 of 9 in __main__.example_18
***Test Failed*** 1 failures.
For whitespace errors, see the file /home/jaap/downloads/sage-3.4.2.alpha0/tmp/.doctest_matrix_symbolic_dense.py
[83.9 s]
exit code: 1024

Now doing a fresh install on this machine.

Still waiting for a fresh install (and test) on Fedora 10, 32 bits.

Jaap


kcrisman

unread,
Apr 30, 2009, 3:26:56 PM4/30/09
to sage-devel
Upgrades fine from 3.4.2.alpha0 on G4 PPC OSX.4.

- kcrisman

Jaap Spies

unread,
Apr 30, 2009, 4:04:59 PM4/30/09
to sage-...@googlegroups.com
Jaap Spies wrote:

>
> Still waiting for a fresh install (and test) on Fedora 10, 32 bits.
>

As a follow up: Fedora 10, 32 bits, fresh install:

./sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"

*** *** Error: TIMED OUT! PROCESS KILLED! *** ***
*** *** Error: TIMED OUT! *** ***
*** *** Error: TIMED OUT! *** ***
[360.2 s]
exit code: 1024

----------------------------------------------------------------------
The following tests failed:


sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
Total time for all tests: 360.3 seconds

In a retry:


[jaap@peace sage-3.4.2.rc0]$ ./sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
**********************************************************************
File "/home/jaap/Download/sage-3.4.2.rc0/devel/sage/sage/matrix/matrix_symbolic_dense.pyx", line 214:
sage: -matrix(SR, 2, range(4))
Exception raised:
Traceback (most recent call last):
File "/home/jaap/Download/sage-3.4.2.rc0/local/bin/ncadoctest.py", line 1231, in run_one_test
self.run_one_example(test, example, filename, compileflags)
File "/home/jaap/Download/sage-3.4.2.rc0/local/bin/sagedoctest.py", line 38, in run_one_example
OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
File "/home/jaap/Download/sage-3.4.2.rc0/local/bin/ncadoctest.py", line 1172, in run_one_example
compileflags, 1) in test.globs
File "<doctest __main__.example_11[2]>", line 1, in <module>
-matrix(SR, Integer(2), range(Integer(4)))###line 214:
sage: -matrix(SR, 2, range(4))
File "matrix0.pyx", line 1497, in sage.matrix.matrix0.Matrix.__repr__ (sage/matrix/matrix0.c:7963)
File "matrix0.pyx", line 1526, in sage.matrix.matrix0.Matrix.str (sage/matrix/matrix0.c:8252)
File "matrix0.pyx", line 172, in sage.matrix.matrix0.Matrix.list (sage/matrix/matrix0.c:2839)
File "matrix_symbolic_dense.pyx", line 349, in sage.matrix.matrix_symbolic_dense.Matrix_symbolic_dense._list (sage/matrix/matrix_symbolic_dense.c:4257)
File "matrix_symbolic_dense.pyx", line 123, in sage.matrix.matrix_symbolic_dense.Matrix_symbolic_dense.get_unsafe (sage/matrix/matrix_symbolic_dense.c:3011)
File "/home/jaap/Download/sage-3.4.2.rc0/local/lib/python2.5/site-packages/sage/calculus/calculus.py", line 10261, in symbolic_expression_from_maxima_string
raise TypeError, "unable to make sense of Maxima expression '%s' in Sage"%s
TypeError: unable to make sense of Maxima expression '(-sage196)[1,1]' in Sage
**********************************************************************
File "/home/jaap/Download/sage-3.4.2.rc0/devel/sage/sage/matrix/matrix_symbolic_dense.pyx", line 413:


sage: M.determinant()
Expected:
4*x - 6
Got:
determinant(sage513)
**********************************************************************

2 items had failures:
1 of 3 in __main__.example_11


1 of 9 in __main__.example_18

***Test Failed*** 2 failures.
For whitespace errors, see the file /home/jaap/Download/sage-3.4.2.rc0/tmp/.doctest_matrix_symbolic_dense.py
[89.0 s]
exit code: 1024

----------------------------------------------------------------------
The following tests failed:


sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
Total time for all tests: 89.0 seconds
[jaap@peace sage-3.4.2.rc0]$

Jaap

John H Palmieri

unread,
Apr 30, 2009, 11:21:02 PM4/30/09
to sage-devel
On Apr 30, 5:23 am, mabshoff <mabsh...@googlemail.com> wrote:
> Hello folks,
>
> here goes 3.4.2.rc0 - a little later than planned, but it seems like
> we fixed all the issues (and more) that needed to be fixed. We finally
> merged the symbolic logic code written well over *18* months ago, but
> "no doctests, no merge" is something we take seriously. On top of that
> we are at
>
> Overall weighted coverage score: 71.1%
> Total number of functions: 22348
> We need 881 more function to get to 75% coverage.
> We need 1999 more function to get to 80% coverage.
> We need 3116 more function to get to 85% coverage.
>
> Note that this is also due to splitting DSage into its own spkg. We
> will discuss what to do with the DSage codebase at SD15 I assume.
>
> The source tarball, upgrade bits as well as a sage.math binary are in
> the usual place at
>
> http://sage.math.washington.edu/home/mabshoff/release-cycles-3.4.2/
>
> I have done some preliminary build testing and it seems to at least
> build. From here on we will only merge truly critical bugs that fix
> doctests and grave issues, otherwise better luck in Sage 4.0 :).
>
> Please build, doctest and report any issues.

Mac OS X 10.5, Intel: on an upgraded version, I had one doctest
failure,

sage -t "devel/sage/sage/interfaces/psage.py"
**********************************************************************
File "/Applications/sage_builds/sage-3.4.2.rc0-upgrade/devel/sage/sage/
interfaces/psage.py", line 35:
sage: print "ignore this"; w # random output
Exception raised:
Traceback (most recent call last):
File "/Applications/sage_builds/sage-3.4.2.alpha0-upgrade/local/
bin/ncadoctest.py", line 1231, in run_one_test
self.run_one_example(test, example, filename, compileflags)
File "/Applications/sage_builds/sage-3.4.2.alpha0-upgrade/local/
bin/sagedoctest.py", line 38, in run_one_example
OrigDocTestRunner.run_one_example(self, test, example,
filename, compileflags)
File "/Applications/sage_builds/sage-3.4.2.alpha0-upgrade/local/
bin/ncadoctest.py", line 1172, in run_one_example
compileflags, 1) in test.globs
File "<doctest __main__.example_0[5]>", line 1, in <module>
print "ignore this"; w # random output###line 35:
sage: print "ignore this"; w # random output
File "/Applications/sage_builds/sage-3.4.2.rc0-upgrade/local/lib/
python2.5/site-packages/sage/interfaces/expect.py", line 1557, in
__repr__
s = s.replace(self._name, self.__dict__['__custom_name'])
KeyError: '__custom_name'
**********************************************************************
1 items had failures:
1 of 6 in __main__.example_0
***Test Failed*** 1 failures.


On a fresh build, all tests passed.

On sage.math (the binary that you provided), I had one doctest
failure:

sage -t sage/sage/modular/hecke/hecke_operator.py
**********************************************************************
File "/scratch/palmieri/sage-3.4.2.rc0-sage.math-only-x86_64-Linux/
devel/sage-misc/sage/modular/hecke/hecke_operator.py", line 106:
sage: t2 - t3
Expected:
Hecke operator on Modular Symbols space of dimension 6 for Gamma_1
(6) of weight 4 with sign 0 and over Rational Field defined by:
(not printing 6 x 6 matrix)
Got:
Hecke operator on Modular Symbols space of dimension 6 for Gamma_1
(6) of weight 4 with sign 0 and over Rational Field defined by:
[ -19 0 0 4/7 -12/7 8/7]
[ 4 -26 0 -17/7 51/7 -34/7]
[ -18 0 7 -12/7 -6/7 18/7]
[ 0 -18 4 -16/7 34/7 -18/7]
[ 0 -18 4 -23/7 41/7 -18/7]
[ 0 -18 4 -23/7 34/7 -11/7]
**********************************************************************
File "/scratch/palmieri/sage-3.4.2.rc0-sage.math-only-x86_64-Linux/
devel/sage-misc/sage/modular/hecke/hecke_operator.py", line 156:
sage: t2 - t3
Expected:
Hecke operator on Modular Symbols space of dimension 6 for Gamma_1
(6) of weight 4 with sign 0 and over Rational Field defined by:
(not printing 6 x 6 matrix)
Got:
Hecke operator on Modular Symbols space of dimension 6 for Gamma_1
(6) of weight 4 with sign 0 and over Rational Field defined by:
[ -19 0 0 4/7 -12/7 8/7]
[ 4 -26 0 -17/7 51/7 -34/7]
[ -18 0 7 -12/7 -6/7 18/7]
[ 0 -18 4 -16/7 34/7 -18/7]
[ 0 -18 4 -23/7 41/7 -18/7]
[ 0 -18 4 -23/7 34/7 -11/7]
**********************************************************************

There were some warnings in building the html version of the reference
manual, and I've just posted a patch at #5951 to fix them.

John

Alex Ghitza

unread,
Apr 30, 2009, 11:24:45 PM4/30/09
to sage-...@googlegroups.com
On a couple of 32-bit machines running archlinux, upgraded and built
fine from 3.4.2.alpha0, and everything under "make ptestlong" passes.

Best,
Alex

mabshoff

unread,
Apr 30, 2009, 11:54:05 PM4/30/09
to sage-devel


On Apr 30, 8:21 pm, John H Palmieri <jhpalmier...@gmail.com> wrote:
> On Apr 30, 5:23 am, mabshoff <mabsh...@googlemail.com> wrote:
>

Hi John,

<SNIP>
Is this reproducible?
I might have had some patch applied that could cause this, i.e. #5736,
since I was playing around with it. I don't think I can reproduce this
at the moment and I am not seeing it with a clean build anyway.

> There were some warnings in building the html version of the reference
> manual, and I've just posted a patch at #5951 to fix them.

Cool, I will have a look.

>   John

Cheers,

Michael

John H Palmieri

unread,
May 1, 2009, 1:06:16 AM5/1/09
to sage-devel
Apparently not: I've been trying to get it again, but unsuccessfully.

John

mabshoff

unread,
May 1, 2009, 1:13:09 AM5/1/09
to sage-devel


On Apr 30, 10:06 pm, John H Palmieri <jhpalmier...@gmail.com> wrote:
> On Apr 30, 8:54 pm, mabshoff <mabsh...@googlemail.com> wrote:

<SNIP>

> > Is this reproducible?
>
> Apparently not: I've been trying to get it again, but unsuccessfully.

Yeah, my experience is that that code is inherently racy and no one
has even been able to reproduce it. I don't see it on sage.math at
all, but I am also use a RAM disk for DOT_SAGE, so I think that has
something to do with it.

>   John

It looks like we should consider merging "#5952: Worksheet downloading
blocks the entire server" since otherwise you can trivially DOS the
whole server. Maybe that option should be disabled per default for
now? No patch has been posted and I am not 100% sure RobertWB is
working on a patch. Comments?

I have also found another trivial documentation formatting problem for
the Victor Miller basis, so I would like that patch (which I should
post shortly at #5953) to go in since it basically adds a could ":"
and a few newline in a few places.

But other than that the deep freeze for 3.4.2 stays in effect :)

Cheers,

Michael

Justin C. Walker

unread,
May 1, 2009, 1:13:24 AM5/1/09
to sage-...@googlegroups.com
Hi, Michael,

On Apr 30, 2009, at 05:23 , mabshoff wrote:

>
> Hello folks,
>
> here goes 3.4.2.rc0 - a little later than planned, but it seems like
> we fixed all the issues (and more) that needed to be fixed. We finally
> merged the symbolic logic code written well over *18* months ago, but
> "no doctests, no merge" is something we take seriously. On top of that
> we are at
>
> Overall weighted coverage score: 71.1%
> Total number of functions: 22348
> We need 881 more function to get to 75% coverage.
> We need 1999 more function to get to 80% coverage.
> We need 3116 more function to get to 85% coverage.
>
> Note that this is also due to splitting DSage into its own spkg. We
> will discuss what to do with the DSage codebase at SD15 I assume.
>
> The source tarball, upgrade bits as well as a sage.math binary are in
> the usual place at
>
> http://sage.math.washington.edu/home/mabshoff/release-cycles-3.4.2/

My first attempt to build this (Mac OS X, 10.5.6, Dual Quad Xeon) blew
up with python-2.5. I had two of these errors:

Compiling /Users/tmp/sage-3.4.2.rc0/local/lib/python2.5/test/
test_multibytecodec.py ...
Sorry: UnicodeError: ("\\N escapes not supported (can't load
unicodedata module)",)

(i.e., twice with the same file), and the final straw was:

Sleeping for three seconds before testing python


Traceback (most recent call last):

File "<string>", line 1, in <module>
File "/tmp/sage-3.4.1/local/lib/python2.5/md5.py", line 6, in
<module>
from hashlib import md5
File "/tmp/sage-3.4.1/local/lib/python2.5/hashlib.py", line
133, in <module>
md5 = __get_builtin_constructor('md5')
File "/tmp/sage-3.4.1/local/lib/python2.5/hashlib.py", line 60,
in __get_builtin_constructor
import _md5
ImportError: No module named _md5
md5 module failed to import

I restarted the build without changing anything, and it completed w/o
complaints.

The log file is at
sage.math.washington.edu:~justin/logs/3.4.2.rc0.log
if you want to paw through it looking for hints; I didn't see anything
obvious, but your younger eyes are probably sharper than mine :-}

Justin

--
Justin C. Walker, Curmudgeon at Large
Institute for the Absorption of Federal Funds
--
Democracy is two wolves and a lamb
voting on what to have for lunch.
Liberty is a well-armed lamb contesting
the vote.

William Stein

unread,
May 1, 2009, 1:18:18 AM5/1/09
to sage-...@googlegroups.com
On Thu, Apr 30, 2009 at 10:13 PM, mabshoff <mabs...@googlemail.com> wrote:
>
>
>
> On Apr 30, 10:06 pm, John H Palmieri <jhpalmier...@gmail.com> wrote:
>> On Apr 30, 8:54 pm, mabshoff <mabsh...@googlemail.com> wrote:
>
> <SNIP>
>
>> > Is this reproducible?
>>
>> Apparently not: I've been trying to get it again, but unsuccessfully.
>
> Yeah, my experience is that that code is inherently racy and no one
> has even been able to reproduce it. I don't see it on sage.math at
> all, but I am also use a RAM disk for DOT_SAGE, so I think that has
> something to do with it.
>
>>   John
>
> It looks like we should consider merging "#5952: Worksheet downloading
> blocks the entire server" since otherwise you can trivially DOS the
> whole server.

> Maybe that option should be disabled per default for
> now? No patch has been posted and I am not 100% sure RobertWB is
> working on a patch. Comments?

I could easily DOS any sage notebook server right now in many ways,
which I won't list here since I don't want to give anybody any ideas.
#5952 only stops the server *while* the directories with the
worksheets are being tar'd up -- once that filesystem operation
finishes, things go back to being nonblocking.

> I have also found another trivial documentation formatting problem for
> the Victor Miller basis, so I would like that patch (which I should
> post shortly at #5953) to go in since it basically adds a could ":"
> and a few newline in a few places.
>
> But other than that the deep freeze for 3.4.2 stays in effect :)
>
> Cheers,
>
> Michael
>
>
> >
>



--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

Robert Bradshaw

unread,
May 1, 2009, 1:30:01 AM5/1/09
to sage-...@googlegroups.com

Yes, it is easily to DOS the notebook if one tries, but I'm wary of a
big button that "accidently" does so. Taring up the directories for a
dozen worksheets can take several minutes.

I'll try to get a patch out tonight, perhaps as a first pass it would
be good to use to the opposite of the accounts option in the notebook
() command.

- Robert

William Stein

unread,
May 1, 2009, 1:35:31 AM5/1/09
to sage-...@googlegroups.com
That's a very good point. Over (some) NFS setups it can really take
a long time indeed.

> I'll try to get a patch out tonight, perhaps as a first pass it would
> be good to use to the opposite of the accounts option in the notebook
> () command.

Yep. Thanks!!

William

>
> - Robert

Jaap Spies

unread,
May 1, 2009, 3:05:54 AM5/1/09
to sage-...@googlegroups.com
Jaap Spies wrote:
> mabshoff wrote:
>> Hello folks,

>
> Now doing a fresh install on this machine.
>


On this fresh install first success, followed by the failure:

SAGE build/upgrade complete!
[jaap@paix sage-3.4.2.rc0]$ ./sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
[125.6 s]

----------------------------------------------------------------------
All tests passed!
Total time for all tests: 125.8 seconds
[jaap@paix sage-3.4.2.rc0]$ ./sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
**********************************************************************
File "/home/jaap/downloads/sage-3.4.2.rc0/devel/sage/sage/matrix/matrix_symbolic_dense.pyx", line 413:


sage: M.determinant()
Expected:
4*x - 6
Got:
determinant(sage513)
**********************************************************************
1 items had failures:
1 of 9 in __main__.example_18
***Test Failed*** 1 failures.

For whitespace errors, see the file /home/jaap/downloads/sage-3.4.2.rc0/tmp/.doctest_matrix_symbolic_dense.py
[74.1 s]
exit code: 1024

----------------------------------------------------------------------
The following tests failed:


sage -t "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
Total time for all tests: 74.1 seconds
[jaap@paix sage-3.4.2.rc0]$

Weird!


Jaap


mabshoff

unread,
May 1, 2009, 3:13:38 AM5/1/09
to sage-devel


On May 1, 12:05 am, Jaap Spies <j.sp...@hccnet.nl> wrote:
> Jaap Spies wrote:
> > mabshoff wrote:
> >> Hello folks,
>
> > Now doing a fresh install on this machine.

Hi Jaap,

> On this fresh install first success, followed by the failure:
>
> SAGE build/upgrade complete!
> [jaap@paix sage-3.4.2.rc0]$ ./sage -t  "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
> sage -t  "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
>          [125.6 s]

<SNIP>

>         sage -t  "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
> Total time for all tests: 74.1 seconds
> [jaap@paix sage-3.4.2.rc0]$
>
> Weird!

Yeah, I don't want to invest any time debugging this - I have not been
able to reproduce this anyway - and I much rather spend time on the
ecl switch. If that is still there after dumping clisp we will take
another stab at this.

While going over the open tickets in 4.0 I noticed this ticket:

#5943 (Sage 3.4.2.a0: prime_pi(2^50) segfaults)

If someone could take a stab at that it would be nice since that is
brand new code and ought to be a little bit more stable than that. I
am waiting on a fix for #5952 in the morning, i.e. Saturday, so some
hero might want to earn some brownie points over night :)

> Jaap

Cheers,

Michael

John Cremona

unread,
May 1, 2009, 4:12:50 AM5/1/09
to sage-...@googlegroups.com
Three successful builds from scratch of 3.4.2.rc0 with all tests
passing (32-bit ubuntu, 32-bit Suse and 64-bit ubuntu).

John

2009/5/1 mabshoff <mabs...@googlemail.com>:

mabshoff

unread,
May 1, 2009, 5:24:53 AM5/1/09
to sage-devel


On May 1, 12:13 am, mabshoff <mabsh...@googlemail.com> wrote:
> On May 1, 12:05 am, Jaap Spies <j.sp...@hccnet.nl> wrote:

<SNIP>

> > SAGE build/upgrade complete!
> > [jaap@paix sage-3.4.2.rc0]$ ./sage -t  "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
> > sage -t  "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
> >          [125.6 s]
>
> <SNIP>
>
> >         sage -t  "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
> > Total time for all tests: 74.1 seconds
> > [jaap@paix sage-3.4.2.rc0]$
>
> > Weird!
>
> Yeah, I don't want to invest any time debugging this - I have not been
> able to reproduce this anyway - and I much rather spend time on the
> ecl switch. If that is still there after dumping clisp we will take
> another stab at this.

Ahhhhhhhhhhh, I jinxed myself. On a 32 bit FC10 box:

sage -t -long "devel/sage/sage/matrix/
matrix_symbolic_dense.pyx"
Total time for all tests: 11440.3 seconds
[mabshoff@cicero sage-3.4.2.rc0-ciero-gcc-4.3.3]$ ./sage -t -long
"devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
sage -t -long "devel/sage/sage/matrix/matrix_symbolic_dense.pyx"
**********************************************************************
File "/home/mabshoff/build-3.4.2.rc0/sage-3.4.2.rc0-ciero-gcc-4.3.3/
devel/sage/sage/matrix/matrix_symbolic_dense.pyx", line 413:
sage: M.determinant()
Expected:
4*x - 6
Got:
determinant(sage513)
**********************************************************************
1 items had failures:
1 of 9 in __main__.example_18
***Test Failed*** 1 failures.
For whitespace errors, see the file /home/mabshoff/build-3.4.2.rc0/
sage-3.4.2.rc0-ciero-gcc-4.3.3/tmp/.doctest_matrix_symbolic_dense.py
[113.0 s]
exit code: 1024

So I guess we will take a stab at this in the morning.

Cheers,

Michael

Alex Ghitza

unread,
May 1, 2009, 8:25:55 AM5/1/09
to sage-...@googlegroups.com
For the record, I'm also getting this on an older laptop running
32-bit arch linux.

Alex
--
Alex Ghitza -- Lecturer in Mathematics -- The University of Melbourne
-- Australia -- http://www.ms.unimelb.edu.au/~aghitza/

Kiran Kedlaya

unread,
May 1, 2009, 9:54:04 AM5/1/09
to sage-devel
Clean build on 64-bit Fedora 10 (Opteron) fails one doctest:

sage -t "devel/sage/sage/sets/primes.py"
**********************************************************************
File "/opt/sage/sage-3.4.2.rc0/devel/sage/sage/sets/primes.py", line
80:
sage: P>x^2+x
Expected:
True
Got:
False
**********************************************************************
1 items had failures:
1 of 12 in __main__.example_4
***Test Failed*** 1 failures.
For whitespace errors, see the file /opt/sage/sage-3.4.2.rc0/
tmp/.doctest_primes.py
[1.7 s]
exit code: 1024

----------------------------------------------------------------------
Kiran

mabshoff

unread,
May 1, 2009, 10:00:33 AM5/1/09
to sage-devel


On May 1, 6:54 am, Kiran Kedlaya <ksk...@gmail.com> wrote:
> Clean build on 64-bit Fedora 10 (Opteron) fails one doctest:

Hi Kiran,

> sage -t  "devel/sage/sage/sets/primes.py"
> **********************************************************************
> File "/opt/sage/sage-3.4.2.rc0/devel/sage/sage/sets/primes.py", line
> 80:
>     sage: P>x^2+x
> Expected:
>     True
> Got:
>     False

I can only see this fail if Maxima failed to start up. Even though the
doctest is a little silly IMHO, i.e. why on earth would be set of
primes be greater than a symbolic polynomial. In fact the a primeset
should be greater than *anything* but another primeset.

>
> ----------------------------------------------------------------------
> Kiran

Cheers,

Michael

mabshoff

unread,
May 1, 2009, 10:01:44 AM5/1/09
to sage-devel


On May 1, 5:25 am, Alex Ghitza <aghi...@gmail.com> wrote:
> For the record, I'm also getting this on an older laptop running
> 32-bit arch linux.

Thanks, I have seen this no on Linux for x86, x86-64 as well as
Itanium, so we ought to get a handle on this today. It is now #5957.

> Alex

Cheers,

Michael

kcrisman

unread,
May 1, 2009, 10:38:54 AM5/1/09
to sage-devel
> >     sage: P>x^2+x
> > Expected:
> >     True
> > Got:
> >     False

Yes, I just added that test to show that Primes() has comparison. I
was a little surprised to see the result, but it passed testing, so...

If someone posts a ticket with a suggested fix that makes Primes()
greater than everything, I can try to implement it - I but for now I
just wanted to make sure it was clear Primes() did compare to weird
things and not throw an error.

Note in the docstring for comparing (current) symbolics that
Some comparisons are fairly arbitrary but
consistent:: sage: cmp(SR(3), x) #random due to
architecture dependence -1 sage: cmp(x, SR(3))
#random due to architecture dependence 1 """
so perhaps it's not a Maxima thing but a symbolic comparison thing.

- kcrisman

mabshoff

unread,
May 1, 2009, 10:45:00 AM5/1/09
to sage-devel


On May 1, 7:38 am, kcrisman <kcris...@gmail.com> wrote:
> > >     sage: P>x^2+x
> > > Expected:
> > >     True
> > > Got:
> > >     False
>
> Yes, I just added that test to show that Primes() has comparison.  I
> was a little surprised to see the result, but it passed testing, so...
>
> If someone posts a ticket with a suggested fix that makes Primes()
> greater than everything, I can try to implement it - I  but for now I
> just wanted to make sure it was clear Primes() did compare to weird
> things and not throw an error.

Well, what I wrote was nonsense. __cmp__ always returns -1 unless both
are Primes(), i.e what is greater depends on what is left and right
respectively. I don't care too much about that now, but if someone has
a better suggestion I am interested in hearing it.

> Note in the docstring for comparing (current) symbolics that
>         Some comparisons are fairly arbitrary but
> consistent::                    sage: cmp(SR(3), x) #random due to
> architecture dependence            -1            sage: cmp(x, SR(3))
> #random due to architecture dependence            1        """
> so perhaps it's not a Maxima thing but a symbolic comparison thing.

Never write a doctest to be #random if you can trivially work around
it. In both cases the point is that cmp() should not return "0" since
x and SR(3) aren't equal. Should that ever be broken the doctest would
not catch it, so something like

sage: cmp(SR(3), x) in [-1,1]
True

is much better in that case.

> - kcrisman

Cheers,

Michael

William Stein

unread,
May 1, 2009, 10:49:11 AM5/1/09
to sage-...@googlegroups.com
On Fri, May 1, 2009 at 7:45 AM, mabshoff <mabs...@googlemail.com> wrote:
>
>
>
> On May 1, 7:38 am, kcrisman <kcris...@gmail.com> wrote:
>> > >     sage: P>x^2+x
>> > > Expected:
>> > >     True
>> > > Got:
>> > >     False
>>
>> Yes, I just added that test to show that Primes() has comparison.  I
>> was a little surprised to see the result, but it passed testing, so...

You should change the doctest to

sage: P != x^2 + x
True

The comparison is completely arbitrary and will be machine specific.
However equality or not is not arbitrary.

William

>>
>> If someone posts a ticket with a suggested fix that makes Primes()
>> greater than everything, I can try to implement it - I  but for now I
>> just wanted to make sure it was clear Primes() did compare to weird
>> things and not throw an error.
>
> Well, what I wrote was nonsense. __cmp__ always returns -1 unless both
> are Primes(), i.e what is greater depends on what is left and right
> respectively. I don't care too much about that now, but if someone has
> a better suggestion I am interested in hearing it.
>
>> Note in the docstring for comparing (current) symbolics that
>>         Some comparisons are fairly arbitrary but
>> consistent::                    sage: cmp(SR(3), x) #random due to
>> architecture dependence            -1            sage: cmp(x, SR(3))
>> #random due to architecture dependence            1        """
>> so perhaps it's not a Maxima thing but a symbolic comparison thing.
>
> Never write a doctest to be #random if you can trivially work around
> it. In both cases the point is that cmp() should not return "0" since
> x and SR(3) aren't equal. Should that ever be broken the doctest would
> not catch it, so something like
>
> sage: cmp(SR(3), x) in [-1,1]
> True
>
> is much better in that case.
>
>> - kcrisman
>
> Cheers,
>
> Michael
> >
>



Jason Grout

unread,
May 1, 2009, 10:57:33 AM5/1/09
to sage-...@googlegroups.com
mabshoff wrote:
>
>
> On May 1, 7:38 am, kcrisman <kcris...@gmail.com> wrote:
>>>> sage: P>x^2+x
>>>> Expected:
>>>> True
>>>> Got:
>>>> False
>> Yes, I just added that test to show that Primes() has comparison. I
>> was a little surprised to see the result, but it passed testing, so...
>>
>> If someone posts a ticket with a suggested fix that makes Primes()
>> greater than everything, I can try to implement it - I but for now I
>> just wanted to make sure it was clear Primes() did compare to weird
>> things and not throw an error.
>
> Well, what I wrote was nonsense. __cmp__ always returns -1 unless both
> are Primes(), i.e what is greater depends on what is left and right
> respectively. I don't care too much about that now, but if someone has
> a better suggestion I am interested in hearing it.


In the future of python, things that don't have a sensible order throw a
TypeError when comparing:

http://docs.python.org/3.0/whatsnew/3.0.html#ordering-comparisons

Why don't we just throw an error?

Jason

--
Jason Grout

kcrisman

unread,
May 1, 2009, 11:15:49 AM5/1/09
to sage-devel
> You should change the doctest to
>
> sage: P != x^2 + x
> True
>
> The comparison is completely arbitrary and will be machine specific.
> However equality or not is not arbitrary.
>
> > sage: cmp(SR(3), x) in [-1,1]
> > True
>


Okay, those both seem like reasonable suggestions - patch is up with
William's fix at # 5959.

> In the future of python, things that don't have a sensible order throw a TypeError when comparing:
> http://docs.python.org/3.0/whatsnew/3.0.html#ordering-comparisons
> Why don't we just throw an error?

As for throwing an error, probably that should be done Sage-wide for
things that don't make sense, not piecemeal. At least I don't feel
comfortable opening that can of worms!

- kcrisman

Robert Bradshaw

unread,
May 1, 2009, 3:16:39 PM5/1/09
to sage-...@googlegroups.com
On May 1, 2009, at 8:15 AM, kcrisman wrote:

>> In the future of python, things that don't have a sensible order
>> throw a TypeError when comparing:
>> http://docs.python.org/3.0/whatsnew/3.0.html#ordering-comparisons
>> Why don't we just throw an error?
>
> As for throwing an error, probably that should be done Sage-wide for
> things that don't make sense, not piecemeal. At least I don't feel
> comfortable opening that can of worms!

Yep, this was opened a while back, and it turned into one of the
longest treads ever... I would still like to see conclusion on that.

- Robert

William Stein

unread,
May 1, 2009, 3:31:40 PM5/1/09
to sage-...@googlegroups.com

Despite being a huge thread, the resolution is really easy. I think
we should do the following: "The ordering comparison operators (<, <=,
>=, >) should raise a TypeError exception when the operands don’t have
a meaningful natural ordering. Thus, expressions like 1 < '', 0 > None
or len <= len will no longer be valid. A corollary is that sorting a
heterogeneous list no longer makes sense – all the elements must be
comparable to each other. Note that this does not apply to the == and
!= operators: objects of different incomparable types will always
compare unequal to each other."

The question is now how to do this. Little by little, or all at once.
I prefer little by little whenever possible, so is that possible
here?

-- William

Marshall Hampton

unread,
May 1, 2009, 4:24:22 PM5/1/09
to sage-devel
All tests passed on my intel macbook running 10.5.

-M. Hampton

Nicolas M. Thiery

unread,
May 1, 2009, 5:29:36 PM5/1/09
to sage-...@googlegroups.com
On Fri, May 01, 2009 at 12:31:40PM -0700, William Stein wrote:
>
> On Fri, May 1, 2009 at 12:16 PM, Robert Bradshaw
> <robe...@math.washington.edu> wrote:
> >
> > On May 1, 2009, at 8:15 AM, kcrisman wrote:
> >
> >>> In the future of python, things that don't have a sensible order
> >>> throw a TypeError when comparing:
> >>> http://docs.python.org/3.0/whatsnew/3.0.html#ordering-comparisons
> >>> Why don't we just throw an error?
> >>
> >> As for throwing an error, probably that should be done Sage-wide for
> >> things that don't make sense, not piecemeal.  At least I don't feel
> >> comfortable opening that can of worms!
> >
> > Yep, this was opened a while back, and it turned into one of the
> > longest treads ever... I would still like to see conclusion on that.
>
> Despite being a huge thread, the resolution is really easy. I think
> we should do the following: "The ordering comparison operators (<, <=,
> >=, >) should raise a TypeError exception when the operands don’t have
> a meaningful natural ordering. Thus, expressions like 1 < '', 0 > None
> or len <= len will no longer be valid. A corollary is that sorting a
> heterogeneous list no longer makes sense – all the elements must be
> comparable to each other. Note that this does not apply to the == and
> != operators: objects of different incomparable types will always
> compare unequal to each other."

+1

What's not resolved is how much the system should try to be clever
when comparing, e.g., and element of QQ[sqrt(2)] and of QQ[sqrt(3)].

> The question is now how to do this. Little by little, or all at
> once. I prefer little by little whenever possible, so is that
> possible here?

If this can help measure the progress, I can add a systematic test in
the Sets() category, so that for P a parent, P.check() will check that
P.an_element() < None actually raises an error.

Cheers,
Nicolas
--
Nicolas M. Thiéry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

Robert Bradshaw

unread,
May 1, 2009, 5:50:41 PM5/1/09
to sage-...@googlegroups.com

There was a lot of other stuff in that thread, but yes, lets do this.

> What's not resolved is how much the system should try to be clever
> when comparing, e.g., and element of QQ[sqrt(2)] and of QQ[sqrt(3)].

If a + b works, than a < b should be compared in parent(a+b) (which
may or may not have an ordering). For the moment, the above would
fail. (In fact, QQ[sqrt(a)] doesn't yet come with an embedding...)

Another unresolved issue was how to handle RealField(53).pi() ==
RealField(500).pi()

- Robert

Justin C. Walker

unread,
May 1, 2009, 9:56:22 PM5/1/09
to sage-...@googlegroups.com

On Apr 30, 2009, at 22:13 , Justin C. Walker wrote:

>
> Hi, Michael,
>
> On Apr 30, 2009, at 05:23 , mabshoff wrote:
>
>>
>> Hello folks,
>>
>> here goes 3.4.2.rc0 - a little later than planned, but it seems like
>> we fixed all the issues (and more) that needed to be fixed. We
>> finally
>> merged the symbolic logic code written well over *18* months ago, but
>> "no doctests, no merge" is something we take seriously. On top of
>> that
>> we are at
>>
>> Overall weighted coverage score: 71.1%
>> Total number of functions: 22348
>> We need 881 more function to get to 75% coverage.
>> We need 1999 more function to get to 80% coverage.
>> We need 3116 more function to get to 85% coverage.
>>
>> Note that this is also due to splitting DSage into its own spkg. We
>> will discuss what to do with the DSage codebase at SD15 I assume.
>>
>> The source tarball, upgrade bits as well as a sage.math binary are in
>> the usual place at
>>
>> http://sage.math.washington.edu/home/mabshoff/release-cycles-3.4.2/
>
> My first attempt to build this (Mac OS X, 10.5.6, Dual Quad Xeon) blew
> up with python-2.5. I had two of these errors:

[snip]


> I restarted the build without changing anything, and it completed w/o
> complaints.

"make test" for the second attempt would not work because of seg-
faults very early in the run.

I then rebuilt (3rd time) and retested 3.4.2.rc0 in the above
configuration. This time, it built w/o errors. Testing showed only
one problem, a failure in the dsage test which I've seen before:

=
=
=
=
=
=
=
========================================================================
[ERROR]:
sage.dsage.twisted.tests.test_remote.ClientRemoteCallsTest.testremoteSu
bmitBadJob

Traceback (most recent call last):

Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was
unclean.
Selectables:
<Broker #0 on 60222>
=
=
=
=
=
=
=
========================================================================
[ERROR]:
sage
.dsage
.twisted.tests.test_remote.WorkerRemoteCallsTest.testremote_job_failed

I reran sage-dsage-trial by hand and it passed.

Justin

--
Justin C. Walker, Curmudgeon-At-Large


Institute for the Absorption of Federal Funds

--------
If you're not confused,
You're not paying attention
--------

Alex Ghitza

unread,
May 2, 2009, 5:58:17 AM5/2/09
to sage-...@googlegroups.com
On Fri, May 1, 2009 at 5:13 PM, mabshoff <mabs...@googlemail.com> wrote:
>
> While going over the open tickets in 4.0 I noticed this ticket:
>
>   #5943 (Sage 3.4.2.a0: prime_pi(2^50) segfaults)
>
> If someone could take a stab at that it would be nice since that is
> brand new code and ought to be a little bit more stable than that. I
> am waiting on a fix for #5952 in the morning, i.e. Saturday, so some
> hero might want to earn some brownie points over night :)
>

Brownies are overrated.

I played around with prime_pi() for a while, both on sage.math and on
my laptop at the office (macbook running 32-bit archlinux). I didn't
manage to get a segfault on either machine with prime_pi(2^50). I
guess that's the good news? Anyway, the bad news is that the answers
returned do not agree between the two machines, e.g.

{{{
# on sage.math
sage: time prime_pi(2^50)
CPU times: user 4854.46 s, sys: 3.17 s, total: 4857.63 s
Wall time: 4857.73 s
33483379603407
#
# on my laptop
sage: time prime_pi(2^50)
CPU times: user 5555.60 s, sys: 7.81 s, total: 5563.40 s
Wall time: 5598.64 s
21969300962685
}}}

I'm pretty sure that the sage.math answer is more likely to be the
right one. You can maybe guess from the timings why I didn't try
prime_pi(2^51). I have, however, tried smaller values. I'm going to
put that data up on the trac ticket.

I have absolutely no idea what's wrong, but something definitely is,
and hopefully this will help someone fix it.


Best,
Alex

mabshoff

unread,
May 2, 2009, 4:00:22 PM5/2/09
to sage-devel


On May 2, 2:58 am, Alex Ghitza <aghi...@gmail.com> wrote:
> On Fri, May 1, 2009 at 5:13 PM, mabshoff <mabsh...@googlemail.com> wrote:
>
> > While going over the open tickets in 4.0 I noticed this ticket:
>
> >   #5943 (Sage 3.4.2.a0: prime_pi(2^50) segfaults)
>
> > If someone could take a stab at that it would be nice since that is
> > brand new code and ought to be a little bit more stable than that. I
> > am waiting on a fix for #5952 in the morning, i.e. Saturday, so some
> > hero might want to earn some brownie points over night :)
>
> Brownies are overrated.

:)

> I played around with prime_pi() for a while, both on sage.math and on
> my laptop at the office (macbook running 32-bit archlinux).  I didn't
> manage to get a segfault on either machine with prime_pi(2^50).  I
> guess that's the good news?  

Somewhat. As it turns out 3.4.2.a0 does not blow up, but 3.4.1 does:

mabshoff@sage:/scratch/mabshoff/sage-3.4.2.final$ sage
----------------------------------------------------------------------
| Sage Version 3.4.1, Release Date: 2009-04-21 |
| Type notebook() for the GUI, and license() for information. |
----------------------------------------------------------------------
sage: time prime_pi(2^50)
/usr/local/sage/local/bin/sage-sage: line 198: 13649 Segmentation
fault sage-ipython "$@" -i
mabshoff@sage:/scratch/mabshoff/sage-3.4.2.final$ ./sage
----------------------------------------------------------------------
| Sage Version 3.4.2.rc0, Release Date: 2009-04-30 |
| Type notebook() for the GUI, and license() for information. |
----------------------------------------------------------------------
sage: time prime_pi(2^50)
<SNIP>

> Anyway, the bad news is that the answers
> returned do not agree between the two machines, e.g.
>
> {{{
> # on sage.math
> sage: time prime_pi(2^50)
> CPU times: user 4854.46 s, sys: 3.17 s, total: 4857.63 s
> Wall time: 4857.73 s
> 33483379603407
> #
> # on my laptop
> sage: time prime_pi(2^50)
> CPU times: user 5555.60 s, sys: 7.81 s, total: 5563.40 s
> Wall time: 5598.64 s
> 21969300962685
>
> }}}

Interesting. I tried to find a similar counter example with Solaris/
x86 and Solaris/Sparc, but it seems I wasn't patient enough and I did
also use two 32 bit versions.

What to do? Put a cap on the arguments prime_pi gives you an answer
and throw NotImplementedError otherwise, i.e. for some 2^n we feel
comfortable enough.

> I'm pretty sure that the sage.math answer is more likely to be the
> right one.  You can maybe guess from the timings why I didn't try
> prime_pi(2^51).  I have, however, tried smaller values.  I'm going to
> put that data up on the trac ticket.

Thanks.

> I have absolutely no idea what's wrong, but something definitely is,
> and hopefully this will help someone fix it.

The code uses doubles and a "pseudo" long double sqrt (with build in
error checking for the pseudo long double function). I am not really
surprised that we are seeing issues for large inputs and I am glad you
detected the problem :)

> Best,
> Alex

Cheers,

Michael

mabshoff

unread,
May 2, 2009, 4:17:35 PM5/2/09
to sage-devel


On May 2, 1:00 pm, mabshoff <mabsh...@googlemail.com> wrote:
> On May 2, 2:58 am, Alex Ghitza <aghi...@gmail.com> wrote:

<SNIP>

> > I played around with prime_pi() for a while, both on sage.math and on
> > my laptop at the office (macbook running 32-bit archlinux).  I didn't
> > manage to get a segfault on either machine with prime_pi(2^50).  I
> > guess that's the good news?  
>
> Somewhat. As it turns out 3.4.2.a0 does not blow up, but 3.4.1 does:
>
> mabshoff@sage:/scratch/mabshoff/sage-3.4.2.final$ sage
> ----------------------------------------------------------------------
> | Sage Version 3.4.1, Release Date: 2009-04-21                       |
> | Type notebook() for the GUI, and license() for information.        |
> ----------------------------------------------------------------------
> sage: time prime_pi(2^50)
> /usr/local/sage/local/bin/sage-sage: line 198: 13649 Segmentation
> fault      sage-ipython "$@" -i
> mabshoff@sage:/scratch/mabshoff/sage-3.4.2.final$ ./sage
> ----------------------------------------------------------------------
> | Sage Version 3.4.2.rc0, Release Date: 2009-04-30                   |
> | Type notebook() for the GUI, and license() for information.        |
> ----------------------------------------------------------------------
> sage: time prime_pi(2^50)
> <SNIP>

Ok, in hindsight it is pretty obvious why this doesn't segfault in
3.4.2.a0 any more:

mabshoff@sage:/scratch/mabshoff/sage-3.4.2.final$ ./sage
----------------------------------------------------------------------
| Sage Version 3.4.2.rc0, Release Date: 2009-04-30 |
| Type notebook() for the GUI, and license() for information. |
----------------------------------------------------------------------
sage: len(prime_range(2^50))
/scratch/mabshoff/sage-3.4.2.final/local/bin/sage-sage: line 198:
13833 Segmentation fault sage-ipython "$@" -i

So I am rewriting the tickets: #5943 is about the still existing crash
in 3.4.2.final while #5963 is about the wrong results for prime_pi()
on some platforms.

Cheers,

Michael

Dr. David Kirkby

unread,
May 2, 2009, 5:44:46 PM5/2/09
to sage-...@googlegroups.com

Mathematica 6 (on a Sun SPARC) gives an answer in far less time than Sage:


In[3]:= PrimePi[2^50]

PrimePi::largp:
Argument 1125899906842624 in PrimePi[1125899906842624]
is too large for this implementation.

Out[3]= PrimePi[1125899906842624]


Well, perhaps not really an answer!

mabshoff

unread,
May 2, 2009, 5:50:11 PM5/2/09
to sage-devel


On May 2, 2:44 pm, "Dr. David Kirkby" <david.kir...@onetel.net> wrote:
> Alex Ghitza wrote:

Hi David,

<SNIP>

> > I'm pretty sure that the sage.math answer is more likely to be the
> > right one.  You can maybe guess from the timings why I didn't try
> > prime_pi(2^51).  I have, however, tried smaller values.  I'm going to
> > put that data up on the trac ticket.
>
> Mathematica 6 (on a Sun SPARC) gives an answer in far less time than Sage:
>
> In[3]:= PrimePi[2^50]
>
> PrimePi::largp:
>     Argument 1125899906842624 in PrimePi[1125899906842624]
>       is too large for this implementation.
>
> Out[3]= PrimePi[1125899906842624]
>
> Well, perhaps not really an answer!

:)

Could you figure out what the upper bound is that MMA allows? I have
discussed this with William in IRC and in 3.4.2 we should just throw a
NotImplementedError for some bound where we are comfortable with
knowing the result is correct on 32 and 64 bit. Unfortunately this
isn't something we can doctest with a reasonable amount of time.

The suggestion then was to implement something on top of the range
computed with floats using MPFR for example, but we will see what
happens. I am sure that if I asked if someone needed to compute
prime_pi() for anything larger than 2^48 someone would say yes, so
this ought to be fixed.

Cheers,

Michael

Dr. David Kirkby

unread,
May 2, 2009, 6:01:14 PM5/2/09
to sage-...@googlegroups.com


You should have used Mathematica - it returns its answer very quickly:


Mathematica 6.0 for Sun Solaris SPARC (64-bit)
Copyright 1988-2008 Wolfram Research, Inc.

In[1]:= ?PrimePi
PrimePi[x] gives the number of primes \[Pi] (x) less than or equal to x.

In[2]:= PrimePi[2^50]

PrimePi::largp:
Argument 1125899906842624 in PrimePi[1125899906842624]
is too large for this implementation.

Out[2]= PrimePi[1125899906842624]

William Stein

unread,
May 2, 2009, 6:52:19 PM5/2/09
to sage-...@googlegroups.com, R. Andrew Ohana

Andrew looked into this whole issue a while ago, and told me that the
prime_pi he implemented *should* only work up to about 2^40, and the
algorithm would take far too long above there. I thought he had
included an error message if the input exceeds 2^40, but I guess not.
So +1 to your suggestion above, but with a smaller bound that 2^48.

He told me Mathematica can go up to about 2^45 or so, but not beyond.
The algorithm in Mathematica is completely different (and better) than
what Andrew implemented for Sage. As far as I know the situation for
computing pi(X) using general purpose math software is thus:

* Mathematica -- has the best implementation available

* Sage -- now has the second best implementation available

* Pari, Maple, Matlab -- "stupid" implementations, which all simply
enumerate all primes up to x and see how many there are. Useless, and
can't come close to what Sage or Mathematica do.

-- William

mabshoff

unread,
May 2, 2009, 7:04:25 PM5/2/09
to sage-devel


On May 2, 3:52 pm, William Stein <wst...@gmail.com> wrote:
> On Sat, May 2, 2009 at 2:50 PM, mabshoff <mabsh...@googlemail.com> wrote:

<SNIP>

> > The suggestion then was to implement something on top of the range
> > computed with floats using MPFR for example, but we will see what
> > happens. I am sure that if I asked if someone needed to compute
> > prime_pi() for anything larger than 2^48 someone would say yes, so
> > this ought to be fixed.
>
> Andrew looked into this whole issue a while ago, and told me that the
> prime_pi he implemented *should* only work up to about 2^40, and the
> algorithm would take far too long above there.   I thought he had
> included an error message if the input exceeds 2^40, but I guess not.
>    So +1 to your suggestion above, but with a smaller bound that 2^48.

Cool.

> He told me Mathematica can go up to about 2^45 or so, but not beyond.

At least for MMA 6.0 on linux x86-64 the limit seems to be around
2^47:

MMA Sage

2^44: 18.04 110.88 (597116381732)
2^45: 29.98 207.61 (1166746786182)
2^46: 47.59 389.98 (2280998753949)
2^47: 89.25 728.84 (4461632979717)
2^48: NA :) about an hour - correct?

According to Alex's numbers at least on his laptop 2^46 was correct on
32 bits, but given the length of the test (~6 minutes on sage.math
this isn't really doctestable).

> The algorithm in Mathematica is completely different (and better) than
> what Andrew implemented for Sage.   As far as I know the situation for
> computing pi(X) using general purpose math software is thus:
>
>    * Mathematica -- has the best implementation available
>
>    * Sage -- now has the second best implementation available

Yep, the old implementation was about 1000 times slower than Andrew's
which is about 5 times slower than MMA 6.0 - so great job from
catching us up from 5000 times to only 5 times :).

>    * Pari, Maple, Matlab -- "stupid" implementations, which all simply
> enumerate all primes up to x and see how many there are.  Useless, and
> can't come close to what Sage or Mathematica do.

Well, what should we pick as upper bound? 2^40 seems to be what Andrew
suggests, but maybe 2^42 or 2^43? In that range we can actually add
#long doctests and I would be much more comfortable that we would at
least catch potential issues.

>  -- William

Cheers,

Michael

Dr. David Kirkby

unread,
May 2, 2009, 7:23:17 PM5/2/09
to sage-...@googlegroups.com
mabshoff wrote:
ed smaller values. I'm going to
>>> put that data up on the trac ticket.
>> Mathematica 6 (on a Sun SPARC) gives an answer in far less time than Sage:
>>
>> In[3]:= PrimePi[2^50]
>>
>> PrimePi::largp:
>> Argument 1125899906842624 in PrimePi[1125899906842624]
>> is too large for this implementation.
>>
>> Out[3]= PrimePi[1125899906842624]
>>
>> Well, perhaps not really an answer!
>
> :)
>
> Could you figure out what the upper bound is that MMA allows? I have
> discussed this with William in IRC and in 3.4.2 we should just throw a
> NotImplementedError for some bound where we are comfortable with
> knowing the result is correct on 32 and 64 bit. Unfortunately this
> isn't something we can doctest with a reasonable amount of time.
>
> The suggestion then was to implement something on top of the range
> computed with floats using MPFR for example, but we will see what
> happens. I am sure that if I asked if someone needed to compute
> prime_pi() for anything larger than 2^48 someone would say yes, so
> this ought to be fixed.
>
> Cheers,
>
> Michael

Hi Micheal,

Yes, I can figure it out.

The upper limit is PrimePi[249999999999999] (2.5x10^14), which
Mathematica gives as 7783516108362. It took 20 minutes or so on a
heavily loaded machine.

In[95]:= PrimePi[249999999999999]

Out[95]= 7783516108362

It can not manage PrimePi[250000000000000]

2^47 is 1.41x10^14,
2^48 is 2.81x10^14.

Since the maximum that can be handled is just under 2.5x10^14,
Mathematica can compute PrimePi[2^47], but not PrimePi[2^48]

Here's a table of PrimePi[2^n], with n ranging from 0 to 47. It took
roughly 20 minutes or so to compute the table.

In[19]:= Table[{n,PrimePi[2^n]},{n,0,47}]

Out[19]= {{0, 0}, {1, 1}, {2, 2}, {3, 4}, {4, 6}, {5, 11}, {6, 18}, {7, 31},

> {8, 54}, {9, 97}, {10, 172}, {11, 309}, {12, 564}, {13, 1028},

> {14, 1900}, {15, 3512}, {16, 6542}, {17, 12251}, {18, 23000},

> {19, 43390}, {20, 82025}, {21, 155611}, {22, 295947}, {23, 564163},

> {24, 1077871}, {25, 2063689}, {26, 3957809}, {27, 7603553},

> {28, 14630843}, {29, 28192750}, {30, 54400028}, {31, 105097565},

> {32, 203280221}, {33, 393615806}, {34, 762939111}, {35, 1480206279},

> {36, 2874398515}, {37, 5586502348}, {38, 10866266172},

> {39, 21151907950}, {40, 41203088796}, {41, 80316571436},

> {42, 156661034233}, {43, 305761713237}, {44, 597116381732},

> {45, 1166746786182}, {46, 2280998753949}, {47, 4461632979717}}


PS, Mathematica computes PrimePi[some_negative_number] as 0. Does Sage
handle that case ok?

Dave

mabshoff

unread,
May 2, 2009, 8:45:47 PM5/2/09
to sage-devel


On May 1, 6:54 am, Kiran Kedlaya <ksk...@gmail.com> wrote:
> Clean build on 64-bit Fedora 10 (Opteron) fails one doctest:
>
> sage -t  "devel/sage/sage/sets/primes.py"
> **********************************************************************
> File "/opt/sage/sage-3.4.2.rc0/devel/sage/sage/sets/primes.py", line
> 80:
>     sage: P>x^2+x
> Expected:
>     True
> Got:
>     False

<SNIP>

For the record: This is now #5966 and will be fixed in 3.4.2.final.

Cheers,

Michael

kcrisman

unread,
May 2, 2009, 9:39:56 PM5/2/09
to sage-devel

>
> For the record: This is now #5966 and will be fixed in 3.4.2.final.

It also has been #5959, with a patch, since yesterday morning -
figured if I caused the trouble, I should fix it :) That doesn't
address "needlessly starting Maxima" but unfortunately I won't be able
to address that til at least Monday.

HTH,
- kcrisman

mabshoff

unread,
May 2, 2009, 9:48:54 PM5/2/09
to sage-devel
Thanks for the heads up, I closed it as dupe of #5966 since it is
trivial to fix. Note that #5959 was not marked properly to have a
patch.

> HTH,
> - kcrisman

Cheers,

Michael

Dr. David Kirkby

unread,
May 3, 2009, 2:42:28 AM5/3/09
to sage-...@googlegroups.com
mabshoff wrote:

>> He told me Mathematica can go up to about 2^45 or so, but not beyond.
>
> At least for MMA 6.0 on linux x86-64 the limit seems to be around
> 2^47:

As I said in the other post, the limit is PrimePi[249999999999999].

> MMA Sage
>
> 2^44: 18.04 110.88 (597116381732)
> 2^45: 29.98 207.61 (1166746786182)
> 2^46: 47.59 389.98 (2280998753949)
> 2^47: 89.25 728.84 (4461632979717)
> 2^48: NA :) about an hour - correct?
>
> According to Alex's numbers at least on his laptop 2^46 was correct on
> 32 bits, but given the length of the test (~6 minutes on sage.math
> this isn't really doctestable).
>
>> The algorithm in Mathematica is completely different (and better) than
>> what Andrew implemented for Sage. As far as I know the situation for
>> computing pi(X) using general purpose math software is thus:

One (admitidly unlikely) possibility is that Wolfram have used the same
algorithm, but used assembly code. I think they claim the kernel is
C/C++, but that permits bits of inline assembly.


>> * Mathematica -- has the best implementation available
>>
>> * Sage -- now has the second best implementation available
>
> Yep, the old implementation was about 1000 times slower than Andrew's
> which is about 5 times slower than MMA 6.0 - so great job from
> catching us up from 5000 times to only 5 times :).

I gather you have managed to get a SPARC version of Sage running. It
would be interesting to compare the performance on SPARC of Mathematica
and Sage. I very much doubt Wolfram would have used hand-optimised
assembly code on SPARC, so if the timings still show Mathematica to be
5x quicker, then yes, I would agree it's a better algorithm. But if the
timings are very similar, it suggests to me that perhaps the better
performance of Mathematica is due to writing assembly code, rather than
using a high-level language.

>> * Pari, Maple, Matlab -- "stupid" implementations, which all simply
>> enumerate all primes up to x and see how many there are. Useless, and
>> can't come close to what Sage or Mathematica do.

> Well, what should we pick as upper bound? 2^40 seems to be what Andrew
> suggests, but maybe 2^42 or 2^43? In that range we can actually add
> #long doctests and I would be much more comfortable that we would at
> least catch potential issues.

Personally, if Sage can go to 249999999999999, then I would use that as
an upper bound. If the algorithm in Sage gives the same answer as
Mathematica

In[95]:= PrimePi[249999999999999]
Out[95]= 7783516108362

then why not use Sage to there? The poor SPARC I used for this was very
heavily loaded and quite old, but it only took 20 minutes or so. Even an
algorithm that is 5x slower should be able to compute
primepi(249999999999999) in under an hour on any half-decent modern
machine.

Of course, a search of the literature for a better algorithm might have
some millage. Unfortunately, I no longer have access to a university
account and so can't search journals without paying for access to
individual ones, which clearly I can't justify. Neither am I
particularly good at maths, having never maths studied beyond that
needed for an engineering degree


One unfortunate side-effect of the closed-source vs open-source code is
the fact that if open-source code (e.g. Sage) has a faster algorithm
than Mathematica, then it's relatively easy for WRI to look at the
algorithm in Sage and use the same one, to bring Mathematica and Sage to
similar performances. The converse is not true of course - it is not
possibly for Sage developers to find out the algorithm Mathematica uses.
However, a look at

http://mathworld.wolfram.com/PrimeCountingFunction.html

might give some clues as to the algorithm Mathematica uses. Although the
algorithm Mathematica is not stated in that Mathworld page, there are a
number of references. Two look interesting:

Mapes, D. C. "Fast Method for Computing the Number of Primes Less than a
Given Limit." Math. Comput. 17, 179-185, 1963.

Gourdon, X. "New Record Computation for pi(x), x=10^21." 27 Oct 2000.
http://listserv.nodak.edu/scripts/wa.exe?A2=ind0010&L=nmbrthry&P=2988

The link
http://listserv.nodak.edu/cgi-bin/wa.exe?A2=ind0010&L=nmbrthry&P=2988
states the algorithm used, but in a way I don't understand. It says:

"This value has been checked by computing pi(10^21+10^8) with
a different parameter y used in the algorithm"

but y is not defined!

William Stein

unread,
May 3, 2009, 2:49:02 AM5/3/09
to sage-...@googlegroups.com, R. Andrew Ohana

This page:

http://reference.wolfram.com/mathematica/note/SomeNotesOnInternalImplementation.html#12788

says what algorithm is used in Mathematica to compute PrimePi, and it
is definitely *not* the one used in Sage. It is a completely
different harder-to-implement algorithm with better asymptotic
complexity.

In any case, we definitely do intend to install Mathematica on our new
Sparc T5240 box, since UW has a site license for Mathematica which
includes a Sparc Solaris version.

-- William

Dr. David Kirkby

unread,
May 3, 2009, 2:49:07 AM5/3/09
to sage-...@googlegroups.com
mabshoff wrote:

>> He told me Mathematica can go up to about 2^45 or so, but not beyond.
>
> At least for MMA 6.0 on linux x86-64 the limit seems to be around
> 2^47:

As I said in the other post, the limit is PrimePi[249999999999999].

> MMA Sage


>
> 2^44: 18.04 110.88 (597116381732)
> 2^45: 29.98 207.61 (1166746786182)
> 2^46: 47.59 389.98 (2280998753949)
> 2^47: 89.25 728.84 (4461632979717)
> 2^48: NA :) about an hour - correct?
>
> According to Alex's numbers at least on his laptop 2^46 was correct on
> 32 bits, but given the length of the test (~6 minutes on sage.math
> this isn't really doctestable).
>
>> The algorithm in Mathematica is completely different (and better) than
>> what Andrew implemented for Sage. As far as I know the situation for
>> computing pi(X) using general purpose math software is thus:

One (admitidly unlikely) possibility is that Wolfram have used the same
algorithm, but used hand-optimised assembly code, rather than a
high-level language I think WRI claim the kernel is C/C++, but that

permits bits of inline assembly.

>> * Mathematica -- has the best implementation available
>>
>> * Sage -- now has the second best implementation available
>
> Yep, the old implementation was about 1000 times slower than Andrew's
> which is about 5 times slower than MMA 6.0 - so great job from
> catching us up from 5000 times to only 5 times :).

I gather you have managed to get a SPARC version of Sage running. It

would be interesting to compare the performance on SPARC of Mathematica
and Sage. I very much doubt Wolfram would have used hand-optimised
assembly code on SPARC, so if the timings still show Mathematica to be
5x quicker, then yes, I would agree it's a better algorithm. But if the
timings are very similar, it suggests to me that perhaps the better
performance of Mathematica is due to writing assembly code, rather than
using a high-level language.

>> * Pari, Maple, Matlab -- "stupid" implementations, which all simply


>> enumerate all primes up to x and see how many there are. Useless, and
>> can't come close to what Sage or Mathematica do.

> Well, what should we pick as upper bound? 2^40 seems to be what Andrew
> suggests, but maybe 2^42 or 2^43? In that range we can actually add
> #long doctests and I would be much more comfortable that we would at
> least catch potential issues.

Personally, if Sage can go to 249999999999999, then I would use that as

an upper bound. If the algorithm in Sage gives the same answer as
Mathematica

In[95]:= PrimePi[249999999999999]
Out[95]= 7783516108362

then why not use Sage to there? The poor SPARC I used for this was very
heavily loaded and quite old, but it only took 20 minutes or so. Even an
algorithm that is 5x slower should be able to compute
primepi(249999999999999) in under an hour on any half-decent modern
machine.

Of course, a search of the literature for a better algorithm might have
some millage. Unfortunately, I no longer have access to a university
account and so can't search journals without paying for access to
individual ones, which clearly I can't justify. Neither am I
particularly good at maths, having never maths studied beyond that
needed for an engineering degree

A look at the Mathworld entry for the Prime Counting Function

http://mathworld.wolfram.com/PrimeCountingFunction.html

might give some clues as to the algorithm Mathematica uses. Although the
algorithm Mathematica is not stated in that Mathworld page, there are a
number of references. Two look interesting:

Mapes, D. C. "Fast Method for Computing the Number of Primes Less than a
Given Limit." Math. Comput. 17, 179-185, 1963.

Gourdon, X. "New Record Computation for pi(x), x=10^21." 27 Oct 2000.
http://listserv.nodak.edu/scripts/wa.exe?A2=ind0010&L=nmbrthry&P=2988

The link
http://listserv.nodak.edu/cgi-bin/wa.exe?A2=ind0010&L=nmbrthry&P=2988
states the algorithm used, but in a way I don't understand. It says:

"This value has been checked by computing pi(10^21+10^8) with
a different parameter y used in the algorithm"

But y is not defined! I'll try to drop the author of that post an email,
to see if he knows of any fast algorithm that might be applicable to the
general purpose case.

Since PrimePi[n] been computed to n=10^21 and both Mathematica and Sage
can't do PrimePi[10^15], there may well be a *lot* faster algorithm known.


Dave

Alex Ghitza

unread,
May 3, 2009, 3:21:48 AM5/3/09
to sage-...@googlegroups.com
On Sun, May 3, 2009 at 9:23 AM, Dr. David Kirkby
<david....@onetel.net> wrote:
>
> Here's a table of PrimePi[2^n], with n ranging from 0 to 47. It took
> roughly 20 minutes or so to compute the table.
>
> In[19]:= Table[{n,PrimePi[2^n]},{n,0,47}]
>
> Out[19]= {{0, 0}, {1, 1}, {2, 2}, {3, 4}, {4, 6}, {5, 11}, {6, 18}, {7, 31},
>
>  >    {8, 54}, {9, 97}, {10, 172}, {11, 309}, {12, 564}, {13, 1028},
>
>  >    {14, 1900}, {15, 3512}, {16, 6542}, {17, 12251}, {18, 23000},
>
>  >    {19, 43390}, {20, 82025}, {21, 155611}, {22, 295947}, {23, 564163},
>
>  >    {24, 1077871}, {25, 2063689}, {26, 3957809}, {27, 7603553},
>
>  >    {28, 14630843}, {29, 28192750}, {30, 54400028}, {31, 105097565},
>
>  >    {32, 203280221}, {33, 393615806}, {34, 762939111}, {35, 1480206279},
>
>  >    {36, 2874398515}, {37, 5586502348}, {38, 10866266172},
>
>  >    {39, 21151907950}, {40, 41203088796}, {41, 80316571436},
>
>  >    {42, 156661034233}, {43, 305761713237}, {44, 597116381732},
>
>  >    {45, 1166746786182}, {46, 2280998753949}, {47, 4461632979717}}
>

For the record, here's the same thing in Sage. As you can see, it
took slightly less than 30 minutes on a macbook.

sage: time [[n, prime_pi(2^n)] for n in range(48)]
CPU times: user 1727.90 s, sys: 0.78 s, total: 1728.69 s
Wall time: 1737.05 s

[[0, 0],

[47, 4454203917918]]

And as mentioned before, the values up to 46 are correct, and the
value at 47 is wrong.

> PS, Mathematica computes PrimePi[some_negative_number] as 0. Does Sage
> handle that case ok?
>

sage: prime_pi(-20)
0

mabshoff

unread,
May 3, 2009, 4:01:35 AM5/3/09
to sage-devel
Ok, I would suggest we do the following:


* In 3.4.2 cap prime_pi at 2^40 since that is what Andrew suggests as
correct range for his algorithm
* add the following #long doctest (it takes about 25 seconds total in
sage.math):

sage: for n in (30..40):
....: time prime_pi(2^n)
<SNIP results>

* For the range of 2^40+1 to 2^46 I am uncomfortable to have it
available per default, especially if we don't at least spot check the
values via doctesting and it would be getting slow to do so - even for
#long. But we could add some keyword to force the upper limit up to a
2^46 - the issue should be revisited post 3.4.2 - maybe even
implementing a better algorithm?

* In 3.4.1 prime_pi() was completely broken for even 2^35 because len
(prime_range(2^35)) aborts on sage.math since pari fails to allocate
124 *GB* of memory. That implementation is actually about 2000 times
slower than what Andrew wrote. I have no idea why pari's
primes_up_to_n or whatever it is called internally is so extremely
inefficient. Maybe somebody out to check pari-svn and complain? As
mentioned on the other ticket (#5943) the prime_range(2^50) outright
segfaults even in 3.4.2.final.

Thoughts?

Cheers,

Michael

Dr. David Kirkby

unread,
May 3, 2009, 8:54:35 AM5/3/09
to sage-...@googlegroups.com
mabshoff wrote:
> Ok, I would suggest we do the following:
>
>
> * In 3.4.2 cap prime_pi at 2^40 since that is what Andrew suggests as
> correct range for his algorithm
> * add the following #long doctest (it takes about 25 seconds total in
> sage.math):
>
> sage: for n in (30..40):
> ....: time prime_pi(2^n)
> <SNIP results>
>
> * For the range of 2^40+1 to 2^46 I am uncomfortable to have it
> available per default, especially if we don't at least spot check the
> values via doctesting and it would be getting slow to do so - even for
> #long. But we could add some keyword to force the upper limit up to a
> 2^46 - the issue should be revisited post 3.4.2 - maybe even
> implementing a better algorithm?


I personally can't see the point in stopping the algorithm working at
2^40, if it does work at 2^46. I realise you can't specifically test
2^46 on every occasion, but surely that argument could apply to sine,
cosine and every other function that exists. It's impractical to check
every single value that might be thrown at it. Of course, if there is
reason to believe the algorithm will fail, rather than just become very
slow, then there is good reason for putting a limit.


> * In 3.4.1 prime_pi() was completely broken for even 2^35 because len
> (prime_range(2^35)) aborts on sage.math since pari fails to allocate
> 124 *GB* of memory.

Em, but a failure to allocate sufficient memory should not cause a seg
fault - whether or not the memory allocation is sensible or not.

Perhaps this is why you are intending putting a limit. I can understand
that.

mabshoff

unread,
May 4, 2009, 4:49:21 AM5/4/09
to sage-devel
Yeah, tell me. Either pari and/or Sage's pari interface sucks. It is
absolutely ridiculous that pari seems to require that much memory, but
it wouldn't be the first time we find an absolutely insane memory leak
in pari.

> Perhaps this is why you are intending putting a limit. I can understand
> that.

We are talking about two different limits here.

Cheers,

Michael

mabshoff

unread,
May 4, 2009, 4:55:34 AM5/4/09
to sage-devel


On May 3, 5:54 am, "Dr. David Kirkby" <david.kir...@onetel.net> wrote:
> mabshoff wrote:

<SNIP>

Sorry, I forgot to reply to this part of the post since I thought it
was in another independent post.

> >  * For the range of 2^40+1 to 2^46 I am uncomfortable to have it
> > available per default, especially if we don't at least spot check the
> > values via doctesting and it would be getting slow to do so - even for
> > #long. But we could add some keyword to force the upper limit up to a
> > 2^46 - the issue should be revisited post 3.4.2 - maybe even
> > implementing a better algorithm?
>
> I personally can't see the point in stopping the algorithm working at
> 2^40, if it does work at 2^46.

We don't know it works on every for for 2^46 since we basically have a
sample of two computers. The past has taught me that this isn't even
close to what I would consider acceptable. I have put in the limit of
2^40 and the doctests to check correctness for 2^n for n in 30..40 and
that will be widely tested. Once that works we can think about
extending the tests or implementing another algorithm. Given a choice
between shipping potentially broken code or limiting Sage's
capabilities I always chose to disable code. We are beating the pants
off every known open implementation prime_pi() and lack MMA by a small
constant factor - that is an excellent improvement for Sage 3.4.2 - no
need to get greedy ;)

> I realise you can't specifically test
> 2^46 on every occasion, but surely that argument could apply to sine,
> cosine and every other function that exists. It's impractical to check
> every single value that might be thrown at it. Of course, if there is
> reason to believe the algorithm will fail, rather than just become very
> slow, then there is good reason for putting a limit.

No, sin(2^n) and so on for large n is computed via MPFR which is
provably correct (modulo bugs of course) and has been tested by many
people for many, many years and is written by world experts in the
field. I can trust that code as much as I can trust code.

Cheers,

Michael

William Stein

unread,
May 4, 2009, 11:30:05 AM5/4/09
to sage-...@googlegroups.com, R. Andrew Ohana
+1 in general, but prime_pi is very new code written by a new sage dev
who strangely has not made any comments at all about this issue (and
the algorithm he implemented is very sensitive to input size). Until
we get some feedback from him, I like Michael suggestion.

>
> No, sin(2^n) and so on for large n is computed via MPFR which is
> provably correct (modulo bugs of course) and has been tested by many
> people for many, many years and is written by world experts in the
> field. I can trust that code as much as I can trust code.
>
> Cheers,
>
> Michael
>
>
> >
>



--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

Dr. David Kirkby

unread,
May 5, 2009, 3:44:52 AM5/5/09
to sage-...@googlegroups.com
mabshoff wrote:

>>> * In 3.4.1 prime_pi() was completely broken for even 2^35 because len
>>> (prime_range(2^35)) aborts on sage.math since pari fails to allocate
>>> 124 *GB* of memory.
>> Em, but a failure to allocate sufficient memory should not cause a seg
>> fault - whether or not the memory allocation is sensible or not.
>
> Yeah, tell me. Either pari and/or Sage's pari interface sucks. It is
> absolutely ridiculous that pari seems to require that much memory, but
> it wouldn't be the first time we find an absolutely insane memory leak
> in pari.

>> Perhaps this is why you are intending putting a limit. I can understand
>> that.
>
> We are talking about two different limits here.

No, we were not - just a confusing way I wrote it. A memory alloction
issue is completely different to limiting an algorithm due to concerns
about it.

As a matter of interested, what does Sage give for
primepi(249999999999999)? Is it the same as Mathematica (7783516108362)?

It would be interesting if it gave the same result as Mathematica on any
of the computers.

In[95]:= PrimePi[249999999999999]
Out[95]= 7783516108362

Dave

Fredrik Johansson

unread,
May 5, 2009, 4:08:56 AM5/5/09
to sage-...@googlegroups.com
On Sun, May 3, 2009 at 8:42 AM, Dr. David Kirkby
<david....@onetel.net> wrote:
> The link
> http://listserv.nodak.edu/cgi-bin/wa.exe?A2=ind0010&L=nmbrthry&P=2988
> states the algorithm used, but in a way I don't understand. It says:
>
> "This value has been checked by computing pi(10^21+10^8) with
> a different parameter y used in the algorithm"
>
> but y is not defined!

The algorithm (and the definition of y) are given in the paper mentioned
in the same post:

Math Of Comp 1996 by Deleglise & Rivat : Computing Pi(x), the
Meissel, Lehmer, Lagarias, Miller, Odlyzko method.

I found an online copy: http://cr.yp.to/bib/1996/deleglise.pdf

Fredrik

mabshoff

unread,
May 5, 2009, 4:18:44 AM5/5/09
to sage-devel


On May 5, 12:44 am, "Dr. David Kirkby" <david.kir...@onetel.net>
wrote:
> mabshoff wrote:

<SNIP>

Hi Dave,

> > We are talking about two different limits here.
>
> No, we were not - just a confusing way I wrote it. A memory alloction
> issue is completely different to limiting an algorithm due to concerns
> about it.

Well, I certainly am. To recap:

(a) prime_pi() used to use to be trivially implemented as len
(prime_range())
(b) len(prime_range()) sucks memory and performance wise, i.e. eats
in excess of 124GB for input n=2^35 and plainly segfaults for n=2^50.
(c) we have known failures of the new prime_pi() for some inputs,
i.e. n=2^47

So what I am doing is:

(a) cap the new prime_pi() to 2^40 since that is the upper bound we
can verify via doctests in reasonable time. Given the wide testing I
have done this seems to work.
(c) cap prime_range() to a reasonable value and/or rewrite it sanely
- this has not happened yet.

> As a matter of interested, what does Sage give for
> primepi(249999999999999)? Is it the same as Mathematica (7783516108362)?

Sage.math, i.e. 64 bit, gives me:

sage: time prime_pi(249999999999999)
CPU times: user 1241.60 s, sys: 1.07 s, total: 1242.67 s
Wall time: 1243.69 s
7783516108362

Since this is 20 minutes CPU time on a fast box we cannot make this a
doctest, even a long one. I guess we need to implement a better
algorithm :p

> It would be interesting if it gave the same result as Mathematica on any
> of the computers.
>
> In[95]:= PrimePi[249999999999999]
> Out[95]= 7783516108362
>
> Dave

Cheers,

Michael

William Stein

unread,
May 5, 2009, 9:37:58 AM5/5/09
to sage-...@googlegroups.com

Fredrik, Just out of curiosity, is that the sort of algorithm you like
to implement?

William

kcrisman

unread,
May 5, 2009, 9:55:00 AM5/5/09
to sage-devel
Just FYI, Bressoud and Wagon's Computational Number Theory book has a
more elementary (and very readable) account of this type of formula -
starting with Legendre and going through Meissel and Lehmer, from
around page 130 to page 140 - including fairly explicit implementation
ideas.

- kcrisman

Fredrik Johansson

unread,
May 5, 2009, 10:10:49 AM5/5/09
to sage-...@googlegroups.com
On Tue, May 5, 2009 at 3:37 PM, William Stein <wst...@gmail.com> wrote:
> Fredrik, Just out of curiosity, is that the sort of algorithm you like
> to implement?
>
> William

Quite possibly, but I'm not familiar with any of these algorithms so
I'd need some practice to implement anything even remotely optimized.

Perhaps something to look at during SD15? "Number Theory" is on the
official topic list
and this sounds very much like number theory.

Fredrik

William Stein

unread,
May 5, 2009, 10:20:06 AM5/5/09
to sage-...@googlegroups.com, saged...@googlegroups.com
Yep, this *is* number theory.

By the way, now is a good time to start the Sage Days 15 project page.
I've listed you here:

http://wiki.sagemath.org/days15/projects

Anybody who will virtually participate (e.g., Alex Ghitza), please
also add their projects. I want to make an extra effort this time
around to make virtual involvement with Sage Days 15 easier. At a
minimum, I'll post video from talks promptly.

Does anybody know how to live streaming video broadcasts? I.e.,
something like Skype, but where arbitrarily many people can view the
video live all at once? If so, we could setup a laptop like that and
have it broadcast live all the Sage Days talks on Saturday/Sunday.

William

> Fredrik

Mike Hansen

unread,
May 5, 2009, 10:53:32 AM5/5/09
to sage-...@googlegroups.com
On Tue, May 5, 2009 at 7:20 AM, William Stein <wst...@gmail.com> wrote:
> Does anybody know how to live streaming video broadcasts?  I.e.,
> something like Skype, but where arbitrarily many people can view the
> video live all at once?  If so, we could setup a laptop like that and
> have it broadcast live all the Sage Days talks on Saturday/Sunday.

I've used www.ustream.tv (although not on my computer), and it worked very well.

--Mike

William Stein

unread,
May 5, 2009, 12:33:15 PM5/5/09
to sage-...@googlegroups.com

That looks really good / easy: http://www.ustream.tv/get-started

Can 2 or 3 people who won't be at Sage Days respond and say they would actually
watch a live video stream from Sage Days, if I post one? If so, then
I'll do it (if not, I
won't waste the time).

-- William

Mike Hansen

unread,
May 5, 2009, 12:40:22 PM5/5/09
to sage-...@googlegroups.com
On Tue, May 5, 2009 at 9:33 AM, William Stein <wst...@gmail.com> wrote:
> That looks really good / easy:  http://www.ustream.tv/get-started
>
> Can 2 or 3 people who won't be at Sage Days respond and say they would actually
> watch a live video stream from Sage Days, if I post one?  If so, then
> I'll do it (if not, I won't waste the time).

It'd also be helpful for someone to volunteer to say watch IRC and
relay any questions from there to the live event.

--Mike

Craig Citro

unread,
May 5, 2009, 12:42:40 PM5/5/09
to sage-...@googlegroups.com
> It'd also be helpful for someone to volunteer to say watch IRC and
> relay any questions from there to the live event.
>

Good point! Thanks for volunteering! :P

(Sorry, couldn't resist.)

-cc

Nick Alexander

unread,
May 5, 2009, 12:45:19 PM5/5/09
to sage-...@googlegroups.com
> Can 2 or 3 people who won't be at Sage Days respond and say they
> would actually
> watch a live video stream from Sage Days, if I post one?

I am unlikely to watch a live stream but have watched several of the
posted videos after the fact.

Nick

William Stein

unread,
May 5, 2009, 12:50:58 PM5/5/09
to sage-...@googlegroups.com

Cool. One advantage for some people of the live stream is they will be able
to ask questions during the talk.

William

Rob Beezer

unread,
May 5, 2009, 1:38:56 PM5/5/09
to sage-devel
I'd watch Mike Hansen do "Advanced Mercurial" if it happened Sunday
morning when I'm pinned at home prior to working our graduation
ceremony that afternoon - only so I could ask him a few questions via
IRC (which I was going to suggest as a useful adjunct to a live
stream).

But perhaps better would be to just send Mike a few questions in
advance and watch/download a video later. ;-)

Rob

VictorMiller

unread,
May 5, 2009, 2:43:58 PM5/5/09
to sage-devel
As a comparison I just ran my old C program (implementing the
algorithm in my paper with Lagarias and Odlyzko) on my workstation
which is a fast Dell '86 box (sorry I don't have more details) running
Red Hat:

time ./findn 249999999999999
n=249_999_999_999_999
pi(249_999_999_999_999)=7_783_516_108_362

real 0m18.826s
user 0m18.628s
sys 0m0.028s

William Stein

unread,
May 5, 2009, 2:55:08 PM5/5/09
to sage-...@googlegroups.com, R. Andrew Ohana
On Tue, May 5, 2009 at 11:49 AM, William Stein <wst...@gmail.com> wrote:

> On Tue, May 5, 2009 at 11:43 AM, VictorMiller <victor...@gmail.com> wrote:
>>
>> As a comparison I just ran my old C program (implementing the
>> algorithm in my paper with Lagarias and Odlyzko) on my workstation
>> which is a fast Dell '86 box (sorry I don't have more details) running
>> Red Hat:
>
> Wow, I didn't know Dell was selling computers back in 1986.  :-)  [just kiddin']

>
>>
>> time ./findn 249999999999999
>> n=249_999_999_999_999
>> pi(249_999_999_999_999)=7_783_516_108_362
>>
>> real    0m18.826s
>> user    0m18.628s
>> sys     0m0.028s
>
> Wow, nice!    Give my your C code!!
>

By the way, on a core2 2.66Ghz 13MB cache (sage.math) we have:

In[3]:= Timing[PrimePi[249999999999999]]

Out[3]= {155.44, 7783516108362}

So your timings above are significantly better than all general
purpose math software for
computing prime_pi.

William

William Stein

unread,
May 5, 2009, 2:49:38 PM5/5/09
to sage-...@googlegroups.com, R. Andrew Ohana
On Tue, May 5, 2009 at 11:43 AM, VictorMiller <victor...@gmail.com> wrote:
>
> As a comparison I just ran my old C program (implementing the
> algorithm in my paper with Lagarias and Odlyzko) on my workstation
> which is a fast Dell '86 box (sorry I don't have more details) running
> Red Hat:

Wow, I didn't know Dell was selling computers back in 1986. :-) [just kiddin']

>
> time ./findn 249999999999999
> n=249_999_999_999_999
> pi(249_999_999_999_999)=7_783_516_108_362
>
> real    0m18.826s
> user    0m18.628s
> sys     0m0.028s

Wow, nice! Give my your C code!!


>
>

Jason Grout

unread,
May 5, 2009, 8:47:30 PM5/5/09
to sage-...@googlegroups.com

Could you post a list of the sessions that you are planning on having?
I didn't find many things listed at http://wiki.sagemath.org/days15

I might be interested if the timing works out. In that case, it would
be good to have IRC access for questions.

Jason

Alex Ghitza

unread,
May 5, 2009, 9:06:18 PM5/5/09
to sage-...@googlegroups.com
On Wed, May 6, 2009 at 2:33 AM, William Stein <wst...@gmail.com> wrote:
> Can 2 or 3 people who won't be at Sage Days respond and say they would actually
> watch a live video stream from Sage Days, if I post one?  If so, then
> I'll do it (if not, I
> won't waste the time).
>

I can't work out right now what time it will be in Melbourne during
these talks, but if it's not during my teaching/seminars/office hours
then I would happily watch a live video stream.

(So I guess my answer is pretty much isomorphic to Jason Grout's.)


Best,
Alex

Dan Drake

unread,
May 5, 2009, 9:47:29 PM5/5/09
to sage-...@googlegroups.com
On Tue, 05 May 2009 at 09:33AM -0700, William Stein wrote:
> Can 2 or 3 people who won't be at Sage Days respond and say they would
> actually watch a live video stream from Sage Days, if I post one? If
> so, then I'll do it (if not, I won't waste the time).

I'd like to watch live (and ask questions via IRC, that would be nice),
but the time difference is annoying. (I'm 16 hours ahead of Seattle.)
I'd only be able to watch sessions starting late afternoon or later...if
there's a timetable, I can answer more accurately.

Dan

--
--- Dan Drake <dr...@kaist.edu>
----- KAIST Department of Mathematical Sciences
------- http://mathsci.kaist.ac.kr/~drake

signature.asc
Reply all
Reply to author
Forward
0 new messages