benchmarking CAS
rjf
system, concerning how to make it run more efficiently, a number of
issues have been raised that can speed up individual operations. I
was thinking about how to make the testing more meaningful.
In the long term, there are other issues at play that will affect
speed, other than what is revealed by timing a single command.
For example, "professional" lisp systems must be prepared to run
indefinitely, and therefore do things like compacting storage. This
generally means that addresses of cells are not permanent, and
addresses cannot be used as proxies for (say) hashcodes or total
orderings.
On the other hand, Maple, Mathematica, ... may not have such issues:
I doubt that they are used for production 24/7 computing, say for
airline reservation systems; most users run them for a few CPU minutes
at worst. Since they do not have to run indefinitely, and/or have a
much simpler memory model for most of their data, they can use
addresses as proxies for various things. (In particular, Mathematica
uses page locations as a proxy for the age of an object -- X has to be
recomputed if X is older than any of the objects on which it depends).
But if they were run for months, what would happen? (And do we care?)
Note that it is also possible for a "less-professional" lisp system to
have a simple garbage collector that does not move data: this works
for a while (and maybe indefinitely) -- so long as memory
fragmentation does not emerge as a problem. This makes the author of
every brand-new naive lisp system proud. for a little while.
So a good timing result where the whole computation takes a few
minutes is not such a great measure, at least if it is supposed to be
a predictor of longer computations. Running some 10-second test suite
predicts how long it takes to run that 10-second test suite.
One could, I suppose, also have some measure of "memory dispersion"
before cutting off such a benchmark; additionally, one could create a
benchmark that (a) took a long time and (b) created/deleted/created/
huge data objects.
Perhaps running every benchmark repeatedly, say 2000 times, might be
interesting. The first 1000 times one could delete any leftover
results, and the second 1000 times, the results should be retained,
clogging memory. Programs which retain data in hash tables for fast
recomputation, etc, must be considered within the context of truth in
testing: what would a memory-intense sequence of computations do if
they were not just repetitions of already remembered results.
Comments?
RJF
Vladimir Bondarenko
paramenter at the choice of the proper CAS.
In particular, the mean time to crash.
For example, typically, if set to this task, the VM machine
can crash Maple 11 within 1 hour, Mathematica 6 within 1 day,
and Derive within 3 days.
Best wishes,
Vladimir Bondarenko
VM and GEMM architect
Co-founder, CEO, Mathematical Director
http://www.cybertester.com/ Cyber Tester, LLC
http://maple.bug-list.org/ Maple Bugs Encyclopaedia
http://www.CAS-testing.org/ CAS Testing
-----------------------------------------------------------------
"We must understand that technologies
like these are the way of the future."
-----------------------------------------------------------------
SzH
> For example, typically, if set to this task, the VM machine
> can crash Maple 11 within 1 hour, Mathematica 6 within 1 day,
> and Derive within 3 days.
You mean that you are able to run the Mathematica 6 kernel
continuously, performing random symbolic computations (like
integration), for 24 hours, without any problems? That is better than
what I expected. How much memory does your machine have?
Roman Pearce
> On the other hand, Maple, Mathematica, ... may not have such issues:
> But if they were run for months, what would happen? (And do we care?)
In Maple memory is allocated but not returned to the operating system.
The time for garbage collection grows, but eventually memory use and
performance stabilizes. Fragmentation is an issue when large
objects
of various sizes are repeatedly allocated.
For example, I recently tried the following computation:
f := expand((1+x+y+z+t)^30):
g := f+1:
p := expand(f*g):
divide(p,f,'q');
bytes used=991591348816, alloc=1649501176, time=29025.53
Because the polynomials are large, Maple uses interpreted routines
that run with garbage collection. The time was not very good, but
bytesalloc is basically the size of the product, so it looks like
the garbage collector will keep the system up. I have seen and
done other computations that took days and weeks. If something
is going to take more than a month, it would probably make sense
to write it in C.
Vladimir Bondarenko
SzH> kernel continuously, performing random symbolic computations
SzH> (like integration),
Well, the inputs the VM machine feeds into the MathKernel are not
totally random :)
Some of them are brute-force ones, but a tangible fraction of them
comes from some conditions the VM machine calculates trying to crash
the MathKernel.
(The VM machine, if set to such a task, always crashes Maple,
Mathematica, AXIOM, Maxima, Derive, and MuPAD; the concrete timing
depends on the initial parameters.
As of today, Derive 6.1 is the most stable computer algebra system
of the above-listed.
We observed some specific runs when Derive 6.1 had been operational
for about 3 weeks (!) of continuous calculations the VM machine forced
it to perform.)
SzH> for 24 hours, without any problems?
Kind of this, in the average.
SzH> That is better than what I expected.
:)
It was worse in Mathematica 5.2. But we feel that in Mathematica 6
some real improvements are introduced.
SzH> How much memory does your machine have?
RAM is 4 Gb, and total free HDD size of about 500 Gb.
Nasser Abbasi
"Vladimir Bondarenko" <v...@cybertester.com> wrote in message
news:373f7f01-6a15-49c0...@p25g2000hsf.googlegroups.com...
(The VM machine, if set to such a task, always crashes Maple,
Mathematica, AXIOM, Maxima, Derive, and MuPAD; the concrete timing
depends on the initial parameters.
As of today, Derive 6.1 is the most stable computer algebra system of the
above-listed."
It would be interesting if your VM machine can add SAGE to the systems it
tests. SAGE is described here:
http://en.wikipedia.org/wiki/Software_for_Algebra_and_Geometry_Experimentation
I played with SAGE a little, but I find its interface on windows, having to
use a browser, a bit awkward to say the least. I have not tried it text
based interface.
Nasser
Vladimir Bondarenko
Thanks for your idea.
Technically, we could do this. Anyways, now we have a small amount of
resource we can dedicate to a concrete system; but we here hope that
the things could look better reasonably soon, hopefully, this year.
As you understand, if we here dedicate something to SAGE, we subtract
this from the above list I quoted.
Could you please tell why do you feel that testing SAGE could be
really useful? Useful for whom?
By the way, maybe you know, approximately, how many persons use SAGE?
Occasionally? On a constant basis?
Cheers,
Vladimir
On Mar 29, 11:57 pm, "Nasser Abbasi" <n...@12000.org> wrote:
> Hello Vladimir;
>
> "Vladimir Bondarenko" <v...@cybertester.com> wrote in message
>
> news:373f7f01-6a15-49c0...@p25g2000hsf.googlegroups.com...
>
> (The VM machine, if set to such a task, always crashes Maple,
> Mathematica, AXIOM, Maxima, Derive, and MuPAD; the concrete timing
> depends on the initial parameters.
> As of today, Derive 6.1 is the most stable computer algebra system of the
> above-listed."
>
> It would be interesting if your VM machine can add SAGE to the systems it
> tests. SAGE is described here:
>
> http://en.wikipedia.org/wiki/Software_for_Algebra_and_Geometry_Experi...
SzH
> It would be interesting if your VM machine can add SAGE to the systems it
> tests. SAGE is described here:
>
>
> I played with SAGE a little, but I find its interface on windows, having to
> use a browser, a bit awkward to say the least. I have not tried it text
> based interface.
AFAIK Sage is primarily a browser-based notebook interface to other
packages such as Maxima and SciPy. So it is those other packages that
would benefit from testing and not SAGE itself.
Jaap Spies
This is certainly not true! Sage has its own code base and is much, much
more then only a notebook interface.
See http://www.sagemath.org/
Browse through the code:
http://www.sagemath.org/hg/sage-main?cmd=manifest;manifest=-1;path=/sage/
Jaap
Nasser Abbasi
>Could you please tell why do you feel that testing SAGE could be really
>useful? Useful for whom?
Well, since Sage is, I think, is the first open source GPL'ed CAS software?
and so it would be useful to see how its quality stack up to commercial
software. May be for some who are thinking of using open source CAS but not
sure if Sage is good enough to invest in learning it? Having the ability to
read the source code and experiment with it is a very important aspect in
learning about algorithms used in computer algebra, assuming all things are
equal.
>By the way, maybe you know, approximately, how many persons use SAGE?
>Occasionally? On a constant basis?
From
http://thedaily.washington.edu/2008/1/14/sage-insights-open-source-technology/
it says (dated january 14, 2008): (I cut paste the relevent lines)
"Typically SAGE gets 1,000 hits per month and our user base is 10,000; about
300 developers are on our [developer] e-mail list. Maple [a similar program]
has about a million users, he added. Stein attributes low usage among
professionals to the fact that some professors and engineers find it
difficult to transfer their calculations to a new program. He said students
appreciate the advantages SAGE provides, such as compatibility with non-open
source programs and its ease of use."
Nasser
SzH
> SzH wrote:
> > On Mar 30, 8:57 am, "Nasser Abbasi" <n...@12000.org> wrote:
> >> It would be interesting if your VM machine can add SAGE to the systems it
> >> tests. SAGE is described here:
>
> >>http://en.wikipedia.org/wiki/Software_for_Algebra_and_Geometry_Experi...
>
> >> I played with SAGE a little, but I find its interface on windows, having to
> >> use a browser, a bit awkward to say the least. I have not tried it text
> >> based interface.
>
> > AFAIK Sage is primarily a browser-based notebook interface to other
> > packages such as Maxima and SciPy. So it is those other packages that
> > would benefit from testing and not SAGE itself.
>
> This is certainly not true! Sage has its own code base and is much, much
> more then only a notebook interface.
I am not a SAGE user, but the sagemath.org website created the
impression that SAGE is just some kind of front end to other
packages. I wanted to learn more about this system, but there is no
"About SAGE" link on the front page and the FAQ is not very
informative.
So does SAGE have its own symbolic computation engine? Can it e.g.
compute integrals (one thing that Vladimir has been testing)?
> Seehttp://www.sagemath.org/
> Browse through the code:http://www.sagemath.org/hg/sage-main?cmd=manifest;manifest=-1;path=/s...
I don't think that the right way to introduce SAGE to a newbie is to
ask him to browse the system's source code ...
SAGE might be a useful program, but the current website is somehow not
convincing (and not informative) enough to persuade me to attempt a
500 MB download of something that will only run on a linux system
running inside an emulator ... Just compare it with e.g. the orders
of magnitude better Yacas website to see what I mean.
mabshoff
>
> "Vladimir Bondarenko" <v...@cybertester.com> wrote in message
>
> news:373f7f01-6a15-49c0...@p25g2000hsf.googlegroups.com...
>
> (The VM machine, if set to such a task, always crashes Maple,
> Mathematica, AXIOM, Maxima, Derive, and MuPAD; the concrete timing
> depends on the initial parameters.
> As of today, Derive 6.1 is the most stable computer algebra system of the
> above-listed."
Vladimir,
could you speculate what crashes the various systems, i.e. memory
leaks, random bugs and so on?
> It would be interesting if your VM machine can add SAGE to the systems it
> tests. SAGE is described here:
>
> http://en.wikipedia.org/wiki/Software_for_Algebra_and_Geometry_Experi...
>
> I played with SAGE a little, but I find its interface on windows, having to
> use a browser, a bit awkward to say the least. I have not tried it text
> based interface.
>
> Nasser
Hi Nasser,
looking at what the VM tests Sage farms out most of those tasks
[integration, differentiation, ODE] to Maxima, which is already
tested. We are currently in the process of implementing symbolic
caculus in Sage itself and will hopefully in the next year move
differentiation, limits and integration and so on away from Maxima
into the Sage core. This is motivated by the need for performance and
the fact that we want to dump any lisp based code from the core of
Sage in the long term.
Cheers,
Michael
mabshoff
> On Mar 30, 2:14 pm, Jaap Spies <j.sp...@hccnet.nl> wrote:
>
>
>
> > SzH wrote:
> > > On Mar 30, 8:57 am, "Nasser Abbasi" <n...@12000.org> wrote:
> > >> It would be interesting if your VM machine can add SAGE to the systems it
> > >> tests. SAGE is described here:
>
> > >>http://en.wikipedia.org/wiki/Software_for_Algebra_and_Geometry_Experi...
>
> > >> I played with SAGE a little, but I find its interface on windows, having to
> > >> use a browser, a bit awkward to say the least. I have not tried it text
> > >> based interface.
>
> > > AFAIK Sage is primarily a browser-based notebook interface to other
> > > packages such as Maxima and SciPy. So it is those other packages that
> > > would benefit from testing and not SAGE itself.
>
> > This is certainly not true! Sage has its own code base and is much, much
> > more then only a notebook interface.
>
> I am not a SAGE user, but the sagemath.org website created the
> impression that SAGE is just some kind of front end to other
> packages. I wanted to learn more about this system, but there is no
> "About SAGE" link on the front page and the FAQ is not very
> informative.
>
> So does SAGE have its own symbolic computation engine? Can it e.g.
> compute integrals (one thing that Vladimir has been testing)?
No, it currently doesn't have its own engine, but some of us are in
the process of writing one.
> > Seehttp://www.sagemath.org/
> > Browse through the code:http://www.sagemath.org/hg/sage-main?cmd=manifest;manifest=-1;path=/s...
>
> I don't think that the right way to introduce SAGE to a newbie is to
> ask him to browse the system's source code ...
Yeah, not very useful for some newbie. But I guess that link posted by
Jaap was more directed at "Sage is a web frontend".
> SAGE might be a useful program, but the current website is somehow not
> convincing (and not informative) enough to persuade me to attempt a
The website could certainly be improved. For an incomplete list of
what Sage can do check out
http://www.sagemath.org/doc/html/ref/index.html
> 500 MB download of something that will only run on a linux system
> running inside an emulator ...
It runs on various flavors of Linux and also on OSX. Microsoft
Reseatch is paying for a native Windows port and Solaris is getting
close to running. There are many other worthwhile platforms left after
that in my book.
> Just compare it with e.g. the orders
> of magnitude better Yacas website to see what I mean.
Cheers,
Michael
Jaap Spies
>
>> Seehttp://www.sagemath.org/
>> Browse through the code:http://www.sagemath.org/hg/sage-main?cmd=manifest;manifest=-1;path=/s...
>
> I don't think that the right way to introduce SAGE to a newbie is to
> ask him to browse the system's source code ...
>
This was certainly not an introduction to Sage, but just showing
the existence of the the code base!
Try the tutorial: http://www.sagemath.org/doc/html/tut/index.html
> SAGE might be a useful program, but the current website is somehow not
> convincing (and not informative) enough to persuade me to attempt a
> 500 MB download of something that will only run on a linux system
> running inside an emulator ... Just compare it with e.g. the orders
> of magnitude better Yacas website to see what I mean.
You are right! There is no 'about Sage' page on the web site.
Some Sage developers are working hard on a native Windows port of
Sage. This is not a trivial undertaking, but there is support from
Microsoft Research.
Jaap
Vladimir Bondarenko
MA> could you speculate what crashes the various systems,
MA> i.e. memory leaks, random bugs and so on?
This would require some extra work, some extra resource we lack now.
I am very reluctant to waste words.
MA> random bugs
What do you mean by "random bugs"? Couldn't you please formulate?
MA> Sage farms out most of those tasks [integration, differentiation,
MA> ODE] to Maxima, which is already tested.
yeah, to an extent our resource affords.
MA> Sage farms out most of those tasks [integration, differentiation,
MA> ODE] to Maxima, which is already tested.
Why sure, I meant precisely this!
MA> We are currently in the process of implementing symbolic
MA> calculus in Sage itself and will hopefully in the next year
MA> move differentiation, limits and integration and so on away
MA> from Maxima into the Sage core.
Oh, this is a different kettle of fish altogether!
This, along with the drive you seem to push SAGE, will be a good
incentive for Cyber Tester to test SAGE with the VM machine.
Cheers,
Vladimir Bondarenko
VM and GEMM architect
Co-founder, CEO, Mathematical Director
http://www.cybertester.com/ Cyber Tester, LLC
http://maple.bug-list.org/ Maple Bugs Encyclopaedia
http://www.CAS-testing.org/ CAS Testing
-----------------------------------------------------------------
"We must understand that technologies
like these are the way of the future."
-----------------------------------------------------------------
Jaap Spies
> Michael Abshoff writes
>
>
> MA> Sage farms out most of those tasks [integration, differentiation,
> MA> ODE] to Maxima, which is already tested.
>
> Why sure, I meant precisely this!
>
Is this true? How far do you test Maxima? What shows up in the newsgroups
are failures of Mathematica and Maple. But maybe I'm wrong and are you doing
extensive tests in Maxima.
Jaap
caveman
> Hello Nasser,
>
> Thanks for your idea.
>
> Technically, we could do this. Anyways, now we have a small amount of
> resource we can dedicate to a concrete system; but we here hope that
> the things could look better reasonably soon, hopefully, this year.
>
> As you understand, if we here dedicate something to SAGE, we subtract
> this from the above list I quoted.
Not necessarily - you have many options.
1) Release the code under the GPL, letting others adapt it to SAGE.
Of course, if you want to sell your code, which I think you do, this
might not be too attractive.
(As I have said before, my personal belief is your biggest barrier to
commercial exploitation is your unorthodox approach.)
2) Open source the code, but license it under a restrictive license that
expressly forbids usage by commercial users such as Wolfram Research
without payment of a license fee, but allows its free use testing
open-source programs such as SAGE.
Of course WRI could technically ignore your license, but I very much
doubt they would. No WRI manager is going to ask an employee in WRI to
ignore your license.
However, you may be exploited in some ways.
* An employee of WRI could compile your code at home, report bugs at
WRI, not telling the truth how he found them. He would appear a good
employee and gain promotion.
* Normal users would find bugs, report them to WRI, so WRI gain while
others do the work for them.
3) Adapt your code to SAGE (some work on your part), then release a
*binary* package that only works with SAGE and written in such a way it
would not work with the commercial programs Mathematica, Maple, MATLAB etc.
Obviously you could couple this technical restriction with a license
which prevents its adoption by commercial applications.
I've no idea how difficult it might be for someone to write a
SAGE->Mathematica translator, then use your binary code to test
Mathematica by fooling your code.
Again, nobody in WRI is going to officially ask another employee to
write such a translator, but there might be an incentive for some
employees to do this at home.
4) Release a binary package for testing SAGE an use a client/server
approach so that the code needs an internet connection and gets a
license valid for one day only - much the same as FlexLM works. Then if
you thought you were being exploited you could quickly put a stop to it.
PS - you asked me to email you, which I did, but you never replied.
Dave (from the UK).
mabshoff
> Michael Abshoff writes
Hi Vladimir
> MA> could you speculate what crashes the various systems,
> MA> i.e. memory leaks, random bugs and so on?
>
> This would require some extra work, some extra resource we lack now.
>
> I am very reluctant to waste words.
Sure. I considered the possibility that you did collect the kind of
failures you saw.
> MA> random bugs
>
> What do you mean by "random bugs"? Couldn't you please formulate?
Does the same sequence of input always leads reproducibly to the same
internal error/segfault at the same point during the computation or
does the same sequence of input fail at different, unpredictable
points or not at all?
Detecting that the process just ended up eating all the available RAM
while doing the same computation over and over again [i.e. indicating
memory leaks] should also be straight forward compared to either
straight segfaults or "internal error ... exiting" type situations
where the CAS does detect some issue and exits somewhat gracefully.
> MA> Sage farms out most of those tasks [integration, differentiation,
> MA> ODE] to Maxima, which is already tested.
>
> yeah, to an extent our resource affords.
>
> MA> Sage farms out most of those tasks [integration, differentiation,
> MA> ODE] to Maxima, which is already tested.
>
> Why sure, I meant precisely this!
>
> MA> We are currently in the process of implementing symbolic
> MA> calculus in Sage itself and will hopefully in the next year
> MA> move differentiation, limits and integration and so on away
> MA> from Maxima into the Sage core.
>
> Oh, this is a different kettle of fish altogether!
>
> This, along with the drive you seem to push SAGE, will be a good
> incentive for Cyber Tester to test SAGE with the VM machine.
Yeah, automated testing would be excellent, once the code is getting
merged and can do more than trivial things. The Sage developers as a
whole are fanatical about testing in general and I for example will
drop anything to hunt down a reproducible memory leak. We have a rule
that fixes for bugs are only merged if accompanied by a test that
verifies that the bug in question was fixed. Unlike some of the other
CASs you test we welcome bugs reports and our bug tracker is open to
the public since we have nothing to hide :)
Sage's main strength are certainly not in the calculus are yet since
the original developers cared a lot about number theory and calculus
was initially tacked on to be able to use Sage in Calculus classes. We
are now slowly getting more developers from the areas that do care
about efficient symbolic calculus [physics for example] and
consequently things will improve in that area.
> Cheers,
>
> Vladimir Bondarenko
Cheers,
Michael
> VM and GEMM architect
> Co-founder, CEO, Mathematical Director
>
> http://www.cybertester.com/ Cyber Tester, LLChttp://maple.bug-list.org/ Maple Bugs Encyclopaediahttp://www.CAS-testing.org/ CAS Testing
caveman
> It runs on various flavors of Linux and also on OSX. Microsoft
> Reseatch is paying for a native Windows port and Solaris is getting
> close to running. There are many other worthwhile platforms left after
> that in my book.
What is your problem with Solaris? I was at one point working with
William on trying to do some work with Solaris and have spent quite a
bit of time attempting compilation. The problem seems to be the large
number of external packages you use, all of which are written by
different people using very different approaches.
Not sure what your latest status is, but if you need hardware, rather
than man-hours, I can probably help with access via SSH to one or more
multi-processor Suns. I know at one time you were limited by hardware
availability. If that is still so, I can help.
Thomas Richard
> In Maple memory is allocated but not returned to the operating system.
I think this no longer correct in general for Maple 11. Earlier versions
did have this problem, AFAIK - maybe the change was already in 10.
Upon a restart (button press or command), you can see the effect in
the Windows task manager or the Unix equivalents of such a tool.
--
Thomas Richard
Maple Support
Scientific Computers GmbH
http://www.scientific.de
mabshoff
> mabshoff wrote:
> > It runs on various flavors of Linux and also on OSX. Microsoft
> > Reseatch is paying for a native Windows port and Solaris is getting
> > close to running. There are many other worthwhile platforms left after
> > that in my book.
Hi,
> What is your problem with Solaris?
It compiles in 32 bit mode with small modifications, but the test
suite doesn't pass yet. One big issue is finding a self hosted lisp
that compiles with either gcc or Sun Forte's compiler suite [sbcl and
cmucl are not self hosted, but sbcl does work for my purposes],
another one is in certain parts of libSingular.
> I was at one point working with
> William on trying to do some work with Solaris and have spent quite a
> bit of time attempting compilation. The problem seems to be the large
> number of external packages you use, all of which are written by
> different people using very different approaches.
Well, that is not that much of a problem since I have been fixing
[Solaris specific] bugs in all those packages and been pushing them
back up stream. There are certainly small problems left here and
there, but we can handle those. It is mostly all about smoothly
merging fixes back into our packages as well as keeping those build
issues from new patches under control. I don't think it will surprise
anybody from the Open Source world that most code these days usually
is developed and works on Linux while it can be hard to get it working
on Solaris. Ten years ago it was very much a reversed situation.
> Not sure what your latest status is, but if you need hardware, rather
> than man-hours, I can probably help with access via SSH to one or more
> multi-processor Suns. I know at one time you were limited by hardware
> availability. If that is still so, I can help.
Thanks. William did hire me full time to work on the Solaris port for
2008 and I am certain we will get it done in 32/64 bit on Solaris 9/10/
Indiana on x86/x86-64/Sparc. This is a huge test matrix and the
differences between Solaris 9 and 10 are rather large. I am switching
the 64 bit build of Sage to the Sun Forte compiler since it causes way
fewer problems in 64 bit mode compared to gcc. So if you have some
more test machines with say Indiana on Sparc I would be more than
happy to take you up on that offer. Even build and test feedback would
be nice. Since this is getting OT we should take this offlist from now
on or go over to gg:sage-devel.
Cheers,
Michael
rjf
......
>
> Well, since Sage is, I think, is the first open source GPL'ed CAS software?
> and so it would be useful to see how its quality stack up to commercial
> software.
How is it first?
Maxima? Axiom? Jacal?
Vladimir Bondarenko
About some of them we thought. I hope to give you much more details
in private.
I received your letter several days ago, and wrote you two letters to
this address. None of them bounced, and still you cannot read them.
Could they be trapped by your spam filter if you have any?
Dave
> On Mar 30, 4:47 pm, caveman <dev.n...@dev.null.com> wrote:
>> mabshoff wrote:
>>> It runs on various flavors of Linux and also on OSX. Microsoft
>>> Reseatch is paying for a native Windows port and Solaris is getting
>>> close to running. There are many other worthwhile platforms left after
>>> that in my book.
>
> Hi,
>
>> What is your problem with Solaris?
>
> It compiles in 32 bit mode with small modifications, but the test
> suite doesn't pass yet.
> So if you have some
> more test machines with say Indiana on Sparc I would be more than
> happy to take you up on that offer. Even build and test feedback would
> be nice. Since this is getting OT we should take this offlist from now
> on or go over to gg:sage-devel.
I'll drop you a private email.
rjf
(sci.math) and (deliberately?) misinterpret the main question. The
question was whether we could come up with benchmarks that tested the
long-term performance of a CAS (and maybe, whether that mattered or
not). How long it takes to provoke a fatal error by some heuristic
search is a different question.
Thomas Richard actually provided info regarding Maple 11 behavior in
returning memory. It still doesn't answer the more pointed question,
which is... can Maple 11 get into a situation where its memory usage
is so scattered that it runs at "disk speed" even though the actual
data in use is not so voluminous.
Whether SAGE should be tested independently of its components is
hardly a question. Why should SAGE not have bugs in addition to the
bugs in the libraries it uses? On the main question here, as to
whether there are benchmarks that would test efficiency, I assume that
the benchmarks for Maxima (etc) would be relevant to SAGE to the
extent that SAGE is just calling Maxima. To the extent that SAGE has
its own inner loops and memory allocation, it could have its own
problems or successes. I understand that it is essentially a python
program, and thus the python implementation may be a boon or a
disaster, and maybe that is the most prominent factor that can be
identified. (Is there a compacting garbage collector for python?)
I don't understand NA's comment that SAGE is the first GPL CAS, since
it depends on several other predecessors. Or the comment that large
problems in Maple are interpreted.
If we don't raise questions when people continue to compare CAS on
essentially irrelevant grounds, we are in the position of essentially
allowing others who read this newsgroup from accepting these comments
as valid.
rjf
(sci.math) and (deliberately?) misinterpret the main question. The
question was whether we could come up with benchmarks that tested the
long-term performance of a CAS (and maybe, whether that mattered or
not). How long it takes to provoke a fatal error by some heuristic
search is a different question.
Thomas Richard actually provided info regarding Maple 11 behavior in
returning memory. It still doesn't answer the more pointed question,
which is... can Maple 11 get into a situation where its memory usage
is so scattered that it runs at "disk speed" even though the actual
data in use is not so voluminous.
Whether SAGE should be tested independently of its components is
hardly a question. Why should SAGE not have bugs in addition to the
bugs in the libraries it uses? On the main question here, as to
whether there are benchmarks that would test efficiency, I assume that
the benchmarks for Maxima (etc) would be relevant to SAGE to the
extent that SAGE is just calling Maxima. To the extent that SAGE has
its own inner loops and memory allocation, it could have its own
problems or successes. I understand that it is essentially a python
program, and thus the python implementation may be a boon or a
disaster, and maybe that is the most prominent factor that can be
identified. (Is there a compacting garbage collector for python?)
If we don't raise questions when people continue to compare CAS on
par...@domain.invalid
> [integration, differentiation, ODE] to Maxima, which is already
> tested. We are currently in the process of implementing symbolic
> caculus in Sage itself and will hopefully in the next year move
> differentiation, limits and integration and so on away from Maxima
> into the Sage core. This is motivated by the need for performance and
> the fact that we want to dump any lisp based code from the core of
> Sage in the long term.
>
If you want to move away from lisp, what about using the C++ Giac
library instead of rewriting and testing again all the calculus code.
Giac already implements the mrv algorithm for limits, the Risch (pure
transcendental) algorithm for limits, as well as few methods for
definite integration.
Giac was 3rd at the Trophees du Libre that Sage won last year.
http://www-fourier.ujf-grenoble.fr/~parisse/giac.html
Mike Hansen
> program, and thus the python implementation may be a boon or a
> disaster, and maybe that is the most prominent factor that can be
> identified. (Is there a compacting garbage collector for python?)
Python 2.5 will release memory back to the system, but the CPython
implementation guarantees that it will not move objects around.
> If we don't raise questions when people continue to compare CAS on
> essentially irrelevant grounds, we are in the position of essentially
> allowing others who read this newsgroup from accepting these comments
> as valid.
Agreed.
--Mike
mabshoff
Hello,
> Well, VB managed both to post to an additional distracting newsgroup
> (sci.math) and (deliberately?) misinterpret the main question. The
> question was whether we could come up with benchmarks that tested the
> long-term performance of a CAS (and maybe, whether that mattered or
> not). How long it takes to provoke a fatal error by some heuristic
> search is a different question.
I agree with you on that point, i.e. the use of toy examples to
benchmark is wrong. It is amazing how much code out there [non-
mathematical and mathematical alike] falls over when you start pushing
it. And the vast majority of benchmarks do not take long term effects
or even memory consumption into account. The most famous example I can
come up with are Gröbner Basis computations comparing the Buchberger
algorithm with F4 or F5.
> Thomas Richard actually provided info regarding Maple 11 behavior in
> returning memory. It still doesn't answer the more pointed question,
> which is... can Maple 11 get into a situation where its memory usage
> is so scattered that it runs at "disk speed" even though the actual
> data in use is not so voluminous.
Well, I think it mostly matters if that is a case that does happen in
real life or if you can construct some malicious example to prove your
point. I know next to nothing about the internal memory management
systems of Maple or Mathematica, but the choices aren't really between
"dumb" malloc and sophisticated gc enabled lisp with compactification.
People have been using slab allocators in code for decades, so the
fragmentation issue that potentially happens with some "dumb" heap
base allocation algorithm has less of an impact in the real world
IMHO. Heap fragmentation is a serious issue, but if that really is a
problem with the algorithms one wants to implement one needs to adapt
the algorithm or use various tricks like slab allocators. Those aren't
a magic bullet, but one my list of things to do heap fragmentation is
not very high. I can easily construct and example where I exhaust all
available memory using slab allocators, but I have never hit an
example like that in real life.
> Whether SAGE should be tested independently of its components is
> hardly a question. Why should SAGE not have bugs in addition to the
> bugs in the libraries it uses?
Why is that question even relevant to this discussion? Given the
choice between GAP to do some computation related to group theory or
everything else out there what would you choose? Given some
computation related to L-functions would you use John Cremona's eclib,
sympow or lcalc or something else? I am sure that most of the above
computations matter little to most people, but Sage's main goal is
mathematical research in number theory, combinatorics, graph theory
and cryptography and a couple other things. So people tend to push the
code a little harder than you average college student computing some
integrals to do his or her homework.
Sage is more than the sum of its parts and I am under the impression
from previous comments you made in public and private correspondence
that you do not understand the concept behind Sage. Does Sage have
bugs? Yes. But the important thing is that we are fixing those bugs
and I tend to believe that we are moving much more quickly than the
Open Source competition out there.
And regarding memory consumption: I do valgrind the complete Sage test
suite at least once a week and analyze the logs and also investigate
every time somebody does report a potential memory leak. After every
two or three patches I merge I run a test suite consisting of 50,000+
[and steadily growing] inputs consuming about an hour and a half of
CPU time and if things get broken we do fix them. People have been
running computations on sage.math for months at a time with the same
Sage instance and in those particular cases the memory consumption did
not grow. So Sage is not some toy system.
Every component in Sage is audited for memory leaks via the doctests
and I have been slowly but surely auditing all components of Sage with
their own test suite with valgrind for memory leaks. I am not done
yet, but will finish this year. All the issues I found have been fixed
upstream. We take memory leaks and bugs in general very seriously. If
a patch causes memory leaks during review that code is not merge,
regardless of how desirable that feature is. The Sage project did not
set out to do an alright job or come in second best. The declared goal
is to beat Magma on performance and memory efficiency and while we are
not there yet across the board we have made tremendous progress in the
last year and have overtaken Magma in some areas. See
http://sagemath.blogspot.com/2008/02/benchmarketing-modular-hermite-normal.html
for example.
Sage certainly still has its weaknesses (mv poly factorization, no F4/
F5 for anything but boolean rings and a couple other things), but we
are aware of those issues and on the way to fix them. Magma is still
better in a lot of areas, but let's see how things are in two years.
> On the main question here, as to
> whether there are benchmarks that would test efficiency, I assume that
> the benchmarks for Maxima (etc) would be relevant to SAGE to the
> extent that SAGE is just calling Maxima. To the extent that SAGE has
> its own inner loops and memory allocation, it could have its own
> problems or successes. I understand that it is essentially a python
> program, and thus the python implementation may be a boon or a
> disaster,
Sage does combine Python, Cython, C and C++ code (and some Fortran)
and depending on the computation you do the vast amount of memory is
not allocated in the Python instance but in the lower layers. For
example if you allocate a large number of matrices and delete them the
memory is returned to the system since Python's garbage collector is
barely involved in that case. Some high level structure are involved
since we use Python's C API. It raises the potential for heap
fragmentation, but with today's 64 bit CPUs I am more concerned about
the eventual demise of our Sun than the exhaustion of VM space.
> and maybe that is the most prominent factor that can be
> identified. (Is there a compacting garbage collector for python?)
I don't think Python itself can return memory to the system, but it
doesn't matter too much for the reasons cited above.
> If we don't raise questions when people continue to compare CAS on
> essentially irrelevant grounds, we are in the position of essentially
> allowing others who read this newsgroup from accepting these comments
> as valid.
Cheers,
Michael
Vladimir Bondarenko
RJF> long-term performance of a CAS (and maybe, whether that
RJF> mattered or not).
A very good question. The long-term performance is something the
customers need. Personally, I hate when after using Maple 11 for
several days, I must reboot as my machine gets tortoise-like.
In contrast, if Mathematica 6 runs for days, and Maple 11 is not run
over this span of time, there is no tangible performance degradation.
Please note that I speak only about *classic* version as using
Standard Worksheets makes my machine, quickly, running just like a
sleepy Bradypus.
...
The long-term performance does matter as it is not impossible to
imagine a symbolic/hybrid calculation task that takes a week or
a month.
RJF> and (deliberately?) misinterpret the main question.
I am shocked to hear this viewpoint... I hope you are not in
serious...
SzH
> RJF> whether we could come up with benchmarks that tested the
> RJF> long-term performance of a CAS (and maybe, whether that
> RJF> mattered or not).
>
> A very good question. The long-term performance is something the
> customers need. Personally, I hate when after using Maple 11 for
> several days, I must reboot as my machine gets tortoise-like.
>
> In contrast, if Mathematica 6 runs for days, and Maple 11 is not run
> over this span of time, there is no tangible performance degradation.
Here's a very simple experiment with Mathematica 6.0.2 on WinXP. The
results are repeatable on my computer (I ran this several times). I
have 1 GB of RAM.
In[1]:= $HistoryLength = 0
Out[1]= 0
In[2]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
Out[2]= {61.86, Null}
In[3]:= Timing[x =.]
Out[3]= {1.984, Null}
In[4]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
Out[4]= {84.766, Null}
In[5]:= Timing[x =.]
Out[5]= {2.843, Null}
In[6]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
Out[6]= {94.219, Null}
In[7]:= Timing[x =.]
Out[7]= {2.922, Null}
In[8]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
Out[8]= {95.484, Null}
In[9]:= Timing[x =.]
Out[9]= {2.938, Null}
Note how the timing of the Expand[...] increases from ~60 s to ~95 s
in the subsequent evaluations. I don't want to speculate about why
this happens, but I would like to know what other people think.
Two things to watch out for if you start experimenting yourself (I
have /not/ tested these thoroughly, so they might not be repeatable):
1. Each input should be put in a separate input cell. If the
Expand[...] and the x=. are in the same input cell, Mathematica does
not release memory and the system starts swapping ...
2. It appears that if Share[] is executed after Expand[...], but /
before/ x=., then the timing value does not increase.
Szabolcs
SzH
> Two things to watch out for if you start experimenting yourself (I
> have /not/ tested these thoroughly, so they might not be repeatable):
>
> 1. Each input should be put in a separate input cell. If the
> Expand[...] and the x=. are in the same input cell, Mathematica does
> not release memory and the system starts swapping ...
>
> 2. It appears that if Share[] is executed after Expand[...], but /
> before/ x=., then the timing value does not increase.
These two are not correct. Something else must have caused them.
Jaap Spies
>
> If we don't raise questions when people continue to compare CAS on
> essentially irrelevant grounds, we are in the position of essentially
> allowing others who read this newsgroup from accepting these comments
> as valid.
On Dec 6, 2005 rjf wrote a comparison in sci.math.symbolic:
> Elephants are interesting and useful. Feathers are interesting
> and useful.
> Elephants with feathers are a curiosity. Is SAGE an elephant
> with feathers?
>
>
See: http://sci.tech-archive.net/Archive/sci.math.symbolic/2005-12/msg00096.html
This is almost two and a half year ago. Time to reconsider!?
Next release of Sage will be sage-3.0, due in a week or two. Sage is
very much alive. Not as big a an elephant, but growing. And even without
feathers the future looks light and bright.
I invite you, Richard Fateman and others to look at Sage with an open mind.
Jaap
semiopen
> SzH> You mean that you are able to run the Mathematica 6
> SzH> kernel continuously, performing random symbolic computations
> SzH> (like integration),
>
> Well, the inputs the VM machine feeds into the MathKernel are not
> totally random :)
>
> Some of them are brute-force ones, but a tangible fraction of them
> comes from some conditions the VM machine calculates trying to crash
> the MathKernel.
>
> (The VM machine, if set to such a task, always crashes Maple,
> Mathematica, AXIOM, Maxima, Derive, and MuPAD; the concrete timing
> depends on the initial parameters.
>
> As of today, Derive 6.1 is the most stable computer algebra system
> of the above-listed.
>
> for about 3 weeks (!) of continuous calculations the VM machine forced
> it to perform.)
This is another reason that it is so sad that Texas Instruments
stopped supporting Derive. Have you looked at TI's replacement for
Derive? I am not optimistic, but since they had Derive's source code
maybe they were able to preserve some of Derive's nice features.
> SzH> for 24 hours, without any problems?
>
> Kind of this, in the average.
>
> SzH> That is better than what I expected.
>
> :)
>
> It was worse in Mathematica 5.2. But we feel that in Mathematica 6
> some real improvements are introduced.
>
> SzH> How much memory does your machine have?
>
> RAM is 4 Gb, and total free HDD size of about 500 Gb.
>
> On Mar 29, 2:48 pm, SzH <szhor...@gmail.com> wrote:
>
>
>
> > On Mar 29, 9:50 pm, Vladimir Bondarenko <v...@cybertester.com> wrote:
>
> > > For example, typically, if set to this task, the VM machine
> > > can crash Maple 11 within 1 hour, Mathematica 6 within 1 day,
> > > and Derive within 3 days.
>
> > You mean that you are able to run the Mathematica 6 kernel
> > continuously, performing random symbolic computations (like
> > integration), for 24 hours, without any problems? That is better than
> > what I expected. How much memory does your machine have?- Hide quoted text -
>
> - Show quoted text -
Vladimir Bondarenko
SO> Instruments stopped supporting Derive.
I've been very close (or at time inside) the Derive's witchcraft
kitchen and know more than I can say.
But I can state safely that Derive has a big potential for growth.
If one fine day Texas Instruments would sell it, I'd probably could
have a try to persuade Albert Rich to continue pushing Derive. Well
there is not exactly I would like to say, but you can guess more.
And maybe one fine day I can tell more about all this thrilling
story.
SO> Have you looked at TI's replacement for Derive?
In the autumn of 2006 I enjoyed a privilege to be selected one
of its beta testers
................................................................
http://groups.google.com/group/sci.math.symbolic/msg/9672efcab9159d9f
http://www.fedex.com/Tracking?language=english&cntry_code=&tracknumbers=790636569170
http://www.cas-testing.org/index.php?list=3
................................................................
Instead of testing it manually, we forced the VM machine to test
it. :)
With some unique results, as usually.
It is now for the first time that I mentioned this fact publicly.
SO> I am not optimistic, but since they had Derive's source
SO> code maybe they were able to preserve some of Derive's
SO> nice features.
Unfortunately, it's better that you are not optimistic... there
are several reasons for this...
What's good, you and me still can use old good Derive 6.1.
> > - Show quoted text -- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
Roman Pearce
> Thomas Richard actually provided info regarding Maple 11 behavior in
> returning memory.
We need to clear this up. I assume that if you're running programs
for days or weeks then you will have objects in memory that you need
to keep, so you can't use the restart command. This would not be
true for something like a web-based integrator that simply passes
independent problems to the system, but it would be true for a
complex program that maintains an internal state, like an airline
reservation program. Without restarting the kernel, Maple will
not return memory to the OS and the time for garbage collection
will grow (up to a limit).
It looks like Mathematica does the same thing:
SzH wrote:
> In[2]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
> Out[2]= {61.86, Null}
...
> In[8]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
> Out[8]= {95.484, Null}
I would like to say that this is not an entirely unreasonable
design, and that more advanced garbage collection techniques
could possibly reduce the overhead. The difference in time
is largely because the system has more memory allocated, which
means that the garbage collector has to scan more memory each
time a GC occurs. Scanning memory is not free. Computers have
finite memory bandwidth, on the order of a few GB/s. If you are
repeatedly scanning even 200MB, you will notice it. And you will
be hit twice because garbage collection flushes the L2/L3 cache.
Regarding your question:
> can Maple 11 get into a situation where its memory usage
> is so scattered that it runs at "disk speed" even though
> the actual data in use is not so voluminous.
I can think of three situations where this could happen:
1) You have large objects in memory that are not being used.
The garbage collector still has to look at them.
2) You temporarily used a lot of memory. Modern operating systems
swap unused pages to disk, so even with GB allocated (because
memory is not released), the slowdown will be temporary if the
internal memory structures coalesce free space (they do).
You don't scan memory marked as "free".
3) There is a bug in the garbage collector
Also, replying to mabshoff now, I would really like to know what
you meant here:
rjf
Many of the Maple problems cited here are solved by generation
scavenging garbage collectors in modern lisps.
I don't quite understand the "coalesce" comment above. If only
adjacent memory locations are coalesced, that doesn't prevent
fragmentation.
And probably Mathematica also has these problems. At some point I
suspect that the shortcuts taken by these systems in their memory
allocation and associated hacks will come back and bite them (e.g.
when problems run at disk speed); whereas certain more sophisticated
(and perhaps initially slower?) systems will compact their memory and
continue to run at RAM speed.
Whether these problems occur often, sometimes, or never --- presumably
depends on the nature of the computations attempted.
RJF
SzH
> SzH wrote:
> > In[2]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
> > Out[2]= {61.86, Null}
> ...
> > In[8]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
> > Out[8]= {95.484, Null}
>
> I would like to say that this is not an entirely unreasonable
> design, and that more advanced garbage collection techniques
> could possibly reduce the overhead. The difference in time
> is largely because the system has more memory allocated, which
> means that the garbage collector has to scan more memory each
> time a GC occurs. Scanning memory is not free. Computers have
> finite memory bandwidth, on the order of a few GB/s. If you are
> repeatedly scanning even 200MB, you will notice it. And you will
> be hit twice because garbage collection flushes the L2/L3 cache.
I didn't mean that this was a bug, I just found it a bit surprising.
I must say that (not being a programmer) I have no idea how garbage
collection algorithms work. But this page led me believe that
clearing the value of x (x =.) will restore the Mathematica kernel to
its original state (save for the existence of 'x' as a symbol with no
OwnValues):
http://reference.wolfram.com/mathematica/note/SomeNotesOnInternalImplementation.html
"Every piece of memory used by Mathematica maintains a count of how
many times it is referenced. Memory is automatically freed when this
count reaches zero."
If it works like this then why would a large amount of memory need to
be scanned explicitly?
Again, I am not a programmer, so sorry if this is a stupid question.
I read about a similar technique in Bruce Eckel's book: "Thinking in C+
+". From what I have learned there, I do not see why doing the same
calculation a second time would be slower than it was the first time.
Daniel Lichtblau
> On Mar 31, 10:14 pm, Roman Pearce <rpear...@gmail.com> wrote:
>
>
>
> > SzH wrote:
> > > In[2]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
> > > Out[2]= {61.86, Null}
> > ...
> > > In[8]:= Timing[x = Expand[(a + b + c + d + e + f + g)^34];]
> > > Out[8]= {95.484, Null}
>
> > I would like to say that this is not an entirely unreasonable
> > design, and that more advanced garbage collection techniques
> > could possibly reduce the overhead. The difference in time
> > is largely because the system has more memory allocated, which
> > means that the garbage collector has to scan more memory each
> > time a GC occurs. Scanning memory is not free. Computers have
> > finite memory bandwidth, on the order of a few GB/s. If you are
> > repeatedly scanning even 200MB, you will notice it. And you will
> > be hit twice because garbage collection flushes the L2/L3 cache.
>
> I didn't mean that this was a bug, I just found it a bit surprising.
> I must say that (not being a programmer) I have no idea how garbage
> collection algorithms work. But this page led me believe that
> clearing the value of x (x =.) will restore the Mathematica kernel to
> its original state (save for the existence of 'x' as a symbol with no
> OwnValues):
>
>
> "Every piece of memory used by Mathematica maintains a count of how
> many times it is referenced. Memory is automatically freed when this
> count reaches zero."
>
> If it works like this then why would a large amount of memory need to
> be scanned explicitly?
>
> Again, I am not a programmer, so sorry if this is a stupid question.
> I read about a similar technique in Bruce Eckel's book: "Thinking in C+
> +". From what I have learned there, I do not see why doing the same
> calculation a second time would be slower than it was the first time.
I also do not know the reason for this. I can only surmise that it is
an effect of memory fragmentation. My own anecdotal experience with
this type of timing rise on repeated computations is that it tends to
stabilize after a few iterations. Which I guess is mildly encouraging.
Daniel Lichtblau
Wolfram Research
mabshoff
Hi Roman,
<SNIP>
Sorry, I don't follow these groups too closely, but I happened to run
across your question while I was looking for some quote.
> Also, replying to mabshoff now, I would really like to know what
> you meant here:
>
> mabshoff wrote:
> > And the vast majority of benchmarks do not take long term effects
> > or even memory consumption into account. The most famous example I can
> > come up with are Gröbner Basis computations comparing the Buchberger
> > algorithm with F4 or F5.
>
>
I meant the following: If you look at Allan Steel's Gröbner Basis
Timings Page [1] you will see him compare his excellent F4
implementation in Magma with various "classical" Bucherberger
implementations in Singular and M2. But while Magma wipes the floor
with Singular2-0-5 and Macaulay 2 0.9.2 timing wise [note: these are
benchmarks from 2004, so those were "current" releases] if you look at
the amount of memory consumed by Magma and the competition you will
quickly see that Magma's F4 consumes considerably more memory than
either Singular or M2. So an uninformed onlooker will conclude that
Magma is king of the hill [it is when enough hardware, i.e. RAM is
available], but the story changes considerably when you consider some
PhD candidate hunting for cycles and RAM to get his/her computations
done. It is likely that the person will push the available resources
and chances are that you can deal with 10 times the required time much
more easily on a budget than with let's say five time the required
RAM.
I have been asked to do various GB computations where I had trouble to
fit the end result, i.e. the GBasis in 16GB RAM [in fact I never got
to the end, the problem was just too huge - at least with the current
algorithms] , so very few systems on this planet would be able to
compute the same GBasis with F4/F5 assuming I had actually finished.
So while I consider F4/F5 a huge achievement and also the
implementation of it in FGb as well as Magma superb coding it should
not lead people to believe that we should measure progress in
benchmarking CAS problems by time alone, but also consider RAM
consumption.
Cheers,
Michael