VM: A new high-performance engine for MuPAD quality assurance
Vladimir Bondarenko
X-Original-To: cyber...@crimea.com
Delivered-To: cyber...@crimea.com
Received: by mail.crimea.com (Postfix, from userid 86)
id 42497107FE2; Fri, 17 Jun 2005 16:40:22 +0300 (EEST)
Received: from web52103.mail.yahoo.com (web52103.mail.yahoo.com
[206.190.39.72]) by mail.crimea.com (Postfix) with SMTP
id 28161107F04 for v...@cybertester.com ; Fri, 17 Jun 2005
16:40:20 +0300 (EEST)
Received: (qmail 30254 invoked by uid 60001); 17 Jun 2005 13:33:38
-0000
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.
com; h=Message-ID:Received:Date:From:Reply-To:Subject:To:MIME-
Version:Content-Type:Content-Transfer-Encoding;b=B62jauDpOQMp69Sm/
o84FTv+PWvuOFYVuvhIgUZ2Fbr0sTtFbppBLDay7kc/VlDB/9Zcp3tZ1NIvev+zlWn
izFtHTeUMNmsJHpJmH4YiHUhRxdgRTp1KsRhawumlJsf9JFxNB0VRk2X2LnjVdTTxY
FWoVsy7fSOQwGaSNwN/OYc= ;
Message-ID: 2005061713333...@web52103.mail.yahoo.com
Received: from [213.138.112.3] by web52103.mail.yahoo.com via
HTTP; Fri, 17 Jun 2005 06:33:38 PDT
Date: Fri, 17 Jun 2005 06:33:38 -0700 (PDT)
From: Vladimir Bondarenko algebr...@yahoo.com
Reply-To: v...@cybertester.com
Subject: VM: A new high-performance engine for MuPAD quality assurance
To: kl...@sciface.com, oliver...@sciface.com, b...@fuchssteiner.de,
wal...@mupad.de, wal...@uni-paderborn.de , v...@cybertester.com
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="0-1684619800-1119015218=:
30114" Content-Transfer-Encoding: 8bit
Dear Dr Kluge,
Being SciFace's CEO, it is you who bears the ultimate responsibility
for financial success of your company and further MuPAD's fate, for
the destiny of SciFace's customers and your company's staff. Your
signature is set on the MuPAD first prize Beta Contest Certificate
of March 22, 2002 with which SciFace awarded one of us.
http://www.cas-testing.org/SciFace.phtml
The results of that time can be compared with a grammar school level,
not even with a K-12 one. But like your company advances so we do.
We are pleased to inform SciFace that, as of June of 2005, the Cyber
Tester's Team has reached, with our current LIFT/CYCLE technologies
implementation completion degree, a level of undergraduate, and this
is what we propose to round-table, SciFace & Cyber Tester.
We are speaking about the VM machine. A failure prediction testing
oracle that checkmates, over a wide gamut, any human tester however
diligent and gifted and trained. A current level implementation of
a new resource saving paradigm one of us discovered in the end of
December of 1992.
It is understandable that SciFace attempts to deliver the best MuPAD
quality possible to the its customers. Both of us, being professional
software makers, truly enjoy the quality of the large-scale source
code developed by the MuPAD Team and your longing for being the best
on the market. For example, it is more than pleasant to feel that
the new MuPAD's fantastic graphics often, if not almost always,
surpasses that of ALL your competitors, bravo!
Just an example of hundreds to savour.
> plotfunc2d(exp(1/tan(z)), z = 0..2*PI);
// MuPAD 3.1.1: a nice graph of 2 branches
> Plot[Exp[1/Tan[z]], {z, 0, 1}]
// Mathematica 5.1.1: A single vertical line.
> plot(exp(1/tan(z)), z = 0..2*Pi);
// Maple 9.5.2: Floating Point Overflow. Please shorten axes.
Another nice example. Numerical approximation of hypergeometric
functions, one of fundamental cogs in serious calculations, very
often works really good. Just compare:
Maple 9.5.2> evalf(hypergeom([1, 1/4, 1/3],[2, 6], 1));
Execution stopped: Stack limit reached. # 1451 seconds
Error, (in evalf/exp/exp1) too many levels of recursion # 19470 seconds
Execution stopped: Stack limit reached. # 1277 seconds
Execution stopped: Stack limit reached. # 1949 seconds
Error, (in evalf/exp/exp1) too many levels of recursion # 14527 seconds
Execution stopped: Stack limit reached. # 523 seconds
while MuPAD 3.1.1 returns instantly the correct answer
> float(hypergeom([1, 1/4, 1/3],[2, 6], 1));
1.007618856
In integration, we also can show examples when MuPAD 3.1.1 beats
all competitors, here you are:
> int(sin(ln(1+exp(z)))/(1+exp(z))/(1+exp(-z)), z= 0..infinity);
1/4*cos(ln(2)) + 1/4*sin(ln(2))
while Derive 6.1, Maple 9.5.2, and Mathematica 5.1 return the
integral unevaluated.
However, the quoted sweet stuff should not hide from you a truly
discouraging fact that the only proper name for the current SciFace's
QA process is 'disaster'. Indeed, our VM machine witnesses:
1. The MuPAD bug database misses (at least) hundreds distinct bug
manifestations (actually, our estimation is that currently,
thousands distinct bug manifestation are missed).
2. There are dozens bugs in the current MuPAD bug database that had
been fixed in the earlier versions, but have resurfaced in
MuPAD 3.1.1, and thus must be re-opened - but somehow the
SciFace's testing staff cannot observe these facts.
In other words, your engineers are not aware of the existence
of thousands bugs in MuPAD 3.1.1, and therefore, cannot fix them.
Thus, unless the point is not taken seriously, your customers
would tumble over these bugs, and this would be not good both
the customers or for SciFace.
In fact, MuPAD has grown into a large system, and traditional,
human-based, at best semi-automated quality management methods
cannot deal successfully with such huge systems. For example,
being not equipped with new high-performance QA support methods,
Maplesoft had run into trouble and was forced to resort to
selling a raw, beta Maple 9 version under the brand of a Maple
9 commercial version to which fact the Maplesoft's customers
reacted highly negatively.
Coming back to MuPAD quality assurance process.
Just a single totally embarrassing, typical freshman calculus
example of many hundreds variegated bugs in MuPAD 3.1.1 identified
by the VM machine but missed by the SciFace's human beings, and
not cataloguized into MuPAD bug database.
TEST CASE: reset(): assume(a>-1):
int(z^a, z= 0..1);
ACTUAL: (1/2)^a
EXPECTED: 1/(a+1)
TEST CASE: reset(): assume(a<-1):
int(z^a, z= 0..1);
ACTUAL: 1/(a + 1)
EXPECTED: infinity
While working on MuPAD 3.1.1 our VM machine encounters with
hundreds MuPAD error messages like, in particular,
Error: argument must be of 'Type::Arithmetical'
Error: Argument out of range
Error: assumption impossible (property::Null)
Error: Can't convert to double (out of range)
Error: Can't evaluate to boolean
Error: cannot find either side of {[V36=0]} in variable list[U36,V36]
Error: Complex arguments are not allowed in comparisons;
Error: degree too large:
Error: Division by zero
Error: Domain attribute x missing;
Error: expecting an arithmetical expression
Error: illegal alias name
Error: Illegal argument
Error: Illegal coefficient ring
Error: Illegal first argument FAIL
Error: Illegal operand
Error: Invalid index
Error: Invalid left-hand side
Error: Out of memory
Error: PARI failure! Please report to bu...@mupad.de
Error: Recursive definition
Error: singularity
Error: Zero denominator detected
Fatal internal error. [c0000005]
and more, identifies hundreds mathematically invalid answers
and regression bugs. Not nice.
Here is a tiny human-selected collection of MuPAD 3.1.1 bug
manifestations out of a huge stock generated by the VM machine.
...................................................................
TEST CASE: int(exp(-z)*cosh(z), z= 0..infinity);
ACTUAL: undefined
EXPECTED: infinity
...................................................................
TEST CASE: solve(1^x = 1^y, {x, y});
ACTUAL: {[]}
EXPECTED: {[x = x, y = y]} or {[x = y]}
...................................................................
TEST CASE: solve(a*(1 + x)*(1 + y) = x, {a, x, y});
ACTUAL: Error: Zero denominator detected [recursive_normal]
EXPECTED: {a=a, x=x, y=-(a+a*x-x)/a/(1+x)}, {a=0, x=0, y=y}
...................................................................
TEST CASE: reset(): assume(n>=3):
int(int(1/(1+x+y)^n, x=0..infinity),y=0..infinity);
ACTUAL: -infinity
EXPECTED: 1/(n^2-3*n+2)
...................................................................
TEST CASE: solve(ode(y'(z) = sqrt(z * y(z)), y(z)));
ACTUAL: {0} // in about 35 seconds at Athlon XP 1600+
EXPECTED: {solve(5/(25*y(z)-z^5)/(5*(z*y(z))^(1/2)-z^3)*
(z*y(z))^(1/2)+1/(25*y(z)-z^5)/(5*(z*y(z))^(1/
2)-z^3)*z^3-C1 = 0, y(z))}
...................................................................
TEST CASE: sum(1/(n!*(n + 1)), n= 0..infinity);
ACTUAL: Error: Division by zero [_power];
during evaluation of 'Sum::hypergeomeval'
EXPECTED: exp(1)-1
...................................................................
TEST CASE: sum(1/((n - 1)*(n - 2)), n= 1..infinity);
ACTUAL: -1
EXPECTED: undefined or Error: Singularity [Sum::sum]
...................................................................
TEST CASE: sum(3^n-2^n, n= 1..infinity);
ACTUAL: undefined
EXPECTED: infinity
...................................................................
TEST CASE: solve(ode(z*y''(z) + y'(z) - z*y(z) = 1, y(z)));
ACTUAL: Error: Zero denominator detected [recursive_normal]
EXPECTED: {besselI(0,z)*C2+besselK(0,z)*C1+PI/2*z*struveL(0,z
)*(besselI(0,z)*besselK(1,z)+besselK(0,z)*besselI(1
,z))}
...................................................................
TEST CASE: solve(ode(y'(z) = y(z)/(1 - z^2*y(z)), y(z)));
ACTUAL: Error: Recursive definition [See ?MAXDEPTH];
during evaluation of 'recursive_normal'
EXPECTED: {C1-(-z*y(z)^(1/2)*besselK(1,2*y(z)^(1/2))-
besselK(0,2*y(z)^(1/2)))/(z*y(z)^(1/2)*
besselI(1,2*y(z)^(1/2))-besselI(0,2*y(z)^(1/2)))}
...................................................................
TEST CASE: solve(ode(y'(z) = 1/(1 - z^2*y(z)), y(z)));
ACTUAL: MuPAD keeps running after 60000 seconds...
EXPECTED: {solve(z+tan(y(z)*(-y(z))^(1/2)-C1*(-y(z))^(1/2))*
(-y(z))^(1/2)/y(z) = 0, y(z))}
...................................................................
TEST CASE: limit(igamma(z+1,z)/gamma(z+1), z= infinity);
ACTUAL: 0
EXPECTED: 1/2
...................................................................
TEST CASE: float(hypergeom([1/2, 1/2, 1], [5/6, 4/3], 1));
ACTUAL: 2.86065496 in 15000+ seconds
EXPECTED: 2.958744085 in 0.1 seconds
...................................................................
TEST CASE: abs(exp(PI*sqrt(163))-262537412640768744);
ACTUAL: exp(163^(1/2)*PI) - 262537412640768744
EXPECTED: 262537412640768744 - exp(163^(1/2)*PI)
...................................................................
And these examples are maybe not still the most embarrassing ones,
trust us.
With the current massive bug stuff like the quoted, of thousands
distinct bugs unknown to the SciFace developers, your company could
soon be running risk of loosing its customers... Why don't stay
competitive?
Maybe you could consider the stated as a kind of selling propaganda
directed to entice you to acquire Cyber Tester's services.
Firstly, let us make a point of the fact that we have none of plans
to get rich at the expense of SciFace GmbH ;) Our absolute stress
is on efficient considering of mutually beneficial cooperation, to
the full delight of the MuPAD's end consumers.
Secondly, at this point, let us assure you that our company would
collect far much more positive publicity & customer's appreciation,
and in the very near future, pure profit, via public demonstration
of how powerful our unique diagnostic approach is, on the example
of MuPAD.
However, we remember and much appreciate our exciting, extremely
pleasant experience we got 3 years ago while working, shoulder to
shoulder, with the MuPAD Team on MuPAD bugs within the framework
of the MuPAD Beta Contest. This is why, our absolute preference
is to propose SciFace to consider a full streamlined solution
directed at MuPAD smoother and quicker development and quality
management. We need about 3 hours to produce the demonstration.
1. In the process of demonstration, a Cyber Tester's
representative in quality of a gift, as a token of
our respect to the SciFace's Team will present -
for free - 500 distinct bug manifestations.
2. We would be happy to present you a set of carefully
designed ideas about making the whole SciFace's
development process streamlined: less expensive,
more stable and quick.
3. We developed a set of new, novel, unique marketing
ideas for SciFace.
4. Last but not least. We have 2 more topics to discuss
face to face and to find a compromise to our mutual
pleasance.
It is up to SciFace, to accept or reject our proposals, but it
is natural, we expect that before presenting our unique ideas
and data, a typical non-disclosure agreement will be signed in
Paderborn, on the spot.
We are entirely convinced that, given our prior 2002 achievements
demonstrated, which are actually only a small fraction of our
today's powers, you would consider most seriously our attitude
and ultimate determination of providing the MuPAD customers with
a new level of quality which would lead to increase SciFace's
competitiveness level, and a better market position of MuPAD.
Yes, now the summer has set in, a great time to relax, to have a
peaceful vacation, to dive & swim & enjoy the weather...
However, to call a spade a spade, as we, facts at our fingertips,
are feeling the situation, any further delay about the case could
be only beneficial not for MuPAD's consumers and not for SciFace's
owners and workers, but for the SciFace's competitors.
So we were wondering what are you thinking about appointing, in the
most near future, a 1 working day long meeting where we present you
the proof of our concept as it it described above in the items 1-4?
We are hoping to hear from you soon.
Best wishes,
Vladimir Bondarenko, CEO
Alexander Chudinovskikh, Commercial Director
Cyber Tester, LLC
P.S.
A remark about the era of manual MuPAD testing, the era that
has ended with the advent of our LIFT & CYCLE technologies.
The MuPAD bug database has been created by 227 persons. Here
are the main contributors.
------------------------------------------------------------
Author TOTAL INVALID DUPLICATE NOISE
------------------------------------------------------------
Vladimir Bondarenko 579 43 33 13.1%
v...@cybertester.com
v...@mail.strace.net
Juergen Gerhard 155 3 1 2.6%
jnge...@mupad.de
Stefan Wehmeier 94 1 0 1.0%
ste...@mupad.de
Francois Maltey 87 4 10 16.1%
francois.maltey@
enst-bretagne.fr
N. M. Thiery
nth...@users.sf.net 44 3 1 9.1%
Christopher Creutzig 40 1 1 5.0%
c...@mupad.de
chris...@creutzig.de
bug...@creutzig.de
K. Drescher 27 1 2 11.1%
dres...@sciface.com
Ralf Hillebrand 23 0 0 0%
ton...@mupad.de
d-r.s...@t-online.de 21 4 2 28.6%
Florent.Hivert@ 20 2 1 15.5%
univ-mlv.fr
cor...@mupad.de 15 0 0 0%
c.mcguinness@ 15 1 0 6.7%
oceanfree.net /colm.
mcgui...@eircom.net
graebe@informatik. 14 0 1 7.1%
uni-leipzig.de
steve.swanson@ 13 1 0 7.7%
mackichan.com
Bern...@hit.no 13 4 0 30.8%
guenter.braunss@ 12 1 0 8.3%
math.uni-giessen.de
Walter Oevel 12 2 0 16.7%
wal...@mupad.de
A. Sorgatz 11 0 0 0%
sor...@sciface.com
beg...@upb.de 9 4 0 44.4%
Thomas Metzner 4 0 0 0%
t...@mupad.de
b...@mupad.de 6 1 0 16.7%
maje...@emirates.net.ae 6 2 0 33.3%
maavl@mathlabo. 5 1 1 40.0%
univ-poitiers.fr
st...@iipl.snu.ac.kr 3 1 1 66.7%
daniel...@free.fr 2 0 0 0%
Francois. 2 1 0 50.0%
Thom...@inria.fr
P.P.S.
Several months ago, during about a 200 hours long run, the VM
machine also identified many hundreds of distinct bugs in the
famous AXIOM system, and we revealed about 1% only of these
bugs publicly (we do have solid reasons to act in such a way :)
and for that I has been added to a list of contributors of this
most magnificent and unbelievably buggy system.
List of Contributors
The following is a fairly complete list of people who have
contributed to the development of Axiom over it's long history.
http://page.axiom-developer.org/zope/mathaction/AboutAxiom
)credits //<- An AXIOM command to show the contributors list
An alphabetical listing of contributors to AXIOM (to August, 2004):
Cyril Alberga Roy Adler Richard Anderson
George Andrews
Henry Baker Stephen Balzac Yurij Baransky
David R. Barton Gerald Baumgartner Gilbert Baumslag
Fred Blair Vladimir Bondarenko Mark Botch
;)
Alexandre Bouyer Peter A. Broadbery Martin Brock
Manuel Bronstein Florian Bundschuh William Burge
Robert Caviness Bruce Char Cheekai Chin
David V. Chudnovsky Gregory V. Chudnovsky Josh Cohen
Don Coppersmith George Corliss Robert Corless
Gary Cornell Meino Cramer
Timothy Daly Sr. Timothy Daly Jr. James H. Davenport
Jean Della Dora Gabriel Dos Reis Michael Dewar
Claire DiCrescendo Sam Dooley Lionel Ducos
Martin Dunstan Brian Dupee Dominique Duval
Robert Edwards Lars Erickson
Richard Fateman Bertfried Fauser Stuart Feldman
Brian Ford Albrecht Fortenbacher George Frances
Constantine Frangos Timothy Freeman Korrinn Fu
Marc Gaetano Rudiger Gebauer Kathy Gerber
Patricia Gianni Holger Gollan Teresa Gomez-Diaz
Laureano Gonzalez-Vega Stephen Gortler Johannes Grabmeier
Matt Grayson James Griesmer Vladimir Grinberg
Oswald Gschnitzer
Steve Hague Vilya Harvey Satoshi Hamaguchi
Martin Hassner Henderson Antoine Hersen
Pietro Iglio
Richard Jenks
Grant Keady Tony Kennedy Klaus Kusche
Bernhard Kutzler
Larry Lambe Frederic Lehobey Michel Levaud
Rudiger Loos Michael Lucks Richard Luczak
Camm Maguire Bob McElrath Michael McGettrick
Ian Meikle David Mentre Victor S. Miller
Gerard Milmeister Mohammed Mobarak H. Michael Moeller
Michael Monagan Marc Moreno-Maza Scott Morrison
Mark Murray
William Naylor C. Andrew Neff John Nelder
Godfrey Nolan Arthur Norman Jinzhong Niu
Michael O'Connor Kostas Oikonomou
Julian A. Padget Bill Page Jaap Weel
Susan Pelzel Michel Petitot Didier Pinchon
Claude Quitte
Norman Ramsey Michael Richardson Renaud Rioboo
Jean Rivlin Nicolas Robidoux Simon Robinson
Michael Rothstein Martin Rubey
Philip Santas Alfred Scheerhorn William Schelter
Gerhard Schneider Martin Schoenert Marshall Schor
Fritz Schwarz Nick Simicich William Sit
Elena Smirnova Jonathan Steinbach C Y Student
Christine Sundaresan Robert Sutor Moss E. Sweedler
Eugene Surowitz
James Thatcher Baldir Thomas Mike Thomas
Dylan Thurston Barry Trager Themos T. Tsikas
Gregory Vanuxem
Bernhard Wall Stephen Watt Juergen Weiss
M. Weller Mark Wegman James Wen
Thorsten Werther Michael Wester John M. Wiley
Berhard Will Clifton J. Williamson Stephen Wilson
Shmuel Winograd Robert Wisbauer Waldemar Wiwianka
Knut Wolf
David Yun
Richard Zippel Bruno Zuercher Dan Zwillinger
I am very proud and happy to be placed near such cachalots like
Manuel Bronstein, Gregory V. Chudnovsky, James H. Davenport,
Richard Fateman, and Michael Wester!
P.P.P.S.
During the last months, we post to the Google Groops variegated math
tasks difficult for modern computer algebra systems.
The challenges are a tiny fraction of what we hope could become the
definitive, world's largest benchmark test for new powerful computer
algebra systems to be created within the next 20-30 years.
In our opinion, by several reasons, such a benchmark should consist
of *two* components, a static and a variable ones, - and in a year
or so we plan to tell publicly much more about this stuff.
P.P.P.P.S.
We already have calculated enough stuff to be demonstrated publicly
step by step, to claim safely that, within the next 12 months only,
our technologies will have telling impact on the whole CAS market.
Here is beta 0.1 of the first world's document written by a human
in a close cooperation with an automated testing expert system
Man+Machine Review Of Maple Crisis: 1993-2004. Part I
http://maple.bug-list.org/MapleCrisis-Review-01.pdf [16 Mb]
and here you can find a comment about the Review
http://groups-beta.google.com/group/sci.math.symbolic/msg/5cdbf429b23b905d?hl=en
Seldom, if ever, have I encountered a document like Vladimir's
"Man+Machine Review of Maple Crisis: 1993-2004. Part I" before.
[...]
The final version we keep working on includes much new and novel
stuff, in particular, new notions, diagrams, tables, comparisons,
samples, examples, and is planned to be of about 3,500 pages long.