Charlwood Fifty test results

[Albert Rich]

Following is a table comparing the results produced by 6 symbolic integrators on the Charlwood Fifty test-suite:

Charlwood Fifty Test Results
# 1 2 3 4 5 6 7
1 0 2 2 2 2 2 2
2 2 2 2 2 2 2 2
3 0 0 2 1 0 2 -1
4 1 1 1 1 1 1 0
5 1 1 0 1 0 2 0
6 2 2 2 0 1 2 0
7 2 2 2 1 2 2 2
8 1 1 0 0 0 1 0
9 2 2 2 1 2 0 2
10 2 2 2 1 0 2 0
11 0 2 2 2 2 0 0
12 1 1 2 1 0 1 0
13 1 1 2 1 0 0 0
14 0 2 1 2 2 0 0
15 2 2 2 1 2 0 0
16 0 2 1 2 2 0 2
17 2 2 1 1 2 0 0
18 2 2 2 2 2 1 2
19 2 2 2 2 2 2 2
20 2 2 2 2 2 0 2
21 0 2 1 1 0 0 0
22 0 1 2 1 1 0 0
23 0 2 2 2 2 0 0
24 2 2 2 2 2 2 2
25 2 2 1 2 2 0 2
26 2 2 1 2 2 0 0
27 2 2 1 2 2 2 2
28 2 2 1 2 2 0 2
29 2 2 2 2 2 0 0
30 2 2 2 2 2 0 0
31 2 2 1 2 2 0 2
32 2 2 2 2 2 2 2
33 2 2 1 2 2 1 2
34 2 2 2 2 2 2 2
35 0 2 2 2 2 0 0
36 2 2 1 2 2 2 2
37 2 2 1 1 2 2 2
38 1 2 2 1 0 2 0
39 1 2 2 1 0 2 0
40 2 2 1 2 0 2 1
41 0 0 1 1 0 0 0
42 1 1 1 1 1 1 0
43 2 2 1 1 2 2 1
44 1 1 1 0 2 1 0
45 0 0 1 1 0 0 0
46 0 2 2 2 2 1 2
47 1 1 1 1 1 1 0
48 1 2 2 1 2 2 1
49 2 2 1 2 2 0 0
50 2 2 2 2 2 2 0
Tot 65 85 75 73 71 49 40

In summary, the column labels and total scores for the 7 systems tested are as follows:

1. 66 Rubi 4.0
2. 85 Rubi 4.1
3. 75 FriCAS 1.2.1
4. 73 Mathematica 9
5. 71 Derive 6.1
6. 49 Maple 17
7. 40 Maxima 5.28.0-2

The table in slightly more readable format is available as a pdf file at

http://www.apmaths.uwo.ca/~arich/CharlwoodFiftyTestResults.pdf

The 50 integration problems and optimal antiderivatives making up the test-suite are available at

http://www.apmaths.uwo.ca/~arich/CharlwoodIntegrationProblems.pdf

The table uses a numeric grading system to compare the result produced by the built-in integrate function provided by each system with the optimal antiderivative as shown in the above file. Simplification of results using functions and/or domain restrictions on variables is not allowed. In other words, the results a novice user would see are graded as is.

For each problem, if the system gives a result as good as the optimal antiderivative, 2 points are earned. If the system gives a mathematically correct result (i.e. its derivative equals the integrand) but is substantially more complex than the optimal antiderivative (e.g. significantly larger or involves the imaginary unit), 1 point is earned. If the system is unable to integrate the expression in closed-form or times-out after 30 seconds or so, no points are earned. If the system returns a mathematically incorrect result, 1 point is decremented. So a perfect score on the Charlwood Fifty test-suite is 100.

I would be happy to include other systems in the table if a knowledgeable user sends me its version number and numeric grades and/or raw test results for each of the 50 problems using the scoring system described above. Also if you think I have unfairly graded a system, please provide evidence and I will revise the table accordingly.

Of course, the improvement of Rubi from a score of 65 to 85 was due to version 4.1 having been written in light of the Charlwood Fifty test-suite. Hopefully the other systems will quickly improve their scores as well...

Albert

[clicl...@freenet.de]

Albert Rich schrieb:

Nice. Only for Charlwood's problem 44 does Rubi 4.1 (complex domain)
fall behind Derive 6.10 (real domain). I am not surprised that problems
3 and 45 are not handled by Rubi. But why is problem 41 untractable?

Martin.

[Albert Rich]

On Friday, July 5, 2013 11:15:41 AM UTC-10, clicl...@freenet.de wrote:

> Nice. Only for Charlwood's problem 44 does Rubi 4.1 (complex domain)
> fall behind Derive 6.10 (real domain). I am not surprised that problems
> 3 and 45 are not handled by Rubi. But why is problem 41 untractable?
>
> Martin.

Thanks for your compliment of the Charlwood Fifty test results I posted. Problem #41 is

integrate(log(sin(x)) sqrt(1+sin(x)), x)

Rubi starts correctly (I think) using integration by parts that gives the log free integral

integrate(cos(x) cot(x)/sqrt(1+sin(x)), x)

Unfortunately Rubi and the other systems tested cannot determine that its antiderivative is

2 cos(x)/sqrt(1+sin(x)) – 2 arctanh(cos(x)/sqrt(1+sin(x)))

Obviously since its antiderivative is a function of cos(x)/sqrt(1+sin(x)), the log free integral above can be integrated by making the substitution u=cos(x)/sqrt(1+sin(x)) yielding the trivial integral

integrate(2-2/(1-u^2), x)

Any ideas how a rule-based system like Rubi should “know” to make that substitution?

Albert

[clicl...@freenet.de]

Albert Rich schrieb:

>
> On Friday, July 5, 2013 11:15:41 AM UTC-10, clicl...@freenet.de wrote:
>

> > [...] But why is problem 41 untractable?
>
> [...] Problem #41 is

>
> integrate(log(sin(x)) sqrt(1+sin(x)), x)
>
> Rubi starts correctly (I think) using integration by parts that gives
> the log free integral
>
> integrate(cos(x) cot(x)/sqrt(1+sin(x)), x)
>
> Unfortunately Rubi and the other systems tested cannot determine that
> its antiderivative is
>
> 2 cos(x)/sqrt(1+sin(x)) – 2 arctanh(cos(x)/sqrt(1+sin(x)))
>

> [...]

The integrand COT(x)*COS(x)/SQRT(1 + SIN(x)) equals (1 - SIN(x)^2)
/(SIN(x)*SQRT(1 + SIN(x))) which equals (1 - SIN(x))*SQRT(1 + SIN(x))
/SIN(x). That is, the integrand obtained after integration by parts is a
function of SIN(x) only, and of a type that Rubi should be able to
handle.

Martin.

[Nasser M. Abbasi]

Just trying things....

Starting with your final expression above

I1 = (1 - SIN(x))*SQRT(1 + SIN(x)) /SIN(x)

Rub 4.1 uses rule 2072 to integrate it, whichhas quit few
transformations in it.

When doing substitution y=sin(x) in the above first, and obtaining new
integrand as function of y only

I2 = (1 - y) Sqrt[1 + y]/(y Sqrt[1 - y^2])

and integrating this instead of I1, Rubi used rule 17, which seems to be
much simpler.

Both give the same result (after letting y->sin(x) for I2 case),
which is

-2 ArcTanh[Sqrt[1 - Sin[x]]] + 2 Sqrt[1 - Sin[x]]


--Nasser

[clicl...@freenet.de]

Albert Rich schrieb:

>
> Following is a table comparing the results produced by 6 symbolic
> integrators on the Charlwood Fifty test-suite:
>
> Charlwood Fifty Test Results
> # 1 2 3 4 5 6 7
> 1 0 2 2 2 2 2 2
> 2 2 2 2 2 2 2 2
> 3 0 0 2 1 0 2 -1
> 4 1 1 1 1 1 1 0
> 5 1 1 0 1 0 2 0
> 6 2 2 2 0 1 2 0
> 7 2 2 2 1 2 2 2
> 8 1 1 0 0 0 1 0
> 9 2 2 2 1 2 0 2
> 10 2 2 2 1 0 2 0

> [...]

>
> In summary, the column labels and total scores for the 7 systems
> tested are as follows:
>
> 1. 66 Rubi 4.0
> 2. 85 Rubi 4.1
> 3. 75 FriCAS 1.2.1
> 4. 73 Mathematica 9
> 5. 71 Derive 6.1
> 6. 49 Maple 17
> 7. 40 Maxima 5.28.0-2
>

Your results for Mathematica 9.01 (column 4) appear to be incompatible
with Nasser's results for problems 1 to 10 at

<http://www.12000.org/my_notes/ten_hard_integrals/index.htm>

According to Nasser, Mathematica fails entirely on problem 5, and
succeeds on problems 6,7,8,9 only in terms of non-elementary functions
(elliptic integrals). According to your table, Mathematica succeeds
suboptimally on problems 5,7,9 and fails on problems 6,8.

Similarly, Nasser reports Maple 17 to fail on problems 9,10, whereas you
report (column 6) a failure for problem 9 and a full success for problem
10.

Martin.

[clicl...@freenet.de]

"Nasser M. Abbasi" schrieb:

>
> Just trying things....
>
> Starting with your final expression above
>
> I1 = (1 - SIN(x))*SQRT(1 + SIN(x)) /SIN(x)
>
> Rub 4.1 uses rule 2072 to integrate it, whichhas quit few
> transformations in it.
>
> When doing substitution y=sin(x) in the above first, and obtaining new
> integrand as function of y only
>
> I2 = (1 - y) Sqrt[1 + y]/(y Sqrt[1 - y^2])
>
> and integrating this instead of I1, Rubi used rule 17, which seems to
> be much simpler.
>
> Both give the same result (after letting y->sin(x) for I2 case),
> which is
>
> -2 ArcTanh[Sqrt[1 - Sin[x]]] + 2 Sqrt[1 - Sin[x]]
>

If Rubi evaluates the integral

INT((1 - SIN(x))*SQRT(1 + SIN(x))/SIN(x), x)

to

2*SQRT(1 - SIN(x)) - 2*ATANH(SQRT(1 - SIN(x)))

then you have found a bug, since the derivative

DIF(2*SQRT(1 - SIN(x)) - 2*ATANH(SQRT(1 - SIN(x))), x)

equals

COS(x)/SQRT(COS(x)^2)*(1 - SIN(x))*SQRT(1 + SIN(x))/SIN(x)

where the piecewise constant COS(x)/SQRT(COS(x)^2) differs from unity.

Accordingly, a correct evaluation of the integral is

2*COS(x)/SQRT(1 + SIN(x)) - 2*ATANH(COS(x)/SQRT(1 + SIN(x)))

Martin.

[Nasser M. Abbasi]

On 7/6/2013 11:42 AM, clicl...@freenet.de wrote:

>
> If Rubi evaluates the integral
>
> INT((1 - SIN(x))*SQRT(1 + SIN(x))/SIN(x), x)
>
> to
>
> 2*SQRT(1 - SIN(x)) - 2*ATANH(SQRT(1 - SIN(x)))
>
> then you have found a bug, since the derivative
>
> DIF(2*SQRT(1 - SIN(x)) - 2*ATANH(SQRT(1 - SIN(x))), x)
>
> equals
>
> COS(x)/SQRT(COS(x)^2)*(1 - SIN(x))*SQRT(1 + SIN(x))/SIN(x)
>
> where the piecewise constant COS(x)/SQRT(COS(x)^2) differs from unity.
>
> Accordingly, a correct evaluation of the integral is
>
> 2*COS(x)/SQRT(1 + SIN(x)) - 2*ATANH(COS(x)/SQRT(1 + SIN(x)))
>

Ok, please let me backup. Rubi does generate the result you showed

2* T1 - 2*ATANH( T1 )

Where

T1 = COS(x)/SQRT(1 + SIN(x))

But I took it abone myself to just replace T1 above with

T2 = SQRT(1 - SIN(x))

since they seemed the same to me(*), so I wrote in this post
that Rubi gave

2* T2 - 2*ATANH( T2 )

Which you complained about. I see now that T1=T2 only from -Pi/2..Pi/2.

Sorry about this confusion. I need to review this whole process again
to see why it did not work for all x range using this.
---
(*) In T1, let COS(x) = SQRT(1-SIN(X)^2), then

T1 = SQRT( (1-SIN(X)^2) / (1 + SIN(x)) )

Let
SIN(X) = Y
then it is the same as
T1 = SQRT ( (1-Y^2)/(1+Y) )
which is
T1 = SQRT ( 1-Y )
But Y = SIN(X), hence
T1 = SQRT ( 1- SIN(X) )
which is T2

So this is what happened.

--Nasser

[clicl...@freenet.de]

"Nasser M. Abbasi" schrieb:

COS(x) equals SQRT(COS(x)^2) only where -pi/2 < ARG(COS(x)) <= pi/2;
elsewhere one has COS(x) = -SQRT(COS(x)^2). Your replacement of COS(x)
by SQRT(1-SIN(X)^2) = SQRT(COS(x)^2) therfore leads to corresponding
sign flips in the T2-based result.

Since Rubi manages to return the proper T1-based result already, it
shouldn't be hard to make it handle Charlwood's problem 41. It merely
needs to be taught to normalize integrands like COT(x)*COS(x)/
SQRT(1 + SIN(x)) to their sine-only form. This would be just one
normalization among others performed regularly, and would only require
some twisting or minor extension of existing code, I guess.

Martin.

[Albert Rich]

On Friday, July 5, 2013 11:55:54 PM UTC-10, clicl...@freenet.de wrote:

> Your results for Mathematica 9.01 (column 4) appear to be incompatible
> with Nasser's results for problems 1 to 10 at
>
> <http://www.12000.org/my_notes/ten_hard_integrals/index.htm>
>
> According to Nasser, Mathematica fails entirely on problem 5, and
> succeeds on problems 6,7,8,9 only in terms of non-elementary functions
> (elliptic integrals). According to your table, Mathematica succeeds
> suboptimally on problems 5,7,9 and fails on problems 6,8.
>
> Similarly, Nasser reports Maple 17 to fail on problems 9,10, whereas you
> report (column 6) a failure for problem 9 and a full success for problem
> 10.

After having redone the problems in question, I stand by all the grades shown in the Charlwood Fifty test-results table.

On problems 5,7,9 Mathematica returns a mathematically correct antiderivative expressed in terms of elliptic integrals, so they receive the nonoptimal grade of 1. On problems 6,8, Mathematica times-out after 30 seconds on my computer, so they receive a grade of 0, as per the rules given. However if you wait long enough, Mathematica does return a huge, multipage result involving elliptic integrals and the imaginary unit for problems 6,8.

Nasser and I agree that Maple failed to integrate problem 9. On problem 10, I entered the integrand as

x^3*exp(1)^arcsin(x)/sqrt(1-x^2)

whereas he probably entered it as

x^3*exp(arcsin(x))/sqrt(1-x^2)

Because of some bazaar quirk in Maple, it succeeds in integrating the former and not the latter! Perhaps some Maple aficionado can justify, or at least explain, this phenomena...

Albert

[Albert Rich]

On Saturday, July 6, 2013 10:28:30 AM UTC-10, clicl...@freenet.de wrote:

> Since Rubi manages to return the proper T1-based result already, it
> shouldn't be hard to make it handle Charlwood's problem 41. It merely
> needs to be taught to normalize integrands like COT(x)*COS(x)/
> SQRT(1 + SIN(x)) to their sine-only form. This would be just one
> normalization among others performed regularly, and would only require
> some twisting or minor extension of existing code, I guess.

Rather than blindly normalizing such integrands to their sine-only form and hoping for the best, it is safer to drive the degree of the cosine factor toward zero using the following hot new rule: If a^2-b^2=0 and m>0 is even, then

Int[Cos[c+d*x]^m*Sin[c+d*x]^n*(a+b*Sin[c+d*x])^p, x]

goes to

1/a*Int[Cos[c+d*x]^(m-2)*Sin[c+d*x]^n*(a+b*Sin[c+d*x])^(p+1), x] -
1/b*Int[Cos[c+d*x]^(m-2)*Sin[c+d*x]^(n+1)*(a+b*Sin[c+d*x])^(p+1), x]

This rule is based on the trig identity

Cos[z]^2 = (a+b*Sin[z])/a - Sin[z]*(a+b*Sin[z])/b

which is valid if a^2-b^2=0. The new rule will make it possible for the next version of Rubi to integrate a whole new class of expressions including the integrand in question, as well as the much more difficult one

Cos[x]^4/(Sin[x]^3*(a+a*Sin[x])^(3/2))

Albert

[Nasser M. Abbasi]

On 7/6/2013 3:32 PM, Albert Rich wrote:

> Nasser and I agree that Maple failed to integrate problem 9. On problem 10, I entered the integrand as
>
> x^3*exp(1)^arcsin(x)/sqrt(1-x^2)
>
> whereas he probably entered it as
>
> x^3*exp(arcsin(x))/sqrt(1-x^2)
>
> Because of some bazaar quirk in Maple, it succeeds in integrating the former and not the latter!
>Perhaps some Maple aficionado can justify, or at least explain, this phenomena...
>
> Albert
>

That is interesting. I wonder how you discovered this. It would
never have occurred to me to try that.

I updated the table for the 10 integrals, I suppose it is fair to give
this one to Maple now. I am using now Maple 17.01 (version just came out).
So maple now has 9/10 as well.

I also made a full trace of this problem 10, one with

x^3*exp(1)^arcsin(x)/sqrt(1-x^2)

and another trace with

x^3*exp(arcsin(x))/sqrt(1-x^2)

to try to find why Maple gives an answer for one case and not the other.

Will look at the traces more later. But if someone like to try find out,
the traces are uploaded as well in HTML and PDF format. They are under the
Maple download section:

http://www.12000.org/my_notes/ten_hard_integrals/index.htm

ps. HTML export by Maple are little strange, one needed to scroll
down a bit to see start seeing the actual text. PDF export is better.

pps. I did not use full tracing, as that would have generated huge amount
of data.

ppps. I did not do timing on results, so, yes some Mathematica results
took more than 30 seconds.

--Nasser

[Albert Rich]

On Saturday, July 6, 2013 11:53:22 AM UTC-10, Nasser M. Abbasi wrote:

> On 7/6/2013 3:32 PM, Albert Rich wrote:
>> On problem 10, I entered the integrand as
>>
>> x^3*exp(1)^arcsin(x)/sqrt(1-x^2)
>>
>> whereas he probably entered it as
>>
>> x^3*exp(arcsin(x))/sqrt(1-x^2)
>>
>> Because of some bazaar quirk in Maple, it succeeds in integrating the former and not the latter!
>

> That is interesting. I wonder how you discovered this. It would
> never have occurred to me to try that.

It would never have occurred to me either. But the program I wrote that translates the problems from Mathematica to Maple syntax, translates expressions of the form E^z to exp(1)^z. This is because for some inexplicable reason Maple does not define a constant for the base of the natural logarithms, but does for the imaginary unit and pi. Go figure...

Albert

[clicl...@freenet.de]

Albert Rich schrieb:

So, while there is no real discrepancy between the Mathematica results
for problems 7 and 9, your 30-second timeouts explain the (apparent)
failures on problems 6 and 8. What remains to be explained are the
incompatible Mathematica results for problem 5 where Nasser obtained an
unevaluated integral:

<http://www.12000.org/my_notes/ten_hard_integrals/inse5.htm#x6-50005>

The Maple quirk uncovered by problem 10 looks like a case of missing
normalization again :).

Martin.

[Nasser M. Abbasi]

> So, while there is no real discrepancy between the Mathematica results
> for problems 7 and 9, your 30-second timeouts explain the (apparent)
> failures on problems 6 and 8. What remains to be explained are the
> incompatible Mathematica results for problem 5 where Nasser obtained an
> unevaluated integral:
>
> <http://www.12000.org/my_notes/ten_hard_integrals/inse5.htm#x6-50005>
>

This is really, really strange. I can't explain what happened.

I just run #5 now again on fresh kernel, with M 9.01 and now it gives
a result !

I really do not know why and how this has happened. You saw the output
and that it did not evaluate before.

This means M has 10/10 score now (but I do not do penalties for
non-elementary anti-derivatives and for long computation time, so
Albert's more advanced scoring scheme will give different rankings)

--------------------------------------------------------------
In[41]:= Integrate[Cos[x]^2/Sqrt[Cos[x]^4+Cos[x]^2+1],x]

Out[41]= -(2 I Cos[x]^2 EllipticPi[3/2+(I Sqrt[3])/2,I
ArcSinh[Sqrt[-((2 I)/(-3 I+Sqrt[3]))] Tan[x]],(3 I-Sqrt[3])/
(3 I+Sqrt[3])] Sqrt[1-(2 I Tan[x]^2)/(-3 I+Sqrt[3])]
Sqrt[1+(2 I Tan[x]^2)/(3 I+Sqrt[3])])/(Sqrt[-(I/(-3 I+Sqrt[3]))]
Sqrt[15+8 Cos[2 x]+Cos[4 x]])
-------------------------------------------------------------

Just updated page:

http://www.12000.org/my_notes/ten_hard_integrals/index.htm

Sorry again for this. I wish I know why it did not evaluate before
and thanks for spotting this.

--Nasser

[clicl...@freenet.de]

"Nasser M. Abbasi" schrieb:

>
> On 7/6/2013 3:32 PM, Albert Rich wrote:
>
> > Nasser and I agree that Maple failed to integrate problem 9. On
> > problem 10, I entered the integrand as
> >
> > x^3*exp(1)^arcsin(x)/sqrt(1-x^2)
> >
> > whereas he probably entered it as
> >
> > x^3*exp(arcsin(x))/sqrt(1-x^2)
> >
> > Because of some bazaar quirk in Maple, it succeeds in integrating
> > the former and not the latter!
> > Perhaps some Maple aficionado can justify, or at least explain, this
> > phenomena...
> >
>

> That is interesting. I wonder how you discovered this. It would
> never have occurred to me to try that.
>
> I updated the table for the 10 integrals, I suppose it is fair to give
> this one to Maple now. I am using now Maple 17.01 (version just came
> out). So maple now has 9/10 as well.
>

Your choice is defensible since Charlwood employs true exponentiation
and not the exponential function in his paper. But it assumes special
knowledge of Maple and its quirks. A naive user who knows or discovers
that Euler's constant is not available would typically use EXP(...) and
not EXP(1)^(...), as is borne out by many sci.math.symbolic posts, I
think. Since Albert appeals to the "results a novice user would see" he
should then probably use EXP(...) rather than EXP(1)^(...).

I fear this quirk may bear on Maple's failure to do problem 62 from
Timofeev's Chapter 1 as well ...

Martin.

[Albert Rich]

I just posted Charlwood Fifty integration test-suite results at

http://www.apmaths.uwo.ca/~arich/CharlwoodFiftyTestResults.pdf

that includes a column for the recently released version 4.2 of Rubi. The bottom line shows a modest improvement from 85 to 88 in Rubi's grade. This is due to improvements made in the rules for integrating expressions involving inverse trig functions.

Also listed in the pdf file are the 50 Charlwood integrals and corresponding optimal antiderivatives. Note that the antiderivative for problem #35 is slightly simpler than in previous versions.

Albert