bin/test --slow sympy/integrals/tests/test_transforms.py hangs

[Joachim Durchholz]

Title describes the situation.
The problem was introduced in d491d411118f80840f9e0bd38ed0d7d2dfaba4bb

For me, this caused over an hour of
- a wild goose hunt for a nonexistent bug in what I was working on
- bisecting while not being sure whether it's just slow or hanging
Not happy because I spent over an hour for avoidable work, and because
now I can't use slow tests at all. (Well, I'm grumpy anyway because I
can't sleep and already know I'll pay dearly tomorrow.)

Now how to move forward here...

A) Can we change the development process so that nothing gets pulled
unless the slow tests pass, too?
B) Tom, can you disable that test, or fix it up? I'd also love it if you
could fix or disable the failing tests; having to check manually whether
that E pattern is just the known errors or something new isn't the way
unit testing works best.

Regards,
Jo

[Tom Bachmann]

Hm. Actually we knew when pushing that said test seems to never finish.
I will submit a pull request that turns @slow into @skip.

Sorry for the inconvenience.

[Joachim Durchholz]

Am 23.01.2012 09:14, schrieb Tom Bachmann:
> Hm. Actually we knew when pushing that said test seems to never finish.

I once let it run for two hours, and it was consuming 100% of a CPU core
and didn't stop, so I'm pretty sure it's an endless loop.

I did some analysis but didn't retain the workdir (on the mistaken
assumption you already knew what's going on), so I'm recalling this from
memory:

- it's happening after the tenth (or so) call of doit (in mellin, I
think, but could be meijerint)
- the call fails, popping up through several levels as an exception,
then several more levels in the form of functions returning a None result.

Hope this will help you a bit in your own analysis.

> I will submit a pull request that turns @slow into @skip.

Thanks :-)

How about attaching a @skip to the known-to-fail tests as well?

Regards,
Jo

[Aaron Meurer]

Hmm. From what I remember, it was not hanging in the new algorithm,
but in heurisch(). This means that it probably would indeed finish if
you let it run long enough, but that algorithm can be so inefficient
that there's no telling if it could be hours, days, weeks, ...

If it is indeed hanging in the new algorithm, this should be investigated.

That's what XFAIL is for. Or did I misunderstand you?

Aaron Meurer

>
>
> Regards,
> Jo
>
> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To post to this group, send email to sy...@googlegroups.com.
> To unsubscribe from this group, send email to
> sympy+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/sympy?hl=en.
>

[Tom Bachmann]

No it's hanging in heurisch.

(And in fact smichr already merged my fix - i.e. turning the test into
skip.)

[Aaron Meurer]

Exactly. It takes about 20 min. to get to the part of heurisch that
hangs on my computer, but that's where it eventually stays. If you
apply the following patch:

diff --git a/sympy/integrals/risch.py b/sympy/integrals/risch.py
index bb5f6c9..ed30e71 100644
--- a/sympy/integrals/risch.py
+++ b/sympy/integrals/risch.py
@@ -426,6 +426,7 @@ def _integrate(field=None):
coeff, dependent = term.as_independent(*V)
equations[dependent] += coeff

+ print len(equations), len(coeffs)
solution = solve(equations.values(), *coeffs)

if solution is not None:
diff --git a/sympy/integrals/tests/test_transforms.py
b/sympy/integrals/tests/test_transforms.py
index 7cf23cf..8ca02b8 100644
--- a/sympy/integrals/tests/test_transforms.py
+++ b/sympy/integrals/tests/test_transforms.py
@@ -45,7 +45,7 @@ def test_as_integral():
@slow
@XFAIL
def test_mellin_transform_fail():
- skip("Risch takes forever.")
+ #skip("Risch takes forever.")

from sympy import Max, Min
MT = mellin_transform

and run ./bin/test --slow sympy/integrals/tests/test_transforms.py,
after about 20 min. it will print 10120 465. This means that the
heuristic Risch algorithm is trying to solve a system of 10120
equations with 465 unknowns. Our inefficient rref routine simply
can't handle this. As a comparison, the similarly hanging integral
from http://code.google.com/p/sympy/issues/detail?id=1441 tries to
solve ~600 equations in ~450 variables. And if that weren't enough,
the entire algorithm is actually called twice (line 277), and the
second time generally results in an even larger system.

Regarding the first 20 minutes, if you put a print statement at the
top of heurisch(), it executes after about 15 seconds. So I don't
think the Meijer G code is too slow. The other slow part of heurisch
is the call to expand at line 422.

Aaron Meurer