Inconsistent test results

[Joachim Durchholz]

Hi all,

I'm getting assertion errors when running the full test suite, or when
testing just integrals, but no errors if I run the tests in isolation.
The errors are scattered across the test suite. It seems it's always the
same set of errors, all related to str() returning something different
than expected. Hash seed variation does not affect the outcome.

What's the best approach to pinpoint the source of the problem?

With --enhance-asserts I'm getting this:

sympy/external/tests/test_autowrap.py[10] sfsffffsss

[OK]
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "sympy/utilities/runtests.py", line 487, in _test
slow=slow, enhance_asserts=enhance_asserts))
File "sympy/utilities/runtests.py", line 885, in test
self.test_file(f, sort, timeout, slow, enhance_asserts)
File "sympy/utilities/runtests.py", line 939, in test_file
source = self._enhance_asserts(source)
File "sympy/utilities/runtests.py", line 919, in _enhance_asserts
tree = parse(source)
File "/usr/lib/python2.7/ast.py", line 37, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "<unknown>", line 214
print("TEST NOT REMOVED: %s" % work, file=sys.stderr)
^
SyntaxError: invalid syntax

[Julien Rioux]

Hi,

Looks like a code path that wasn't tested. Try adding
from __future__ import print_function
as the first executable statement at the top of the file sympy/external/tests/test_autowrap.py, does it help?

Cheers,
Julien

[Joachim Durchholz]

Am 05.08.2013 23:29, schrieb Julien Rioux:
>> Traceback (most recent call last):
>> File "<string>", line 1, in <module>
>> File "sympy/utilities/runtests.py", line 487, in _test
>> slow=slow, enhance_asserts=enhance_asserts))
>> File "sympy/utilities/runtests.py", line 885, in test
>> self.test_file(f, sort, timeout, slow, enhance_asserts)
>> File "sympy/utilities/runtests.py", line 939, in test_file
>> source = self._enhance_asserts(source)
>> File "sympy/utilities/runtests.py", line 919, in _enhance_asserts
>> tree = parse(source)
>> File "/usr/lib/python2.7/ast.py", line 37, in parse
>> return compile(source, filename, mode, PyCF_ONLY_AST)
>> File "<unknown>", line 214
>> print("TEST NOT REMOVED: %s" % work, file=sys.stderr)
>> ^
>> SyntaxError: invalid syntax
>
> Looks like a code path that wasn't tested. Try adding
> from __future__ import print_function
> as the first executable statement at the top of the file
> sympy/external/tests/test_autowrap.py, does it help?

Heh. Turned out the error happens before the test file name is printed,
so the error looks as if it happened one file early unless you look very
closely.
The file missing that import was sympy/external/tests/test_codegen.py then.

Pull request for that fix and for proper file name reporting in case of
early problems forthcoming.

[Joachim Durchholz]

I'm finally getting somewhere: I found an actual bug, not just
inconsistent behaviour :-)

What happens is this:

Some tests in galgebra look like this:

def test_whatever
GA_Printer.on()
... some lengthy test code here ...
GA_Printer.off()

Now if the timeout interrupts somewhere between GA_Printer.on() and
GA_Printer.off(), the GA_Printer.off() call will not happen.

GA_Printer has this to say:

@staticmethod
def on():
GA_Printer.Basic__str__ = Basic.__str__
Basic.__str__ = lambda self: GA_Printer().doprint(self)
return

@staticmethod
def off():
Basic.__str__ = GA_Printer.Basic__str__
return

So if there's a timeout and off() isn't called, Basic.__str__ will still
be the lambda.
The next call to on() will even overwrite GA_Printer.Basic__str__ with
the lambda.

Since timeouts are done by triggering a Skipped exception, I guess the
right way to handle this would be to turn GA_Printer into a context manager.
1) What are the ramifications?
2) Are there other things that need to be turned into a context manager?

[Aaron Meurer]

There are none that I can think of. Probably on and off should be made
private and the context managers made to be the only way to do things.
Or maybe even just removed. The whole thing can just be

from contextlib import contextmanager

@contextmanager
def GA_Printing():

GA_Printer.Basic__str__ = Basic.__str__
Basic.__str__ = lambda self: GA_Printer().doprint(self)

yield
Basic.__str__ = GA_Printer.Basic__str__

(modulo any typos I made there of course)

The only issue with removing them is that it gets to be more annoying
to use it interactively (at least until the IPython devs come to their
senses with https://github.com/ipython/ipython/issues/3191).

> 2) Are there other things that need to be turned into a context manager?

Probably a lot of stuff. Anything that sets something and then unsets
it when it's done. It's hard to find by grepping.

Also, many try: finally blocks would be easier to read/use/reuse as
context managers.

Aaron Meurer

>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sympy.
> For more options, visit https://groups.google.com/groups/opt_out.

[Joachim Durchholz]

Am 15.08.2013 01:10, schrieb Aaron Meurer:
>> Since timeouts are done by triggering a Skipped exception, I guess the right
>> way to handle this would be to turn GA_Printer into a context manager.
>> 1) What are the ramifications?
>
> There are none that I can think of. Probably on and off should be made
> private and the context managers made to be the only way to do things.
> Or maybe even just removed. The whole thing can just be
>
> from contextlib import contextmanager
>
> @contextmanager
> def GA_Printing():
> GA_Printer.Basic__str__ = Basic.__str__
> Basic.__str__ = lambda self: GA_Printer().doprint(self)
> yield
> Basic.__str__ = GA_Printer.Basic__str__
>
> (modulo any typos I made there of course)
>
> The only issue with removing them is that it gets to be more annoying
> to use it interactively (at least until the IPython devs come to their
> senses with https://github.com/ipython/ipython/issues/3191).

I made a fix that keeps on() and off() as deprecated functions, and does
the actual work in new _on() and _off() functions.
The tests run fine with that :-)
I did that via __enter__ and __exit; the @contextmanager approach looks
cleaner, I'll integrate that and submit a pull request.

I'm wondering whether we should postpone deprecating on() and off()
until after #3191 is finished.
(I can see both side of the ongoing argument actually. I'd go for
enter_context and exit_context in Ipython, plus a diagnostic function to
list all active contexts. It's not ideal, but I don't think the ideal
thing is possible there.)

>> 2) Are there other things that need to be turned into a context manager?
>
> Probably a lot of stuff. Anything that sets something and then unsets
> it when it's done. It's hard to find by grepping.

Darn.
Something of that is probably already in the code that collects test
results for the final test summary. With timeout=1, I'm getting
mutilated or no output.
Somebody with more insight into the test runner code should comment on
that. I have no idea where to look first. (The test runner code could
use a good deal of refactoring and rewriting. It's a bit too much for
me, unfortunately.)

> Also, many try: finally blocks would be easier to read/use/reuse as
> context managers.

Worth doing.
I'll leave that to a GSoC student though :-)