Is there a way to run a specific test with the cache switched off?

[Joachim Durchholz]

I'm currently tutoring PR #9376, which fixes a bug that happens only
when caching is turned off.
We're trying to figure out a way to write a regression test. Our test
suite is run with caching switched on, even on Travis, so we need a way
to make sure that this single test is running with caching switched off.

I gather that the test framework does clear_cache() before each test,
but that's not enough - the test seems to be filling the cache on its
own, and enough to break the infinite recursion that will happen without
caching. So we really need to disable the caching machinery on a
per-test basis.
Does such a feature exist?

[Peter Brady]

Not really, caching is a decision made at import time (there's probably a more precise language for this).  I think there are two options here:

1) Write a context manager that forces calls to f.__wrapped__ rather than f so that the cache is bypassed.  This may not be possible.

2) Have a set of tests run without the cache and add the option to bin/test to select them. This is definitely possible.

[Aaron Meurer]

Can't you do ./bin/test -C <set of tests>?

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sympy.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sympy/cf05bc44-b413-4f27-8e25-3194fa4d71e6%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Peter Brady]

Absolutely.  I was thinking that there was a decently sized set of tests that were going to be run regularly without the cache in which case it might make sense to introduce another mark/option but if it's only a few than specifying them manually is probably better.

You received this message because you are subscribed to a topic in the Google Groups "sympy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sympy/xPAhw28bPDc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sympy+un...@googlegroups.com.

To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6J8gb1ihH4neFhqZjV%2Bx%3DUX%2B%3Dy7W81pExF9a8dJLUyMSw%40mail.gmail.com.

[Joachim Durchholz]

Am 06.05.2015 um 20:58 schrieb Peter Brady:
> Absolutely. I was thinking that there was a decently sized set of tests
> that were going to be run regularly without the cache in which case it
> might make sense to introduce another mark/option but if it's only a few
> than specifying them manually is probably better.

I'd definitely prefer a mark/decorator/whatever on the test itself.
Keeps the information about the test in one place, instead of
distributing it across the test code and the test driver.

We have more conditional testing like that BTW. Slow tests, Travis-only
tests, tests that require numpy or matplotlib, etc. I'm starting to
think that we should unify all these into a single decorator and let the
test driver filter.
Is something like that in Python's test.py? Then we should probably
mimick that.
... right, see pytest.org/latest/example/markers.html .

[Aaron Meurer]

In my experience, you'll want to run the whole test suite without the
cache (I know it's slow). A lot of cache related issues only show up
when the whole test suite is run, because some test will end up
mutating some state that affects another test. It's also very hard to
predict where a cache related failure will pop up.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sympy.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sympy/554A6934.6000107%40durchholz.org.

[Joachim Durchholz]

Am 06.05.2015 um 21:51 schrieb Aaron Meurer:
> In my experience, you'll want to run the whole test suite without the
> cache (I know it's slow).

Hm. Obviously nobody ever did this since 25 Jun 2014.
I.e.
- not ever locally for anybody
- not ever on Travis
- not during testing for two release candidates
- not during testing for a release (0.7.6)

> A lot of cache related issues only show up
> when the whole test suite is run, because some test will end up
> mutating some state that affects another test.

Doesn't the test runner do a clear_cache before each test?
I saw this happen during doctesting, in code that looked like it's run
for both normal and doctesting, so I've been assuming it always happens.

> It's also very hard to
> predict where a cache related failure will pop up.

Sure.
I'm about a regression, i.e. something that's already known to be fragile.
Look at the diff:

@@ -520,7 +520,7 @@ def _check(ct1, ct2, old):
if old == self.base:
return new**self.exp._subs(old, new)

- if old.func is self.func and self.base is old.base:
+ if old.func is self.func and self.base == old.base:
if self.exp.is_Add is False:
ct1 = self.exp.as_independent(Symbol, as_Add=False)
ct2 = old.exp.as_independent(Symbol, as_Add=False)

It's a subtle error that even a Chris Smith can fall prey to, so I think
this line is just fragile and needs to be tested, uncached, on at least
one path before the next release.
Running tests uncached in general on Travis might work (but might be
unworkable).
Requiring an uncached test run on prerelease might work, too. The
disadvantage here is that release blockers might lie undisovered until
right before a release, and the best person to fix the blocker might be
unavailable.
The final option would be to do cache-related regression tests (and
*regression* tests only) without the cache. More infrastructure work
required, but faster catching with little extra overhead (assuming that
tests can be easily trimmed town to minimal examples).

[Peter Brady]

Doesn't the test runner do a clear_cache before each test?

No. Just at the start of each file.

--
You received this message because you are subscribed to a topic in the Google Groups "sympy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sympy/xPAhw28bPDc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sympy+un...@googlegroups.com.

To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/554A7E65.1040400%40durchholz.org.

[Aaron Meurer]

I've done it a few times (although admittedly not in a while), but
there have always been errors that have been "too hard" to fix, even
for a release. See https://github.com/sympy/sympy/issues/5594 (that
issue may be outdated).

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sympy.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sympy/554A7E65.1040400%40durchholz.org.

[Joachim Durchholz]

Am 07.05.2015 um 00:17 schrieb Aaron Meurer:
> I've done it a few times (although admittedly not in a while), but
> there have always been errors that have been "too hard" to fix, even
> for a release.

Ah, I didn't know that.
In the light of this, do we even want to return SymPy to a state that
works without cache?

[Aaron Meurer]

I'm not sure. It is worthwhile to look at these test failures, though,
as they usually indicate bugs that could be triggered even with the
cache if things are run in the right way.

Aaron Meurer

>
> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sympy.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sympy/554AFE05.30707%40durchholz.org.

[Joachim Durchholz]

Am 06.05.2015 um 23:01 schrieb Peter Brady:
>>
>> Doesn't the test runner do a clear_cache before each test?
>
> No. Just at the start of each file.

Ah, I missed that because I ran just a single test where I observed that.

I'm wondering whether we shouldn't do clear_cache() before each test. If
I do that, overall running time of bin/test increases from 5102 to 5244
seconds on my machine, that's an increase of merely 2.8%.