nosetests --with-isolation problems
Ondrej Certik
how is the --with-isolation supposed to work? I thought it should be
able to fix the following problem, but it doesn't.
$ hg clone http://hg.sympy.org/sympy
$ cd sympy/sympy
$ nosetests -x --with-isolation
..........................................................................................................................................................F
======================================================================
FAIL: test_str.test_Derivative
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/lib/python-support/python2.4/nose/case.py", line 203, in
runTest
self.test(*self.arg)
File "/home/ondra/ext/sympy/sympy/core/tests/test_str.py", line 59,
in test_Derivative
assert str(e) == "D(x**2, x)"
AssertionError
----------------------------------------------------------------------
Ran 155 tests in 4.929s
FAILED (failures=1)
So there is a problem discovered. Let's just run this particular test:
$ nosetests -x --with-isolation core/tests/test_str.py
........
----------------------------------------------------------------------
Ran 8 tests in 0.028s
OK
How is that possible? This test should have failed.
ondra@fuji:~/ext/sympy/sympy$ nosetests -x --with-isolation core/
tests/
............................................................................................................................................................................
----------------------------------------------------------------------
Ran 172 tests in 7.089s
OK
And here too....
The real problem is in sympy, due to caching bugs. But that's why I
would like to use sandboxed environment, so that the tests are
reliable and doesn't depend on the previous tests.
So, if the above test was executed alone, it worked. Is there a way to
make nosetest "not to fail" the test when run in the full test suite?
I mean - obviously nosetest changed the environment somehow. BTW,
py.test also has this problem.
Thanks for any suggestions,
Ondrej
Ondrej Certik
On Jan 18, 1:24 am, Ondrej Certik <ond...@certik.cz> wrote:
> Hi,
>
> how is the --with-isolation supposed to work? I thought it should be
> able to fix the following problem, but it doesn't.
>
simple project, but here the isolation plugin works! Try:
$ hg clone http://hg.certik.cz/nose
$ hg up b8bcbadf8c3d
$ nosetests tests/test_sub.py tests/test_add.py
.F
======================================================================
FAIL: test_add.test_add
Traceback (most recent call last):
File "/var/lib/python-support/python2.4/nose/case.py", line 203, in
runTest
self.test(*self.arg)
assert add(3, 2) == 5
AssertionError
----------------------------------------------------------------------
Ran 2 tests in 0.001s
FAILED (failures=1)
This is because the first test modifies the internal state of the
"add" method and it fails in the second test as you can see here:
$ cat cache.py
def add(x, y, cache=[-1]):
cache[0] += 1
return x+y+cache[0]
def sub(x, y):
return x-y
And now try:
$ nosetests --with-isolation tests/test_sub.py tests/test_add.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK
and everything works just fine, because the isolation plugin indeed
creates a clean environment for each test module.
Well, so why it isn't working in the real life example in my previous
email in sympy? I don't know. Do you have some suggestions?
Ondrej
jason pellerin
will leak. If you need to isolate tests more than that, John J Lee has
written an excellent plugin that runs each test in its own process.
You can find it here:
http://pypi.python.org/pypi/nosepipe/
If that doesn't help, it may at least be a guide to what you'd have to
do in a custom plugin to sandbox the tests as you need to.
JP
Ondrej Certik
>
> --with-isolation only resets sys.modules, so any other global state
> will leak. If you need to isolate tests more than that, John J Lee has
> written an excellent plugin that runs each test in its own process.
> You can find it here:
>
> http://pypi.python.org/pypi/nosepipe/
>
> If that doesn't help, it may at least be a guide to what you'd have to
> do in a custom plugin to sandbox the tests as you need to.
Thanks a lot, that's something I need.
I tried it:
ondra@pc232:~/ext/sympy$ nosetests --with-process-isolation -x
E
======================================================================
ERROR: test_gosper.test_normal
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/lib/python-support/python2.4/nose/case.py", line 152, in run
self.runTest(result)
File "/var/lib/python-support/python2.4/nose/case.py", line 170, in runTest
test(result)
File "nosepipe.py", line 151, in __call__
Exception: short message body (want 772418861, got 94)
----------------------------------------------------------------------
Ran 0 tests in 0.242s
FAILED (errors=1)
So I looked into nosepipse.py:151
try:
stdout = popen.stdout
while True:
header = stdout.read(4)
if not header:
break
if len(header) < 4:
raise Exception("short message header %r" % header)
request_len = struct.unpack("!I", header)[0]
data = stdout.read(request_len)
if len(data) < request_len:
raise Exception("short message body (want %d, got %d)" %
(request_len, len(data)))
parts = data.split(":", 1)
So there is some problem with reading from IO in the plugin. CCing to
John J. Lee - do you think I made something wrong? I added the plugin
among builtin plugins, because setuptools don't work reliably on my
computer - nosetools doesn't install with them, but it installs as a
Debian package. But then I cannot add any modules to it using
setuptools, so I just added the module in nose/plugins manually and
adapted nose/builtin by hand to import it.
Then I decided I try without it, but no luck either so far. I
commented out tests, that run alone but not in the whole suite and
then I am getting this weird problem:
$ nosetests -x
..................................................................................................................................................................................................................................................E
======================================================================
ERROR: test_spherical_harmonics.test_Zlm
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/lib/python-support/python2.4/nose/case.py", line 203, in runTest
self.test(*self.arg)
File "/home/ondra/ext/sympy/sympy/functions/special/tests/test_spherical_harmonics.py",
line 73, in test_Zlm
assert Zlm(1, -1, th, ph) == sqrt(3/(4*pi))*sin(th)*sin(ph)
File "/home/ondra/ext/sympy/sympy/functions/special/spherical_harmonics.py",
line 39, in Zlm
zz = simplify(zz)
TypeError: 'module' object is not callable
----------------------------------------------------------------------
Ran 243 tests in 6.964s
FAILED (errors=1)
The problematic part of the code is here:
def Zlm(l, m, th, ph):
from sympy import simplify
if m > 0:
zz = (-1)**m*(Ylm(l, m, th, ph) + Ylm_c(l, m, th, ph))/sqrt(2)
elif m == 0:
return Ylm(l, m, th, ph)
else:
zz = (-1)**m*(Ylm(l, -m, th, ph) - Ylm_c(l, -m, th, ph))/(I*sqrt(2))
zz = zz.expand(complex=True)
zz = simplify(zz)
return zz
The simplify() function is in sympy/simplify/simplify.py, but in
sympy/__init__.py we do "from simplify import *" and in
sympy/simplify/__init__.py we also import the simplify() function, so
this should work and it does work, whene executed individually:
ondra@pc232:~/ext/sympy$ nosetests
sympy/functions/special/tests/test_spherical_harmonics.py
.....
----------------------------------------------------------------------
Ran 5 tests in 0.520s
OK
And it even works when the whole functions submodule of sympy is tested:
ondra@pc232:~/ext/sympy$ nosetests sympy/functions/
.....................................................
----------------------------------------------------------------------
Ran 53 tests in 2.995s
OK
So something is wrong, but not that wrong. Any ideas?
But if you are curious, why I want to use nosetest, here is the
problem I just discovered with py.test by accident:
http://code.google.com/p/sympy/issues/detail?id=627
the whole sympy testsuite run, but if one particular test file is
executed directly, it fails, because py.test skipped some tests when
run on the whole suite.
Due to that, some parts of sympy aren't checked even though we had
tests for them, but py.test simply didn't execute them. And so of
course those unchecked parts are broken now.
So I am trying to find some solution, that doesn't use so much magic,
but that I can trust and that is robust.
So far even though I am getting some weird issues with nosetests, it
looks promising, since it was able to execute the whole sympy
testsuite with only 9 errors out of 585 tests (some very intricate),
that were all written for py.test. So that's good.
So I commented the failing tests with our own @XFAIL decorator, that
simply ignores all exceptions, and it seems to work, but now I got
this problem:
$ nosetests -x
.............................................................................................................................................................................................................................................................................................E
======================================================================
ERROR: Failure: AttributeError ('cStringIO.StringO' object has no
attribute 'encoding')
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/lib/python-support/python2.4/nose/loader.py", line 363,
in loadTestsFromName
module = self.importer.importFromPath(
File "/var/lib/python-support/python2.4/nose/importer.py", line 39,
in importFromPath
return self.importFromDir(dir_path, fqname)
File "/var/lib/python-support/python2.4/nose/importer.py", line 84,
in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/ondra/ext/sympy/sympy/interactive/__init__.py", line 6, in ?
pprint_try_use_unicode()
File "/home/ondra/ext/sympy/sympy/printing/pretty/pretty_symbology.py",
line 100, in pretty_try_use_unicode
s.encode(sys.stdout.encoding)
AttributeError: 'cStringIO.StringO' object has no attribute 'encoding'
----------------------------------------------------------------------
Ran 286 tests in 47.430s
FAILED (errors=1)
This seems like a bug in nosetests, isn't it? The problem is in the
function pprint_try_use_unicode(), that reads sys.stdout.encoding, but
this function isn't tested in any tests, but is included in
__init__.py in the interactive module, but this module is not imported
by default. It's so that
users can just do
from sympy.interactive import *
and they will get nice 2D unicode printing in regular python
interpreter. So nosetests shouldn't try to import it, if it cannot
survive it's import. :)
Maybe one solution could be to tell nosetest exactly which tests to execute.
Ondrej
Ondrej Certik
BTW, nosetest should never import that module, that could also be the
reason for one test that failed above,
because it returned a 2D asciiart output for x**2, while that test is
only testing 1D output, which is the default. But by importing the
interarctive module, it turns on 2D printing globally.
Is there some easy way to tell nosetest not to import some particular
__init__.py? Actually - that could be the problem - if nosetest is
disovering
tests by iterativelly importing __init__.py files, it can screw things
up by it. Yes, that could be it. So I need to figure out how to tell
nosetest
to only look for test_*.py files recursively and that's it. I know
it's somewhere in the docs, but I need to go now. Will continue later.
Ondrej
John J Lee
[...]
>> from sympy.interactive import *
>>
>> and they will get nice 2D unicode printing in regular python
Don't do that. Imports should never have side effects like this.
There's no need for your code to do it, and it'll always end in tears for
somebody.
> up by it. Yes, that could be it. So I need to figure out how to tell
> nosetest
> to only look for test_*.py files recursively and that's it. I know
No, you don't. You need to fix your module not to do stuff at module
import time. :-)
John
John J Lee
[...]
Probably due to your module that fiddles with stdout on import.
John
Ondrej Certik
Right, you are absolutely right. sympy with this crap in sympy.interactive
is not a module but an application then. :)
I thought it's innocent, when this module is *not* imported by default.
But imports should never have side effects, that is absolutely correct.
> [...]
> > Exception: short message body (want 772418861, got 94)
> [...]
>
> Probably due to your module that fiddles with stdout on import.
Right. I deleted that interactive modul, to see if things improve.
I am still getting the same exception with --with-process-isolation.
But I think I know where the rest of the problems could be: we
sometimes use this
in tests (but nowhere else):
from sympy import *
and I was always saying to myself - this is going to bite you once,
rewrite it to explicit imports, before it's too late.
And I didn't rewrite it, and I think I just got bitten by it now. The
problem with "import *" is that when something is broken,
as it is now in sympy, it just do some evil things. Consider for
example the issue I sent a link to above. All I did was that I changed
the "import *" to explicit imports, as this is what I do whenever I
fix something in some module and I notice that it uses implicit
impots, and suddenly py.test started to pick up half of the tests. So
actually, it's probably not py.test, that is broken, but sympy.
And actually, that might also be the problem with py.test not
executing some of the tests.
Thanks very much for your help, I think I am getting on the right
track to fix this mess. But, did anyone of you think
about some robust mechanism of being sure, that your tests are
actually being executed? I know you can check the total
number of tests, that py.test or nosetest reports, but this number is
changing (increasing), and it can just happen, that the testsuite
forgets
to run some tests, for whatever reason. The test suite is there to
point you to the broken part of the tested package. But if the package
is broken in a way, so that the tests aren't executed (as it seems
this is what just happened now in sympy), we weren't allarmed by the
tests
and we happily continued developing a broken package.
Anyway, it's just the second verse in the Zen of Python
Explicit is better than implicit.
So I think I'll remember that now.
Ondrej
Kumar McMillan
>
> On Fri, 18 Jan 2008, Ondrej Certik wrote:
> [...]
> >> from sympy.interactive import *
> >>
> >> and they will get nice 2D unicode printing in regular python
> [...]
>
> Don't do that. Imports should never have side effects like this.
> There's no need for your code to do it, and it'll always end in tears for
> somebody.
I agree with John but to be fair this is very common practice for
"demos," especially in the python gaming world. sigh. I do think
nose needs to change its ways to avoid importing everything and I am
reminded again of issue116:
http://code.google.com/p/python-nose/issues/detail?id=116
>
>
> > up by it. Yes, that could be it. So I need to figure out how to tell
> > nosetest
> > to only look for test_*.py files recursively and that's it. I know
>
> No, you don't. You need to fix your module not to do stuff at module
> import time. :-)
or ... I believe you can simply add --match='^test_.*\.py' and that
will short-circuit discovery via import. I think.
>
>
> John
>
>
>
> >
>
Ondrej Certik
>
> On Jan 18, 2008 1:36 PM, John J Lee <j...@pobox.com> wrote:
> >
> > On Fri, 18 Jan 2008, Ondrej Certik wrote:
> > [...]
> > >> from sympy.interactive import *
> > >>
> > >> and they will get nice 2D unicode printing in regular python
> > [...]
> >
> > Don't do that. Imports should never have side effects like this.
> > There's no need for your code to do it, and it'll always end in tears for
> > somebody.
>
> I agree with John but to be fair this is very common practice for
> "demos," especially in the python gaming world. sigh. I do think
> nose needs to change its ways to avoid importing everything and I am
> reminded again of issue116:
> http://code.google.com/p/python-nose/issues/detail?id=116
Yep, see my reply before in the thread.
>
> >
> >
> > > up by it. Yes, that could be it. So I need to figure out how to tell
> > > nosetest
> > > to only look for test_*.py files recursively and that's it. I know
> >
> > No, you don't. You need to fix your module not to do stuff at module
> > import time. :-)
>
> or ... I believe you can simply add --match='^test_.*\.py' and that
> will short-circuit discovery via import. I think.
Yes, I tried this simple regexp already, but unfortunately this fails
to find any tests, because they are not in the top level directory,
but in nested "test/" directories, usually in each module.
Ondrej
Ondrej Certik
But I just discovered, that this does the trick:
$ nosetests --match='test_*'
But isn't this rather supposed to find tests in files (not filenames)?
BTW, the process isolation plugin still isn't working with this.
But then I found that the following trick does the job:
$ nosetests sympy/*/tests/test_*.py
it doesn't execute all tests (some are more nested), but almost all.
Ondrej