Names for generated tests

[Vicki]

Hi folks,

I'm having some trouble with the names of generated tests & hope
you can help me out.

The problem I'm trying to solve is twofold:
- generate tests that run the same test but loop over various
parameters to feed the test
- name the test something sensible involving those parameters

From the docs, this looked like the best way to solve that problem:

def testPhotonRate():

def checkrate(em,total,funits):
em.convert(funits)
test=em.photonrate()
assert abs(test-total) < 0.0001

ema=S.GaussianSource(100,1000,2,fluxunits='photlam')
emh=S.GaussianSource(100,1000,2,fluxunits='photnu',
waveunits='hz')
emc=S.GaussianSource(100,1000,2,fluxunits='counts')
uset={ema:['photlam','flam','stmag','vegamag'],
emh:['photnu', 'fnu', 'abmag','jy','mjy'],
emc:['counts','obmag']}
for em in (ema,emh,emc):
for u in uset[em]:
checkrate.description="%s.testPhotonRate.test%sfunc"%
(__name__,u)
yield checkrate, em, 100, u


When I run this test set, the tests seem to have the right names when
they are run, in verbose mode; but the error report that shows the
results for the failing tests just repeats the name of the last test
that was run:
(See below).

Is this a bug, or am I doing something wrong?

thanks,
Vicki

{{non-generated tests snipped}}
v05_tickets.testPhotonRate.testphotlamfunc ... ok
v05_tickets.testPhotonRate.testflamfunc ... ok
v05_tickets.testPhotonRate.teststmagfunc ... ok
v05_tickets.testPhotonRate.testvegamagfunc ... ERROR
v05_tickets.testPhotonRate.testphotnufunc ... FAIL
v05_tickets.testPhotonRate.testfnufunc ... FAIL
v05_tickets.testPhotonRate.testabmagfunc ... FAIL
v05_tickets.testPhotonRate.testjyfunc ... FAIL
v05_tickets.testPhotonRate.testmjyfunc ... FAIL
v05_tickets.testPhotonRate.testcountsfunc ... ok
v05_tickets.testPhotonRate.testobmagfunc ... ok

======================================================================
ERROR: v05_tickets.testPhotonRate.testobmagfunc
----------------------------------------------------------------------
Traceback (most recent call last):
File "/user/laidler/pyinstall//lib/python/nose/case.py", line 182,
in runTest
self.test(*self.arg)
File "/data/gaudete1/dg1/laidler/ssb/checkout/pysynphot/test/
v05_tickets.py", line 185, in checkrate
test=em.photonrate()
File "/user/laidler/pyinstall/pysynphot/spectrum.py", line 300, in
photonrate
self.convert(funits.linunit)
File "/user/laidler/pyinstall/pysynphot/spectrum.py", line 343, in
convert
nunits = units.Units(targetunits)
File "/user/laidler/pyinstall/pysynphot/units.py", line 34, in Units
return factory(uname)
File "/user/laidler/pyinstall/pysynphot/units.py", line 547, in
factory
key=uname.lower()
AttributeError: 'NoneType' object has no attribute 'lower'

======================================================================
FAIL: v05_tickets.testPhotonRate.testobmagfunc
----------------------------------------------------------------------
Traceback (most recent call last):
File "/user/laidler/pyinstall//lib/python/nose/case.py", line 182,
in runTest
self.test(*self.arg)
File "/data/gaudete1/dg1/laidler/ssb/checkout/pysynphot/test/
v05_tickets.py", line 186, in checkrate
assert abs(test-total) < 0.0001
AssertionError

======================================================================
FAIL: v05_tickets.testPhotonRate.testobmagfunc
----------------------------------------------------------------------
Traceback (most recent call last):
File "/user/laidler/pyinstall//lib/python/nose/case.py", line 182,
in runTest
self.test(*self.arg)
File "/data/gaudete1/dg1/laidler/ssb/checkout/pysynphot/test/
v05_tickets.py", line 186, in checkrate
assert abs(test-total) < 0.0001
AssertionError

======================================================================
FAIL: v05_tickets.testPhotonRate.testobmagfunc
----------------------------------------------------------------------
Traceback (most recent call last):
File "/user/laidler/pyinstall//lib/python/nose/case.py", line 182,
in runTest
self.test(*self.arg)
File "/data/gaudete1/dg1/laidler/ssb/checkout/pysynphot/test/
v05_tickets.py", line 186, in checkrate
assert abs(test-total) < 0.0001
AssertionError

======================================================================
FAIL: v05_tickets.testPhotonRate.testobmagfunc
----------------------------------------------------------------------
Traceback (most recent call last):
File "/user/laidler/pyinstall//lib/python/nose/case.py", line 182,
in runTest
self.test(*self.arg)
File "/data/gaudete1/dg1/laidler/ssb/checkout/pysynphot/test/
v05_tickets.py", line 186, in checkrate
assert abs(test-total) < 0.0001
AssertionError

======================================================================
FAIL: v05_tickets.testPhotonRate.testobmagfunc
----------------------------------------------------------------------
Traceback (most recent call last):
File "/user/laidler/pyinstall//lib/python/nose/case.py", line 182,
in runTest
self.test(*self.arg)
File "/data/gaudete1/dg1/laidler/ssb/checkout/pysynphot/test/
v05_tickets.py", line 186, in checkrate
assert abs(test-total) < 0.0001
AssertionError

----------------------------------------------------------------------
Ran 33 tests in 0.113s

FAILED (errors=1, failures=5)

[jason pellerin]

Hi,

The problem is that checkrate is defined outside of the loop, so
there's only one checkrate function, whose description is changed each
time through the loop. Since nose isn't caching the description (which
is probably a bug), when the description is looked up again when error
output is printed at the end of the test run, the description for each
test will (as you saw) be the last thing that checkrate.description
was set to.

In the near term, you can fix this by moving the checkrate function
definition inside the innermost loop. Longer term, if you could report
this as a bug in nose's tracker, I'll try to get it fixed for the next
release.

JP

[Vicki]

Hi Jason,

Thanks for the quick reply.

>
> In the near term, you can fix this by moving the checkrate function
> definition inside the innermost loop.

Cool, that worked. Thanks. It will restrict some flexibility, but as a
workaround it's not bad.

>Longer term, if you could report
> this as a bug in nose's tracker, I'll try to get it fixed for the next
> release.

Sure. Can I just copy in my original message, or do you need a
standalone case that demonstrates it?

While I'm talking about names, I have a question about test names as
nose understands them. I usually start out by running all tests in a
file, eg "nosetests mytest.py", and then want to subsets or individual
tests.

Suppose I have a file mytest.py, which is in a directory on my path,
and contains

class Test1(object):
def testa(self):
def testb(self):

def TestStuff():

def TestGenerator():
for i in range:
def check_something(foo):
...
yield check_something, i

If I do "nosetests mytest", they all get run. Cool.

"nosetests mytest.TestStuff" runs nothing. I would expect it to run
the function TestStuff().

"nosetests mytest.Test1" runs nothing. I would expect it to run both
the test methods defined in the Test1 class.

"nosetests mytest.Test1:testa" has sometimes worked, but it is not
working for me right now. (possibly the difference between inheriting
from a unittest testcase and not?)

"nosetests mytest.TestGenerator" runs nothing. I would expect it to
run all the generated tests.

Are my expectations incorrect, or is this a bug?

And, I have no idea whether there is a way to name my generated tests
such that I could give nosetest that name; but if there is, how would
I do that?

thanks,
Vicki

[jason pellerin]

On Wed, Jul 2, 2008 at 5:54 PM, Vicki <lai...@stsci.edu> wrote:

> Sure. Can I just copy in my original message, or do you need a
> standalone case that demonstrates it?

The message is fine.

> "nosetests mytest.TestStuff" runs nothing. I would expect it to run
> the function TestStuff().

nose requires that you explicitly indicate the module:callable
boundary in a test name with a colon, so the way to do what you want
to do is:

nosetests mytest:TestStuff
nosetests mytest:Test1.testa

There are complicated reasons for this, but the simple one is that
given the much wider variety of things that nose considers a valid
test name relative to unittest, it winds up being much simpler for
users (and nose developers) if nose doesn't try to guess at what part
of a name is probably a file on disc and what part is probably
something in that file.

JP

[Vicki]

>
> > "nosetests mytest.TestStuff" runs nothing. I would expect it to run
> > the function TestStuff().
>
> nose requires that you explicitly indicate the module:callable
> boundary in a test name with a colon, so the way to do what you want
> to do is:
>
> nosetests mytest:TestStuff
> nosetests mytest:Test1.testa

Aha! Thanks. From the examples, I had thought that the dots defined
namespace in the usual way, and the colon indicated the specific test
that was supposed to be run.

>
> There are complicated reasons for this, but the simple one is that
> given the much wider variety of things that nose considers a valid
> test name relative to unittest, it winds up being much simpler for
> users (and nose developers) if nose doesn't try to guess at what part
> of a name is probably a file on disc and what part is probably
> something in that file.

Sure, that makes sense.

thanks again,
Vicki