Exclude base class from tests?

[Vicki]

Hi folks,

A pattern I frequently use is to define a base class that contains all
the actual tests and a "finish-setup" routine, then define real test
cases that inherit from that base class and specify only the setUp().
I've pasted in a toy example below.

When I use nose to test the file containing these classes, it
invariably tries to run MyBase.test1 and MyBase.test2, which of course
fail because MyBase doesn't have the data values defined.

Is there a way to tell nose to exclude this class? So far the only
workaround I've found is to put the base class in a different file,
import that module, and inherit from otherfile.MyBase, which isn't
really desirable, and it still fails if I'm letting nose discover all
files in the directory.

thanks,
Vicki

class MyBase(unittest.TestCase):
def finishsetup(self):
self.z=self.x+self.y

def test1(self):
self.assertEqual(self.z, self.x+self.y)

def test2(self):
self.assert_(self.x > self.y)

class RealCase1(MyBase):
def setUp(self):
self.x=10
self.y=5
MyBase.finishsetup(self)

class RealCase2(MyBase):
def setUp(self):
self.x=42
self.y=13
MyBase.finishsetup(self)

[Kumar McMillan]

By default nose will run a class if a) it descends from
unittest.TestCase or b) its name has "test" in it. So I generally
solve the problem you're up against by using multiple inheritance,
like this:

class MyBase(object):

def finishsetup(self):
self.z=self.x+self.y

def test1(self):
self.assertEqual(self.z, self.x+self.y)

def test2(self):
self.assert_(self.x > self.y)

class RealCase1(MyBase, unittest.TestCase):

def setUp(self):
self.x=10
self.y=5
MyBase.finishsetup(self)

class RealCase2(MyBase, unittest.TestCase):

def setUp(self):
self.x=42
self.y=13
MyBase.finishsetup(self)

But I suppose you could also use the __test__ property. If it is
False then nose will not run it as a test.

class MyBase(unittest.TestCase):
__test__ = False
# ...

class RealCase1(MyBase):
__test__ = True
# ...

this of course means every concrete class *must* override __test__
which someone might forget to do.

[Vicki]

Thanks Kumar,

On May 20, 11:21 am, Kumar McMillan <kumar.mcmil...@gmail.com> wrote:
> By default nose will run a class if a) it descends from
> unittest.TestCase or b) its name has "test" in it.

Huh, I did not realize that the former was true.

> So I generally
> solve the problem you're up against by using multiple inheritance,
> like this:

(snip)

I see why that would work. It feels more like a workaround to me than
a
solution.

The piece that feels wrong to me is that test cases function as
namespace containers, but nose does not seem to have a way to match or
exclude based on the case name. It operates only on the test name, and
apparently only on the naked name of the test method (test1) rather
than the fully specified name (RealCase1.test1, MyBase.test1).

Is that a correct statement, and if so, might it be possible to modify
the behavior to work
on fully specified names?

>
> But I suppose you could also use the __test__ property.  If it is
> False then nose will not run it as a test.
>
> class MyBase(unittest.TestCase):
>     __test__ = False
>     # ...
>
> class RealCase1(MyBase):
>     __test__ = True
>     # ...
>
> this of course means every concrete class *must* override __test__
> which someone might forget to do.

Yes, I thought about that solution too, and disliked it both because
it's easy to forget, and because it is repeated code that I can't
stash in the base class.

Vicki

[Kumar McMillan]

On Wed, May 20, 2009 at 1:00 PM, Vicki <lai...@stsci.edu> wrote:
>
> Thanks Kumar,
>
> On May 20, 11:21 am, Kumar McMillan <kumar.mcmil...@gmail.com> wrote:
>> By default nose will run a class if a) it descends from
>> unittest.TestCase or b) its name has "test" in it.
>
> Huh, I did not realize that the former was true.
>
>> So I generally
>> solve the problem you're up against by using multiple inheritance,
>> like this:
>
> (snip)
>
> I see why that would work. It feels more like a workaround to me than
> a
> solution.
>
> The piece that feels wrong to me is that test cases function as
> namespace containers, but nose does not seem to have a way to match or
> exclude based on the case name. It operates only on the test name, and
> apparently only on the naked name of the test method (test1) rather
> than the fully specified name (RealCase1.test1, MyBase.test1).

I'm not sure what you mean by this. Nose will match on the class
name. For example:

class TestStuff(object):
def test_it(self):
pass

class Other(object):
def test_it(self):
pass

In these classes above, only TestStuff.test_it will be matched as a
test, not Other.test_it because the class name `Other' does not have
test in the title.

[Vicki]

On May 20, 2:22 pm, Kumar McMillan <kumar.mcmil...@gmail.com> wrote:

> On Wed, May 20, 2009 at 1:00 PM, Vicki <laid...@stsci.edu> wrote:
>
> > The piece that feels wrong to me is that test cases function as
> > namespace containers, but nose does not seem to have a way to match or
> > exclude based on the case name. It operates only on the test name, and
> > apparently only on the naked name of the test method (test1) rather
> > than the fully specified name (RealCase1.test1, MyBase.test1).
>
> I'm not sure what you mean by this.  Nose will match on the class
> name.  For example:
>
> class TestStuff(object):
>     def test_it(self):
>         pass
>
> class Other(object):
>     def test_it(self):
>         pass
>
> In these classes above, only TestStuff.test_it will be matched as a
> test, not Other.test_it because the class name `Other' does not have
> test in the title.

Sorry, I meant that I cannot use the command line switches to match or

exclude
based on the case name.

nosetests basetest.py -e MyBase

still runs the MyBase tests. So does specifying MyBase.test1,
"MyBase*", and so on.
Likewise trying to match with -m "RealCase*" doesn't work.

I can however -m test1 or -e test2, to select or exclude every
instance of the
named test in all cases in the file.

That's what I meant by fully qualified test name vs. naked test name.

I gather that the behavior you describe is the test discovery phase,
whereas I'm wanting to interact at the, er, test selection phase?

[Kumar McMillan]

On Wed, May 20, 2009 at 2:07 PM, Vicki <lai...@stsci.edu> wrote:
>
> Sorry, I meant that I cannot use the command line switches to match or
> exclude
> based on the case name.
>
> nosetests basetest.py -e MyBase

ok, I see what you mean. It's not exactly what you want but I have
found that using attributes (or "tags") are a useful for this kind of
use case. What I've been done in the past for when I want to exclude
classes is to create a metaclass that sets some attributes on test
methods. It's a bit heavyweight and the new class decorators will
solve this nicer.

The following code probably doesn't work but the general idea is :

class UITestType(type):
def __new__(cls, bases, dictionary):
# add an attribute to each class method,
# something like this (I forget)
for method in dictionary.values():
method.ui = True
return type.__new__(cls, bases, dictionary)

class UITest(unittest.TestCase):
__metaclass__ = UITestType

class TestLoginUI(UITest):
# the metaclass does the equivalent of:
# @attr('ui')
def test_login_then_logout(self):
pass

Then on the command line you could do:

$ nosetests -a 'ui'

to run all tests descending from UITest and

$ nosetests -a '!ui'

to exclude all tests descending from UITest

More info on the attribute selector plugin:
http://somethingaboutorange.com/mrl/projects/nose/0.11.1/plugins/attrib.html

[Vicki]

Thanks - I will bear this in mind for the general case of including or
excluding
classes; although I think it will fail for the specific case of "run
the subclasses, but not
the base class", won't it?

In any event, all of these possible workarounds seem like very
complicated solutions
to what ought in principle to be a simple problem.

I'd like to understand whether it is in fact _not_ a simple problem,
or whether this is just a use case that's missing from the UI.

Vicki

On May 20, 4:38 pm, Kumar McMillan <kumar.mcmil...@gmail.com> wrote:

> More info on the attribute selector plugin:http://somethingaboutorange.com/mrl/projects/nose/0.11.1/plugins/attr...

[Kumar McMillan]

On Wed, May 20, 2009 at 3:49 PM, Vicki <lai...@stsci.edu> wrote:
>
> Thanks - I will bear this in mind for the general case of including or
> excluding
> classes; although I think it will fail for the specific case of "run
> the subclasses, but not
> the base class", won't it?

yes, it would

>
> In any event, all of these possible workarounds seem like very
> complicated solutions
> to what ought in principle to be a simple problem.

keep in mind you can always just run all tests in a specific class like this:

$ nosetests myapp/tests/test_something.py:TheClass

>
> I'd like to understand whether it is in fact _not_ a simple problem,
> or whether this is just a use case that's missing from the UI.

it sounds like you are after something more like:

$ nosetests --run-tests-from-subclass MyBaseTest

?

(with a shorter command line option obviously)

I think that would be tricky because class names are not unique in
Python. So you might need to provide a full module path to the class.
I suppose you could try it out -- it could be done as a plugin with
about 10 lines of code.

[Vicki]

On May 20, 5:13 pm, Kumar McMillan <kumar.mcmil...@gmail.com> wrote:

> On Wed, May 20, 2009 at 3:49 PM, Vicki <laid...@stsci.edu> wrote:
>
> > Thanks - I will bear this in mind for the general case of including or
> > excluding
> > classes; although I think it will fail for the specific case of "run
> > the subclasses, but not
> > the base class", won't it?
>
> yes, it would

Ah ok - I'm not familiar with metaclasses so wasn't sure I was reading
it correctly.

>
> > In any event, all of these possible workarounds seem like very
> > complicated solutions
> > to what ought in principle to be a simple problem.
>
> keep in mind you can always just run all tests in a specific class like this:
>
> $ nosetests myapp/tests/test_something.py:TheClass

Good to know; of course it works for matching but not for excluding.

>
>
> > I'd like to understand whether it is in fact _not_ a simple problem,
> > or whether this is just a use case that's missing from the UI.
>
> it sounds like you are after something more like:
>
> $ nosetests --run-tests-from-subclass MyBaseTest
>
> ?

That would solve this particular problem, but it seems needlessly
specific.
I really want to be able to match or exclude based on the
CaseName.testname
pattern:

nosetests -e "MyBase.*"

The -exclude switch takes regular expressions. That's a legal regular
expression
that should match MyBase.testfoo. (Or some variant of it should, MyBase
\..* or something.)

More generally I might want to do

nosetests -m "ClassM.*" to select a group of classes and all their
tests.


I don't understand why this doesn't work in the first place. The fact
that it doesn't makes me think there must be a reason: maybe there's
something going on behind the scenes that I don't understand, or some
restriction on what part of the test name is available to the piece of
code that's processing those match and exclude flags.

thanks,
Vicki

[Garrett Cooper]

IIRC re-module enabled regexps aren't supported in the default selector plugin.
-Garrett