Naming convention for integration test case module names

[Baiju M]

Hi,

Can we use a naming convention for integration test case module names.
We do have a naming convention all unit test modules, they are ending with
_test.py The same convention can be used for integration tests.

There is test module named `cmd.py` inside `tests/integration/states`,
due to this, I cannot use pdb in any other modules there. I get
these error:

Traceback (most recent call last):
File "tests/integration/states/host.py", line 25, in test_present
import pdb;pdb.set_trace()
File "/usr/lib/python2.6/pdb.py", line 59, in <module>
class Pdb(bdb.Bdb, cmd.Cmd):
AttributeError: 'module' object has no attribute 'Cmd'

Well, we can easily fix this issue by renaming that cmd.py to somthing
else. But still having a common naming convention would be better.

If this proposal is accepted, I can volunteer to send a pull request
with renaming of all integration test modules :)

Regards,
Baiju M

[Thomas S Hatch]

We ran into this with the cmd module. This is why we renamed it to cmdmod and used the virtual function to fix it. I think we should just do the same with the state module. This same problem would be evident in all of the modules themselves, not just the tests, so I think that making a _test convention would not correctly parallel the underlying problem. 

[Jeff Schroeder]

On Wed, May 30, 2012 at 10:01 AM, Thomas S Hatch <that...@gmail.com> wrote:
> We ran into this with the cmd module. This is why we renamed it to cmdmod
> and used the virtual function to fix it. I think we should just do the same
> with the state module. This same problem would be evident in all of the
> modules themselves, not just the tests, so I think that making a _test
> convention would not correctly parallel the underlying problem.

Yeah ${name}mod seems good for the actual filenames.

> On Wed, May 30, 2012 at 10:50 AM, Baiju M <baiju....@gmail.com> wrote:
>>
>> Hi,
>>
>> Can we use a naming convention for integration test case module names.
>> We do have a naming convention all unit test modules, they are ending with
>> _test.py  The same convention can be used for integration tests.
>>
>> There is test module named `cmd.py` inside `tests/integration/states`,
>> due to this, I cannot use pdb in any other modules there.  I get
>> these error:
>>
>> Traceback (most recent call last):
>>  File "tests/integration/states/host.py", line 25, in test_present
>>    import pdb;pdb.set_trace()
>>  File "/usr/lib/python2.6/pdb.py", line 59, in <module>
>>    class Pdb(bdb.Bdb, cmd.Cmd):
>> AttributeError: 'module' object has no attribute 'Cmd'
>>
>> Well, we can easily fix this issue by renaming that cmd.py to somthing
>> else.  But still having a common naming convention would be better.
>>
>> If this proposal is accepted, I can volunteer to send a pull request
>> with renaming of all integration test modules :)
>>
>> Regards,
>> Baiju M
>
>

--
Jeff Schroeder

Don't drink and derive, alcohol and analysis don't mix.
http://www.digitalprognosis.com

[Baiju M]

The main reason for renaming test module to have a common suffix
(preferably prefix)
is for automatic test discovery which is available in modern
test-runners including unittest2.
http://www.voidspace.org.uk/python/articles/unittest2.shtml#test-discovery

For example, now all salt unit-tests can be run like this:

unit2 discover -s tests/unit -p *test.py

If the unit-tests module named were starting with test_, this command
would be sufficient:

unit2 discover -s tests/unit

This will also reduce lots of boilerplate code. For example, I
removed this boiler plate
from tests/unit/modules/gem_test.py and run the test cases and it
recovered all test cases!

-if __name__ == "__main__":
- loader = TestLoader()
- tests = loader.loadTestsFromTestCase(TestGemModule)
- TextTestRunner(verbosity=1).run(tests)

$ unit2 discover -s tests/unit/modules/ -p gem_test.py -v
test__gem (gem_test.TestGemModule) ... ok
test_list (gem_test.TestGemModule) ... ok
test_sources_list (gem_test.TestGemModule) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.003s

[Thomas S Hatch]

well, I don't want to rename a bunch of files, only the occasional ones like cmd that we need to rename. The names should directly reference what they are doing

[Thomas S Hatch]

Right, but will that translate into the integration tests? And the recent addition made by Dan Colish covers this problem with the new -n option passed to the runtests script

[Baiju M]

The auto-discovery feature based on module names will not work for
integration tests now. Loading of master and minions should be changed
to make use test-fixture mechanism provided by unittest2 (setUp & tearDown).
Yes, this need some work. So, just renaming is not sufficient for
auto-discovery
and running tests.

The -n option add the functionality to run tests based on names:
http://docs.python.org/library/unittest.html#unittest.TestLoader.loadTestsFromName
For example: python tests/runtests.py -n tests.unit.modules.gem_test
But this option will not auto-discover entire test suite based on
naming convention.

[Dan Colish]

On Wednesday, May 30, 2012 at 11:05 AM, Baiju M wrote:
> The auto-discovery feature based on module names will not work for
> integration tests now. Loading of master and minions should be changed
> to make use test-fixture mechanism provided by unittest2 (setUp & tearDown).
> Yes, this need some work. So, just renaming is not sufficient for
> auto-discovery
> and running tests.
>
> The -n option add the functionality to run tests based on names:
> http://docs.python.org/library/unittest.html#unittest.TestLoader.loadTestsFromName
> For example: python tests/runtests.py -n tests.unit.modules.gem_test
> But this option will not auto-discover entire test suite based on
> naming convention.
>

You're right. There would need to be some extra logic to handle this correctly. I often use nose to avoid issues like this when running tests. Seeing that the tests are for developers only, it might make sense to choose a dependency that already does what we want instead of recreating it.

--Dan