Code Management
Jens
I'm writing some modules in Python, and I'm also using unittests. I'm
wondering about some things:
1) Should I put my unittests in a subdirectory? Does the subdirectory
have to be a package?
2) Does the main folder /myproject have to be a package? Should I put
my modules directly under /myproject, or should I create a subfolder,
for example /myproject/modules
Does anyone have any "best practices" as to how to manage your code?
Thanks!
brz...@gmail.com
> Dear Reader,
>
> I'm writing some modules in Python, and I'm also using unittests. I'm
> wondering about some things:
I'd love to hear how others manage this sort of thing as well. I'll
describe what I've found works best for me, and if others do the same,
maybe we can all learn from each other.
> 1) Should I put my unittests in a subdirectory? Does the subdirectory
> have to be a package?
I put them in a subdirectory and make it a package. The biggest
advantage I see of using a subdirectory is that I can have lots of
test scripts and it's easy to keep them organized separately from
production code. The biggest disadvantage is that if I run the test
scripts from inside that subdirectory, they need to import modules
from their parent directory, and I'm using Python 2.4 which doesn't
have relative imports. So I put a .pth file in my site-packages
directory that adds the top-level package of my project to the
pythonpath. Then the test modules can import production code using
the fully qualified package.subpackage.module name for each production
module being tested.
> 2) Does the main folder /myproject have to be a package? Should I put
> my modules directly under /myproject, or should I create a subfolder,
> for example /myproject/modules
I make all non-trivial projects into packages. It allows me to do the
trick with a .pth file I described above. It makes it easier to reuse
all or part of my project as a component of a second project. And it
makes creating documentation with epydoc easier.
I typically lay out a project directory something like this:
projectname/
setup.py
dev_install.py # automatically creates .pth file in site-packages
alltests.py # runs all unit tests
main_package_name/
__init__.py
module1.py
module2.py
doc/
img/
subpackage1/
__init__.py
module3.py
module4.py
tests/
__init__.py
test1.py
test2.py
subpackage2/
etc....
With this setup, tests are organized by subpackage, and each test
script can be run by itself, or all together from alltests.py. If a
module in subpackage2 needs to import a module from subpackage1
(generally a sign of poor design, but it happens), it would need to
use the fully qualified "import
main_package_name.subpackage1.module1". Each time I check a new copy
of the project out from version control to a new location, I have to
make that the "active" version by running dev_install.py, which puts
a .pth file in site-packages that adds the newly checked out path to
the pythonpath. As long as I remember that step, this approach works
well.
> Does anyone have any "best practices" as to how to manage your code?
>
> Thanks!
If anyone does, I'd love to hear about them.
-Casey Raymondson
Sergio Correia
> 1) Should I put my unittests in a subdirectory? Does the subdirectory
> have to be a package?
Sure, to avoid cluttering put the tests in a folder called 'tests'
(and any input the tests require, like mock files, output files used
to test if the output is correct, etc etc, should be put in that
folder or in a subfolder of 'tests'.
I like this setup:
myproject
../ package1
../../ tests
../../../ mock_files
../package2
and so on
If you want to be able to import the tests from python (say, 'from
myproject.tests import test_all'), then yes, you need to make the
tests folder a subpackage (just drop an empty __init__.py file)
> 2) Does the main folder /myproject have to be a package? Should I put
> my modules directly under /myproject, or should I create a subfolder,
> for example /myproject/modules
Of course... make /myproject a package. Now, if your modules are kinda
small you can just dump them in /myproject, but if what you are doing
is fairly complex, it's better to move each module to its own folder.
> Does anyone have any "best practices" as to how to manage your code?
I want 'best practices' too... anyone?
Sergio
PS: Casey, neat idea with the 'alltests.py'. Btw, I think relative
imports can be helpful:
http://www.python.org/dev/peps/pep-0328/#guido-s-decision
Ben Finney
> 1) Should I put my unittests in a subdirectory?
I find that it is best to do so even for single-module packages,
because it makes writing the 'setup.py' easier ("don't install
anything in the unit test directory").
> Does the subdirectory have to be a package?
Doing so allows you to address individual test modules and cases by
imported name, which (IIRC) enables some features of the unittest
library module. If you intend to use 'setuptools', this allows you to
specify the test suite by name, which will be run before building the
distribution files.
> 2) Does the main folder /myproject have to be a package? Should I put
> my modules directly under /myproject, or should I create a subfolder,
> for example /myproject/modules
I find this structure to be best::
fooproject/
setup.py
LICENSE
Makefile
foo/
__init__.py
spam.py
eggs.py
bar/
__init__.py
beans.py
baz/
__init__.py
bacon.py
test/
suite.py
test_foo.py
test_spam.py
test_eggs.py
test_bar_beans.py
test_baz_bacon.py
doc/
README
blah.txt
mumble.txt
Each of the unit test modules is executable (invoking
'unittest.main()' if run as a program), and corresponds to one
importable module from the code under test; this isn't strictly
necessary but makes it much easier to manage. There is no need to
mimic the hierarchical structure of the implementation modules, so
long as each unit test module clearly shows what it's testing.
The 'test/suite.py' module is executable (again, invoking
'unittest.main()') and defines a 'unittest.TestSuite' instance for all
the test cases, named 'suite'.
The (setuptools-enabled) 'setup.py' is configured to not install the
'test' package, and to use 'test.suite.suite' as the test suite
object. It will include the documentation and LICENSE in source and
"binary" distributions.
The Python package will be installed as an importable package named
'foo', with the hierarchy of sub-packages as shown; this is pretty
automatic via setuptools (as per the documentation, anyway) with the
structure defined above.
The 'Makefile' contains targets for 'build', 'install', 'clean', etc.
and a 'test' target that runs 'python ./test/suite.py'.
--
\ "Never use a long word when there's a commensurate diminutive |
`\ available." -- Stan Kelly-Bootle |
_o__) |
Ben Finney
Jens
Thank you very much for your ideas! I'm trying to make it work, and
have created a small example:
dummy/
dummy_package/
__init__.py
moduleA.py
tests/
__init__.py
test.py
I'm using Python 2.5.1. When I'm trying to call a function in
'moduleA' from 'test' it won't work unless I make the 'dummy' folder a
package as well. That's pretty weird. Does 'dummy_package' have to be
in my pythonpath or something? How do I reference moduleA from test?
I would like to avoid making 'dummy' into a package as well.
Jens
Problem solved. I added 'dummy' to the PYTHONPATH. (Do I really have
to do that for every project I create?) Anyway, it works the way I'd
like it to now.
Ben Finney
> On 21 Nov., 04:16, Jens <j3n...@gmail.com> wrote:
> > On 21 Nov., 01:46, brzr...@gmail.com wrote:
> > dummy/
> > dummy_package/
> > __init__.py
> > moduleA.py
> > tests/
> > __init__.py
> > test.py
To avoid confusion, the directory that is the package should be named
as you want the imports to appear; e.g. if you want to 'import
foo.module_a', then name the directory containing 'module_a.py' as
'foo/'.
This often results in::
foo/
setup.py
foo/
__init__.py
module_a.py
test/
__init__.py
test_module_a.py
That is, the *project* directory (containing all the files) is named
'foo/'; the *package* directory (where all the implementation code is
found) is named 'foo/foo/', and the unit tests are found in the
directory 'foo/test/'.
That's normal, though if it confuses you you might want to rename the
project directory. I'm often doing development on multiple
version-control branches, so each project directory is named for the
branch it contains; within each of those, the same 'foo/' name is used
for the package directory.
> > I'm using Python 2.5.1. When I'm trying to call a function in
> > 'moduleA' from 'test' it won't work unless I make the 'dummy'
> > folder a package as well. That's pretty weird. Does
> > 'dummy_package' have to be in my pythonpath or something? How do I
> > reference moduleA from test?
You should install the package into a place where Python's import path
will find it. Read up on using the standard library 'distutils'
mechanism for this; it involves writing a 'setup.py' file to define
the parameters for installation of your package.
> > I would like to avoid making 'dummy' into a package as well.
Yes. The top-level directory is used for containing a number of files,
including 'setup.py', that should not be part of the installed
package.
> Problem solved. I added 'dummy' to the PYTHONPATH. (Do I really have
> to do that for every project I create?) Anyway, it works the way I'd
> like it to now.
I hope the above makes it clearer what I prefer for this situation.
--
\ "True greatness is measured by how much freedom you give to |
`\ others, not by how much you can coerce others to do what you |
_o__) want." --Larry Wall |
Ben Finney
Sergio Correia
with pythonpath. If you are not familiar with PTH files, what I do is
this
1) Go to "C:\Program Files\Python25\Lib\site-packages" or whatever is
appropiate in your case.
2) Create a text file, name it something like "MyProjects.PTH" (note
the extension!)
3) in the file, just write the path of the folder that contains all
your projects (in my case, C:/docs/python)
The idea is to keep the main python installation separated from the
modules you are currently developing. Your python installation goes to
"program files/python" or "bin/python", and your personal projects go
somewhere else (usually inside your 'user' folder). This smooths many
things, like working with different versions of a package you are
developing.
Jens
wrote:
>
> I hope the above makes it clearer what I prefer for this situation.
>
It does. You've been most helpful - thanks a lot! I'll bookmark this
thread and keep it under my pillow :-)
BlueBird
> As a side note, I find much easier to drop a PTH file than messing
> with pythonpath. If you are not familiar with PTH files, what I do is
> this
>
> 1) Go to "C:\Program Files\Python25\Lib\site-packages" or whatever is
> appropiate in your case.
> 2) Create a text file, name it something like "MyProjects.PTH" (note
> the extension!)
> 3) in the file, just write the path of the folder that contains all
> your projects (in my case, C:/docs/python)
>
> The idea is to keep the main python installation separated from the
> modules you are currently developing. Your python installation goes to
> "program files/python" or "bin/python", and your personal projects go
> somewhere else (usually inside your 'user' folder). This smooths many
> things, like working with different versions of a package you are
> developing.
>
Hi,
If I understand you correctly, you have the following directory
organisation
[Python installation]/site-packages/[MyProject.pth pointing to /home/
user/python-dev/]
And then you do your development in python-dev. But how do you manage
multiple development branches of the same program ?
My directory structure looks like this:
python-dev:
+ prog-branch1/
| + foo
| | + foo.py
| | + tests
| | + test_foo.py
| + bar
| + bar.py
| + tests
| + test_bar.py
+ prog-branch2/
+ foo
| + foo.py
| + tests
| + test_foo.py
+ bar
+ bar.py
+ tests
+ test_bar.py
bar/bar.py needs to import symbols from foo/foo.py . And bar/tests/
test_bar.py needs some symbols from both bar/bar.py and foo/foo.py
I don't understand how having python-dev in the .pth file solves the
problem.
In my case, I make all my imports relative to the root of my project:
bar/bar.py:
from foo.foo import some_foo
bar/tests/test_bar.py
from foo.foo import some_foo
from foo.bar import some_bar
The way I managed to make it work is by extending sys.path but I would
be happy to find a better solution:
bar/bar.py is actually:
import os, sys
sys.path.append( '..')
from foo.foo import some_foo
and bar/tests/test_bar.py is actually:
import os, sys
sys.path.append( os.path.join('..','..') )
from foo.foo import some_foo
from foo.foo import some_bar
What is not nice with this method is that every runnable test script
must extend unconditionally the sys.path . That creates some clutter.
Philippe
A.T.Hofkamp
> On Nov 21, 7:05 am, "Sergio Correia" <sergio.corr...@gmail.com> wrote:
> And then you do your development in python-dev. But how do you manage
> multiple development branches of the same program ?
If you are using SVN, you may want to check out 'combinator' by Divmod
(divmod.org). That tool manages SVN branches, and also switches Python when you
switch branches. Very handy.
Albert
Sergio Correia
If you are using python 2.5, relative imports are no longer an issue:
http://docs.python.org/whatsnew/pep-328.html
That problem solved, what you sometimes want is to change the version
of your package. I just change the text in the PTH file, to point to
another version, and voilá (no need to move files around or to mess
with the code in the package itself). Probably you can write a script
that changes the PTH file from python itself.
Albert:
Thanks, that looks useful.
mi...@hl.id.au
> have to be a package?
As others have suggested, this is a good way to organise your tests.
To avoid problems with the import path, look at nosetests [1]. This
allows you to run::
nosetests
main_package_name.subpackage1.test.test1:TestSomeClass.test_func
to run the test_func function in the TestSomeClass test case.
Or alternatively::
nosetests main_package_name.subpackage1
to run all tests contained in the main_package_name.subpackage1
module.
This allows you to remove all of your 'if __name__ ==
"__main__":unittest.main()' code, as well as gets rid of the problem
with relative imports (which others solved using .pth files).
The only downside for me is that running nosetests takes at least half
a second (eg: import nose), compared to making the tests runnable
which can be _very_ quick. For test driven development it's nice to
have tests really really quick.
One gotcha; if you're converting from a 'if __name__ ==
"__main__":...' system; nosetests by default ignores files that are
executable (so you'll need to chmod -x main_package_name/subpackage1/
test/test1.py).
[1] nose: a discovery-based unittest extension - (http://
somethingaboutorange.com/mrl/projects/nose/).
BlueBird
> Bluebird:
>
> If you are using python 2.5, relative imports are no longer an
> issue:http://docs.python.org/whatsnew/pep-328.html
It does not solve my problem, or I missed something:
User@Phil_vaio /cygdrive/d/work/work/vy-dev/foo $ find . -name '*.py'
./bar/bar.py
./bar/tests/test_bar.py
./bar/tests/test_bar1.py
./bar/tests/__init__.py
./bar/__init__.py
./foo/foo.py
./foo/tests/__init__.py
./foo/__init__.py
./__init__.py
d:\work\work\vy-dev\foo>C:\Python25\python.exe bar/tests/test_bar.py
d:\work\work\vy-dev\foo
['d:\\work\\work\\vy-dev\\foo\\bar\\tests', 'C:\\Python25\\lib\\site-
packages\\h
gsvn-0.1.4-py2.5.egg', 'C:\\WINDOWS\\system32\\python25.zip', 'C:\
\Python25\\DLL
s', 'C:\\Python25\\lib', 'C:\\Python25\\lib\\plat-win', 'C:\\Python25\
\lib\\lib-
tk', 'C:\\Python25', 'C:\\Python25\\lib\\site-packages', 'C:\\Python25\
\lib\\sit
e-packages\\win32', 'C:\\Python25\\lib\\site-packages\\win32\\lib', 'C:
\\Python2
5\\lib\\site-packages\\Pythonwin', 'C:\\Python25\\lib\\site-packages\
\wx-2.8-msw
-ansi']
Traceback (most recent call last):
File "bar/tests/test_bar.py", line 8, in <module>
from ..bar import bar_f
ValueError: Attempted relative import in non-package
BlueBird
> On Nov 26, 5:07 pm, "Sergio Correia" <sergio.corr...@gmail.com> wrote:
>
> >Bluebird:
>
> > If you are using python 2.5, relative imports are no longer an
> > issue:http://docs.python.org/whatsnew/pep-328.html
>
> It does not solve my problem, or I missed something:
>
Any idea what could be wrong in my code (in the previous message). Or
did I misunderstand relative import seriously ?
Philippe
Marc Christiansen
Does this give you an idea?
0:tolot:/tmp> mkdir t1
0:tolot:/tmp> touch t1/__init__.py
0:tolot:/tmp> echo print __name__ >t1/test.py
0:tolot:/tmp> python t1/test.py
__main__
0:tolot:/tmp> python -c "import t1.test"
t1.test
Marc