How python source code in large projects are organized?

[Peng Yu]

Hi,

I am wondering what is the best way of organizing python source code
in a large projects. There are package code, testing code. I'm
wondering if there has been any summary on previous practices.

Regards,
Peng

[Daniel Fetchinson]

> I am wondering what is the best way of organizing python source code
> in a large projects. There are package code, testing code. I'm
> wondering if there has been any summary on previous practices.

I suggest looking at the source code of large projects like twisted,
PIL, django, turbogears, etc. These have different styles, pick the
one you like best.

HTH,
Daniel

--
Psss, psss, put it down! - http://www.cafepress.com/putitdown

[Peng Yu]

On Sun, Sep 20, 2009 at 11:31 AM, Daniel Fetchinson
<fetch...@googlemail.com> wrote:
>> I am wondering what is the best way of organizing python source code
>> in a large projects. There are package code, testing code. I'm
>> wondering if there has been any summary on previous practices.
>
> I suggest looking at the source code of large projects like twisted,
> PIL, django, turbogears, etc. These have different styles, pick the
> one you like best.

Is there a webpage or a document that describes various practices?

Regards,
Peng

[exa...@twistedmatrix.com]

I wrote very briefly on this topic, hope it helps:

http://jcalderone.livejournal.com/39794.html

Jean-Paul

[Christopher Arndt]

On 20 Sep., 22:10, exar...@twistedmatrix.com wrote:

> On 07:10 pm, pengyu...@gmail.com wrote:
>
> >On Sun, Sep 20, 2009 at 11:31 AM, Daniel Fetchinson
> ><fetchin...@googlemail.com> wrote:
> >>>I am wondering what is the best way of organizing python source code
> >>>in a large projects. There are package code, testing code. I'm
> >>>wondering if there has been any summary on previous practices.
>
> >>I suggest looking at the source code of large projects like twisted,
> >>PIL, django, turbogears, etc. These have different styles, pick the
> >>one you like best.

Start by having a standard for structuring the single Python files. I
like to have mine organized like this:

- (shebang)
- encoding declaration
- module docstring
- __all__ symbol export list
- imports
- standard library
- third-party
- project specific
- author, version, date and copyright information
- globals (logging setup, constants, etc.)
- utility functions
- classes
- "main" function (some like to have this at the top)
- if __name__ == '__main__': clause

For sub-packages, like to have the following layout:

mainpackage/
subpackage2/
test/test_subpackage2.py
__init__.py
base.py
exceptions.py
module1.py
module1.py
subpackage2/
...
__init__.py
mainpackage.py

If a package is not too big, the test may also go into a "test" sub-
directory of the main package.

I have an __all__ symbol export list in all module file, so then in in
__init__.py I can do:

from base import *
from exception import *
from module1 import this, that
from module2 import foo, bar

without worrying about import too much and in my main package I can
use:

from package import this, bar

And in e.g. package.module1 I can do

from package.exceptions import SomeError

without having to worry about circular imports.

If my package has any code to be called directly from the command
line, I create a script in the root of the distribution or in a "bin"
subdirectory:

mydistro/
bin/
script1
mainpackage/
...

and in that script:

#!/usr/bin/env python

from mainpackage import run
# or
# from mainpackage.command.mycommand import run
run()

you can also add a command line script entry point to your setup
script to let this script be automatically created on installation by
setuptools/easy_install. Look at the TurboGears 1.1 branch for
examples.

HTH, Chris

[Jonathan Gardner]

(Sorry for the late reply.)

My advice: Don't write big projects. Not that Python can't do it, but
because it's a bad idea in general. Instead, write lots of small,
independent projects that have well-defined, simple interfaces and
work together.

First step is to map out what components you have and what their
interfaces are. (The interface is what functions and classes exist and
what their behavior is.) Nail this down early on.

Try to reuse components from other projects. If something exists but
it isn't up to par with what you need, adapt your interface
requirements or consider contributing to the project.

Next step is to build your scaffolding. For each component you'll be
building, build up the interface and leave out the implementation. If
possible, put in something reasonable but incorrect. For instance:

def twice(x): return 1

If that's not possible:

def twice(x): raise NotImplementedError

(If you think of 'twice' as something complicated like
'standard_deviation', then you'll appreciate this more.)

Next, get the stack "working". Although it will be incorrect, it
should do something visible.

Finally, flesh out the scaffolding by replacing the wrong
implementations with correct ones, working from basic components to
higher level components. Keep the stack working at all times.

When writing unit tests, test only the code in the component and not
code from another component.

The structure of individual components or modules should be standard,
not just for your project but for the vast majority of projects out
there. Use paster to get your code setup. This will give you a good
standard template to work from. Create a virtualenv to get your
working environment. Then run "python setup.py develop" from each of
the modules to install them in the working environment. If you have
the dependencies mapped out, it should install all the stuff you need.

In the end, you should get something that looks more like Pylons and
less like Django. Projects built in this way tend to have more
replaceable components and the components tend to be more useful
outside of the project. That means you write less code and get more
work done.