How to improve the usability of nested packages

[Michael Schwarz]

I need some guidance on how to structure a large library with lots of
packages and sub-packages. Using the library should be as effortless
as possible. It's your average beast of a catch-all for sharing code
across applications at our company. Let's call it "the_library". In my
attempt to structure the library, I was following two principles:

- The complete library is contained in a single package. This is to
avoid polluting the top-level namespace.
- Only modules and sub-packages directly under the top-level package
should be imported directly. This means that any class or function in
the library is accessed using the same qualified name everywhere
inside the library or the application. This makes moving code around
easier.

Following this, using a module from the library is pretty
straight-forward. A typical file in the application code could start
with:

from the_library import sip, rtp, sdp

This works from any module or script in the library or application.
Then I decided to split the "sip" module into smaller modules, e.g.
"message", "transaction", "dialog", all contained in a package named
"sip". Ideally, an application would still import the sip package
using the import above and then, for example, access the "DialogID"
class using "sip.dialog.DialogID". Currently this is only possible
when also explicitly importing the "dialog" module:

from the_library import sip
import the_library.sip.dialog

This is ugly and seems unnecessary to me as, for example, having all
the modules in the "sip" package available using a single import would
not pollute the local namespace. So I tried to enable this by
importing all the modules in the "sip" package from the package's
"__init__.py":

from . import message, transaction, dialog

… which doesn't work. Some of the modules reference other modules in
the same package. I'm not talking about cyclic references, but, for
example, the "dialog" module uses the "transaction" module. The
problem is that the "dialog" module uses the same mechanism shown
above to import the other modules from it's package. This means that
modules and packages are imported in this order:

- Application code executes "from the_library import sip"
- the_library/__init__.py is executed. No imports here.
- the_library/sip/__init__.py executes "from . import [...], dialog"
- the_library/sip/dialog.py executes "from the_library import sip"

During the last import a problem arises: The module object for the
package "the_library" does not yet have a "sip" member (as it is still
executing the import) and so the import fails. It is still possible to
import the "transaction" module directly from the "dialog" module
using:

from . import transaction

But this would make the "transaction" module available under a
different qualified name as anywhere else (where it's accessed using
"sip.transaction").

What path would you take to circumvent this problem? Would you break
the rule that any module should be accessed using the same way, no
matter from where it is accessed, or would you maybe structure the
library entirely different?

Thanks for any suggestions!

Michael

[Terry Reedy]

nested package == subpackage

> or would you maybe structure the library entirely different?

Based on my limited experience with subpackages* plus reports on this
list about problems, such as yours, I have concluded that subpackages
are an attractive nuisance that are generally more trouble than they are
worth. I suggest you consider sticking with your original flat (no
subpackage) design. (But maybe someone knows better than me how to make
subpackages work ;-).

* In my current project, I started with all modules in subpackages but
have since moved them into the package and deleted the subpackages.

--
Terry Jan Reedy

[Stefan H. Holek]

Hi Michael,

What we have learned from creating the Zope Toolkit (formerly Zope 3), is that __init__.py files in namespace packages should be empty, and imports should be absolute. [1]

That said, there are ways to avoid import cycles. One is to very carefully craft your modules so they do not have to import from each other. Another is to not have imports at the module level, but move them into the functions where they are required. Third, large libraries like the Zope Toolkit usually have mechanisms to defer imports to some point after initial loading. You may want explore this direction as well. [2]

(Not trying to plug the ZTK here, it just happens to be a large, namespace-using library I know.)

Hope this helps,
Stefan

[1] http://docs.zope.org/zopetoolkit/
[2] http://pypi.python.org/pypi/zope.deferredimport

--
Stefan H. Holek
ste...@epy.co.at

[Rouslan Korneychuk]

On 11/02/2012 12:11 PM, Michael Schwarz wrote:
> … which doesn't work. Some of the modules reference other modules in
> the same package. I'm not talking about cyclic references, but, for
> example, the "dialog" module uses the "transaction" module. The
> problem is that the "dialog" module uses the same mechanism shown
> above to import the other modules from it's package. This means that
> modules and packages are imported in this order:
>
> - Application code executes "from the_library import sip"
> - the_library/__init__.py is executed. No imports here.
> - the_library/sip/__init__.py executes "from . import [...], dialog"
> - the_library/sip/dialog.py executes "from the_library import sip"

In a way, you do have a cyclic reference. If you think of "import sip"
as "from sip import __init__ as sip", it should become apparent.


Anyway, I see two ways around this. One is to rename the package and
create a module under the_library with the old package name that imports
what you want. To keep using the same names, every module in the package
can import siblings like so (where sippkg is the new package name):
import the_library.sippkg.transaction
sip = the_library.sippkg


The second way is to not use import at all to load sibling modules
(except in __init__.py), and instead use:
sip = sys.modules['the_library.sip']

This will work as long as you are careful to load the modules from
__init__.py in the correct order, where each module's dependencies must
appear before the module, in the import list.