In writing some non-trivial amount of Python code I keep running into
an organizational issue. I will try to state the problem fairly
generally, and follow up with a (contrived) example.

The root cause of my difficulties is that by default, the relationship
between a module hierarchy and the structure of files on disk is too
strong for my taste. I want to separate the two as much as possible,
but I do not want to resort to non-conventional "hacks" to do it. I am
posting this in an attempt to present what I perceive to be a
practical problem, and to get suggestions for solutions, or opinions
on the most practical policy for how to deal with it.

Like I said, I would like a weaker relationship between file system
structure and module hierarchy. In particular there are two things I
would like:

  * Least importantly, I don't like jamming code into __init__.py,
    as a personal preference.
  * Most importantly, I do not like to jam large amounts of code
    into a single source file, just for the purpose of keeping
    the public interface in the same package.

An contrived but hopefully illustrative example:

We have an organization "Org", which has a library, and as part of
that library is code that relates to doing something with animals. As
a result, the interesting top-level package for this example is:

   org.lib.animal

Suppose now that I want an initial implementation of the most
important animal. I want to create the class (but see [1]):

   org.lib.animal.Monkey

The public interface consists of that class only (and possibly a small
handful of functions). The implementation is quite significant however
- it is 500 lines of code long.

At this point, we had to jam those 500 lines of code into
__init__.py. Let's ignore my personal preference of not liking to put
code in __init__.py; the fact remains that we have 500 lines of code
in a single source file.

Now, we want to continue working on this library, adding ten
additional animals.

At this point, we have these choices (it seems to me):

  (1) Simply add these to __init__.py, resulting in
      __init__.py being 5000 lines long[2].

  (2) Put each animal into its own file, resulting in
      org.lib.animal.Monkey now becoming
      org.lib.animal.monkey.Monkey, and animal X becoming
      org.lib.animal.x.X.

The problem I have is that both of these solutions are, in my opinion,
very ugly:

* (1) is ugly from a source code management perspective, because jamming
  5000 lines of code for ten different animals into a single file
  is bad for obvious reasons.

* (2) is ugly because we introduce org.lib.animal.x.X for
  animal X, which:
    (a) is redundant in terms of naming
    (b) redundant in function since we have a single package for
        each animal containing nothing but a single class of
        the same name

Clearly, (1) is bad due to file/source structure reasons, and (2) is
bad for module organizational reasons. So we are back to my original
wish - I want to separate the two, so that I can solve (1)
indepeendently of (2).

Now, I realize that __init__.py can contain arbitrary code, and that
one can override __import__. However, I do not want to resort to
"hacks" just to solve this problem; I would prefer some established
convention in the community, or at least something that is elegant.

Does are people's thoughts on this problem?

Let me just shoot down one possible suggestion right away, to show you
what I am trying to accomplish:

I do *not* want to simply break out X into org.lib.animal.x, and have
org.lib.animal import org.lib.animal.x.X as X. While this naively
solves the problem of being able to refer to X as org.lib.animal.X,
the solution is anything but consistent because the *identity* of X is
still org.lib.animal.x.X. Examples of way this breaks things:

  * X().__class__.__name__ gives unexpected results.
  * Automatically generated documentation will document using the "real"
    package name.
  * Moving the *actual* classes around by way of this aliasing would
    break things like pickled data structure as a result of the change
    of actual identity, unless one *always* pre-emptively maintains
    this shadow hierarchy (which is a problem in and of itself).

Thus, it's not clean. It breaks the module abstraction and as a result
has unintended consequences. I am looking for some kind of clean
solution. What do people do about this in practice?

[1] Optionally, we might introduce an "animals" package such that it
would become org.lib.animal.animals.Monkey, if we thought we were
going to have a lot of public API outside of the animals themselves.
This does not affect this dicussion however, as the exact same thing
would apply to org.lib.animal.animals as applies to org.lib.animal in
the above example.

[2] Ignoring for now that it may not be realistic that every animal
implementation would be that long; in many cases a lot of code would
be in common. But feel free to substitude for something else (a Zoo
say).

--
/ Peter Schuller

Please note that this is not any kind of crazy-brained idea; lots of
languages have absolutely zero relationship between file location and
modules/namespaces.

I realize that technically Python does not have this either. Like I
said in the original post, I do realize that I can override __import__
with any arbitrary function, and/or do magic in __init__. But I also
did not want to resort to hacks, and would prefer that there be some
kind of well-established solution to the problem.

Although I was originally hesitant to use an actual example for fear
of giving the sense that I was trying to start a language war, your
answer above prompts me to do so anyway, to show in concrete terms
what I mean, for those that wonder why/how it would work.

So for example, in Ruby, there is no problem having:

File monkey.rb:

module Org
  module Lib
    module Animal
      class Monkey ...
        ..
      end
    end
  end
end

File tiger.rb:

module Org
  module Lib
    module Animal
      class Tiger ...
        ..
      end
    end
  end
end

This is possible because the act of addressing code to be loaded into
the interpreter is not connected to the namespace/module system, but
rather to the file system.

Some languages avoid (but does not eliminate) the problem I am having
without having this disconnect. For example, Java does have a strong
connection between file system structure and class names. However the
critical difference is that in Java, everything is modeled around
classes, and class names map directly to the file system structure. So
in Java, you would have the class

   org.lib.animal.Monkey

in

   <wherever>/org/lib/animal/Monkey.java

and

   org.lib.animal.Tiger

in

   <wherever>/org/lib/animal/Tiger.java

In other words, introducing a separate file does not introduce a new
package. This works well as long as you are fine with having
everything related to a class in the same file.

The problem is that with Python, everything is not a classes, and a
file translates to a module, not a class. So you cannot have your
source in different files without introducing as many packages as you
introduce files.

--
/ Peter Schuller

from org.lib.animal.monkey import Monkey
Monkey.__module__ = 'org.lib.animal'

(Which, I must admit, is not a bad idea in some cases.)

py> from xml.etree import ElementTree
py> import xml.etree.ElementTree as ET2
py> import xml.etree
py> ET3 = getattr(xml.etree, 'ElementTree')
py> ElementTree is ET2
True
py> ET2 is ET3
True

Ok, there is one exception: the main script is loaded as __main__, but if  
you import it using its own file name, you get a duplicate module.
You could confuse Python adding a package root to sys.path and doing  
imports from inside that package and from the outside with different  
names, but... just don't do that!

Java does still suffer from the same problem since it forces "class ==
file" (well, "public class == file"). However it is less of a problem
since you tend to want to keep a single class in a single file, while
I have a lot more incentive to split up a module into different files
(because you may have a lot of code hiding behind the public interface
of a module).

So essentially, Java and Python have the same problem, but certain
aspects of Java happens to mitigate the effects of it. Languages like
Ruby do not have the problem at all, because the relationship between
files and modules is non-existent.

--
/ Peter Schuller

The module where org.lib.animal.Monkey is actually defined should be
an implementation detail of the library, but simply importing Monkey
into org.lib.animal doesn't quite make it one.

If a user pickles a Monkey class, and then the OP decides to refactor
the Monkey class into a new module (say
org.lib.animal.primate.monkey), then the user would not be able to
unpickle it.  Because, you see, pickles record the module a class is
defined in.  So, now the user has to worry about where Monkey is
actually defined.  It is not an implementation detail.

The solution is to modify the class's __module__ attribute as well as
importing it, as I've already pointed out:

from org.lib.animal.monkey import Monkey
Monkey.__module__ = 'org.lib.animal'

This should be enough to satisfy the OP's requirements, at least for
classes, without softening the one-to-one module-to-file relationship,
or using "hacks".

In fact, I'd say this is good practice.

--
/ Peter Schuller

I will probably have to read up more on the semantics of __import__
and related machinery.

--
/ Peter Schuller

If you set __module__, the only remaining discernable difference is
that the global variables accessed from the Monkey class will be in
org.lib.animal.monkey instead of org.lib.animal.  This has no ill
effects when unpickling or instantiating the class from
org.lib.animal.

[animals]$ ls
animals
[animals]$ rm animals/*.pyc
[animals]$ ls
animals
[animals]$ ls animals
__init__.py  monkey.py
[animals]$ cat animals/monkey.py
class Monkey(object):
     pass
[animals]$ cat animals/__init__.py
from animals.monkey import Monkey
Monkey.__module__ = 'animals'
[animals]$ python
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> from animals import Monkey
 >>> import inspect
 >>> inspect.getsource(Monkey)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/inspect.py ",
line 629, in getsource
     lines, lnum = getsourcelines(object)
   File
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/inspect.py ",
line 618, in getsourcelines
     lines, lnum = findsource(object)
   File
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/inspect.py ",
line 494, in findsource
     raise IOError('could not find class definition')
IOError: could not find class definition
 >>>

--
Robert Kern

The other follow-up to your E-Mail points out a possible problem for
example. I would not have come up with that, but that does not mean
the effect does not exist. And committing to a solution that "seems to
work", only to break massively for some particular use case in the
future, is exactly why I don't want a "hack" for a solution.

I don't know Python internals enough to state of believe with any
authority wither, let's say, stomping __module__ and hacking
sys.modules would be enough to *truly* do it correctly in a proper way
such that it is entirely transparent. This is why I care about whether
it truly changes the real identity of the class; it's not about
satisfying my particular list of examples (because they *were* just
examples).

--
/ Peter Schuller

I call that a bug in the inspect module. In fact, looking at the source
for the findsource() function, I can see no fewer than two bugs, just in
the way it handles classes:

(1) it assumes that the only way to create a class is with a class
statement, which is wrong; and

(2) it assumes that the first occurrence of "class <name>" must be the
correct definition, which is also wrong.

It isn't hard to break the inspect module. Here's an example:

Here's the source of broken.py:

$ cat broken.py
"""Here is a doc string, where I happen to discuss the
class Parrot which will be defined later.
"""
class Parrot:
    pass

class Wensleydale: # THIS IS GONE
    pass

del Wensleydale
class Wensleydale(object):  # but this exists
    pass

It isn't often that I would come right out and say that part of the
Python standard library is buggy, but this is one of those cases.

There is a very good reason to care *in practice*: if there is code out
there that assumes that the source code from Monkey is in the file it was
found in. In practice, you might be stuck needing to work around that.
But that's not a good reason to care *in principle*. In principle, the
actual location of the source code should be an implementation detail of
which we care nothing. It's possible that the source for Monkey doesn't
exist *anywhere*.

It is important to deal with buggy tools. But the correct way to do so is
to fix the bugs, not to throw away perfectly good abstractions.

I think you're slipping to a level of paranoia that's more harmful
that helpful now.

The ironic thing is, breaking the one-to-one module-to-file
relationship is more likely to have "unintended consequences", by a
very large margin.  Python has always been one-to-one module-to-file
(excepting modules built into the interpretter), and many codes and
tools have come to depend on it.

Thanks!

--
/ Peter Schuller

I'll have a closer look at the suggested practice of modifying __module__.

For this particular use case we probably won't end up doing that, but it
may come to be useful in the future.

--
/ Peter Schuller

After that, nothing depends on the physical location on disk, unless
it's buggy. Imported modules are available from 'sys.modules', and
that's where subsequent 'import' statements will find them, with no
reference to file locations.

I would agree with Carl that modifying __module__ in the way he
suggests is legitimate - if it breaks the inspect module, then it is
the inspect module that needs fixing (and/or better support from the
interpreter to help find the real source code).