Imports & PythonPath

[Steadicat]

Right now all the Django modules refer to each other via full module
names:

module = "myproject.settings.admin"

And somewhere inside Django's internals this module gets loaded:

__import__(module)

This system has a few disadvantages:

* All projects must be in the PythonPath.

* Projects cannot be easily renamed, cause every module reference
contains the project name.

* Command line scripts need the user to specify the starting module
explicitly every time thorugh the --settings switch (or even worse,
using the DJANGO_SETTINGS_MODULE environment variable which is giving
folks so many headaches).

* Any non-python dependency (such as templates, media...) has to be
specified in some other way with the *full path*: this means that a
site cannot be easily renamed, moved, moved to another computer (aargh
deployment nightmares!).

A more common model which works quite nicely is based on the idea of a
"site directory". The root folder of a project is specified in some way
(see later):

cwd = "/web/myproject"

Then all of the imports can be based on this directory without the need
to specify the project name explicitly:

sys.path.append(cwd)
import settings.admin

Given the site directory, it is easy to locate all the parts of the
site (configuration files, code, templates, media) relative to this
directory, using naming conventions where necessary.

For command line scripts, including the stand-alone server, the best
and most common way to specify the site directory is... the current
working directory. Just cd to it and you're set.

For mod_python you could have something like this:

SetEnv DJANGO_SITE /web/myproject

which is much simpler than what we have currently:

SetEnv DJANGO_SETTINGS_MODULE myproject.settings.main
PythonPath sys.path + ["/web/myproject"]

Advantages:

* No need to tamper with the PythonPath, one less step for newbies, and
a very error-prone one it seems (i still haven't managed to make
mod_python use PYTHONPATH on Windows).

* Moving or renaming projects is just about moving or renaming the
directory and updating the single reference to it in the httpd.conf.
This would no doubt simplify installation, development and deployment
in many scenarios. Yay for checkout & run!

* No need to set and update environment variables or specify verbose
command line switches such as "--settings myproject.settings.admin".

* There would be no need for things such as:

MEDIA_ROOT = "/home/html/djangoproject.com/m/"

All paths to media and templates could be relative to the site
directory. They could even have default locations (such as "./media"
and "./templates") which would make things simpler for simple cases.

Rails, CherryPy and pretty much every other Python web framework gets
this right. I've tried most of them without having to touch neither the
PythonPath nor the environment variables. Now Django requires me to do
both...

If there are no serious objections, I think this should be fixed before
too many people start relying on the current model.

Cheers,

Stefano J. Attardi

[Simon Willison]

I really, really like this idea. Anyone think of a reason it wouldn't
work?

Cheers,

Simon

[Jacob Kaplan-Moss]

On Jul 20, 2005, at 7:33 PM, Simon Willison wrote:

> I really, really like this idea. Anyone think of a reason it
> wouldn't work?
>

I really do too; I'm just wanted to let it ruminate a bit while I
think about it. There's something in this thing on top of my neck
that's trying to make itself known, so I'm going to give it a few
more hours to work itself out.

It's a good idea, though, and certainly the right idea.

Jacob

[robin...@gmail.com]

If you're willing to require Python 2.4 or above, there's always the
option of doing relative imports:
http://www.python.org/peps/pep-0328.html. Even if you go with the "site
directory" approach, you're still going to want to be aware of this
PEP, because by Python 2.5, non-relative imports are going to start
generating DeprecationWarnings.

Relative imports, for those who haven't yet read PEP 328, look like
this:

# Import a "sister" module (in the same directory as me)
from . import othermod
# Import various functions from a "sister" module
from .othermod import spam, eggs
# Import a module from a parent package
from .. import settings
# Go up one package level, then down a different tree path
from ..settings.admin import ROOT_URLCONF

Is this what your brain was trying to remind you about?

[Adrian Holovaty]

This is an interesting solution. Some thoughts...

* I'm always wary of altering the PYTHONPATH. It opens a can of worms
in terms of potential namespace collisions. Look at the directories
inside a Django project: "settings", "apps", "views", "models". All are
generic terms, and I wouldn't be comfortable assuming there wasn't a
top-level "settings" or "views" module elsewhere on the PYTHONPATH. It
just seems really dirty to assume that.

You're saying Django code should do things like "from settings import
FOO" and "from views.polls import poll_detail"? Please correct me if
I'm misunderstanding this.

* Apps should be able to refer to other apps. This has been built-in to
the system from the beginning. The philosophy behind apps are that
they're "pluggable" -- they should be able to be dropped into another
installation. Granted, the default directory structure created by
"django-admin.py startapp" puts "apps/" *within* the project -- we
should probably change that, so that apps' package names don't have
project names in them. Any suggestions on a different directory
structure?

In the code we (World Online) use in production, we don't have any
concept of "projects" -- we just have a separate package of settings
and URL configurations ("site-packages/worldonline_settings"), and all
our Django apps go within their own package
("site-packages/ellington"). So it's "from ellington.polls.views.polls
import poll_detail", anywhere on the system. All apps can refer to
other apps.

Here's an example of the directory structure we use.

site-packages/
ellington/
polls/
models/
views/
news/
models/
views/
worldonline_settings/
lawrence.py
ljworld.py
urls/
lawrence.py
ljworld.py
templates/
lawrence.com/
ljworld.com/

As you can see, there are three distinct ideas here: apps, settings and
templates. The intention of "django-admin.py startproject" and
"startapp" was to make it easy to get up and running, but it looks like
the setup could be improved.

What's the best way of making it easy for people to get up and running
right away, while maintaining a clear separation among settings (which
are site-specific), apps (which should be able to be "dropped" into any
Django installation) and templates (which are site-specific but also
tied to a particular app)? I'm convinced that the concept of "projects"
isn't the best solution.

Adrian

[Jacob Kaplan-Moss]

On Jul 21, 2005, at 8:52 PM, Adrian Holovaty wrote:

> * I'm always wary of altering the PYTHONPATH. It opens a can of worms
> in terms of potential namespace collisions. Look at the directories
> inside a Django project: "settings", "apps", "views", "models". All
> are
> generic terms, and I wouldn't be comfortable assuming there wasn't a
> top-level "settings" or "views" module elsewhere on the PYTHONPATH. It
> just seems really dirty to assume that.
>
> You're saying Django code should do things like "from settings import
> FOO" and "from views.polls import poll_detail"? Please correct me if
> I'm misunderstanding this.
>

I should point out that Django -- before it really was Django -- used
to use this; code used to do "import settings" and rely on PYTHONPATH
being pointed to the correct settings module. This turned out not to
work, mostly because you can -- and we do -- use a single admin
interface to manage multiple sites.

For the interested, ADMIN_FOR in the admin settings module came out
of this same thinking; it's incredibly useful for an admin interface
to "know" about the sites it supports. Without this "knowledge",
neither the template validators nor the automatic tag/filter/model/
view documentation would work.

> What's the best way of making it easy for people to get up and running
> right away, while maintaining a clear separation among settings (which
> are site-specific), apps (which should be able to be "dropped" into
> any
> Django installation) and templates (which are site-specific but also
> tied to a particular app)? I'm convinced that the concept of
> "projects"
> isn't the best solution.
>

I'll also point out that templates don't have to be site-specific; we
use a "default" template directory with default templates for certain
content types; this rocks for rapidly rolling out features across
multiple sites.

In general, when it comes to things like this, I'm all for making
things easier for first-time developers, but it absolutely can't
impeed advanced users. Remember, PHP is extremely easy for first-
timers, but can you imagine writing Django in PHP? My point is that
there's a necessary cutoff in skill-level for Django users, and I'd
rather that cutoff be higher rather than losing any of the power I've
grown addicted to.

Jacob

[Robin Munn]

Whoops, ignore my suggestion earlier about relative imports. PEP 328
says they made it into Python 2.4, but that's false --
http://python.org/doc/2.4/whatsnew/node10.html says "no relative
imports until Python 2.5 at the earliest," and so does Guido in
http://mail.python.org/pipermail/python-dev/2005-February/051762.html:
"the absolute/relative import part of PEP 328 didn't make it into 2.4
..."

So relative imports won't work. Not yet, anyway.

--
Robin Munn
Robin...@gmail.com
GPG key 0xD6497014