modularity of apps
Todd O'Bryan
coupling and the modularity of apps.
In particular, how plugable are apps supposed to be? It's clear that
at the top level, a project should be able to access all of its
parts, but it seems like it should be the case that each individual
app should be as self-contained as possible, ideally relying only on
its own code and code supplied by django.
But adding an app to a project isn't as simple as just adding it. In
addition to adding it to INSTALLED_APPS, you also have to figure out
if it's self-contained or relies upon other apps. If it's templates
depend on a context_processor having been called, you have to add
that to your project's TEMPLATE_CONTEXT_PROCESSORS. Also, an app may
need to know where it's located in the projects URL space so that it
can create links to other parts of itself (at least, if you don't
want a bunch of "../../../lots/of/updirectory.html" kinds of URLs.
So, as I've been thinking about this stuff, I've been wondering if
some of this could be made less piecemeal and more systematic.
Here are some thoughts, which are pretty raw, but I'd love people's
feedback.
1. Allow people to set a prefix for each app in the settings.py file.
This would basically tell the app what URLs that are mapped to it
start with, and views and templates inside the app could use this
when they want to create internally mapped URLs. By default, it would
also pass control of any URL starting with the prefix to the given
app's urls.py module, much as already happens, I imagine, in most
people's implementations, but the prefix would be an attribute of the
app, rather than some unconnected string as it is now.
2. Allow each app to optionally specify a context_processor(request)
function in their __init__.py file. By default, all the context
processors that exist would be called in the order that the apps are
listed in the INSTALLED_APPS attribute. Obviously, this could be
dangerous because apps could try to define different values in the
context using the same key. Maybe we could do some kind of fake
namespacing ( {{ app.key }} to avoid that.
3. Develop some kind of explicit way that apps can declare
dependencies on other apps so that apps can just be dropped in with
any dependencies, their prefixes set, and they be good to go.
Or maybe we don't need any of this and I'm just being overly anal
retentive.
Todd
Malcolm Tredinnick
On Sat, 2006-07-29 at 19:42 -0400, Todd O'Bryan wrote:
> As our application is growing larger, we're noticing some issues with
> coupling and the modularity of apps.
>
> In particular, how plugable are apps supposed to be? It's clear that
> at the top level, a project should be able to access all of its
> parts, but it seems like it should be the case that each individual
> app should be as self-contained as possible, ideally relying only on
> its own code and code supplied by django.
I don't completely agree with this, and I would wager that you aren't
really doing this either. Rather, a well-defined or easily
understandable dependency chain is really what we (as deployers) desire.
If I am installing any piece of software, an early question is going to
be "what else do I need to get this working?"
[...]
> So, as I've been thinking about this stuff, I've been wondering if
> some of this could be made less piecemeal and more systematic.
You indicated above that you are encountering some of these problems in
practice. With the current infrastructure, what have you found works
well or poorly at the moment?
> Here are some thoughts, which are pretty raw, but I'd love people's
> feedback.
>
> 1. Allow people to set a prefix for each app in the settings.py file.
> This would basically tell the app what URLs that are mapped to it
> start with, and views and templates inside the app could use this
> when they want to create internally mapped URLs. By default, it would
> also pass control of any URL starting with the prefix to the given
> app's urls.py module, much as already happens, I imagine, in most
> people's implementations, but the prefix would be an attribute of the
> app, rather than some unconnected string as it is now.
I'm not sure that a setting in settings.py is required, since you can
(and should) already do the same thing in urls.py. You configure the URL
prefix and use include(...) to include the app's own URL configuration.
However, that doesn't take care of the reverse URL mapping, as you
indicate above. Work on that area is being done by Adrian with his
reverse URL mapping stuff (there have been some commits on this
recently, too), including a template tag to allow templates easy access
to this information. Over time, we will no doubt enhance and extend the
initial work to accomodate more and more deviously twisted ... er...
important use cases. So, whilst this point is probably the most limiting
of the three you mention, it is also, fortunately, the one that is
having some real brain power applied to it.
> 2. Allow each app to optionally specify a context_processor(request)
> function in their __init__.py file. By default, all the context
> processors that exist would be called in the order that the apps are
> listed in the INSTALLED_APPS attribute. Obviously, this could be
> dangerous because apps could try to define different values in the
> context using the same key. Maybe we could do some kind of fake
> namespacing ( {{ app.key }} to avoid that.
I have some generic worries about whether this is too simple to work.
But based on my approximately zero experience with multiple apps
containing multiple processors, they don't have a lot of credibility.
I am wondering, though, if one solution might be to allow certain things
(like settings.TEMPLATE_CONTEXT_PROCESSORS) to be writable so that an
app's __init__.py could amend the config. The idea of mapping app names
to context processors lists and iterating through in the INSTALLED_APPS
order warrants some thought.
Slightly unspecific worries aside, I think a practice of namespacing any
"local" context changes would be a good idea to try.
> 3. Develop some kind of explicit way that apps can declare
> dependencies on other apps so that apps can just be dropped in with
> any dependencies, their prefixes set, and they be good to go.
I think this is possibly best solved by just importing the other
applications in __init__.py. That way, "manage.py validate" and any use
of the application will fail at the very first import. So if my weblog
app depends on my tag app, putting "import tagging" in
weblog/__init__.py enforces the dependency.
Just my personal feeling, but I think using Python's built-in dependency
enforcement like this is preferable to adding any extra infrastructure
to Django. I can't imagine we could do anything that would be much more
simple than "import foo" without imposing a burden on the app developer.
> Or maybe we don't need any of this and I'm just being overly anal
> retentive.
Not at all. Particularly as people start trying to write more Django
applications that are designed to be sharable beyond their immediate
development group, we are going to see more and more of this.
I was having a conversation at OSCON yesterday with a Drupal developer
and discussing how Drupal handles things like dependencies and
"installation profiles" (as they call them) and I mentioned to him that
we probably don't have the experience yet in Django to really know what
are "best practices" for the framework.
We have some good ideas and there is some consistency already and the
benefit of the substantial experiences of a bunch of guys from Kansas,
but are there better ideas available? What problems are people seeing in
other circumstances Are we converging on a "local" best, rather than
*the* best? Hearing different ideas -- particularly those backed up by
real-world deployment experience can only help, even if they only
provide anti-use-cases.
Regards,
Malcolm
Martin Glueck
I'm pretty new to django, so please forgive me if I missed something.....
> However, that doesn't take care of the reverse URL mapping, as you
> indicate above. Work on that area is being done by Adrian with his
> reverse URL mapping stuff (there have been some commits on this
> recently, too), including a template tag to allow templates easy access
> to this information. Over time, we will no doubt enhance and extend the
> initial work to accomodate more and more deviously twisted ... er...
> important use cases. So, whilst this point is probably the most limiting
> of the three you mention, it is also, fortunately, the one that is
> having some real brain power applied to it.
Can you give me pointer to this new template tag? becasue this reverse
url maping is one of my current problems and I'm not really happy the
way I implemented it right now...
> > 2. Allow each app to optionally specify a context_processor(request)
> > function in their __init__.py file. By default, all the context
> > processors that exist would be called in the order that the apps are
> > listed in the INSTALLED_APPS attribute. Obviously, this could be
> > dangerous because apps could try to define different values in the
> > context using the same key. Maybe we could do some kind of fake
> > namespacing ( {{ app.key }} to avoid that.
Using the `include` for the url-config allows to decouple the
application from the url it is `installed` to (assuming that the
reverse mapping issue is solved).
But I would like to see a simmilar approch for some other `settings`
as well, like
TEMPLATE_CONTEXT_PROCESSORS
TEMPLATE_DIRS
MIDDLEWARE_CLASSES
I know that for some of these the order in this list's is important.
So maybe a new way of specifing/adding suff to these settings has to
be defined.
But I think it would help if an application can define which
additional middlewares or context precessor it uses. This would make
the application itself more self-contained.
> > 3. Develop some kind of explicit way that apps can declare
> > dependencies on other apps so that apps can just be dropped in with
> > any dependencies, their prefixes set, and they be good to go.
>
> I think this is possibly best solved by just importing the other
> applications in __init__.py. That way, "manage.py validate" and any use
> of the application will fail at the very first import. So if my weblog
> app depends on my tag app, putting "import tagging" in
> weblog/__init__.py enforces the dependency.
>
> Just my personal feeling, but I think using Python's built-in dependency
> enforcement like this is preferable to adding any extra infrastructure
> to Django. I can't imagine we could do anything that would be much more
> simple than "import foo" without imposing a burden on the app developer.
I agree that using the python import as a dependency enforcement is a
good thing, as long as there are dependencies to other `applications`.
But what still is missing is a way to express which additional
`templatetags` an application needs.
I have only started working with django a few weeks ago, but one of
the first thing I noticed is that defining new template tags makes
writing yout templates way more easier...
And currently you don't have an possibilty to specify what addition
templatetag `libraries` an application needs (besides the `load` in
the templates itself).
And since I normaly don't look at the templates that much, it would
help if this dependancy could be expressed in the `application` as
well...
Just m2c,
Martin
Malcolm Tredinnick
> Malcom,
>
> I'm pretty new to django, so please forgive me if I missed something.....
>
> > However, that doesn't take care of the reverse URL mapping, as you
> > indicate above. Work on that area is being done by Adrian with his
> > reverse URL mapping stuff (there have been some commits on this
> > recently, too), including a template tag to allow templates easy access
> > to this information. Over time, we will no doubt enhance and extend the
> > initial work to accomodate more and more deviously twisted ... er...
> > important use cases. So, whilst this point is probably the most limiting
> > of the three you mention, it is also, fortunately, the one that is
> > having some real brain power applied to it.
>
> Can you give me pointer to this new template tag? becasue this reverse
> url maping is one of my current problems and I'm not really happy the
> way I implemented it right now...
This is still work in progress. Unfinished as yet. I don't think the tag
exists yet, but the django.core.urlresolvers.reverse() and
django.db.models.permalink() methods are part of this.
> > > 2. Allow each app to optionally specify a context_processor(request)
> > > function in their __init__.py file. By default, all the context
> > > processors that exist would be called in the order that the apps are
> > > listed in the INSTALLED_APPS attribute. Obviously, this could be
> > > dangerous because apps could try to define different values in the
> > > context using the same key. Maybe we could do some kind of fake
> > > namespacing ( {{ app.key }} to avoid that.
>
> Using the `include` for the url-config allows to decouple the
> application from the url it is `installed` to (assuming that the
> reverse mapping issue is solved).
> But I would like to see a simmilar approch for some other `settings`
> as well, like
>
> TEMPLATE_CONTEXT_PROCESSORS
> TEMPLATE_DIRS
> MIDDLEWARE_CLASSES
TEMPLATE_CONTEXT_PROCESSORS we talked about earlier and that probably
does bear more thinking about.
TEMPLATE_DIRS is for project-wide templates, so I'm not sure why an
application would want/need to change that. If you depend on some other
template being available, there should be a way to say that (although a
re-usable app is normally not going to do much beyond extending
base.html or something). Uf you supply templates, then they are in your
own template directory and will thus be available as soon as the app is
installed. Installing templates outside your own application is probably
not particularly maintainable. It is necessary in those cases to follow
something like the directory hierarchy from [1], since otherwise
template name clashes result.
[1] http://code.djangoproject.com/wiki/DosAndDontsForApplicationWriters
MIDDLEWARE_CLASSES is a tricky one. My gut feeling is to say we should
explicitly install stuff in there, because it is *very* order dependent.
But it needs more thought, because until you posted this, I, for one,
hadn't considered programmatic changes to that list. We might need to
tweak the middleware processing hierarchy slightly anyway (there are a
couple of tickets open about this; making it more stack-like,
essentially), because there are some slightly flaws in the way it works
currently. Something to keep in mind for when we do that.
[...]
> But what still is missing is a way to express which additional
> `templatetags` an application needs.
> I have only started working with django a few weeks ago, but one of
> the first thing I noticed is that defining new template tags makes
> writing yout templates way more easier...
> And currently you don't have an possibilty to specify what addition
> templatetag `libraries` an application needs (besides the `load` in
> the templates itself).
> And since I normaly don't look at the templates that much, it would
> help if this dependancy could be expressed in the `application` as
> well...
You can do this already, I suspect: Template tags are imported through a
"load" tag in the template. The file that is loaded must be an
importable Python file. So I think it would work to specify this as for
other applications -- "from other_app.templatetags import special_tags"
-- already. Have I overlooked a problem you have in mind here?
Regards,
Malcolm
Steven Armstrong
one may want to define per app. configuration and dependencies
There are usually two kinds of configuration settings an app may need.
Required ones that the user must set for the app to work, and optional
ones for which sane defaults are pre-configured or simply none are needed.
In pyblosxom [1] plugins, we do it like this:
Every plugin can define a 'verify_installation' function which is called
by the engine when you run it in validation mode.
In this function the author can check if required settings are available
and inform the user about the used defaults for optional ones.
One can also check for dependencies in this function, or just print out
messages what's needed for the plugin to work.
I think this is a good approach, as it allows one to give the user hints
what he needs to do to get it working.
Instead of:
ImportError: No module named foo
you get:
This application requires the 'foo' library to be installed.
You can download it from http://www.foo.com/download/.
Documentation is available at http://www.foo.com/docs/.
or:
You must add 'foo.context_processors.bar' to your
TEMPLATE_CONTEXT_PROCESSORS for this application to work.
So maybe something like this would be useful:
- myapp/validate.py, which get's run when the user uses 'manage.py
validate'. From here the user could get a step by step guide what he
needs to do to get everything working.
- myapp/settings.py, for per app settings.
These could maybe be merged with myproject/settings.py in some clever way.
Todd O'Bryan
Actually, I wonder if we couldn't limit the project's urls.py to urls
handled by the project's views, i.e., those not handled by the
INSTALLED_APPS. It seems like you need to do two different things,
add an app in settings.py and then add the URLs that map to it in
urls.py. If the prefix is best thought of as an attribute of the app
(which, I think, is the best OO way of thinking of it), then this
sort of violates DRY.
Unfortunately, what I'm suggesting brings back some magic. You put a
module in INSTALLED_APPS, set a prefix, and magically all the URLs
with that prefix are sent to that app. On the other hand, maybe
that's not bad magic to have, because it seems to work the way you
would expect.
I'll wait for Adrian's work on backward URL mapping, but this seems
like a completely cheap and simple way to handle the problem of
knowing, at least, what the prefix was of the URL that called you.
>> 2. Allow each app to optionally specify a context_processor(request)
>> function in their __init__.py file. By default, all the context
>> processors that exist would be called in the order that the apps are
>> listed in the INSTALLED_APPS attribute. Obviously, this could be
>> dangerous because apps could try to define different values in the
>> context using the same key. Maybe we could do some kind of fake
>> namespacing ( {{ app.key }} to avoid that.
>
> I have some generic worries about whether this is too simple to work.
As do I. We'll see.
>> 3. Develop some kind of explicit way that apps can declare
>> dependencies on other apps so that apps can just be dropped in with
>> any dependencies, their prefixes set, and they be good to go.
>
> I think this is possibly best solved by just importing the other
> applications in __init__.py.
That makes perfect sense; I'm just too much of a Python newbie to
have thought of it. :-)
Thanks for the response,
Todd
Martin Glueck
> This is still work in progress. Unfinished as yet. I don't think the tag
> exists yet, but the django.core.urlresolvers.reverse() and
> django.db.models.permalink() methods are part of this.
Thanks, I check this out....
> TEMPLATE_DIRS is for project-wide templates, so I'm not sure why an
> application would want/need to change that. If you depend on some other
> template being available, there should be a way to say that (although a
> re-usable app is normally not going to do much beyond extending
> base.html or something). Uf you supply templates, then they are in your
> own template directory and will thus be available as soon as the app is
> installed. Installing templates outside your own application is probably
> not particularly maintainable. It is necessary in those cases to follow
> something like the directory hierarchy from [1], since otherwise
> template name clashes result.
>
> [1] http://code.djangoproject.com/wiki/DosAndDontsForApplicationWriters
OK, I guess should have read ths documentaion more carfully....
> You can do this already, I suspect: Template tags are imported through a
> "load" tag in the template. The file that is loaded must be an
> importable Python file. So I think it would work to specify this as for
> other applications -- "from other_app.templatetags import special_tags"
> -- already. Have I overlooked a problem you have in mind here?
Looks a bit akware to import something you don't need in your view.model.
And the error-message would be a bit messy.
But at least a `manage.py validate` would detect the problem.
Martin
Bill de hÓra
> Hi Todd,
>
> On Sat, 2006-07-29 at 19:42 -0400, Todd O'Bryan wrote:
>> 3. Develop some kind of explicit way that apps can declare
>> dependencies on other apps so that apps can just be dropped in with
>> any dependencies, their prefixes set, and they be good to go.
>
> I think this is possibly best solved by just importing the other
> applications in __init__.py. That way, "manage.py validate" and any use
> of the application will fail at the very first import. So if my weblog
> app depends on my tag app, putting "import tagging" in
> weblog/__init__.py enforces the dependency.
That would be leave mechanisms; python __init__.py and INSTALLED_APPS.
INSTALLED_APPS works for me. I don't see the problem in forcing apps to
explicitly register centrally. It's preferable to implicit bilateral
dependencies which will become unmaintainable in short order. Ok, you
have an __init__.py, but INSTALLED_APPS at least obliges app developers
to think about the deploy/runtime context.
> Just my personal feeling, but I think using Python's built-in dependency
> enforcement like this is preferable to adding any extra infrastructure
> to Django. I can't imagine we could do anything that would be much more
> simple than "import foo" without imposing a burden on the app developer.
I can :) It'll tend be a burden on whomever manages the site.
> I was having a conversation at OSCON yesterday with a Drupal developer
> and discussing how Drupal handles things like dependencies and
> "installation profiles" (as they call them) and I mentioned to him that
> we probably don't have the experience yet in Django to really know what
> are "best practices" for the framework.
Plone (products) and Eclipse (osgi) do a good job structurally. But any
kind of plugin framework models suggest Django shift from being a web
framework to an app server. It's quite a difference, imo.
> We have some good ideas and there is some consistency already and the
> benefit of the substantial experiences of a bunch of guys from Kansas,
> but are there better ideas available? What problems are people seeing in
> other circumstances Are we converging on a "local" best, rather than
> *the* best? Hearing different ideas -- particularly those backed up by
> real-world deployment experience can only help, even if they only
> provide anti-use-cases.
Real world cases suggest plugin frameworks are the way to go, combined
with an understanding what the runtime dependencies really are between
apps. Especially dealing with the following issues - 1) where the two
apps you want require conflicting libraries (that's dll/jar hell); 2)
where you can't upgrade django core/contribs, because one of your apps
will break (that's plone/eclipse hell). These are very real, very nasty
problems that come with component/plugin models.
cheers
Bill
Malcolm Tredinnick
OK, maybe it's just because it's early in the morning or because I'm a
bit stupid, but I'm not sure that I am understanding what you are
suggesting in some cases here. A few requests for clarification below...
On Wed, 2006-08-02 at 21:39 +0100, Bill de hÓra wrote:
> Malcolm Tredinnick wrote:
> > Hi Todd,
> >
> > On Sat, 2006-07-29 at 19:42 -0400, Todd O'Bryan wrote:
> >> 3. Develop some kind of explicit way that apps can declare
> >> dependencies on other apps so that apps can just be dropped in with
> >> any dependencies, their prefixes set, and they be good to go.
> >
> > I think this is possibly best solved by just importing the other
> > applications in __init__.py. That way, "manage.py validate" and any use
> > of the application will fail at the very first import. So if my weblog
> > app depends on my tag app, putting "import tagging" in
> > weblog/__init__.py enforces the dependency.
>
> That would be leave mechanisms; python __init__.py and INSTALLED_APPS.
>
> INSTALLED_APPS works for me. I don't see the problem in forcing apps to
> explicitly register centrally. It's preferable to implicit bilateral
> dependencies which will become unmaintainable in short order. Ok, you
> have an __init__.py, but INSTALLED_APPS at least obliges app developers
> to think about the deploy/runtime context.
Are you in agreement or disagreement with checking for dependencies via
__init__.py? I wasn't suggesting we don't also install them in
INSTALLED_APPS -- I was just trying to show there were ways of checking
dependencies using existing Python infrastructure.
Thinking about my suggestion a bit more, though, after I wrote it, I
thought it might be useful to have a helper function that at least
checked the required app was in INSTALLED_APPS already (not just Python
importable). Is that what you are getting at as well?
>
>
> > Just my personal feeling, but I think using Python's built-in dependency
> > enforcement like this is preferable to adding any extra infrastructure
> > to Django. I can't imagine we could do anything that would be much more
> > simple than "import foo" without imposing a burden on the app developer.
>
> I can :) It'll tend be a burden on whomever manages the site.
So, I wrote a poorly constructed sentence, with lots of negatives in it.
And you agree with a smiley.
Oh (30 seconds later)... you're suggesting the "do nothing" approach is
easier: easy for developer, hard for site maintainer? OK, I probably
need to get a humour gland replacement.
> > I was having a conversation at OSCON yesterday with a Drupal developer
> > and discussing how Drupal handles things like dependencies and
> > "installation profiles" (as they call them) and I mentioned to him that
> > we probably don't have the experience yet in Django to really know what
> > are "best practices" for the framework.
>
> Plone (products) and Eclipse (osgi) do a good job structurally. But any
> kind of plugin framework models suggest Django shift from being a web
> framework to an app server. It's quite a difference, imo.
Agreed. This was something I was having to explain in my chat with the
Drupal guy as well. That Django was a little more generic than Drupal.
> > We have some good ideas and there is some consistency already and the
> > benefit of the substantial experiences of a bunch of guys from Kansas,
> > but are there better ideas available? What problems are people seeing in
> > other circumstances Are we converging on a "local" best, rather than
> > *the* best? Hearing different ideas -- particularly those backed up by
> > real-world deployment experience can only help, even if they only
> > provide anti-use-cases.
>
> Real world cases suggest plugin frameworks are the way to go, combined
> with an understanding what the runtime dependencies really are between
> apps. Especially dealing with the following issues - 1) where the two
> apps you want require conflicting libraries (that's dll/jar hell); 2)
> where you can't upgrade django core/contribs, because one of your apps
> will break (that's plone/eclipse hell). These are very real, very nasty
> problems that come with component/plugin models.
At the end of the day, this stuff is really hard. I think we can try to
get to a point where the easy hurdles are maybe handled automatically
and the more subtle stuff requires the installer to actually read the
README that comes with an application. There will still be problems
because of slight API incompatibilities. It's analogous to trying to
maintain perfect API/ABI compatibility in C libraries across version
upgrades, and that stuff's serious rocket science. Either we formally
describe all the APIs that are exposed from an app (yuck!) or we have a
few bumpy patches in the installation path (a little more palatable in
rapid development/deployment situations, I feel).
Maybe we need to encourage putting a __version__ string or somesuch in
an app's __init__.py to enable compatibility checking (combined with
searching for the app in INSTALLED_APPS at validate- and run-time). Then
we educate people in what it means to bump the minor version number
(compatibility maintained) version the major number. There are lots of
examples to draw on from other projects. Getting some concrete ideas to
try out is probably worth it.
Do you have any concrete ideas here, Bill? Other than "everything has
drawbacks"? :-)
Regards,
Malcolm
Gary Wilson
> > Or maybe we don't need any of this and I'm just being overly anal
> > retentive.
>
> Not at all. Particularly as people start trying to write more Django
> applications that are designed to be sharable beyond their immediate
> development group, we are going to see more and more of this.
First off, I think it would be good for applications to have their own
settings file. Currently, if I want to install another django
application I have to add it to INSTALLED_APPS, make sure it has the
TEMPLATE_CONTEXT_PROCESSORS it needs, the MIDDLEWARE_CLASSES it needs,
the AUTHENTICATION_BACKENDS it needs, the cronjobs it needs, and edit
the project's urls.py. And in the future, we might also have escaping
settings (when auto-escaping lands), database settings (when multi db
support lands), and so on.
The reason why these need to be specified per app is because
TEMPLATE_CONTEXT_PROCESSORS can clash, MIDDLEWARE_CLASSES can clash,
etc. I recently ran into a problem where the above clashes happened.
I have an app that uses it's own user table. Well, the multi auth
login function sets the cookie session data to the id of the
authenticated user. But the id is an id for my app's user table, and
if I were to authenticate to my app and then go to django admin app,
things break. Things also break if I want a context processor to use
the template variable 'user' since django.core.context_processors.auth
also uses the 'user' variable. Same goes for twe middleware classes
both setting request.user.
Yes, I could use different template variable and class attribute names,
but now extra work is being done because my apps context processors are
running as well as the admin's context processors. The more apps you
have, the worse things get. Maybe an application should be able to
specify whether the middleware class it uses should be applied globaly
throughout the project or applied only to urls handled by that app?
Would that be too complex? Or maybe the solution is to keep your
django projects small and only use a couple applications per project.
Then you essentially have "per app" settings.
Per app settings I think would go a long way in making applications
more "packaged" though. Maybe settings not specified in the app would
carry over from the project's settings? So if I want to use the same
database then I wouldn't specify one in my app's settings.
Then there is the issue, which I think was mentioned earlier, about how
do you insert the app's middleware classes into the project's
middleware classes since order does matter. Or maybe instead you
specify all the middleware classes to be used in the app and don't do
any inserting into the project's middleware classes.
Sorry if the above isn't very coherent. It's late and I was just
trying to throw some ideas out there. I also have some beef with the
template directories and loading, but I will save that for another
email.
Gary Wilson
Malcolm Tredinnick
> Malcolm Tredinnick wrote:
> > > Or maybe we don't need any of this and I'm just being overly anal
> > > retentive.
> >
> > Not at all. Particularly as people start trying to write more Django
> > applications that are designed to be sharable beyond their immediate
> > development group, we are going to see more and more of this.
>
> First off, I think it would be good for applications to have their own
> settings file. Currently, if I want to install another django
> application I have to add it to INSTALLED_APPS, make sure it has the
> TEMPLATE_CONTEXT_PROCESSORS it needs, the MIDDLEWARE_CLASSES it needs,
> the AUTHENTICATION_BACKENDS it needs, the cronjobs it needs, and edit
> the project's urls.py. And in the future, we might also have escaping
> settings (when auto-escaping lands), database settings (when multi db
> support lands), and so on.
You've got a long way to go before I'm convinced this is a good idea,
but it will be interesting to hear what others think. :-)
One point of view (mine!) is that if an application can't play nicely
with other applications and wants to do its own authentication, etc,
then it's not an application, it's a whole other project. The project
developer should have complete control over what the policies are for
things like this. Otherwise it's too hard to understand what each app is
doing when incorporating it.
App-specific context processors are interesting, though.
I'm not a fan of infinite flexibility as solutions for undefined future
problems, so it's good that you've put in some specific cases you've
encountered. Food for thought here.
Malcolm
Ahmad Alhashemi
> I'm not a fan of infinite flexibility as solutions for undefined future
> problems, so it's good that you've put in some specific cases you've
> encountered. Food for thought here.
+1 on that.
All these suggestions are going to make Django much fatter. It will
make the learning curve steeper too.
The good thing is that it is really easy to implement such features for
our own use without the need to touch a single line of Django's core.
For example, if you want to implement this idea of a context_processors
function in every application, you can easily do this as a "meta"
context_processor that you will add to your settings file as a context
processor.
Then when your "meta" context processor is called, you can have it go
through INSTALLED_APPS, call whatever function you want and collect the
responses whatever format you want, then return the whole thing from
your "meta" context processor, and everything will be available in the
templates automatically.
The same thing for middleware. You can just create some kind of a
"meta" middleware that will use a systematic way of calling other
predefined middleware functions in your INSTALLED_APPS.
I really like these features, they can suddenly make applications much
more powerful. But I'm always afraid that they might make applications
a bit too powerful. I think of them as explosive ideas. I'm working on
a currently experimental extension to Django that adds all sorts of
things like this. I call it Dynamite. If it goes well, I will release
the code for it.
In the end, if a certain feature proves itself popular and useful over
time, we can easily add it to Django's core.
Gary Wilson
> One point of view (mine!) is that if an application can't play nicely
> with other applications and wants to do its own authentication, etc,
> then it's not an application, it's a whole other project. The project
> developer should have complete control over what the policies are for
> things like this. Otherwise it's too hard to understand what each app is
> doing when incorporating it.
So if am using the admin application with its (contrib.auth)
authentication and then the rest of my site using a custom user model
with its own authentication, you are saying that I should make them
separate projects? That just doesn't seem right.
I don't think anyone has answered the OP's question of how plugable are
apps supposed to be? Currently, the only way you can achieve full
modularity is to make every installed app its own project. Only then
can I set the app's TEMPLATE_CONTEXT_PROCESSORS, MIDDLEWARE_CLASSES,
AUTHENTICATION_BACKENDS, etc. and be sure that applications won't stomp
over each other.
d...@simon.net.nz
> One point of view (mine!) is that if an application can't play nicely
> with other applications and wants to do its own authentication, etc,
> then it's not an application, it's a whole other project. The project
> developer should have complete control over what the policies are for
> things like this. Otherwise it's too hard to understand what each app is
> doing when incorporating it.
Just musing a bit here -
I'm wondering about this too - I'm working on a few applications, and
was not sure if it was best to distribute them as django _projects_ or
django _apps_. In fact, what is the distinction between an app and a
project? Most of the examples I've seen have been a one-app-one-project
style. If this is the way people are going to do things, is there any
benefit in having an app/project distinction?
I *do* like the idea of settings per application - I've got a lot of
use out of the main settings file by adding an APPNAME dictionary and
adding user configuration vars into that, but this does mean that I
have to distribute a _project_, instead of an _app_. Would it be a good
idea for startapp to create an empty settings.py in each application
dir? Django.conf.settings could then load get this as settings.APPNAME
Per-app settings would also give you some method of directly
over-riding any other settings.
Finally - I'm not entirely sure that Django needs to try to solve
dependencies, it's up to the app/project developers to describe what's
needed to run their software. This is what readme files are for.
--Simon
Malcolm Tredinnick
> Malcolm Tredinnick wrote:
> > One point of view (mine!) is that if an application can't play nicely
> > with other applications and wants to do its own authentication, etc,
> > then it's not an application, it's a whole other project. The project
> > developer should have complete control over what the policies are for
> > things like this. Otherwise it's too hard to understand what each app is
> > doing when incorporating it.
>
> So if am using the admin application with its (contrib.auth)
> authentication and then the rest of my site using a custom user model
> with its own authentication, you are saying that I should make them
> separate projects? That just doesn't seem right.
Project administrators (or whatever we want to call them) having their
decision usurped by individual apps feels like bad system
administrations practice. If a project sets up a particular
authentication scheme (particularly authentication, which is
security-related), an application should not be able to avoid that.
I'm not really sure that there is a true "right" answer here, Gary.
Again, this is a thread where a lot of untested ideas are being bounced
around. I do know how I would be thinking as a Systems Administrator or
Operations Manager in cases like this (even if it just on my own
personal production machine(s)). Most of what we come up with is going
to be suggestions or best practice ideas and we need to think about that
side of things as well when coming up with ideas.
> I don't think anyone has answered the OP's question of how plugable are
> apps supposed to be?
Except that wasn't Todd's original question. :-)
He was floating a few ideas about how to manage multiple applications.
We've had quite a few productive emails on those lines with various
ideas in this thread. And some of the consensus that came out was that
it was going to require some thinking on the part of the project creator
and that isn't necessarily a bad thing.
There is also the quite valid position, that you're obviously an
advocate for, of having a lot of automated configuration stuff as well.
But it feels like we are going around in circles with semantics at some
points in this thread. Words like "pluggable" or "modular" or
"application" or "project" are somewhat irrelevant: what we are really
talking about is "how can I use some of that guy's code and use it in my
code".
> Currently, the only way you can achieve full
> modularity is to make every installed app its own project. Only then
> can I set the app's TEMPLATE_CONTEXT_PROCESSORS, MIDDLEWARE_CLASSES,
> AUTHENTICATION_BACKENDS, etc. and be sure that applications won't stomp
> over each other.
Stop viewing your applications as warring parties and start thinking of
them as cooperative contributors to your project as whole. You're not
trying to prevent them from stomping over each other, you are trying to
get them to work together. :-)
I can think of lots of potential problems with per-app configurations
like this and I can see that in some cases it might make life easier as
well. It feels a lot like you are moving request-level things down into
applications, for example. (So at which point in the request handling do
they get processed?). But try it out for a while. See if it does make
your application sharing easier. I have zero experience with your
approach, so maybe it is easier and I'm just old and inflexible.
Like I said in my first or second post to this thread, we can only
benefit from people reporting back real-world experiences: things they
have tried that worked or didn't work and why the new thing wasn't
handled nicely by our existing infrastructure.
Best wishes,
Malcolm
Malcolm Tredinnick
> > Malcolm Tredinnick wrote:
> > One point of view (mine!) is that if an application can't play nicely
> > with other applications and wants to do its own authentication, etc,
> > then it's not an application, it's a whole other project. The project
> > developer should have complete control over what the policies are for
> > things like this. Otherwise it's too hard to understand what each app is
> > doing when incorporating it.
>
> Just musing a bit here -
>
> I'm wondering about this too - I'm working on a few applications, and
> was not sure if it was best to distribute them as django _projects_ or
> django _apps_. In fact, what is the distinction between an app and a
> project? Most of the examples I've seen have been a one-app-one-project
> style. If this is the way people are going to do things, is there any
> benefit in having an app/project distinction?
Be careful not to generalise from a few examples to the whole universe
of Django users. :-)
The Lawrence guys have mentioned on this list previously that they have
many different applications they pull together in various projects. If
you look at the source code repository that Ian Holsman periodically
posts for Zyons and his other projects, you can see he's using multiple
apps to do different things. I'm doing the same in a few places
(including sharing applications between projects).
There are plenty of "silent" users of Django, so you should assume
maximum variation. :-)
Regards,
Malcolm
Paul Sargent
this.
Todd O'Bryan wrote:
> 1. Allow people to set a prefix for each app in the settings.py file.
> This would basically tell the app what URLs that are mapped to it
> start with, and views and templates inside the app could use this
> when they want to create internally mapped URLs. By default, it would
> also pass control of any URL starting with the prefix to the given
> app's urls.py module, much as already happens, I imagine, in most
> people's implementations, but the prefix would be an attribute of the
> app, rather than some unconnected string as it is now.
I not a fan of the idea that apps must map to a particular tree of URLs
(i.e.
the blog app is always under /blog/). It's a common implementation, and
a good
one, but I don't necessarily think it fits all projects. I'd rather we
could
say "write your apps using relative references, and here's a toolkit to
help
you"
Basically I like the how powerful and how specific the current urls.py
scheme
is (sidenote: you're never going to get RoR's 'trigger a nasty script
from a
URL because I was trying to be clever' hole in django :-) ). I'd be
sorry to
see it usurped by side-effects of other processes.
In fact, I'd rather this sort of conversation focused more on 'best
practices', and utilising what is already available, rather than
introducing
extra mechanisms (especially 'magic' ones) unless really required.
btw: It's always bugged me that the tutorial on de-coupling app urls
[1]
moves the app specific urls into the app directory, but still has to
reference
the 'mysite' project name in it's urlconf.
[1]
http://www.djangoproject.com/documentation/tutorial3/#decoupling-the-urlconfs
Malcolm Wrote:
> Bill Wrote:
> > Plone (products) and Eclipse (osgi) do a good job structurally. But any
> > kind of plugin framework models suggest Django shift from being a web
> > framework to an app server. It's quite a difference, imo.
>
> Agreed. This was something I was having to explain in my chat with the
> Drupal guy as well. That Django was a little more generic than Drupal.
+1. Django is not just an app server!
For the dependancy/registration stuff, I like the idea of putting some
boiler
plate code in __init__.py. If INSTALLED_APPS is iterated over at some
point
during startup, it could inspect some predefined attributes to see if
dependancies are met.
e.g. __init__.py:
APPNAME = "Lebowski"
APPVERSION = "0.3"
REQUIRES = (("White Russian", ">=3.0"),
("Rug", "=1.0"))
def register(**kwargs):
settings.TEMPLATE_DIRS.append(kwargs['app_path']+"/templates/bowling")
in settings.py:
INSTALLED_APPS = (blah,
blah,
mysite.lebowski,
blah)
APP_ARGUMENTS = {mysite.lebowski : {'app_path' :
"/home/django_apps/lebowski",
'something_else' : 42}}
If any of these things aren't there, no harm, no foul. If they are they
get
checked/called.
It's pretty simple to understand, and pretty flexible. I'm in two minds
about
the register function though. I think if you have somewhere where
people
can do things like modify settings then they're likley to abuse it,
especially
if they start trying to add things which are ordered. Maybe they just
need a
separate 'requires list' like the one above for apps.
e.g. __init__.py:
REQUIRES_MIDDLEWARE =
('django.contrib.auth.middleware.AuthenticationMiddleware',)
And then it's just down to the person installing that they're on the
list and in
a good order.
Paul
James Bennett
> The Lawrence guys have mentioned on this list previously that they have
> many different applications they pull together in various projects.
:)
If we couldn't modularize, we wouldn't be able to sell different
"editions" of Ellington.
The more I think about it, the more I like the idea of using imports
in __init__.py to specify an application's dependencies; the thing
that'd make it especially nice would be some method -- either
conventional or built straight in to Django -- of checking for
"importable" versus "installed" and for throwing nicer error messages.
Something like this, maybe:
from django.conf import settings
dependencies = (
{'name': 'tags', 'module': 'foo.tags'},
{'name': 'todo', 'module': 'bar.todo'},
{'name': 'comments', 'module': 'django.contrib.comments'},
)
dependency_errors = []
for app in dependencies:
try:
__import__(app['module'])
except ImportError:
dependency_errors.append("The '%s' application is required,
but is not present on this system" % app['name'])
finally:
if not app['module'] in settings.INSTALLED_APPS:
dependency_errors.append("The '%s' application is
required, but is not listed in Django's installed applications" %
app['name'])
if dependency_errors:
### echo them to console, or pass along to a management function
For extra bonus points, we'd make it use the stuff in
django/core/management to verify that the needed app, if importable
and in INSTALLED_APPS, is either already installed into the DB or
scheduled to be during the current 'syncdb' run.
Anything more complex would be tending a bit too much toward the "app
server", I think.
--
"May the forces of evil become confused on the way to your house."
-- George Carlin
Gary Wilson
> On 8/13/06, Malcolm Tredinnick <mal...@pointy-stick.com> wrote:
> > The Lawrence guys have mentioned on this list previously that they have
> > many different applications they pull together in various projects.
>
> :)
>
> If we couldn't modularize, we wouldn't be able to sell different
> "editions" of Ellington.
Maybe they could enlighten us on how they modularize.
Gary Wilson
> The more I think about it, the more I like the idea of using imports
> in __init__.py to specify an application's dependencies; the thing
> that'd make it especially nice would be some method -- either
> conventional or built straight in to Django -- of checking for
> "importable" versus "installed" and for throwing nicer error messages.
> Something like this, maybe:
>
> from django.conf import settings
>
> dependencies = (
> {'name': 'tags', 'module': 'foo.tags'},
> {'name': 'todo', 'module': 'bar.todo'},
> {'name': 'comments', 'module': 'django.contrib.comments'},
> )
IMO, the dependency checking is the easy part. In the README or
something, I say MyCoolApp requires the admin app. It's the
configuration settings of the admin app that's hairy.
So let's say MyCoolApp needs the admin app to function. (Let's ignore
the fact that, by default, Django adds everything to settings.py that
the admin app needs to run even though I don't need most of them if I'm
not using the admin app.) So, in MyCoolApp.__init__.py I put the
following:
dependencies = (
{'name': 'admin', 'module': 'django.contrib.admin'},
)
What has this bought me?
Not much considering I still need to make sure I have
ADMIN_MEDIA_PREFIX set, SECRET_KEY set,
'django.template.loaders.app_directories.load_template_source' in my
TEMPLATE_LOADERS,
"django.contrib.sessions.middleware.SessionMiddleware" and
"django.contrib.auth.middleware.AuthenticationMiddleware" and
"django.middleware.doc.XViewMiddleware" in my MIDDLEWARE_CLASSES,
'django.core.context_processors.auth' and
'django.core.context_processors.i18n' and
'django.core.context_processors.request' in my
TEMPLATE_CONTEXT_PROCESSORS,
'django.contrib.auth.backends.ModelBackend' in my
AUTHENTICATION_BACKENDS, "django.contrib.admin" in my INSTALLED_APPS,
and (r'^admin/', include('django.contrib.admin.urls')) in my urls.py.
Somebody help me if I ever wanted to bring down an application for
maintenance. Just removing it from INSTALLED_APPS ain't going to do
it. You can check the importing of an app or it's presence in
INSTALLED_APPS, but that app could still be a good ways from being
"installed."
> Anything more complex would be tending a bit too much toward the "app
> server", I think.
What do you mean a bit too much toward the app server? Isn't that what
we are all doing with django? Building and serving blog apps and forum
apps and news publishing apps and ...
Malcolm Tredinnick
> James Bennett wrote:
> > The more I think about it, the more I like the idea of using imports
> > in __init__.py to specify an application's dependencies; the thing
> > that'd make it especially nice would be some method -- either
> > conventional or built straight in to Django -- of checking for
> > "importable" versus "installed" and for throwing nicer error messages.
> > Something like this, maybe:
> >
> > from django.conf import settings
> >
> > dependencies = (
> > {'name': 'tags', 'module': 'foo.tags'},
> > {'name': 'todo', 'module': 'bar.todo'},
> > {'name': 'comments', 'module': 'django.contrib.comments'},
> > )
>
> IMO, the dependency checking is the easy part. In the README or
> something, I say MyCoolApp requires the admin app. It's the
> configuration settings of the admin app that's hairy.
And that is part of setting up the admin app, not part of the modularity
of some other application. James' suggestion (and a lot of this thread)
has been addressing the identification of the dependencies, not being an
automatic project adiministrator.
> So let's say MyCoolApp needs the admin app to function.
[...]
> So, in MyCoolApp.__init__.py I put the
> following:
>
> dependencies = (
> {'name': 'admin', 'module': 'django.contrib.admin'},
> )
>
> What has this bought me?
Quite a lot, in terms of checking that the right pre-req is installed.
If you don't have the admin app installed, you will find out that you
need it as soon as you run "./manage.py validate" or "...syncdb".
It's not meant to save you the work of installing the admin app, which
seems to be the bulk of your post, but it automatically checks that you
have it installed (or at least attempted to install it).
Regards,
Malcolm
James Bennett
> IMO, the dependency checking is the easy part. In the README or
> something, I say MyCoolApp requires the admin app. It's the
> configuration settings of the admin app that's hairy.
That's what application documentation is for. In theory it's also
possible to use this mechanism to ensure that any additional settings
it requires exist in the project settings file and are non-empty;
something like
from django.conf import settings
try:
my_cool_app_setting = settings.COOL_APP_SETTING
except AttributeError:
dependency_errors.append("the setting 'COOL_APP_SETTING' must be
specified in the project's settings file in order to use this
application")
But as Malcolm has pointed out, the idea here is not to provide a
mechanism for automatically configuring applications -- it's to
provide a way for applications to specify the things they need.
> What do you mean a bit too much toward the app server? Isn't that what
> we are all doing with django? Building and serving blog apps and forum
> apps and news publishing apps and ...
In my mind, at least, an "app server" is a system whose job is to take
many different applications which may be written using wildly
different frameworks, or even no frameworks at all, and mediate
between them (assuming, for example, that they implement some common
interface for communication with the app server). WSGI is a good
example of this sort of thinking in the Python world -- the idea is
that it doesn't matter what you use to write your application, so long
as it exposes the appropriate WSGI-compliant interfaces for its
intended role.
That seems to be far and away a larger and more complex task than what
Django aims for -- yes, Django provides facilities for Django-based
applications to work with one another, but Django is first and
foremost a tool for _writing_ applications, not a tool for _running_
applications.
Gary Wilson
> On 8/17/06, Gary Wilson <gary....@gmail.com> wrote:
> > IMO, the dependency checking is the easy part. In the README or
> > something, I say MyCoolApp requires the admin app. It's the
> > configuration settings of the admin app that's hairy.
>
> That's what application documentation is for. In theory it's also
> possible to use this mechanism to ensure that any additional settings
> it requires exist in the project settings file and are non-empty;
> something like
>
> from django.conf import settings
>
> try:
> my_cool_app_setting = settings.COOL_APP_SETTING
> except AttributeError:
> dependency_errors.append("the setting 'COOL_APP_SETTING' must be
> specified in the project's settings file in order to use this
> application")
>
> But as Malcolm has pointed out, the idea here is not to provide a
> mechanism for automatically configuring applications -- it's to
> provide a way for applications to specify the things they need.
Yes I think that, at the least, a mechanism for checking that apps have
everything they need would be great. I was more trying to point out
that simply checking INSTALLED_APPS is not sufficient. It would be a
good start though.
I am more concerned about when Djangoers start sharing apps more. (For
instance, a Django application repository has been brought up a couple
times on the list. Continuing to add applications to contrib is not
going to scale.) I wouldn't want a context processor of an application
I grabbed from some shared repository to be able to interfere with a
context processor of an app I already have installed. Or if it did
interfere, then it would be nice if this could be caught by some sort
of validation check.
I also think that the check should be only a warning. The admin app
has checks in its code that makes sure that
django.core.context_processors.auth is in TEMPLATE_CONTEXT_PROCESSORS,
but I had another application where I wanted to set a context variable
with the same name. To do this you have to remove the
django.core.context_processors.auth. But then you can't use the admin
app because it will raise an error, even if I have a custom context
processor that runs either my application's context processor or the
auth context processor depending on which application was "handling"
the request. Applications shouldn't be able to so easily polute the
context variable namespace for every installed application.
The default template loaders suffer from this same type of polution.
Let's say I want the admin app installed, which in the template
directory namespace uses admin, admin_doc, registration, and widget.
Now if I later made a registration app (or any application that wanted
to use a folder named registration to hold its templates), I would be
in trouble. One application shouldn't go looking in all the template
directories of every other installed application. That's just wrong.
Many users want to extend or alter the admin or authentication
applications, but can't because they are so intertwined with each other
(and other Django core) that it makes things hard to change. By making
applications more modular, you also make them more extensible.
> > What do you mean a bit too much toward the app server? Isn't that what
> > we are all doing with django? Building and serving blog apps and forum
> > apps and news publishing apps and ...
>
> In my mind, at least, an "app server" is a system whose job is to take
> many different applications which may be written using wildly
> different frameworks, or even no frameworks at all, and mediate
> between them (assuming, for example, that they implement some common
> interface for communication with the app server). WSGI is a good
> example of this sort of thinking in the Python world -- the idea is
> that it doesn't matter what you use to write your application, so long
> as it exposes the appropriate WSGI-compliant interfaces for its
> intended role.
>
> That seems to be far and away a larger and more complex task than what
> Django aims for -- yes, Django provides facilities for Django-based
> applications to work with one another, but Django is first and
> foremost a tool for _writing_ applications, not a tool for _running_
> applications.
I would never expect Django to handle apps written in other frameworks,
but surely Django should be able to run it's own apps well. I would
argue that Django is just as much a tool for running Django
applications as it is a tool for writing Django applications. What
good is writing a Django application if I don't have some equally good
way to run it?