Proposal: modular django configuration

[bur...@gmail.com]

Hi everybody,

Everyone loves the way templates are discovered in django.
No one loves settings.py that much.

This is proposal on how to improve situation significantly.

Configuration facility is suggested in addition to django.conf.settings.
Configuration is going to be put into conf/ directory of the project
(or settings/ -- just set config.global.CONFIG).

:: settings.py ::
from django.conf import config
from os.path import dirname, abspath, join
ROOT = dirname(abspath(__name__)) # type 'str'
INSTALLED_APPS = ...

config.global.ROOT = ROOT
config.global.CONFIG = join(ROOT, 'conf')
config.autodiscover()
# config.global is in fact a synonym for settings

:: conf/global.py ::
# runs before the app settings
from django.conf import config
from os.path import dirname, abspath
ROOT = config.global.ROOT # type 'str', empty if not set.
config.global.JQUERY.default = JQUERY = ROOT +
'static/js/jquery-1.3.2-min.js' # type 'dict', setting default value
for missing items
config.global.JQUERY['1.3.2'] = JQUERY # type 'unordered dict with
default value', now setting arbitrary key
config.global.MEDIA += [ROOT + 'static/js/'] # type 'ordered set with
default value'
config.global.DATABASES['default'] = {...} # backward-compatibility,
so using 'default' not .default!
# Note: after type is set for constant, the type can't be changed.
# Note: please set to tuple not list if you need a clear sign the
option won't be additive any more.

:: conf/global_overrides.py ::
# runs after all other settings but before <site>_overrides, see below
# is now empty

:: conf/apps/myapp.py ::
# runs after all app-specific settings
app.DATABASES['db3'] = {...}
app.ROUTERS += ['Db3_is_readonly']

:: conf/www_server_com.py
# runs before app-specific settings
from django.conf import config
config.global.MEDIA_ROOT = '/var/site/www.server.com/site_media/'
config.global.MEDIA_URL = 'media.server.com'
app.MIDDLEWARE += ['caching.smart_caching_app.SmartCacher']

:: conf/www_server_com_overrides.py
# runs after app-specific settings
config.global.JQUERY['1.3.2'] = 'static/packed.js'
config.global.JQUERY['1.4.2'] = 'static/packed.js'

:: myapp/conf.py ::
# runs in order specified in INSTALLED_APPS
from django.conf import config
app = config.apps.myapp
app.DEPENDS += ['django.contrib.auth']
app.STATIC = app.global.ROOT + 'media/myapp/'
app.IMAGES = app.global.ROOT + 'media/uploads/images/'
app.THUMBS = app.global.ROOT + 'media/uploads/thumbs/'
config.global.MEDIA += [app.IMAGES, app.THUMBS, app.JSES, app.CSSES]
config.global.JQUERY['1.4.2'] = STATIC + 'js/'
config.global.TAGS += ['app1.templatetags.mytags']

:: myapp/forms.py ::
from django.conf import config

app = config.apps['myapp']
class MyForm:
class Media:
css = app.STATIC + 'css/base.css'
js = config.global.JQUERY['1.4.2']

The ultimate order:

django/conf/global.py
conf/__init__.py
conf/global.py # -- you can also set your own personal order there
conf/<site*>.py
app1/conf.py # -- single pass is enough, cause applications can both
provide callbacks for later configuration stages.
app2/conf.py
...
appN/conf.py
conf/apps/app1.py
conf/apps/app2.py
conf/apps/appN.py
conf/global_overrides.py
conf/<site*>_overrides.py

*<site> for www.my-site.com is www_my__site_com (dots replaced with
underlines, dash with double underline).
socket.getfqdn() is used for determining current site.

The motivation is simple:
the bigger your list of application grows, the larger configuration
you will have!
Django has more than 100 of different settings options.
They are not even grouped now.
I hope such django "built-in" type of configuration will suit 99% of
the possible Django projects, and will make django community much
stronger.

I'm going to create a prototype.

Expected benefits:
- 3rd-party applications can be used without a bit of touching and
still customized perfectly.
- Application can connect to each other in dynamic way, or provide
different kinds of plugin points.
- Fixed models dependencies can be replaced with dynamic (i.e, each
application might ask for personal User or UserProfile replacement)
- Really simple media setup for both development and production servers.
- A number of development and production configurations can coexist
in the project, without single 'if'
- Per-application configuration for middlewares, databases, routers,
context processors and other "additive" options
- Preconditions for visual application settings (Needs another proposal)
- Django core settings will be moved to namespaces and grouped semantically.
- Sparse config is better than dense.

Why it needs to be in the django core, not just 3rd-party plugin:
- Because "Namespaces are one honking great idea -- let's do more of those!"
- Because config ubiquity between projects is the main project benefit.

--
Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com

[bur...@gmail.com]

Hi all,

I've written a prototype, and put it on
http://github.com/buriy/django-configurator.
It has few good design decisions, and few maybe not that good.
Anyway, I consider it as a good addition to app loading GSoC Proposal,
which is currently being worked on.

The key highlights of the implementation are the following:

The first concept is the options object. Options can be global or
belong to applications:

>>> from configurator import options
>>> options.KEY = 1
>>> options.apps['myapp'].KEY = 2
>>> type(options)
<class 'configurator.Config'>
>>> type(options.apps)
<class 'configurator.AppList'>
>>> type(options.apps['myapp'])
<class 'configurator.Config'>

If the key is missing, DictList is provided:

from configurator import options
>>> KEY = options.SOME_KEY
>>> type(KEY)
<class 'configurator.containers.DictList'>
>>> id(KEY) == id(options.SOME_KEY)
True

Next concept is DictList. DictList is options holder and it allows one
to set a global option in advance, providing data from one module to
another, not having to deal with applications order in INSTALLED_APPS.

Can be used as an ordered set:

>>> KEY += 'xxx'
>>> KEY
['xxx']
>>> KEY -= 'yyy'
>>> KEY += 'yyy'
>>> KEY
['xxx', 'yyy']
>>> KEY -= 'xxx'
['yyy']

In Django, this can be used for AUTHENTICATION_BACKENDS,
TEMPLATE_CONTEXT_PROCESSORS, MIDDLEWARE_CLASSES, ADMINS, LANGUAGES and
other settings.

Or as an ordered dict:
>>> D = DictList()
>>> D['abc'] = 'def'
>>> D['key'] = 'value'
>>> D.default = '(default)'
>>> D
{default=(default), 'abc': 'def', 'key': 'value'}
In Django, this can be used for DATABASES setup, and, i hope, in 1.3,
for LOGGING setup and APP_MEDIA setup.

Currently I'm not using SortedDict for DictList implementation to not
depend on Django.

The next concept is autodiscover. It allows you to have different
configuration bits for every application or for different servers.
For basic use in django applications, you can put this call in the end
of the settings.py:

autodiscover(locals())

It does the following:
1) puts all written in caps settings from settings.py into options,
wrapping lists, tuples and dicts into DictList
2) iterates over options.DISCOVERY_ORDER, defined by default as the following:

options.DISCOVERY_ORDER = [
ListFiles(options.lazy.CONF, 'global'),# conf/global.py
ListFiles(options.lazy.CONF, options.lazy.SITE), # conf/<site>.py
ListFiles(options.lazy.INSTALLED_APPS, 'conf'), # app1/conf.py
ListFiles(options.lazy.APP_CONF, options.lazy.INSTALLED_APPS), #
conf/app1.py
ListFiles(options.lazy.CONF, 'global', '_overrides'), #
conf/global_overrides.py
ListFiles(options.lazy.CONF, options.lazy.SITE, '_overrides'), #
conf/<site>_overrides.py
]
Every application can update any option in such files.

Please note, that this DISCOVERY_ORDER is lazy, and you can set
options.CONF to your own folder in settings.py, or do the similar
thing with other options before DISCOVERY_ORDER will be evaluated.

3) writes updated settings back to settings.py

If you don't want to change anything in settings.py, you can do
autodiscover(), update_options() and update_back_settings() manually
in any moment of time.

Exceptions, raised in configuration modules, are isolated. If one of
your conf files failed to load, console message will appear, or, in
addition to the message, application will fail with traceback if
console is not a tty.

Known issues: since global django settings are read and set
independently from settings.py, they don't know anything about each
other. So if you do AUTHENTICATION_BACKENDS += 'logins.MyBackend', no
django.contrib.auth.backends.ModelBackend will be added.

[Russell Keith-Magee]

On Thu, Jun 3, 2010 at 11:19 PM, bur...@gmail.com <bur...@gmail.com> wrote:
> Hi all,
>
> I've written a prototype, and put it on
> http://github.com/buriy/django-configurator.
> It has few good design decisions, and few maybe not that good.
> Anyway, I consider it as a good addition to app loading GSoC Proposal,
> which is currently being worked on.

Hi Yuri,

I might be missing something really obvious, but I can't make any
sense out of your proposals. After two quite lengthy emails, I haven't
even been able to work out what what problem you are actually trying
to solve.

Your initial proposal consisted of pages of snippets of configuration,
but no explanation of how those pieces fit together (or at least, I
couldn't follow it). It might be helpful if you step back from the
details and try to give a big picture view of what you're trying to
do.

Yours,
Russ Magee %-)

[bur...@gmail.com]

Hi Russell,

My writing style sometimes is really clumsy. I'm sorry about that.

You might look at the end of the first message in the thread.
Or maybe the thread topic.

The problem is that half of third party plugins write: "after install,
add this and this and this to settings.py".
The problem is that I have reusable applications and they have their
bits of default configuration, have their own options.

Problem is that some of such settings are additive, like
MIDDLEWARE_CLASSES, DATABASES or DATABASE_ROUTERS, and you wanted to
give an app the possibility to configure their part of global
configuration, but not override settings which are set by another
applications.

Problem is that, like everybody, I have different machines where i run
my projects, and they have different settings (DEBUG, DATABASES, etc).

I'm not the first here. The problem and the common ways to solve it
are already observed in Django wiki under
http://code.djangoproject.com/wiki/SplitSettings section.

Probably, the problem with additive settings is the hardest one here.
Currently I don't see any alternative solution better than my.

However, GSoC app loading project could change situation drastically,
and maybe in the end of summer I'll change my mind of the better
solutions for configuration problems. But currently it aims to change
the way how per-application-instance settings are set, not global
settings. In the opposite, I'm interesting mainly in splitting global
settings into different modules.

Also please note, that the more different applications you want to
reuse, the more recognizable the problem is to you. I don't try to
accuse you that you don't seem to have a lot of reusable applications
in your projects -- you might already use some of the methods
described in SplitSettings wiki page, or you're just a fan of big
settings.py file -- I'm just telling you that I'm a big fan of modular
projects, and have dealt with them a lot, and so I wanted to have
really good and obvious solution for the configuration problem.

> --
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

[Simone Federici]

I think it is best that each application has its own configuration files.

The problem is usually the opposite, how to use a pluggable application without changing the source code. And that is why usually the application writes the global settings.

I'm using this snippet of code in order to separate local application settings with global project settings.

Simone

***

from django.conf import settings as django_settings

class ModuleSettings(object):

    """

    This class is used in the pluggable applications

    in order to separate local application settings 

    from global project settings

    

    So each application can have the own settings in own configuration 

    file in the app/settings.py

    >>> SAMPLE_CHOICES = ["1", "b", "c"]

    

    this class is used in app/__init__py to replace the local settings module with an object:

    >>> from commons.utils import ModuleSettings

    >>> settings = ModuleSettings('MYAPP', settings)

    

    in this way the application can read and write the own configuration from 

    the own settings file, but you can always override the application settings configurations 

    by writing in the global project settings the costants with the application prefix.

    >>> MYAPP_SAMPLE_CHOICES = ["A", "B", "C"]

    """

    

    

    def __init__(self, prefix, settings_module):     

        self.__prefix = prefix

        self.settings_module = settings_module   

        

        # replace the local settings module with an object,

        # overriding the values writted in the global settings

        for name in dir(settings_module):                

            if name == name.upper():

                setattr(self, name,  

                        getattr(django_settings, "%s_%s" % ( prefix, name ), 

                                getattr(settings_module, name)))

[Russell Keith-Magee]

On Fri, Jun 4, 2010 at 3:30 PM, bur...@gmail.com <bur...@gmail.com> wrote:
> Hi Russell,
>
> My writing style sometimes is really clumsy. I'm sorry about that.
>
> You might look at the end of the first message in the thread.
> Or maybe the thread topic.
>
> The problem is that half of third party plugins write: "after install,
> add this and this and this to settings.py".
> The problem is that I have reusable applications and they have their
> bits of default configuration, have their own options.
>
> Problem is that some of such settings are additive, like
> MIDDLEWARE_CLASSES, DATABASES or DATABASE_ROUTERS, and you wanted to
> give an app the possibility to configure their part of global
> configuration, but not override settings which are set by another
> applications.
>
> Problem is that, like everybody, I have different machines where i run
> my projects, and they have different settings (DEBUG, DATABASES, etc).
>
> I'm not the first here. The problem and the common ways to solve it
> are already observed in Django wiki under
> http://code.djangoproject.com/wiki/SplitSettings section.

And, as the wiki says:

"""
We don't need a default solution for this. It's not within the scope
of this project to tell people how they should organize their settings
files. Take that opportunity to showcase your individualism.
"""

I agree with Adrian -- this isn't an area where a complex set of
builtin tools will improve the situation. Every situation will have
slightly different requirements. The good thing about Django using
Python for configuration is that just about anything is possible.

That said, there are some common patterns, and we should do a better
job at documenting these patterns -- if only to highlight how flexible
the configuration system can be.

> Probably, the problem with additive settings is the hardest one here.
> Currently I don't see any alternative solution better than my.

The wiki page you mention lists half a dozen approaches, and still
doesn't mention the obvious option: from submodule.settings import *

I'm simply not convinced that this is as big a problem as you make
out. I have some large projects with lots of apps and lots of
settings, but with some organizational discipline and the use of
python imports, I've been able to manage that complexity.

The only area in this that I might concede that there is need for
major improvement is in applications providing default values for
their own settings (i.e., contrib.admin default settings should be
defined by contrib.admin, not in global_settings.py). However, I don't
think this requires the introduction of a complex setting
autodiscovery process -- it just requires a convention about where
default values can be placed in an app so that they will be picked up
by the settings initialization process.

In fact, I would almost argue that it's a *good* thing that it's hard
for applications to add default settings -- I'm not a big believer in
the dictum that more settings necessarily make things better. Settings
are required to point applications at specific services, or to
configure specific extension points. However, IMHO, an application
that requires dozens of carefully configured settings is often an
indication of a poorly designed app.

As for 'application configuration files' -- I think this is a
misnomer. Whenever you deploy a Django project, you're deploying a
project -- an amalgam of multiple apps. There's no such thing as the
configuration for a single app. All apps are potentially dependent on
other apps -- consider the potential impact of middleware and context
processors on the operation of any application. When you configure a
Django project, you really are configuring the entire project,
specifying the interaction of the many parts as a whole; having a
conceptual single project-level configuration file drives home this
fact.

Yours,
Russ Magee %-)

[bur...@gmail.com]

Hi Russell,

I strongly disagree with your and Adrian vision of whether conventions
are good or not.
But I won't comment that any further. There are your political
decisions, and I have no single bit of control on them.
I know that it's impossible to persuade you, so why should I spend my
time doing this.

However, you didn't tell anything on the problem that was the main
reason why I wrote this app.
Additive variables.

DATABASES, DATABASE_ROUTERS, MIDDLEWARE, etc.
Do you think situation will change with them?

In example, I want few of my apps to work with their own databases.
They need to install their database into DATABASES and their router
into DATABASE_ROUTERS.

How would you do that?

[Tom Evans]

On Fri, Jun 4, 2010 at 4:54 PM, bur...@gmail.com <bur...@gmail.com> wrote:
> Hi Russell,
>

> I strongly disagree with your and Adrian vision of whether conventions
> are good or not.
> But I won't comment that any further. There are your political
> decisions, and I have no single bit of control on them.
> I know that it's impossible to persuade you, so why should I spend my
> time doing this.
>
> However, you didn't tell anything on the problem that was the main
> reason why I wrote this app.
> Additive variables.
>
> DATABASES, DATABASE_ROUTERS, MIDDLEWARE, etc.
> Do you think situation will change with them?
>
> In example, I want few of my apps to work with their own databases.
> They need to install their database into DATABASES and their router
> into DATABASE_ROUTERS.
>
> How would you do that?
>

Apps shouldn't configure what databases they use, the project should
configure what database an app uses.

If the app is multi-db aware, and wants its own database, it should
use a value from settings to determine the DB to use, and the project
manager configures both the DB name, and the entry in DATABASES to
give it the right configuration.

Eg:

settings.py:

FOO_APP_DB_NAME = 'fooapp'
DATABASES = { 'default': { .. }, 'fooapp': { ... }, }

Cheers

Tom

[Russell Keith-Magee]

On Fri, Jun 4, 2010 at 11:54 PM, bur...@gmail.com <bur...@gmail.com> wrote:
> Hi Russell,
>

> I strongly disagree with your and Adrian vision of whether conventions
> are good or not.
> But I won't comment that any further. There are your political
> decisions, and I have no single bit of control on them.
> I know that it's impossible to persuade you, so why should I spend my
> time doing this.
>
> However, you didn't tell anything on the problem that was the main
> reason why I wrote this app.
> Additive variables.
>
> DATABASES, DATABASE_ROUTERS, MIDDLEWARE, etc.
> Do you think situation will change with them?
>
> In example, I want few of my apps to work with their own databases.
> They need to install their database into DATABASES and their router
> into DATABASE_ROUTERS.
>
> How would you do that?

Like Tom said - you don't solve it by configuring the app. You
configure the way a project uses an app, not the way an app should be
used in a project. His example for configuring DATABASES is right on
the money.

As an example of why the 'app configuration' approach fails, consider
the case of reusable apps. A reusable app can suggest defaults for
settings, but once a reusable app you wrote is in my project, I need
to configure it to conform to my local conditions. Your app *cannot*
know what database it needs to use when it's in *my* project. That's a
project configuration issue for me.

Yours,
Russ Magee %-)

[bur...@gmail.com]

Actually, some of my apps know what database they need to use, cause
they always use the same databases at the same database server unless
these apps are used for development. These are well established
applications and they have fixed names for databases (you can override
the names, but the names are very specific so you will never need to
override the names).

This is exactly why such app sets its own database options in global.py .
And for each development machine I have
conf/development_machine1__overrides.py which contains overrides.
I believe it's better than having "if socket.getfqdn() ==
'development_machine1':" conditions in settings.py

But, if *my* app doesn't know what database to use in *your* project, then:
1. It will provide sensible and customizable defaults for app-specific
database and app name (let's call them app.LABEL and app.DB) in its
own options namespace (yes, GSoC app loading feature will handle
that).
2. That variable will be used for its router.
3. Router will add itself into DATABASE_ROUTERS
4. No any settings will be added into DATABASES.
And if your used django-configurator, then the rules are pretty simple:
5. Database will be added into the list of the databases in conf/global.py
6. For development machine(s), this database will be overridden in
conf/development_machine1__overrides.py

In your case the configuration will be like this:

FOO_APP_DB_NAME = 'fooapp'
FOO_APP_APP_LABEL = 'fooapp'
DATABASES = { 'default': { ... }, 'fooapp': { ... }, }
if socket.getfqdn() == 'development_machine1':
DATABASES['fooapp'] = { ... }
DATABASE_ROUTERS = ('fooapp.routers.Router',...)

I had this before django-configurator, and I don't think it was any
better than it is now.

You can't even do

from foo_app.settings import *

cause that won't let you to add a router or set up a database!

(well, you can do
import_from_file(locals(), socket.getfqdn().replace('.','_').replace('-','__'))
but it's effectively the same I do in configurator.autodiscover(locals())
hm... do you think this one is a common boilerplate for importing
machine-dependent settings?
)

As alternative, maybe, do we need a registry for routers instead of
settings variable?
The same question should be asked for other additive variables in
settings.py of course.

[Russell Keith-Magee]

... until the very first time that you *do* need to. Seriously - you
simply *cannot* make any assumptions about the deployment environment
in which a user will be using your app. Every single time in my life I
have made the statement "Nobody will ever want/need to..." I have been
proven wrong. Consider it a corollary of Rule 34 :-)

> This is exactly why such app sets its own database options in global.py .
> And for each development machine I have
> conf/development_machine1__overrides.py which contains overrides.
> I believe it's better than having "if socket.getfqdn() ==
> 'development_machine1':" conditions in settings.py
>
> But, if *my* app doesn't know what database to use in *your* project, then:
> 1. It will provide sensible and customizable defaults for app-specific
> database and app name (let's call them app.LABEL and app.DB) in its
> own options namespace (yes, GSoC app loading feature will handle
> that).
> 2. That variable will be used for its router.
> 3. Router will add itself into DATABASE_ROUTERS
> 4. No any settings will be added into DATABASES.
> And if your used django-configurator, then the rules are pretty simple:
> 5. Database will be added into the list of the databases in conf/global.py
> 6. For development machine(s), this database will be overridden in
> conf/development_machine1__overrides.py

And to all of this, I'd fall back on the Zen of Python: explicit is
better than implicit.

IMHO, this is doubly true when dealing with something as critical as
configuration -- personally, I want my configuration files to do
*nothing* surprising or automatic.

For example - you say that your application needs to run in a separate
database. My DB Admin (or hosting provider) won't let me have another
database instance. How does your automated "add this extra database"
approach handle the case where I don't want you to add another
database?

> As alternative, maybe, do we need a registry for routers instead of
> settings variable?
> The same question should be asked for other additive variables in
> settings.py of course.

Again, I don't see why. IMHO we should be letting the end user
configure their site, not try to impose a semi-automated partial
configuration gathering scheme, and then provide a scheme for the end
user to override the default autoconfigured setup (and trust me - no
matter what scheme you propose, there *must* be a way to override it,
because it doesn't matter how smart you are - someone out there will
have needs that step outside the capabilities of your implementation).

Yours
Russ Magee %-)

[Vinay Sajip]

On Jun 5, 8:08 am, Russell Keith-Magee <russ...@keith-magee.com>
wrote:

> in which a user will be using your app. Every single time in my life I
> have made the statement "Nobody will ever want/need to..." I have been
> proven wrong. Consider it a corollary of Rule 34 :-)
>

That's why the YAGNI principle needs to be applied with care ;-)

On this topic, though, I fully agree with you. App configuration
should always be at the mercy of site-wide (project) configuration,
otherwise the app's usability is severely restricted. Maybe that's
fine for some of Yuri's apps, but as a general principle, hardwired
connections between specific databases and apps sounds broken to me,
too.

Regards,

Vinay Sajip

[bur...@gmail.com]

On Sat, Jun 5, 2010 at 2:08 PM, Russell Keith-Magee

I can remember "sparse is better than dense" and a line that
namespaces are a good idea.

> IMHO, this is doubly true when dealing with something as critical as
> configuration -- personally, I want my configuration files to do
> *nothing* surprising or automatic.

It's just a tradeoff between DRY and explicitness, as everything else.
Suggested approach doesn't force you to use automatic configuration.
You may not use it. But why can't I use automatic configuration,
and why 3rd-party plugins creators can't?
This is a question of creditability to this configuration model at all.

> For example - you say that your application needs to run in a separate
> database. My DB Admin (or hosting provider) won't let me have another
> database instance. How does your automated "add this extra database"
> approach handle the case where I don't want you to add another
> database?

Well, in some alternative reality, an app creator might really want to
define a router and a special database because it's cool, not because
it really needs to use its legacy database with its legacy table
names.

If you're afraid of using configurations incorrectly, you might either:
- turn off auto configuration as a whole
- redefine the DISCOVERY_ORDER as you want.
options.lazy.INSTALLED_APPS is a function.
- override the application option in conf/apps/app.py or
conf/global_overrides.py .

Anyway, it's like saying "we did 4 mistakes already, please don't ask
us to make 5th".
Because, generally, anything implementing observer pattern (django
signals, app templates, templatetags, admin.py files), has the same
problems:
- installation order can make sense
- plugins can inject wrong/unreliable components.

Hehe, by the way, I want another list of fields for
django.contrib.auth.admin by default, what should I do? :)

So, it's usually a step forward when you have a component architecture
to use observers for components configuration.
And I agree that observers are one of the most popular ways to make
large software projects overcomplicated, and Java listeners, C++
constructors and Ruby method injections are known leaders there.

So I just don't understand why settings can't be implemented as
provider/subscriber if most users will benefit from it.

>> As alternative, maybe, do we need a registry for routers instead of
>> settings variable?
>> The same question should be asked for other additive variables in
>> settings.py of course.
>
> Again, I don't see why. IMHO we should be letting the end user
> configure their site, not try to impose a semi-automated partial
> configuration gathering scheme, and then provide a scheme for the end
> user to override the default autoconfigured setup (and trust me - no
> matter what scheme you propose, there *must* be a way to override it,
> because it doesn't matter how smart you are - someone out there will
> have needs that step outside the capabilities of your implementation).

Sure, there is a way to override it. Please see above.
(But does this change anything?)

[bur...@gmail.com]

Hardwired connections between specific databases and apps sounds
broken to me too.
But I'm talking about providing sensible defaults.

[Tom Evans]

I'm not going to reply again, Russell has quite clearly ruled this out
but to reply to these points:

As the project manager, you can already use automatic configuration -
its python, you can do whatever the heck you like. Your scope is not
limited in any way, and if you want to auto-discover default settings
from your installed apps, do so. You could even formalize this into a
django app, publish it on googlecode and see how widely it is used by
people without requiring any changes to django itself. If such a
feature where to become standardized, and popular, then it could
potentially then be included into django.contrib, but it's completely
orthogonal to django itself - this is the benefit of using python for
configuration.

In our deployments, a project has a default configuration, which is
then overridden by a site specific settings_local.py, which is checked
out on a site by site basis alongside settings.py.

This is then imported into our settings with code like so:

try:
from settings_local import *
except:
# It's ok to not have any local settings.
pass

# Convert INSTALLED_APPS and MIDDLEWARE_CLASSES into lists,
# so we can append any extra ones supplied in settings_local.py
if EXTRA_APPS:
INSTALLED_APPS = list(INSTALLED_APPS) + EXTRA_APPS
if EXTRA_MIDDLEWARE:
MIDDLEWARE_CLASSES = list(MIDDLEWARE_CLASSES) + EXTRA_MIDDLEWARE

It is hardly a stretch to see how this kind of code could be extended
to try to import settings_auto or settings_default from each app in
installed apps.

>> For example - you say that your application needs to run in a separate
>> database. My DB Admin (or hosting provider) won't let me have another
>> database instance. How does your automated "add this extra database"
>> approach handle the case where I don't want you to add another
>> database?
> Well, in some alternative reality, an app creator might really want to
> define a router and a special database because it's cool, not because
> it really needs to use its legacy database with its legacy table
> names.
>
> If you're afraid of using configurations incorrectly, you might either:
>  - turn off auto configuration as a whole
>  - redefine the DISCOVERY_ORDER as you want.
> options.lazy.INSTALLED_APPS is a function.
>  - override the application option in conf/apps/app.py or
> conf/global_overrides.py .

It's not fear, its confidence. As the project manager, you should be
in control of what configurations are set, not application developers.
How do you propose that an application insert middleware? How do you
solve the ordering issues for settings like that? These problems
aren't insurmountable, but it is just easier to be in control of these
settings yourself. How hard is it to read a README and configure the
right settings for your project.

[bur...@gmail.com]

On Sat, Jun 5, 2010 at 5:07 PM, Tom Evans <teva...@googlemail.com> wrote:
Hi Tom,

> I'm not going to reply again, Russell has quite clearly ruled this out
> but to reply to these points:
>
> As the project manager, you can already use automatic configuration -
> its python, you can do whatever the heck you like. Your scope is not
> limited in any way, and if you want to auto-discover default settings
> from your installed apps, do so. You could even formalize this into a
> django app, publish it on googlecode and see how widely it is used by
> people without requiring any changes to django itself. If such a
> feature where to become standardized, and popular, then it could
> potentially then be included into django.contrib, but it's completely
> orthogonal to django itself - this is the benefit of using python for
> configuration.

I did: http://github.com/buriy/django-configurator

> In our deployments, a project has a default configuration, which is
> then overridden by a site specific settings_local.py, which is checked
> out on a site by site basis alongside settings.py.
>
> This is then imported into our settings with code like so:
>
> try:
>  from settings_local import *
> except:
>  # It's ok to not have any local settings.
>  pass
>
> # Convert INSTALLED_APPS and MIDDLEWARE_CLASSES into lists,
> # so we can append any extra ones supplied in settings_local.py
> if EXTRA_APPS:
>  INSTALLED_APPS = list(INSTALLED_APPS) + EXTRA_APPS
> if EXTRA_MIDDLEWARE:
>  MIDDLEWARE_CLASSES = list(MIDDLEWARE_CLASSES) + EXTRA_MIDDLEWARE

These are exactly my hate patterns.
The goal of django-configurator was to get rid of them in the projects I have.

Btw, you are not yet using the "state of the art" python-based solution:
storing different local_settings in the version control system,
so do a dynamic import based on computer DNS name.

> It is hardly a stretch to see how this kind of code could be extended
> to try to import settings_auto or settings_default from each app in
> installed apps.

Sorry, I'm not native speaker. Can't understand what's "hardly a
stretch" neither from context nor from dictionary.

>>> For example - you say that your application needs to run in a separate
>>> database. My DB Admin (or hosting provider) won't let me have another
>>> database instance. How does your automated "add this extra database"
>>> approach handle the case where I don't want you to add another
>>> database?
>> Well, in some alternative reality, an app creator might really want to
>> define a router and a special database because it's cool, not because
>> it really needs to use its legacy database with its legacy table
>> names.
>>
>> If you're afraid of using configurations incorrectly, you might either:
>>  - turn off auto configuration as a whole
>>  - redefine the DISCOVERY_ORDER as you want.
>> options.lazy.INSTALLED_APPS is a function.
>>  - override the application option in conf/apps/app.py or
>> conf/global_overrides.py .
>
> It's not fear, its confidence. As the project manager, you should be
> in control of what configurations are set, not application developers.
> How do you propose that an application insert middleware? How do you
> solve the ordering issues for settings like that? These problems
> aren't insurmountable, but it is just easier to be in control of these
> settings yourself. How hard is it to read a README and configure the
> right settings for your project.

I am of control. That's why god gave us ability to override.
It seems you are not aware what I suggest.

[Vinay Sajip]

On Jun 5, 11:22 am, "burc...@gmail.com" <burc...@gmail.com> wrote:
> > It is hardly a stretch to see how this kind of code could be extended
> > to try to import settings_auto or settings_default from each app in
> > installed apps.
>
> Sorry, I'm not native speaker. Can't understand what's "hardly a
> stretch" neither from context nor from dictionary.

"Hardly a stretch" == "Not exactly difficult"

Regards,

Vinay Sajip

[Chris]

> try:
>   from settings_local import *
> except:
>   # It's ok to not have any local settings.
>   pass

This pattern is used by almost everybody it seems. Is there any reason
why this pattern hasn't been adopted by "django-admin.py startproject"
yet?

[Tom Evans]

On Sat, Jun 5, 2010 at 11:22 AM, bur...@gmail.com <bur...@gmail.com> wrote:
....

> These are exactly my hate patterns.
> The goal of django-configurator was to get rid of them in the projects I have.

So? Problem solved, you can run your projects in precisely the manner
you choose, using your app.

Why then do you also want to change how everyone else does their
deployments to your system? Why, because you implicitly trust them,
should we allow pluggable application developers to define settings?!

>
> Btw, you are not yet using the "state of the art" python-based solution:
> storing different local_settings in the version control system,
> so do a dynamic import based on computer DNS name.
>

Are you proposing to determine the host name and then dynamically
import settings from that named configuration file? What a kludge -
that would require having every configuration file for all your sites
checked out in the same place.

Each of my deployments has its location specific configuration
deployed from administration RCS repo by cfengine to
settings_local.py, you can keep your 'state of the art'.

>> It is hardly a stretch to see how this kind of code could be extended
>> to try to import settings_auto or settings_default from each app in
>> installed apps.
> Sorry, I'm not native speaker. Can't understand what's "hardly a
> stretch" neither from context nor from dictionary.

How about this - 'It is easy to see how this kind of code could be
extended to try to import settings_auto or ....'

> I am of control. That's why god gave us ability to override.
> It seems you are not aware what I suggest.

I fully understand what you are suggesting, you cannot seem to
comprehend someone understanding what you are asking for, and
disagreeing with it.

One of your use cases for this is to allow a pluggable application to
specify it's own database settings, which is unthinkable to me.
Setting up, configuring and allocating databases is work for the
project manager, not for the application developer.

Similarly for any number of other settings that can have profound
effects on the functionality of the app, eg:
TEMPLATE_LOADERS
MIDDLEWARE_CLASSES
ROOT_URLCONF
INSTALLED_APPS
TEMPLATE_CONTEXT_PROCESSORS

and any number of project specific settings that an application may be
unaware of. The role of configuring a project is for the project
manager, not for the application developer. This is hardly rocket
science!

Cheers

Tom

[bur...@gmail.com]

On Mon, Jun 7, 2010 at 4:39 PM, Tom Evans <teva...@googlemail.com> wrote:
> Are you proposing to determine the host name and then dynamically
> import settings from that named configuration file? What a kludge -
> that would require having every configuration file for all your sites
> checked out in the same place.

I'm not requiring anything.
You can set your own DISCOVERY_ORDER for your settings files.
You can not use autodiscovery at all.

I'm talking not about requirements but abilities.

> Each of my deployments has its location specific configuration
> deployed from administration RCS repo by cfengine to
> settings_local.py, you can keep your 'state of the art'.

How about development?
Everyone's copy-pasting required settings from settings_local.py.template ?

>
>>> It is hardly a stretch to see how this kind of code could be extended
>>> to try to import settings_auto or settings_default from each app in
>>> installed apps.
>> Sorry, I'm not native speaker. Can't understand what's "hardly a
>> stretch" neither from context nor from dictionary.
>
> How about this - 'It is easy to see how this kind of code could be
> extended to try to import settings_auto or ....'

I still don't understand what do you mean by this.
Adding extra configuration files?
You are still able to do this, nothing changed for you unless you want
better autodiscovery.

>> I am of control. That's why god gave us ability to override.
>> It seems you are not aware what I suggest.
>
> I fully understand what you are suggesting, you cannot seem to
> comprehend someone understanding what you are asking for, and
> disagreeing with it.

You said a point that wasn't correct.
I doubted you understood what I suggested.

> One of your use cases for this is to allow a pluggable application to
> specify it's own database settings, which is unthinkable to me.

Ok, don't think about that use case then.
With every weapon you can shoot yourself in a foot.

Can't you do "from yourapp.settings import *"
and can't that settings.py set a database for you?

> Setting up, configuring and allocating databases is work for the
> project manager, not for the application developer.
>
> Similarly for any number of other settings that can have profound
> effects on the functionality of the app, eg:
> TEMPLATE_LOADERS
> MIDDLEWARE_CLASSES
> ROOT_URLCONF
> INSTALLED_APPS
> TEMPLATE_CONTEXT_PROCESSORS
>
> and any number of project specific settings that an application may be
> unaware of. The role of configuring a project is for the project
> manager, not for the application developer. This is hardly rocket
> science!

You have special person for configuring a project.
I'm configuring my projects myself.

You are happy with your "solution", I'm not.
I'm proposing a change that will make both of us happy.

I'm telling that every project configuration have the same steps in
its development.
single file -> two files -> machine dependent config files.

The same is for the app development:
no app-specific options -> app specific options

This evolution does never change its direction.

I suggest a solution that I think might spread.
Of the problem which is already widespread, but for which every
project config developer uses different solutions.

The solution is not obligatory, it's optional.
Like everything in Django.
You will still be happy.

[Tom Evans]

On Mon, Jun 7, 2010 at 12:28 PM, bur...@gmail.com <bur...@gmail.com> wrote:
> You have special person for configuring a project.

I wish; just different hats for different tasks.

> You will still be happy.

Only because this sort of auto configuration will not be going into
django, that much has been made abundantly clear.

Cheers

Tom

[Alex Gaynor]

> --
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>

For one thing I'd say because most *really* large projects realize
this is a really bad way of handling settings. From my experiences
I'd say it makes a lot of sense to invert the dependencies, localized
settings require the global defaults, not the reverse.

Alex

--
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me

[Chris]

On Jun 7, 10:45 am, Alex Gaynor <alex.gay...@gmail.com> wrote:

> On Mon, Jun 7, 2010 at 1:15 AM, Chris <cp368...@ohio.edu> wrote:
> >> try:
> >>   from settings_local import *
> >> except:
> >>   # It's ok to not have any local settings.
> >>   pass
>
> > This pattern is used by almost everybody it seems. Is there any reason
> > why this pattern hasn't been adopted by "django-admin.py startproject"
> > yet?
>

> For one thing I'd say because most *really* large projects realize
> this is a really bad way of handling settings.

Worse than one giant settings.py file?

> From my experiences
> I'd say it makes a lot of sense to invert the dependencies, localized
> settings require the global defaults, not the reverse.

The way I do it (and I'm assuming a lot of people do it this way too)
is to have global settings tracked in their source control, while
local_settings is something they have to write manually because it is
untracked (due to having passwords and all). So it's way lore likely
that the local settings file is missing instead of the global
settings. Also, it really doesn't matter if one is more likely to be
missing because both have to be present in order for the project to
run. Does it really matter if the global or the local settings file
raises the import error?

[Tom Evans]

Having posted the first way, I can appreciate that Alex's technique
gives slightly more control to the person writing the per-instance
config, without some of the cruft. For instance, my 'settings imports
settings_local' technique looks like this:

try:
from settings_local import *
except:
# It's ok to not have any local settings.
pass

# Convert INSTALLED_APPS and MIDDLEWARE_CLASSES into lists,

# so we can append any extra ones supplied in settings_local.py
if EXTRA_APPS:
INSTALLED_APPS = list(INSTALLED_APPS) + EXTRA_APPS

where as if settings.py is the 'local' file, with it loading default
settings from settings_default.py, it could look like this:

from settings_default import *
INSTALLED_APPS = list(INSTALLED_APPS) + [
'some_app',
]

The project will always have default settings, so no need for a
try/except block, and we can directly manipulate settings without
having to pass around extra arguments, it's much neater.

OTOH, it works exactly the same once deployed :)

Cheers

Tom

[Gabriel Hurley]

For the record, the "from settings_default..." method is exactly how
my team's largest projects are deployed. It works just fine for us!

Even more fun (though bordering on too tricky) is converting
INSTALLED_APPS to a set instead of a list so you can add *or* subtract
items.

This whole argument has gotten silly, though, since Russ (rightly)
rejected it about 20 posts back ;-)

All the best,

- Gabriel

On Jun 7, 9:47 am, Tom Evans <tevans...@googlemail.com> wrote:

[Jeliuc Alexandr]

Hello...
Modular or divided is not so good as You can feel it now, I think...
For example second web framework I use is symfony. And I hate it
because of this as You say "modularity".
While developing big project I have more then 50 open buffers in emacs
only, plus gvim buffers, plus console windows and console tabs. These
all are on concentrated on 4 desktops.
Q: Do I need 10 or more configuration files in addition to these?

> *<site> forwww.my-site.comis www_my__site_com (dots replaced with

[dffdgsdfgsdfhjhtre]

I think this is how it should go when settings are loaded:

1. goes through all installed apps, and tried to load settings.py.
2. then it loads the django defaults. This way, if an app settings
file tries to override DATABASES or something, it will get reverted to
the django defaults
3. then it loads the project's root settings.py.

This way each app has a settings.py file that acts as a central
location for all settings the app uses. Most apps these days relies
solely on "off module" documentation (README.txt, etc). IMO it's
always better to have documentation as part of the code itself.

--OR--

Just tell users to add "from app.settings import *" to the bottom of
their project settings file, then below that line, override whatever
you need. Of course for this to work, each app that uses settings
needs to provide a settings.py file to begin with (most apps do not).
I think the former method is better because it's more automatic and
less error prone. Either way is better than how it's done now though...