proposal: settings refactor
Carl Meyer
As my first post on pinax-core-dev ;-) I'd like to propose that Pinax
adopt Transifex' split settings solution. [1] It's flexible, it's
conceptually simple, and it solves many of the problems inherent in
import-based split-settings solutions.
The pattern is familiar from e.g. a Debian-based installation of
Apache (or many other projects), which has a conf.d/ directory
containing a list of configuration files like 05_something.conf,
10_something_else.conf, etc: the files are loaded in order (initial
number provides predictable lexicographic ordering) and each
configuration file is free to add to, modify or tweak the
configuration provided from previous files.
For Pinax, this would look like a top-level conf/ directory containing
something like 05_base.py, 10_pinax.py, etc (or whatever they might
be). The actual settings module itself becomes a simple wrapper that
lists the contents of the conf/ directory, orders them, and calls
execfile(fn) on each one in turn; or exec(open(fn).read()) to be
forward-thinking, since execfile has been removed in Python 3.
Because the code from each conf/ file is executed in the same local
namespace, each conf file is able to see and modify all of the
settings from the previous files, which allows plenty of flexibility
in appending to or tweaking settings like MIDDLEWARE_CLASSES, the new
DATABASES, etc.
It also makes it easy to use various systems for deployment-specific
settings: a VCS-ignored 90_local.py file (which doesn't even need to
exist), or a sub-directory of host-specific conf files, one of which
can be symlinked into the conf/ directory on a given deployment.
It's also trivial to provide a 99_lazy.py file which can contain
"lazy" settings, whose value ought to be based on the value of some
other setting, which may be modified by other settings file (including
a deployment-specific file).
The implementation of this is quite simple, but I do have it
implemented for all my projects and would be happy to contribute the
implementation to Pinax if the core devs approve.
cheers,
Carl
[1] http://code.djangoproject.com/wiki/SplitSettings#UsingalistofconffilesTransifex
Wes Winham
manage settings. It's especially difficult while taking advantage of a
project like Pinax where you're managing settings for Pinax
applications that might change from version to version, settings for
3rd-party applications that might change and settings for any project-
specific applications, all of which might need overridden in complex
ways depending on the environment. I definitely think it's something
with room for improvement.
I also love the debian-style configurations for my system in general.
I guess the question I have about borrowing that style for
configuration files in python is whether or not there's a burden when
debugging and managing settings. Just brainstorming, but all of the
configuration setups I've seen with multiple files and overrides have
been totally declarative. Is giving someone the flexibility of python
across multiple files but in one namespace too much? I suppose we're
already kind of in that situation with application-level defaults and
such, though.
From a deployment standpoint, I definitely see a big advantage to this
multiple-file style of configuration. Right now I have some hacky
fabric tasks that use a mixture of string templating and sed to cajole
my settings_local to the specific combination of settings for
continuous integration, test, staging, beta, live, local etc.
deployments. It kind of sucks.
-Wes
> [1]http://code.djangoproject.com/wiki/SplitSettings#Usingalistofconffile...
Carl Meyer
complex to manage than, say, Apache Debian config files. With the full
power of Python, clearly it would be possible for a user to do all
sorts of insane things in their config files; but that's true of
settings.py to begin with. And yes, this system is a big improvement
for deployment scenarios.
So I've got a demo implementation of this in Pinax in my settings-
proposal branch: http://github.com/carljm/pinax/commit/1ca3865a40fd04388d542dd468486183b7367d64
So far it's only implemented in basic_project, and I more or less just
moved the current settings.py to conf/05_pinax.py; I didn't yet do any
refactoring into multiple files to take advantage of the new system. I
think that piece would be better done by someone more familiar with
Pinax.
The settings.py file in the project is quite small; most of the real
functionality is imported from pinax.utils.conf (where I arbitrarily
placed it; let me know if I should move it elsewhere). There's a bit
of a locals() dance that has to be done in order to move the actual
exec-ing into imported code, but get the resulting settings back into
the settings.py local namespace.
Any thoughts or objections? If the core devs are +1 on moving forward
with this, I'll implement it in the other projects.
Carl
Jannis Leidel
For the record, I'm generally +1 on this proposal and have used the concept myself in a few projects outside of Pinax and enjoyed using it when working with Transifex.
It requires a little attention in case a project needs to be distributed since the conf files are package data to distutils because if we don't add a ``__init__.py`` to the ``conf/`` dir. A ``recursive-include path/to/conf/*.ext`` in the manifest template of the project (if it exists) suffices.
Regarding the file extension, I'm torn between using the ``.conf`` file extension, which might be a source for confusion since it hides the fact that "it's just Python", and ``.conf.py`` which is more verbose but would highlight the fact that it's a non-importable config file and would still conveniently apply Python code highlighting.
Jannis
James Tauber
On Feb 25, 2010, at 4:00 AM, Jannis Leidel wrote:
> For the record, I'm generally +1 on this proposal and have used the concept myself in a few projects outside of Pinax and enjoyed using it when working with Transifex.
Carl and I discussed this in person and I'm +1 too.
> It requires a little attention in case a project needs to be distributed since the conf files are package data to distutils because if we don't add a ``__init__.py`` to the ``conf/`` dir. A ``recursive-include path/to/conf/*.ext`` in the manifest template of the project (if it exists) suffices.
I don't think we should add a __init__.py but should document the need for a recursive-include for packaging.
> Regarding the file extension, I'm torn between using the ``.conf`` file extension, which might be a source for confusion since it hides the fact that "it's just Python", and ``.conf.py`` which is more verbose but would highlight the fact that it's a non-importable config file and would still conveniently apply Python code highlighting.
.conf.py feels a little odd to me.
My preferences is just .py but I do prefer .conf.py to just .conf (for the code highlighting)
James
Brian Rosner
On Feb 25, 2010, at 12:08 AM, Carl Meyer wrote:
> So I've got a demo implementation of this in Pinax in my settings-
> proposal branch: http://github.com/carljm/pinax/commit/1ca3865a40fd04388d542dd468486183b7367d64
Thanks so much for working on this Carl!
>
> So far it's only implemented in basic_project, and I more or less just
> moved the current settings.py to conf/05_pinax.py; I didn't yet do any
> refactoring into multiple files to take advantage of the new system. I
> think that piece would be better done by someone more familiar with
> Pinax.
We can start up a branch on pinax/pinax repo where we can collaborate in a more official manner. I can start that based on your branch. I'll do this tomorrow.
> The settings.py file in the project is quite small; most of the real
> functionality is imported from pinax.utils.conf (where I arbitrarily
> placed it; let me know if I should move it elsewhere). There's a bit
> of a locals() dance that has to be done in order to move the actual
> exec-ing into imported code, but get the resulting settings back into
> the settings.py local namespace.
This little bit has left me a bit uneasy. I spent some time thinking about it on my back from PyCon and think I have a slightly better solution.
We have a lot of duplication of environment bootstrapping in many of the project-level files. manage.py and deployment files. It occurred to me this can be cleaned up. I started a new branch for this clean-up [1]. What we can do based on this branch is set DJANGO_SETTINGS_MODULE to something like pinax.conf.settings where we can run the code to setup project-level settings. We likely will need to set an environment variable to store the location of conf directory. I find this a decent improvement in terms of setup. Any obvious issues or thoughts on this?
[1]: http://github.com/brosner/pinax/compare/master...new_env
Brian Rosner
http://oebfare.com
http://twitter.com/brosner
Brian Rosner
An obvious issue I didn't consider at first is handling of multiple settings files. If we hijack DJANGO_SETTINGS_MODULE this breaks all cases where users might be relying on it or --settings (to many management commands) to point to another settings file. I am going to need to go back to the drawing board on that bit. The rest of my work on new_env branch seems sane enough and cleans up the code some to make it more maintainable.
Joan Miller
disaster and it's going not maintainable as has been showed in Django.
My solution was to use YAML, and then to use a class to convert it to
Python. Features:
* The python configuration is only re-built if yaml config. has been
changed.
* It can add some configuration depending of some variable, as debug.
See in lines where is '?'.
* It adds the absolute paths where it's necessary.
- YAML config. initial => http://pastebin.com/CVqGD1eL
- Python generated => http://pastebin.com/AB4FaBgy
> [1]http://code.djangoproject.com/wiki/SplitSettings#Usingalistofconffile...
Brian Rosner
On Mar 10, 2010, at 3:21 AM, Joan Miller wrote:
> Whatever configuration system using variables of a language is a
> disaster and it's going not maintainable as has been showed in Django.
I strongly disagree. It may not have worked *for you*, but there are a metric ton of projects out there using Django that get along just fine.
>
> My solution was to use YAML, and then to use a class to convert it to
> Python. Features:
>
> * The python configuration is only re-built if yaml config. has been
> changed.
> * It can add some configuration depending of some variable, as debug.
> See in lines where is '?'.
> * It adds the absolute paths where it's necessary.
>
> - YAML config. initial => http://pastebin.com/CVqGD1eL
> - Python generated => http://pastebin.com/AB4FaBgy
I'm not sold on using something like YAML to convert into Python. I want Pinax to be as much like Django as possible. This is would take in the opposite direction even if it is generating the Python. The problem we are addressing is not configuration format, but rather loading order and scope. Your solution doesn't give us anything to solve the problems we are after.