An improved settings.py configuration (revisited)

[VernonCole]

Several months ago, I floated an idea on this forum for re-structuring the settings.py environment.

It generated a fair amount of discussion.  The consensus was that something really ought to be done, but no one was sure just what.
One strong opinion was that any new changes must involve a major move toward 12-factor configuration.
Another opinion was that 12-factor was an okay idea, but is not the best answer to many problems, and simply cannot replace settings.py in any practical way.

So, I have been fiddling with the problem on and off, and I think I have a solution which would be acceptable to pretty much everybody.  Here is a not-so-short description:

1) Some people (like me, for example) really like the idea of structured settings and using DJANGO_SETTINGS_MODULE to switch their configurations around easily.
  So I created a new module directory, and named it "preset" (because the scripts in it are imported _before_ settings.py).  We can't call it "settings" because that would conflict with the name of "settings.py", which Python does not allow. (I was thinking about the six preset buttons on the radio of my Dad's 1957 Chevy.) These "preset" scripts import each other as in a classic J.K-M. structured settings design, until finally they import the common base settings.
  For convenience sake, I added preset/test_* to my .gitignore file so that I can create test scripts willy-nilly as needed.  I can happly switch between them using a simple
"./manage.py somecommand --settings=myproject.preset.test_whatever"
  Also, the preset directory is pre-populated with several example scripts.  Rather than trying to explain in settings.py how to create a DATABASES dictionary of dictionaries, there are several working examples for new guys to copy.  One even opens two databases using a router. This would have helped me a lot when I was getting started. There are also example intermediate-level presets to define local-ish things like time zone and email addresses and similar whatnot, with some of the verbose explanations from settings.py moved into them.

2) Some people want a pure 12-factor configuration with no messing around with configuration files.
  They are in luck.  The default preset is "<project_name>.preset.default_settings".  So if you don't select any (other) preset you get this one.  It is special, because it is a reworked version of Kenneth Reitz's fine dj_database_url module.  His idea was that you need about the same information to open a distant database as to access a distant file, so rather than burn up four or five environment variables, just lump all that information into a URL.  So you can say:
export DATABASE_URL="mysql://vernon:let...@my.server.info:3606/vernons_big_db?production=true;secret_key=gobbldygook"
  The query keys are passed back as a dictionary, so your settings modules can do whatever you want with them. You have two local-programmed presets,either staging.py or production.py, depending on the value of the "?production=" query.  They can be as different or as similar as you wish, or one import the other. Do you really want to set the time zone and the administrator email-address using environment variables?  Go right ahead.

3)  Some people like things fine just the way they are.  If it ain't broke, don't fix it.
   Relax, settings.py is right where you expect it, and still does everything it did before.  If you really want to keep editing this one file, rather than moving your site specific information down into the presets, we will not stop you.  All  presets must eventually import settings.py. Therefore, settings.py will always maintain its most important function: it defines which applications are part of this project. If proper presets are maintained, all implementations of a project will use identical settings.py files.

  One potential problem which I have been made aware of is: if settings.py imports local_settings (and expects those overrides to be final) there could be an unpleasant surprise when a preset overrides them. I added code to preset/defalt_settings.py to issue a warning if it sees that module local_settings has been imported -- but I don't know whether I like that solution.

You can see the present version of the code at https://github.com/vernondcole/formhub in branch make_presets_neater , if you really want to see how it would look in a production system.  There are a few rough things in the code which would be better with help from the core -- like the need to trap and modify ImportError exceptions in order to get proper stack traces for errors in settings modules. Also, I wonder if DATABASE_URL should be more like DJANGO_DATABASE_URL?

So -- is this worth building a Patch so that people can _really_ tear it apart?
--
Vernon Cole

[Carl Meyer]

Hi Vernon,

On 03/07/2014 07:44 PM, VernonCole wrote:
> Several months ago, I floated an idea on this forum for re-structuring
> the settings.py environment.
>
> It generated a fair amount of discussion. The consensus was that
> something really ought to be done, but no one was sure just what.

There was not consensus that "something really ought to be done." I was
part of that discussion, and my opinion was (and still is) that nothing
needs to be done (in Django core).

(For reference, a link to that discussion:
https://groups.google.com/d/topic/django-developers/O89Qb_9pwmo/discussion )

Your proposed technique covers a lot of options - that's great. It works
fine without any changes to Django, and you can package it up as a
startproject template, use it, and promote it.

I think the default startproject template should remain as simple as
possible (that is, a single-file settings.py) and not try to provide all
possible ways of handling multiple-deployment settings.

Carl

[Russell Keith-Magee]

I agree with Carl. It takes a very selective reading of that discussion to say that there was consensus on anything. 

I also agree that the default start project template should be kept simple, and any changes in project structure such as you have proposed should be maintained on a per-project basis, depending on the needs of individual projects.

Yours,

Russ Magee %-)

[VernonCole]

Russ & Carl:

Thanks for the course correction. I guess one tends to read what one wants to see, rather than what the writer wrote.

Yes, this would make a good package, and I can do some different things with it that way.  For example, I really want to type:

>dj -s test_vernon <command>

rather than:

>python manage.py <command> --settings=myproject.preset.test_vernon

I may call it something like django-quick-settings.

Again, thank you for your always kind and valuable input.

--

Vernon Cole