Add strutctured settings module to django 1.7?

[VernonCole]

I find myself using up lots of time and keystrokes explaining about the benefits and methods of a structured settings module in django. 
For example: Using-a-Structured-Settings-environment

It occurs to me that life would be easier if django shipped already set up for structured settings, rather than having to retro-fit it.  The pull request I sent to formhub could be a starting point for the conversion. (formhub/pull/1240)  The changes to core are simple enough that I could do it.

Your  opinions please, Would such a change be good for django 1.7?
--
Vernon Cole
eHealth Africa

[Russell Keith-Magee]

Based on conversations that I had at the sprints at DjangoCon US, I'd say probably not -- because the winds are blowing in another direction.

The primary reason to need structured settings like this is because your development environment is different to your deployment environment (e.g., different passwords, paths and so on). 

However, the emerging best practice for this sort of thing is best described by the "12 factor" approach:

http://12factor.net

Factors 10 and 3 are the most relevant to this discussion -- 10 says that development and production should be the same; 3 says that anything that needs to vary should be set as an environment variable, and consumed from there.

So - I'd expect to see Django moving towards better support for a 12 factor environment, rather than embedding separate settings files as a deployment practice.

Yours,

Russ Magee %-)

[Michael Mior]

Given that, what about incorporating something like DJ-Database-URL[1] into Django? It would be great if this could be the default if DATABASES was left unspecified.

[1] https://github.com/kennethreitz/dj-database-url

[Jannis Leidel]

Speaking of which (apologies if I hijack the thread), django-configurations now has Value classes that can inspect the process environment to set the values of settings: http://django-configurations.readthedocs.org/en/latest/values/

Since I've really enjoyed using the URL-based approach for some settings following the 12 factor methodology, I've also added some special Value subclasses to setup DATABASES, CACHES and the EMAIL* settings (each using the respective 3rd party app to do the actual heavy lifting, like the also mentioned dj-database-url).

Just a side note..

Best,
Jannis

[VernonCole]

Okay, I read it.  I think that it argues for structured settings, not against them.

What I read was a set of guidelines, rather like the Zen of Python. They look like good, reasonable rules-of-thumb to me, and I have been around long enough to recognize such.  One of the reasons I adopted Python was that I found my self in such complete agreement with the Zen the first time I read it. I particularly liked its built-in admission that there will be exceptions to its principles. 

Let's talk about three of these twelve.  I have cut-and-paste-ed them from the web page.

I. Codebase

One codebase tracked in revision control, many deploys

III. Config

Store config in the environment

X. Dev/prod parity

Keep development, staging, and production as similar as possible

Note that number three says "in the environment" not "in a large collection of operating system environment variables". Those are different things. More about that later.

First, note that rule number ten works against rule number one.  The reason that the codebase we call "django" gets deployed so often is because it is so flexible.  The same codebase runs on a Windows laptop and a Java server in a dark server room.  "As similar as possible" gets stretched a very long way.  Yet the fact that I am actually running the same codebase in both situations means that there are a _lot_ of similarities - more than there are differences -- because the codebase is programmed to minimize the differences.   But there must be a way to tell that large, flexible codebase what we need it to do today -- and today in django, that is done in settings.py.

Other products have other methods of specifying their configurations.  The database system I helped write in 1982 had a database-of-databases: you switched environments by specifying (using a command line switch or an operating system environment variable) which root database you were using at that moment.  It eventually worked on eight different operating systems from RT-11 to Windows-32.  Microsoft dotNET uses xml files stored alongside the executable files for their configurations.  If you have ever had to modify one of those monsters, you will appreciate the simplicity of settings.py.  At times, I have written Python scripts to modify the dotNET config files in place. Ugly! Then there is the Windows registry, lets say no more about that.  The point is: configurations have to be stored somewhere.

So how do we store them "in the environment"? 

1) As multiple operating systems environment variables.  Each operating system will need to be separately documented as to how to set environment variables.  (Have you ever tried to explain that to a beginning Windows user?)  Each *nix shell will also require separate documentation, and a separate shell script to set them.  Here there be dragons! Or, perhaps, someone will write a Python script to set them all.  :-o

2) In a pre-defined database-database, perhaps overridden by an environment variable or command line switch.

3) In a generic file with a known name, perhaps overridden by an environment variable or command line switch, in a format as yet to be defined and fought over.

4) In a Python file with a known name, perhaps overridden by an environment variable or command line switch, which can import more generic configuration files, which can import others, until a base standard configuration is reached.

5) In a Python file with a known name, perhaps overridden by an environment variable or command line switch, which may import another Python file with a known name, which cannot be stored in version control.

Numbers three or four could be included in version control, and selected at run time as needed.  I think that is a much more secure "environment" than a motley set of instructions of how to set environment variables in shell xyz, or how to encode some kind of URL giving the same information.

Numbers four and five use a format already familiar to django users, and which is extensible so that more advanced methods of configuration setting (such as the two mentioned elsewhere in this thread) can easily be implemented, if desired.

Number five is what we have today.

I am suggesting number four.

Did that sell anyone?
--
Vernon

[Marc Tamlyn]

No, when 12 factor says in the environment, it means exactly that. *NOT* checked into version control, and in the environment of the shell. 12-factor is talking *only* about number 1 for differences in your environments.

Yes windows is not very good at environment variables in general, but I would argue if you're developing on windows and deploying to *nix you're already doing it wrong. I refer you to point 10. You may however be interested in Jannis' python port of envdir - https://github.com/jezdez/envdir - which works on Windows.

Personally, I find the multiple settings files (local_settings.py, structured settings) to be a pattern I'm moving away from, and I'm unlikely to support changes to Django encouraging it more. Jannis' configurations is a bit different, I find it's a nicer way of grouping your settings together, making them overrideable from external apps etc. The end game is still one settings file, with environment switches.

The advantage of the current settings.py is that it's very simple and flexible, which allows these patterns to all work together. At some point we may change to another system, but I don't feel we should be actively encouraging any particular path at present. There isn't a clear consensus as to the best way forwards.

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

[VernonCole]

(Memo to self: always think about it for a day before replying.)

Okay, I see.  And I certainly agree that a few strategically placed instances of (for example)
>>> psw = os.environ.get('DBPASSWORD', 'DumbDefault')
would help security in most cases.

I will get back to the drawing board, then, and see if I can still get the optimization I am looking for without making the system more difficult for those who don't want it.

My use case, is, I'm afraid, even more wrong than you suggest, Marc.  I develop on Linux, but have to plan for instances where unfortunate individuals must deploy on Windows.  Administrators of those systems are often neophytes at systems administration in general and Python in particular.  Installation of something like envdir would be far beyond hope.   More frequent will be deployments by people who may know Windows systems administration, but this will be their first experience with Linux.  Simplicity is everything.

Thank you, everyone, for the quality input.
--
Vernon

[Andres Osinski]

Honestly, I find the idea of *not* using a settings file to be inconvenient, and the notion of using environment variables for doing anything aside from pointing to a settings file to be pretty disgusting.

Here are my thoughts, in no particular order:

- Pretty much everyone out there uses some form of settings + local_settings.py for configuration, keeping the installation site-specific in the local settings, and checking in a local_settings.example.py file for instructions. This has never really been an issue for me except for database-specific content settings, which is not something environment variables solve.

- The system environment variables is a flat namespace, completely polluted by different shell's variables, with high potential for unintended name clashes.

- For development, reloading an env var does not reset the development server; I usually *want* the dev server to be reset whenever there's a config change.

- You have to set your env vars from somewhere, adding an extra layer of indirection and complexity.

- Not all shells have facilities to easily navigate settings; it's easier if they're stored in a startup shell file, but that offers no advantage from a Django file

- I like the fact that you can use Python facilities like os.path.join to generate config variables on startup.

--
Andrés Osinski
http://www.andresosinski.com.ar/

[Michael Manfre]

I don't think there will ever be a true consensus. I predict that the option that would leave the least number of people unhappy is a settings API that allows for 3rd party settings backends. Having settings in the environment is ideal for deployments, but what each project and admin consider to be the environment will probably not be the same.

Adding support for a settings API shouldn't be too difficult (tm). LazySettings gives us a nice place to hook in and import a 3rd party SettingsBackend. DJANGO_SETTINGS_MODULE already defines a workable way of choosing which settings to load. We'd need to define a way of structuring its value to specify which backend to load and provide it with enough information to get going. A URL format would work well for almost every type of settings backend that I could think of. In addition to the same basic settings behaviors, a SettingsBackend would need to add reload() and save().

Is rough idea worth pursing? If so, I'll be happy to spend time brainstorming and prototyping it. Settings have been a very annoying pain for me for many years.

Regards,

Michael Manfre

[Carl Meyer]

On 09/26/2013 09:02 AM, Michael Manfre wrote:
> I don't think there will ever be a true consensus. I predict that the
> option that would leave the least number of people unhappy is a settings
> API that allows for 3rd party settings backends. Having settings in the
> environment is ideal for deployments, but what each project and admin
> consider to be the environment will probably not be the same.
>
> Adding support for a settings API shouldn't be too difficult (tm).
> LazySettings gives us a nice place to hook in and import a 3rd party
> SettingsBackend. DJANGO_SETTINGS_MODULE already defines a workable way
> of choosing which settings to load. We'd need to define a way of
> structuring its value to specify which backend to load and provide it
> with enough information to get going. A URL format would work well for
> almost every type of settings backend that I could think of. In addition
> to the same basic settings behaviors, a SettingsBackend would need to
> add reload() and save().

I agree that there's unlikely to be consensus anytime soon on One Right
Way to handle configuration (as evidenced by this very thread).

I think your proposal needs stronger justification, though. We already
_have_ a "settings API": you can do whatever you want to load your
settings, as long as in the end you provide Django with a Python module
object populated with names and values. There are several existing
third-party projects building on this "API" to provide more opinionated
and featureful configuration handling (django-configurator,
django-configglue, django-configurations).

This API has limitations, sure, but it also has two advantages: it's
quite simple, and everyone's already adjusted to it. I think any
proposal to build a more complex system into Django core needs to
clearly explain what it allows you to do better than the current API.

Carl

[Tom Evans]

On Thu, Sep 26, 2013 at 2:21 PM, Andres Osinski
<andres....@gmail.com> wrote:
> Honestly, I find the idea of *not* using a settings file to be inconvenient,
> and the notion of using environment variables for doing anything aside from
> pointing to a settings file to be pretty disgusting.

+1

The idea that before you start working on a project you need to infer
and set the 50 or so environment variables required is anathema to me.

The most important concern is that these settings are not part of your
code, they are part of your configuration. Consistency of
configuration is one of the most important things in DevOps, and so
IMHO the most important thing in your project is tracking and managing
the changes in your configuration.

To me, that means one thing. Configuration files live in VCS
repositories - this is the only way to track changes to your
configuration, to validate that there are no unexpected changes to the
file.

Cheers

Tom

[Carl Meyer]

Hi Tom,

On 09/27/2013 09:51 AM, Tom Evans wrote:
> On Thu, Sep 26, 2013 at 2:21 PM, Andres Osinski
> <andres....@gmail.com> wrote:
>> Honestly, I find the idea of *not* using a settings file to be inconvenient,
>> and the notion of using environment variables for doing anything aside from
>> pointing to a settings file to be pretty disgusting.
>
> +1
>
> The idea that before you start working on a project you need to infer
> and set the 50 or so environment variables required is anathema to me.

FWIW, this is a straw man; I don't believe anyone is doing that. I've
recently switched to handling more configuration via env vars, and what
it actually looks like in my case is a normal (single) settings file
with a lot of settings values optionally read from the environment, with
defaults if the env var is not present. Typically a dev environment can
be run with zero env vars, a production environment will have a few
(secret key, database url, sentry dsn, api keys, etc).

> The most important concern is that these settings are not part of your
> code, they are part of your configuration. Consistency of
> configuration is one of the most important things in DevOps, and so
> IMHO the most important thing in your project is tracking and managing
> the changes in your configuration.
>
> To me, that means one thing. Configuration files live in VCS
> repositories - this is the only way to track changes to your
> configuration, to validate that there are no unexpected changes to the
> file.

Totally agreed, but I'm often in a situation where its not possible for
the primary codebase to be kept appropriately private, meaning there
needs to be some solution for injecting "secret" site-specific
configuration into a production instance. The only two solutions I've
used for that are local_settings.py and env vars, and having used both
I'm much happier with env vars. (Even with env vars, I track them in a
simple shell file in a separate, private deployment VCS repo.)

Carl

[andres....@gmail.com]

What is the difference between privately tracking a shell file and a Python config file?
Enviado desde mi BlackBerry de Movistar (http://www.movistar.com.ar)

[Carl Meyer]

On 09/27/2013 10:52 AM, andres....@gmail.com wrote:
> What is the difference between privately tracking a shell file and a Python config file?

Not a lot of difference, but the env vars give you more flexibility. In
a single settings file using env vars, I can easily have things like a
"mode switch" in a single env var that affects the defaults for a number
of related settings, without having to repeat myself on redefining all
those settings. That kind of thing is harder to do cleanly when you're
"inheriting" one settings file from another using "from settings.base
import *", or using local_settings.py.

(Since I don't think Django core is going to take any steps to endorse
any particular approach to handling settings anytime soon, I think this
discussion is really off-topic and should move somewhere else. I should
have considered that before replying to Tom.)

Carl

[VernonCole]

Not off topic at all, Carl, this back-and-forth is helping me, at least, consider some new things.

Here's what I have learned from this discussion so far:

1. Clearly, any change needs to avoid use of the word "structured", since that is now yesterday-ish.

2. Any change needs to lead toward and encourage and/or ease use of 12 factor methodology.

3. The settings.py file needs to stay where it is and do what it does without much change.

4. There is a quiet wish for something better than the status quo, but it must  be flexible enough to allow new and different advanced solutions.

5. It needs to be simple to use for development, demo, test, tutorial, and semi-skilled implementations.

6. A URL-like scheme might be a good default option.

The original idea with which I started this thread miss-fired on points one and three, and I was completely unaware of point two. Point six was something I had rattling in the back of my mind for years, but could not envision a way to do it -- especially with something as simple and elegant as Kenneth Reitz's work.

So I started mucking around and I like the emerging design much better.  I have received a bit of positive feedback on it already, and I think I may have found a decent compromise where practicality and purity can co-exist as multiple example files, and the implementer can pick from them his own best options.  I have several example files there now, which effectively replace the not-verbose-enough commentary in settings.py which tries to tell how to set up the DATABASES dictionary. They include working examples which I use for actual testing against multiple db engines. (My intranet is like my house: just enough security that breaking in is not trivial, and nothing inside worth stealing.)   Since each of the small, individual examples is simple, I have started to add 12-factor features (like getting a password from the environment) to some of them.  I will add samples of at least one "pure" 12-factor method.  I may decide to use a version of dj-database-url as the default.

I moved settings.py back where it belongs.  Since the folder with the examples cannot also be named "settings", I renamed it "preset". So, presently, my modified manage.py sets the default value of DJANGO_SETTINGS_MODULE to "formhub.preset.default_settings".  

As always: any suggestion is welcome.
--
Vernon

[Tino de Bruijn]

To all of you who don't see benefit in putting certain (definitely not all!) settings in the environment, I would like to say: try it. It will finally make sense that you set DJANGO_SETTINGS_MODULE in the first place. I never understood why anybody would want that until this :D.

I was sceptical first as well, but tried out Heroku and therefore this method. I have since converted all projects to a structure where environment specific variables are stored, well, in the environment. Still my settings is a module, where prod.py and dev.py inherit from common.py, but that is mainly because certain apps are not necessary on either dev of production servers. So that is all checked into the repository. I don't see any necessity to store those environment vars in a repository, as they contain login keys and static files buckets etc.

On the production server I find it logical that you define those vars in the config files that also make the app run (those of gunicorn or supervisor for example). Changing an AWS bucket or database is not something related to the code history in the repo, but it is to the configuration of the app in the environment.

On the development side, I currently have alias wenv="eval $(echo '$(cat .env) $1' | tr '\n' ' ')", all variables in a .env-file, and run commands like wenv python manage.py syncdb. I recently stumbled upon envdir[1], but I still miss some features to make it complete. Honcho[2], a python port of Foreman (that what Heroku uses) is a procfile based application manager that also loads variables from .env files is also a way to pick those up, if you run on Heroku for production for example.

Tino

[1]https://github.com/jezdez/envdir

[2]https://github.com/nickstenning/honcho 

--

[Tom Evans]

On Tue, Oct 1, 2013 at 6:42 PM, Tino de Bruijn <tin...@gmail.com> wrote:
> To all of you who don't see benefit in putting certain (definitely not all!)
> settings in the environment, I would like to say: try it. It will finally
> make sense that you set DJANGO_SETTINGS_MODULE in the first place. I never
> understood why anybody would want that until this :D.
>
> I was sceptical first as well, but tried out Heroku and therefore this
> method. I have since converted all projects to a structure where environment
> specific variables are stored, well, in the environment. Still my settings
> is a module, where prod.py and dev.py inherit from common.py, but that is
> mainly because certain apps are not necessary on either dev of production
> servers. So that is all checked into the repository. I don't see any
> necessity to store those environment vars in a repository, as they contain
> login keys and static files buckets etc.

Tracking what login keys are assigned to a specific instance of an
application is the entire purpose of configuration management. The
reason we keep code in VCS is so that we can be assured what is
deployed is coherent and consistent, the same is true for
configuration, it must be known and verifiable.

The entire basis of my position is that configuration is essential
code which must be tracked and be verifiable with an audit history of
changes. If you don't feel that to be the case, you are unlikely to
see the benefit in your configuration.

The second benefit is in consistency. Both approaches take the same
overall method, one file contains secrets/host specific configuration
for this server, the second contains generic configuration, and the
two are combined. With the environment variables way, where is your
host specific configuration? It could be in .login, in a gunicorn
launch script, in a shell script - anywhere. How is the configuration
defined? Are you "setenv foo bar" because this host is Solaris and
you're running csh, or is this an "export foo=bar". Perhaps they are
in your supervisord configuration.

Compare to using a python file - you have full power of the python
interpreter to do whatever you need. Everything is in one language,
the same language as your code. All configuration will apply
regardless of how you start the server, whether it is supervisord
launching a wsgi process or you running runserver. There is no
requirement for this file to belong to the same repository as the
code, or even be in a repository.

There is absolutely nothing that using environment variables as a
source of configuration gives you that you cannot do using a python
configuration file and it makes your configuration management messy
and chaotic.

Cheers

Tom

[Jonathan Slenders]

There's nothing that prevents anyone to track the configuration of the environment in a VCS.

The guideline that it shouldn't be in the repository of the application source code is not against that.

Deployment systems should be able to set-up the environment automatically, (by generating a settings_local.py, gunicorn configuration or anything else.) 

We use our own system [1], but I guess that puppet or chef have configuration files that can be tracked in a VCS as well. 

Personally, I'm also not yet entirely convinced that environment variables are always the solution, but I'm convinced that machine specific configuration files don't belong in the application's repository.

[1] https://python-deploy-framework.readthedocs.org

[Tino de Bruijn]

On Wed, Oct 2, 2013 at 11:16 AM, Tom Evans <teva...@googlemail.com> wrote:

Tracking what login keys are assigned to a specific instance of an

application is the entire purpose of configuration management. The
reason we keep code in VCS is so that we can be assured what is
deployed is coherent and consistent, the same is true for
configuration, it must be known and verifiable.

The entire basis of my position is that configuration is essential
code which must be tracked and be verifiable with an audit history of
changes. If you don't feel that to be the case, you are unlikely to
see the benefit in your configuration.

Ah oké, our requirements differ I see. But I do think there is a slight difference between configuration (what apps to use, middleware, etc) and variables that are specific to the environment, although they are both specified in the settings at the moment. I think this is also what the 12factor approach is. 

The second benefit is in consistency. Both approaches take the same
overall method, one file contains secrets/host specific configuration
for this server, the second contains generic configuration, and the
two are combined. With the environment variables way, where is your
host specific configuration? It could be in .login, in a gunicorn
launch script, in a shell script - anywhere. How is the configuration
defined? Are you "setenv foo bar" because this host is Solaris and
you're running csh, or is this an "export foo=bar". Perhaps they are
in your supervisord configuration.

That is dependent on how the environment is set up. Maybe you tested your app on a free Heroku instance, but decide it is not worth the money and put it on a managed server. Or you switch from your own Solaris box to something that the client owns. In my setup this does not require code changes. 

 

Compare to using a python file - you have full power of the python
interpreter to do whatever you need. Everything is in one language,
the same language as your code. All configuration will apply
regardless of how you start the server, whether it is supervisord
launching a wsgi process or you running runserver. There is no
requirement for this file to belong to the same repository as the
code, or even be in a repository.

I am confused. Do you like the structure proposed by Vernon in his first post? Because that is entirely possible and easy to set up, I also use it (accompanied by env vars). There is one 'but', you do have to set the DJANGO_SETTINGS_MODULE env var. :)

 

There is absolutely nothing that using environment variables as a
source of configuration gives you that you cannot do using a python
configuration file and it makes your configuration management messy
and chaotic.

Well, it makes my code deployable anywhere where I can run python and have the necessary resources, no matter what they are named, without checking in code that might break somebody else's environment.  

Cheers

Tom

Regards,

Tino