Making Django more PaaS-friendly

[James Bennett]

Apologies for how late in the process this is; job hunt + moving cross-country ate a lot of my time.

At Django Under the Hood I proposed that for Django 1.10 we try to find some common best practices for deploying Django on popular platform-as-a-service (PaaS) solutions, and building support for them into Django. The biggest one of these is probably having the ability to read configuration from environment variables instead of hard-coding things into settings files.

At the very least I'd like to propose (assuming Kenneth is on board with it) integrating dj-database-url[1] or something like it directly into Django, so that there's no longer a third-party dependency for reading the database configuration from an environment variable. Whether this is just porting dj-database-url itself in, or making the DATABASES setting understand URLs, I'm unsure of yet and would be interested to hear feedback on. Either way I'm willing to put in the time to develop the patch.

More generally, I'd like to see Django grow helpers for specifying settings that should be read from environment variables, and which environment variables to use (the email settings are another common case, as is anything that requires an API key to access).

There are a few ways to design this. One option would be just a minimal wrapper around os.getenv, perhaps taking an optional type or type-coercing function, so that it would be possible in a settings file to do:

    SECRET_NUMBER_SETTING = env_setting('SECRET_NUMBER', int)

However, this is not much better than the current practice of calling os.getenv. A better solution might be the ability to specify a group of settings which will be read from the environment, and have Django automatically read and set them. For example:

    ENV_SETTINGS = [

        ('SECRET_NUMBER_SETTING', int),

        ('ACME_API_KEY', str),

        ('VOLCANO_LAIR_PASSWORD', str),

    ]

would read the named settings from those environment variables, and coerce them to the appropriate types using the function provided.

The main problem with this is that it's really not very elegant. But at the moment I can't think of anything better, and so I'd like to throw the floor open to ideas on nicer approaches to this. If one can't be found, I do think Django 1.10 should at least figure out how to handle database config from env since that's such a common use case nowadays, but ideally we'd be able to pin down a good API for generically pulling configuration from the environment.

[1] https://github.com/kennethreitz/dj-database-url

[Marc Tamlyn]

I like the idea of integrating dj-database-url. There's a similar package for CACHES[0], which is a neat idea but slightly more tricky to justify as I don't know if PaaS tend to send the cache details in that format necessarily.

I'm not sure about the rest of the mail, although I'd definitely be up for modifying the SECRET_KEY line in particular to make it strongly suggest that this should be different per environment.

M

[0] https://github.com/ghickman/django-cache-url

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAL13Cg85fYj0i0ysxJGo2SDesqELMeA%2BtnKJ9cdpqNHmQ%3DX3Pg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Shai Berger]

Django-database-url is very nice, but AFAICT has no way to support 3rd-party backends. I think it needs to grow this support before it can be used in Django. We can adapt the backend API to help, but it's a little tricky if we don't want to import unused backend.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Aymeric Augustin]

We have a chicken’n’egg problem to support third party database backends with dj-database-url’s current API:

- It is each backend’s responsibility to parse the URL and extract the configuration for a given database.

- Parsing the URL is a prerequisite to figure which database backend to pick for parsing the URL.

I think we need a specific convenience API such as:

DATABASES = {

    ‘default’: {

        ‘NAME’: ‘python.path.to.backend’,

        ‘URL’: ‘backend://...',

    }

}

-- 

Aymeric.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/62C8122A-77C0-434A-A143-23C690E511BA%40platonix.com.

[Josh Smeaton]

I kind of like the idea of making all settings configurable via the environment by prefixing with DJANGO_SETTINGNAME. Sort of like how click allows environment variables for options: http://click.pocoo.org/5/options/#values-from-environment-variables. Ideally configuring settings from the environment should be as transparent as possible to the settings file, so that users aren't required to os.getenv('NAME', 'default') all over the place to derive value.

Django uses quite a few dicts and lists though, so consideration would need to be made for those. Perhaps double underscores could represent keys within a dict?

DJANGO_CACHES__default__BACKEND='redis'

For lists we could split on whitespace (again like Click): 

DJANGO_INSTALLED_APPS="contrib.admin contrib.auth".

This does not address the type of the value though. 

[Tim Graham]

Here is some past work on "Minimize the risk of SECRET_KEY leaks" [0] that was never completed: https://github.com/django/django/pull/2714

[0] https://code.djangoproject.com/ticket/20081

[Raphaël Barrois]

Hi James,

From the experience on our projects, the ``CONFIG.getstr('db.password')`` format works well in settings files:
- It makes it clear that this field is loaded from the environment, and what type is expected
- The setting is set and loaded in the same place
- It allows for type checking
- It handles loading other values

This is easier than automagically altering settings based on environment variables, as the user might set a value in
some part of its settings file, yet find it overridden by some later declaration of ``ENV_SETTINGS``.



In my opinion, a few important questions to address would be:
- What is the format of those environment variables? For instance, using a 'DJANGO_' prefix will avoid name conflicts
- Can developers use this "environment configuration parser" tool to parse extra, custom settings; or are we limited to
Django's needs?
- Should we only support "loading from the environment", or is it useful to allow loading from a ini-style file
(typically to override some values on a local development environment)?


We wrote a small helper to load configuration from various sources (environment and configuration files); this could be
helpful for designing this new feature: http://getconf.readthedocs.org/en/latest/

Feel free to ping me if you're interested in more feedback from building and using that kind of setup :)

[Kristian Glass]

Very very glad to hear this. All too frequently in #django, "please show us your settings (and remove any sensitive data)" ends up with a "Now you need to reset your SECRET_KEY" etc.

I wrote django12factor to do something similar. One of the things I like least about it is the process of actually using it from your settings.py - there's things in there I'd love to see in the generated default.

Right now it handles most settings individually and only addresses "the most common ones" - a simple boolean handler for DEBUG, string splitting for ALLOWED_HOSTS etc., and outsources CACHES / DATABASES / EMAIL* to django-cache-url, dj-database-url and dj-email-url respectively. This isn't ideal, BUT even without getting into generically useful helpers, I think a lot of those could be useful in the standard template...

https://github.com/doismellburning/django12factor/pull/19/files is a branch where I prototyped something similar to your type coercion idea; you may find it more elegant, ymmv - I'd be interested in your thoughts.

[Ryan Hiebert]

I'd love to see better support for PaaS configuration, especially 12-factor. We use Heroku, and I've been directly exposed to the challenges and had to come up with some of my own solutions. Here's some thoughts I have on the matter, in no particular order.

The SECRET_KEY needs to _not_ be required for the `collectstatic` command. I haven't tracked down why it currently _is_ required, but it means that I have added this line to my project, solely so that my project can be built on Heroku.

SECRET_KEY = os.environ.get('SECRET_KEY', 'not-so-secret')

SECRET_KEY is effectively optional (though I'm careful to make sure it's set), and I don't see a good reason that should be. Note that environment variables aren't normally available during slug build, since changing environment variables doesn't re-build the project.

I believe that the biggest current impediment to finding a great solution to loading Django settings is that the settings are based on a Module. Django-Configurations has done an excellent job of making that into a class-based API instead, which allows for mixins and inheritance. Having built-in support for this would be an excellent way to allow 3rd party solutions to build up settings programatically. In the case we are discussing, from the environment.

I believe this would make it far easier for projects like Kristian's django-12factor and dj-email-url to work well. dj-email-url is a good idea (put all the settings in 1 variable, like dj-database-url), except that all the settings are split up at the top-level of the module, making what's conceptually one multi-facted setting a multi-line pain.

One possibility would be extending `DJANGO_SETTINGS_MODULE` with an optional class, separated with a colon, so to get a django-configurations style class you'd use `DJANGO_SETTINGS_MODULE=myproject.settings:MySettings`. Or we could change the variable name `DJANGO_SETTINGS`, which would allow us to avoid the colon: `DJANGO_SETTINGS=myproject.settings.MySettings`. Or paint the bikeshed some other color.

I'd love to see some of the support for environment configurations brought into Django, but IMO moving away from a module based settings API is a more pressing concern that would enable 3rd party libraries to come up with elegant solutions.

[Sean Brant]

https://github.com/joke2k/django-environ has helpers for loading most of the backends from urls. We use it with a docker based deployment at the moment.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/2D66F14E-DD71-41B6-A84D-A0C679FD72F5%40ryanhiebert.com.

[Curtis Maloney]

Just want to throw my 3c in here...

Firstly, for reference, I wrote django-classy-settings to help with
various settings related problems, one of which being env-based setting.

https://github.com/funkybob/django-classy-settings

As my env decorators are written to be class properties, they don't
quite fit this discussion... however, things to consider:

1. All env vars can have a fallback value (the method in my case)
2. You can mark an env var as not having a fallback value, and it will
raise an error at start up if not set.
3. Non-trivial value types are not easy.

The URL series of settings helpers deal with many of these cases fairly
well, as they allow us to specify a backend (protocol), username,
password, host, port, and positional and keyword arguments.

Registering more protocols is actually quite easy, all told.

However, it becomes tricky when we allow multiples, such as database and
cache backends. Do we settle on, for instance, a pattern of
"DJANGO_DATABASE_{name}="? Or some other scheme?

Email settings are certainly another big one, IME... as well as AWS keys...

--
C

[Aymeric Augustin]

> On 11 Apr 2016, at 19:39, Curtis Maloney <cur...@tinbrain.net> wrote:
>
> 1. All env vars can have a fallback value (the method in my case)
> 2. You can mark an env var as not having a fallback value, and it will raise an error at start up if not set.

There’s an additional complexity here.

Usually, it’s best for settings read from environment variables:

a. To use a default value in dev, even if that means a small degradation in
functionality. This allows developers to start working on the project without
adding dozens of exports to their $VIRTUAL_ENV/bin/postactivate, and to add
just the values they need when they work on specific parts like integrations
with third-party systems.

b. To crash if no value is provided in prod, in order to catch configuration
errors upfront.

One might think of switching the behavior depending on settings.DEBUG, but
that won't work because the switch is required to load settings properly.

I’ve seen lots of wrapper that don’t add enough value to be worth it. These
days I just write:

# settings/base.py

...

# settings/dev.py

FOO_TOKEN = os.environ.get('FOO_TOKEN', '')

# settings/prod.py

FOO_TOKEN = os.environ['FOO_TOKEN']

It isn’t as dry as it could be, but at least it’s simple.

--
Aymeric.

[Curtis Maloney]

On 12/04/16 04:26, Aymeric Augustin wrote:
>> On 11 Apr 2016, at 19:39, Curtis Maloney <cur...@tinbrain.net> wrote:
>>
>> 1. All env vars can have a fallback value (the method in my case)
>> 2. You can mark an env var as not having a fallback value, and it will raise an error at start up if not set.
>
> There’s an additional complexity here.
>
> Usually, it’s best for settings read from environment variables:
>
> a. To use a default value in dev, even if that means a small degradation in
> functionality. This allows developers to start working on the project without
> adding dozens of exports to their $VIRTUAL_ENV/bin/postactivate, and to add
> just the values they need when they work on specific parts like integrations
> with third-party systems.
>
> b. To crash if no value is provided in prod, in order to catch configuration
> errors upfront.
>
> One might think of switching the behavior depending on settings.DEBUG, but
> that won't work because the switch is required to load settings properly.

This is all true.... it's generally not an issue when using
django-classy-settings as it was engineered to make it easy to switch
between settings environments such as dev/staging/testing/production.

> I’ve seen lots of wrapper that don’t add enough value to be worth it. These
> days I just write:
>
> # settings/base.py
>
> ...
>
> # settings/dev.py
>
> FOO_TOKEN = os.environ.get('FOO_TOKEN', '')
>
> # settings/prod.py
>
> FOO_TOKEN = os.environ['FOO_TOKEN']
>
> It isn’t as dry as it could be, but at least it’s simple.

This is one thing d-c-s helps you avoid... by letting you declare it all
in one file, inherit complex/dependent settings through classes, and
switch between those with an env var [or whatever mechanism you like]

However, this is slightly tangential to the actual discussion

--
C

[Joey Wilhelm]

Just to throw another voice in here, I'm currently using django-environ[1], as mentioned by Sean Brant. In addition, I'm using a settings setup based on cookiecutter-django[2]. This means having my settings split into `config.settings.common`, `config.settings.local`, `config.settings.production`, for example. In common, I have things like the following:

    

    DATABASES = {

        # Raises ImproperlyConfigured exception if DATABASE_URL not in os.environ

        'default': env.db("DATABASE_URL"),

    }

    EMAIL_BACKEND = env('DJANGO_EMAIL_BACKEND', default='django.core.mail.backends.smtp.EmailBackend')

    # ...etc...

These would obviously propagate up and be used in the production settings. Then in the local settings, I have things like:

    SECRET_KEY = env('DJANGO_SECRET_KEY', default='CHANGEME!!!')

    EMAIL_BACKEND = env('DJANGO_EMAIL_BACKEND', default='django.core.mail.backends.console.EmailBackend') 

    DATABASES = {

        'default': env.db('DATABASE_URL', default='postgis://django:supersecretpassword@localhost:5432/django'),

    }

    # ...etc...

I don't know if this is all necessarily the best way to do it, but it has been working quite well for me. Everything is read out of a .env file, and I provide a template.env file in my project to show which values need to be set in there. `DJANGO_SETTINGS_MODULE` defaults to production, so that missing environment keys will raise alarms by default, and then when possible, those keys become optional in development settings.

-Joey

[1]: https://github.com/joke2k/django-environ

[2]: https://github.com/pydanny/cookiecutter-django

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/570BE142.3000209%40tinbrain.net.

[Martin Owens]

I look after many different django websites and I'd like to add something conceptual.

There is a difference between settings intended by developers to be used by other developers in order to enable and configure their app's behaviour so it work with other code. (I call these developer settings)
And settings used by systems administrators to define the environment they are deployed to as well as any deployment specific controls. (I call these sysadmin settings)

Django has always mixed the two into one file, and it causes problems. I've seen keys and passwords committed to github, extravagant json or os.env based solutions with piles of lines of messy code to what I see most which is a local_settings.py with a template version which is copied in at runtime with sane defaults.

I'd be happy to see a canonical decision made for django, but I'd focus on "who should be editing this setting" not just what it's for, if it's a secret or if it should be different or the same per instance.

Best regards, Martin Owens

On Monday, April 11, 2016 at 7:33:54 AM UTC+1, James Bennett wrote:

[aRkadeFR]

I like the try to move things towards this kind of settings more PaaS-friendly.

As a Django user, I'd like to see 'DSN' or 'URL' for the database as a built-in in Django. I don't
mind setting it in the dict as so:
```
DATABASES = {
    'default': {
        'DSN': 'postgres://user:password@ip:port/name_db',
    }
}
```

and I use the same configuration layout as aymeric augustin, with most of the env name prefix
with ``DJANGO_``.

I'm not exactly a PaaS user, but I package my code with dh-virtualenv under .deb which has
some similar constraints as the PaaS out there.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAL13Cg85fYj0i0ysxJGo2SDesqELMeA%2BtnKJ9cdpqNHmQ%3DX3Pg%40mail.gmail.com.

[Carl Meyer]

Hi James et al,

In general, I like the idea of adding a helper to Django to read
settings from the environment. I think this helper should be kept as
simple and non-magical as is reasonable. Thus:

- I'm in favor of a flexible helper function that can be manually used
in a settings file to read an explicitly-named env var and coerce it
using any arbitrary callable (that gives enough power to "coerce" using
something as simple as `int` or something as complex as
`parse_database_url` that turns a DSN string into a Django db dict),
with some handling of defaults. Below I give some usage examples of the
function I use for that, and link to its code.

- I'm opposed to anything more magical than that, e.g. anything that
tries to set up an all-dancing automatic mapping or translation between
env var names and setting names, or tries to enforce some particular env
var prefix. The minor gains in conciseness with this kind of thing
aren't nearly worth the losses in flexibility in how you construct your
settings file or name your env vars.

- I'm also opposed to conflating the issue of reading env vars with
orthogonal stuff like class-based settings or patterns for using
multiple settings files.

More comments (and sample code) below:

On 04/11/2016 12:26 PM, Aymeric Augustin wrote:
>> On 11 Apr 2016, at 19:39, Curtis Maloney <cur...@tinbrain.net> wrote:
>>
>> 1. All env vars can have a fallback value (the method in my case)
>> 2. You can mark an env var as not having a fallback value, and it will raise an error at start up if not set.
>
> There’s an additional complexity here.
>
> Usually, it’s best for settings read from environment variables:
>
> a. To use a default value in dev, even if that means a small degradation in
> functionality. This allows developers to start working on the project without
> adding dozens of exports to their $VIRTUAL_ENV/bin/postactivate, and to add
> just the values they need when they work on specific parts like integrations
> with third-party systems.
>
> b. To crash if no value is provided in prod, in order to catch configuration
> errors upfront.
>
> One might think of switching the behavior depending on settings.DEBUG, but
> that won't work because the switch is required to load settings properly.

I totally agree with all this. My projects have a separate MODE setting
which is either 'dev' or 'prod', and I use a utility to read settings
from the environment which supports variable defaults (or lack of
default) by mode, exactly as you outline above. Specifically, it works
like this:


env = EnvParser('PRJ_MODE')

SECRET_KEY = env('PRJ_SECRET_KEY', default={'dev': 'dev-secret'})

The effect of this is that the mode ('dev' or 'prod') will be read from
the `PRJ_MODE` env var (usually `PRJ` would be a short code specific to
the project name). Then if the mode is 'dev', no `PRJ_SECRET_KEY` env
var is required, and 'dev-secret' will be used as the default. But if
the mode is 'prod', the server will fail to start up unless
`PRJ_SECRET_KEY` is found in the environment.

I wrote this to allow for any number of arbitrarily-named modes, but in
practice I've never used anything but 'dev' and 'prod' (staging and demo
sites are generally set up as 'prod'). Hardcoding to those two modes
would perhaps allow a nicer syntax for specifying defaults, by using
separate kwargs instead of the dict.

Anyway, FWIW, here's the code I use:
https://gist.github.com/carljm/69b8e351dac87f4c3f5b440632727fdb

Carl

[Curtis Maloney]

On 14/04/16 05:57, Carl Meyer wrote:
> Hi James et al,
>
> In general, I like the idea of adding a helper to Django to read
> settings from the environment. I think this helper should be kept as
> simple and non-magical as is reasonable. Thus:
>
> - I'm in favor of a flexible helper function that can be manually used
> in a settings file to read an explicitly-named env var and coerce it
> using any arbitrary callable (that gives enough power to "coerce" using
> something as simple as `int` or something as complex as
> `parse_database_url` that turns a DSN string into a Django db dict),
> with some handling of defaults. Below I give some usage examples of the
> function I use for that, and link to its code.

+1

This I've found works well in d-c-s...

> - I'm opposed to anything more magical than that, e.g. anything that
> tries to set up an all-dancing automatic mapping or translation between
> env var names and setting names, or tries to enforce some particular env
> var prefix. The minor gains in conciseness with this kind of thing
> aren't nearly worth the losses in flexibility in how you construct your
> settings file or name your env vars.

+1

Adding more policy will only complicate matters... if you just provide
the mechanism, people will surprise you with innovative policies :)

> - I'm also opposed to conflating the issue of reading env vars with
> orthogonal stuff like class-based settings or patterns for using
> multiple settings files.

+1

Once again, there's no need to dictate more than necessary.

--
C

[Mike Bryant]

On 11 April 2016 at 07:33, James Bennett <ubern...@gmail.com> wrote:

> The main problem with this is that it's really not very elegant. But at the
> moment I can't think of anything better, and so I'd like to throw the floor
> open to ideas on nicer approaches to this. If one can't be found, I do think
> Django 1.10 should at least figure out how to handle database config from
> env since that's such a common use case nowadays, but ideally we'd be able
> to pin down a good API for generically pulling configuration from the
> environment.

Sorry for a rather belated response, but I'd like to share our approach to this.
We wrote django-autoconfig [1] to help with our settings. For some
background, it's main role is to allow reusable apps to define
settings that they must have to work effectively (e.g. additional
middleware),

Of relevance though is, to aid in our deployments into AWS we went
with pulling in environment variables named DJANGO_{setting_name} and
coercing them into valid settings.

I mention this here as it seems to be of relevance to the audience,
and I'd welcome any feedback. Apologies if this is just noise.

Mike

[1] https://github.com/mikebryant/django-autoconfig

--


Notice: This email is confidential and may contain copyright material of
members of the Ocado Group. Opinions and views expressed in this message
may not necessarily reflect the opinions and views of the members of the
Ocado Group.



If you are not the intended recipient, please notify us immediately and
delete all copies of this message. Please note that it is your
responsibility to scan this message for viruses.



Fetch and Sizzle are trading names of Speciality Stores Limited, a member
of the Ocado Group.



References to the “Ocado Group” are to Ocado Group plc (registered in
England and Wales with number 7098618) and its subsidiary undertakings (as
that expression is defined in the Companies Act 2006) from time to time.
The registered office of Ocado Group plc is Titan Court, 3 Bishops Square,
Hatfield Business Park, Hatfield, Herts. AL10 9NE.

[and...@dietchef.co.uk]

We deploy django use AWS Elastic Beanstalk and use credstash for secrets:

https://github.com/fugue/credstash

So we can just do PASSWORD = credstash.getSecret('password') which works pretty well with the AWS Key Management Service.

What is a pain is that the AWS Health Check does not set the Host header and we don't know until runtime what the IP address of the server will be. We have to do a call to the local metadata server to find out the IP address then add it to ALLOWED_HOSTS which is really quite hideous. There is an explanation here:

https://dryan.com/articles/elb-django-allowed-hosts/

Personally I would quite like to see an equivalent of the @csrf_exempt decorator perhaps @allowed_hosts_exempt that could be use to mark the health check method.

Andrew

[Hanne Moa]

On 12 April 2016 at 05:35, Martin Owens <doct...@gmail.com> wrote:
> There is a difference between settings intended by developers to be used by

> other developers /../ And settings used by systems administrators to define

> the environment they are deployed to as well as any deployment specific
> controls.
>

> Django has always mixed the two into one file, and it causes problems. I've
> seen keys and passwords committed to github, extravagant json or os.env
> based solutions with piles of lines of messy code to what I see most which
> is a local_settings.py with a template version which is copied in at runtime
> with sane defaults.

I used to use extravagant json messes[*] but have (on non-PaaS)
switched to having a directory /etc/django-sites.d, which is on the
python path, containing one python package per site on that host:

/etc/django-sites.d/MyProjectConf/sitetype.py

MyProject.settings.sitetype is referenced by wsgi-scripts. It first
star-imports MyProject.settings.base, then star-imports
MyProjectConf.sitetype inside a try/except.

That way I can have have important overrides/option-merging for a
sitetype in version control, and still have
puppet/chef/ansible/whatever control deploy-and-site-specific options
without touching anything inside the source tree.

> I'd be happy to see a canonical decision made for django, but I'd focus on
> "who should be editing this setting" not just what it's for, if it's a
> secret or if it should be different or the same per instance.

A tag in the settings-docs would be a good start, or a check list of
more site dependent settings. Hey, the docs are restructured text; the
tag could generate an extra index, et voila :)


--
HM

[*] I *know* why json doesn't survive that spurious comma at the end
of a list/dict, but I like me that comma so much that I no longer use
json for hand-written config in new stuff.

[Marcin Nowak]

Hi all,

On Wednesday, April 13, 2016 at 9:58:18 PM UTC+2, Carl Meyer wrote:

Hi James et al,

In general, I like the idea of adding a helper to Django to read
settings from the environment. I think this helper should be kept as
simple and non-magical as is reasonable. Thus:

This sounds great. Django should not do anything magical and just in one way. 

Just give the tool to the developers.

But I would like to say my thoughts about "settings" itself.

They were good (simple) before incuding nested dictionaries.

After switching to dicts the settings handling went harder way.

As a example I can tell about reconfiguring LOGGING for dev/live settings.

We're tuning them via direct changes like LOGGING['loggers']['some-logger']['level']='...'

which is very error prone. The flat settings are easier to maintain.

Nested settings are good for framework internals and should stay in that form, 

but maybe solution for end users and ENV integration is flattening settings again, at the high-level?

Let's imagine something like INI file which is parsed somehow and converted to internal settings object.

The INI file has a shallow structure (options grouped in sections), 

can be easily extended / overrided / included (look at Buidout as an example),

and ENV keys will be related to flat INI instead of nested settings.py.

Let's consider an example:

[databases]
default.url = postgres+psycopg2://user:pwd@localhost:5432/exampledb
default.engine = django.db.backends.postgres_psycopg2

which can be simply reconfigured by ENV vars like:

DJANGO_DATABASES_DEFAULT_URL="postgres+psycopg2://user:pwd@localhost:5432/exampledb"

DJANGO_DATABASES_DEFAULT_ENGINE="django.db.backends.postgres_psycopg2"

OR add some preprocessing and flexibility:

[databases]
default.url = ${env:SOME-USER-KEY1}
default.engine = ${env:SOME-USER-KEY2}


[env]
SOME_USER_KEY1=a default value for default database URL
SOME_USER_KEY2=a default value for default database engine

where [env] section will by filled automatically by Django from os.environ.

This also gives a possibility to extend settings.ini via development.ini, 

where [databases] can be overridden with developer settings (without env access)

Thanks for reading and sorry for my English. I hope you understood the idea.

Marcin

 

[Aymeric Augustin]

Hello,

If anything, this discussion shows that people have come up with a wide
variety of customized solutions to meet their use cases. That supports
Django’s historical decision not to be too prescriptive in this area.

I still think James’ proposal is valuable because it will nudge developers
towards best practices when deploying on a PaaS, a path often taken
by those least familiar with deployment best practices in general.

Best regards,

--
Aymeric.

[Carl Meyer]

Hi Marcin,

On 04/26/2016 08:12 PM, Marcin Nowak wrote:
> But I would like to say my thoughts about "settings" itself.
> They were good (simple) before incuding nested dictionaries.
> After switching to dicts the settings handling went harder way.

I agree that settings grouped into dicts are harder to manage than flat
settings. Django has not made any "switch to dicts" for settings; in
fact, the last time we had to make a decision between a group of flat
settings with a common prefix vs a dict, we chose flat (the SECURE_*
settings).

> As a example I can tell about reconfiguring LOGGING for dev/live settings.
> We're tuning them via direct changes like
> LOGGING['loggers']['some-logger']['level']='...'
> which is very error prone. The flat settings are easier to maintain.

LOGGING is a special case; this dict format is not defined by Django,
but by Python's logging module and its "dictConfig" function.

> Nested settings are good for framework internals and should stay in that
> form,
> but maybe solution for end users and ENV integration is flattening
> settings again, at the high-level?
>
> Let's imagine something like INI file which is parsed somehow and
> converted to internal settings object.
> The INI file has a shallow structure (options grouped in sections),
> can be easily extended / overrided / included (look at Buidout as an
> example),
> and ENV keys will be related to flat INI instead of nested settings.py.
>
> Let's consider an example:
>
> |
> [databases]
> default.url =postgres+psycopg2://user:pwd@localhost:5432/exampledb
> default.engine =django.db.backends.postgres_psycopg2
>
> |
>
> which can be simply reconfigured by ENV vars like:
>
> DJANGO_DATABASES_DEFAULT_URL="postgres+psycopg2://user:pwd@localhost:5432/exampledb"
> DJANGO_DATABASES_DEFAULT_ENGINE="django.db.backends.postgres_psycopg2"
>
> OR add some preprocessing and flexibility:
>
> |
>
>
> [databases]
> default.url =${env:SOME-USER-KEY1}
> default.engine =${env:SOME-USER-KEY2}
>
>
> [env]
> SOME_USER_KEY1=a defaultvalue fordefaultdatabase URL

> SOME_USER_KEY2=a defaultvalue fordefaultdatabase engine

>
> |
>
> where [env] section will by filled automatically by Django from os.environ.
>
> This also gives a possibility to extend settings.ini via development.ini,
> where [databases] can be overridden with developer settings (without env
> access)

I've done something similar before, and I think it's fine, but this is
an example of precisely the sort of more-magical opinionated approach to
settings that I don't think belongs in Django core; it's perfectly
possible to implement it as a third-party package on top of Django's
current settings handling, for those who like it.

Carl

[James Bennett]

Whee, this thread got big!

The takeaway I'm getting here is that we should be careful about what we adopt into core and when; I'm going to take a couple days and write up some notes on what I think a good conservative approach would look like, and ask for feedback on that.

(spoiler: probably the majority of it is going to be adding support for parsing database URLs in the DATABASES setting)

[Shai Berger]

On Monday 09 May 2016 05:06:47 James Bennett wrote:
> Whee, this thread got big!
>
> The takeaway I'm getting here is that we should be careful about what we
> adopt into core and when; I'm going to take a couple days and write up some
> notes on what I think a good conservative approach would look like, and ask
> for feedback on that.
>

Looking at what we just had in other threads, this is probably DEP-worthy;
even if the results end up being a relatively small modification, it would be
helpful to record why we decided to reject the other suggestions.

Shai.

[Anssi Kääriäinen]

I'm very much afraid we are in a situation where we have a documented
DEP process which isn't actually enforced, nor does it document what
we actually do.

Python's PEP process is useful as the document evolves while the
design evolves. What we do with DEPs is different, we first design
something, then we document the design decision, often mostly because
DEPs are a required document. We don't get what we could from DEPs as
they are written too late.

Could we try an approach where we *first* require a DEP, then have the
design discussion?

- Anssi

[Carl Meyer]

Hi Anssi,

On 05/09/2016 06:53 AM, Anssi Kääriäinen wrote:
> I'm very much afraid we are in a situation where we have a documented
> DEP process which isn't actually enforced, nor does it document what
> we actually do.
>
> Python's PEP process is useful as the document evolves while the
> design evolves. What we do with DEPs is different, we first design
> something, then we document the design decision, often mostly because
> DEPs are a required document. We don't get what we could from DEPs as
> they are written too late.

I'm curious to hear more about this - could you give some example DEPs
where this was a problem? In the case of the two DEPs I was most closely
involved with (multiple template engines and rethinking middleware), I
don't think this is an accurate description of what happened. Both those
DEPs were written relatively early in the design process and evolved as
we learned from discussion and implementation.
They were done quite similarly to how the PEPs I've worked on were done.

I'm also curious about your use of the word "enforced." I don't really
see the DEP process as something that needs "enforcement" (and I'm
curious where you see it as having been lacking in "enforcement"). IMO
the only relevant "enforcement," really, is that the implementation of a
DEP which is rejected by the technical board (has never happened yet)
should not be merged.

DEPs, like PEPs, aren't supposed to be a legalistic policy: there is no
hard and fast rule about which features should have DEPs, and I don't
think we need a hard and fast rule. DEPs should happen when someone
observes that the discussion around a feature is getting quite complex
or contentious, and suggests that a DEP would be helpful. DEPs should be
a tool, not a barrier: a tool for helping to focus discussions on
contentious issues, and record the perspectives expressed in those
discussions, both for posterity and to help keep the discussion from
going around in circles.

> Could we try an approach where we *first* require a DEP, then have the
> design discussion?

In the PEP process, it's very typical to start into design discussions
before realizing that the issue is contentious/complex enough to require
a PEP, then someone collects the knowledge generated from that early
discussion into a first draft PEP, which then receives further
discussion. (In the Python world most of this pre-PEP discussion happens
on the python-ideas mailing list, so if you aren't subscribed there you
might not see it.) I think this thread is a great example of that
working precisely the way it should work: first post some general
thoughts, collect ideas and responses to those thoughts, then work that
into a DEP with a specific proposal, followed by more detailed
discussion of that specific proposal.

Carl

[Anssi Kääriäinen]

On Tue, May 10, 2016 at 1:48 AM, Carl Meyer <ca...@oddbird.net> wrote:
> Hi Anssi,
>
> On 05/09/2016 06:53 AM, Anssi Kääriäinen wrote:
> I'm curious to hear more about this - could you give some example DEPs
> where this was a problem? In the case of the two DEPs I was most closely
> involved with (multiple template engines and rethinking middleware), I
> don't think this is an accurate description of what happened. Both those
> DEPs were written relatively early in the design process and evolved as
> we learned from discussion and implementation.
> They were done quite similarly to how the PEPs I've worked on were done.
>
> I'm also curious about your use of the word "enforced." I don't really
> see the DEP process as something that needs "enforcement" (and I'm
> curious where you see it as having been lacking in "enforcement"). IMO
> the only relevant "enforcement," really, is that the implementation of a
> DEP which is rejected by the technical board (has never happened yet)
> should not be merged.

It's not so much the DEPs that have been written (though the channels
DEP is going to be post-design instead of supporting the design). It's
more about those features that don't have a DEP at all. For example
database schemas and subquery expressions come to mind (these are
ongoing work, PRs available at GitHub). There are various design
choices and the features are somewhat significant, yet the design
hasn't been at all DEP driven.

Also, having DEP requested for channels only at this point, and Andrew
(a technical board member) assuming a DEP wasn't required should point
out there might be a problem. This is not to criticize Andrew but the
way DEPs are sometimes required, sometimes not.

I'm not seeing a problem with the DEP idea in general, with any
particular DEP or with any single feature having or not having a DEP.
I'm criticizing the approach where sometimes patches are called out
randomly for DEP, often at a point where the DEP is just documenting
what has been already decided, not supporting the decision process
itself.

I probably picked the wrong thread to complain about this - if we are
going to want DEPs for features similar to the ideas discussed in this
thread, then the timing for requesting a DEP was accurate.

- Anssi

[Carl Meyer]

Hi Anssi,

Thanks for the clarification, that all makes sense. A few comments below:

On 05/10/2016 12:00 AM, Anssi Kääriäinen wrote:
> It's not so much the DEPs that have been written (though the channels
> DEP is going to be post-design instead of supporting the design). It's
> more about those features that don't have a DEP at all. For example
> database schemas and subquery expressions come to mind (these are
> ongoing work, PRs available at GitHub). There are various design
> choices and the features are somewhat significant, yet the design
> hasn't been at all DEP driven.

I would say that a DEP is generally a good idea if the people working on
the feature feel that it would be helpful to them to organize and record
the various design decisions, or if people in the community are
expressing concern about the feature or there's significant disagreement
about the design decisions that isn't quickly and easily resolved. When
there's a proposal to change a core aspect of Django in a pretty
fundamental way (like with multiple-template-engines or middleware),
that's a good case to just start out right away with a DEP, because it's
important to make really sure the community is on board with the change.
Again, I think DEPs ideally should be a useful tool, not a hoop to jump
through.

> Also, having DEP requested for channels only at this point, and Andrew
> (a technical board member) assuming a DEP wasn't required should point
> out there might be a problem. This is not to criticize Andrew but the
> way DEPs are sometimes required, sometimes not.

I didn't intend to be "requiring" a DEP for channels as some kind of
pro-forma dotting of the I's and crossing of the T's for a decision that
in practice had already been made; if that's how my message in that
thread came across, then I miscommunicated. I (strongly) suggested a
DEP, because when I looked back at e.g the discussion thread from
December I saw a number of concerns expressed about channels that still
didn't seem to me to have been resolved (and it also seemed clear from
messages from you and Mark that they weren't) and it seemed to me that
channels still needed the sort of focused public resolution of questions
and concerns that a DEP discussion can provide.

But it's quite possible I was just wrong in that suggestion. If in fact
all the important decisions are already made, and the only thing
channels really needs to address the concerns is real-world proving and
testing, then perhaps a DEP at this point is a waste of time. (Though a
DEP could also be a useful place to record and summarize the discussion
around the results of such real-world testing.)

> I'm not seeing a problem with the DEP idea in general, with any
> particular DEP or with any single feature having or not having a DEP.
> I'm criticizing the approach where sometimes patches are called out
> randomly for DEP, often at a point where the DEP is just documenting
> what has been already decided, not supporting the decision process
> itself.

Yeah, I certainly agree that the way it worked out for channels wasn't
ideal.

> I probably picked the wrong thread to complain about this - if we are
> going to want DEPs for features similar to the ideas discussed in this
> thread, then the timing for requesting a DEP was accurate.

Agreed.

Carl

[Aymeric Augustin]

On 10 May 2016, at 19:18, Carl Meyer <ca...@oddbird.net> wrote:

> I would say that a DEP is generally a good idea if the people working on
> the feature feel that it would be helpful to them to organize and record
> the various design decisions

DEPs have the additional advantage or drawback, depending on your perspective,
of raising the bar for arguing about the design. The first step is to read 10
to 20 pages, and perhaps a bunch of other sources they reference.

Compared to the traditional process, which simply consists in ranting on this
mailing-list based on partial data and approximate assumptions, DEPs take all
the fun away ;-)

(More seriously: discussions are rarely very bad here, however I believe that
a DEP makes them much more efficient once a certain amount of complexity is
reached, because the DEP provides a consolidated vision of the discussion.)

--
Aymeric.

[Andrew Godwin]

I'd like to agree with this point strongly - having a centralised place that outlines the entire approach and all the discussion around it seems invaluable for larger features, and would have been Nice To Have given recent events.

Andrew