Making startproject's settings more 12-factor-y

[Kit La Touche]

I'm interested in reducing the distance from running `startproject` to having a deployable Django app, having recently had a number of interactions with junior devs interested in using Django but finding deployment a nightmare. Some of this is of course on the other side, on the PaaS of choice, but some of it can (and I think should) be done on the Django side.

I'd like to start small. If I made a PR to replace a few key settings with "get from the env, or fall back to this default", would people be interested in approving that? I hope so!

--Kit

[Tim Graham]

Have you read past discussions on the mailing list?

For example:

Making Django more PaaS-friendly

https://groups.google.com/d/topic/django-developers/BAGhOKXGj4I/discussion

Searching the archives for "12-factor" yields some other results.

That may help you to craft your proposal.

[Kit La Touche]

I have, thank you! I tried to do my research before I wrote this email and this PR, but there's a ton of context to all those conversations that I'm sure I'd be missing, so I'm never gonna reach the point of feeling confident that I have a handle on what's gone before, alas.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a45204c6-5475-4ee9-a467-78c66736cfado%40googlegroups.com.

[Kit La Touche]

I have put in a PR (https://github.com/django/django/pull/13029) that I think matches pretty well with the comments from Carl Meyer in the conversation you linked. Please let me know what you think of it.

It adds a small helper function that will read an env var, optionally coercing it to a type, optionally falling back to a default. It also makes two of the settings in the default settings.py template (DEBUG and SECRET_KEY) use this function, with their previous values set as defaults.

Thanks!

[Javier Buzzi]

I think its great, but id add a little more customizability. I've added an example to your PR.

[Kit La Touche]

I saw what you added; I tried to make this a minimal change, and was conscious also Carl's expressed preference (which I share, tbh) in the linked discussion for less-magic. I figured a function that just parsed values out of the env would be the smallest possible first step, so I'm inclined to keep with this, for now.

On Tue, Jun 23, 2020 at 4:15 PM Javier Buzzi <buzzi....@gmail.com> wrote:

I think its great, but id add a little more customizability. I've added an example to your PR.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8dc94197-fa0c-442a-8f97-3d7990a3a2c2o%40googlegroups.com.

[Bobby Mozumder]

I also have my own version of “startproject" that creates several additional files, including my web server config file (I use H20 web server, not Nginx) as well as a uWSGI vassals config file. I create the following preconfigured files and directories using a command-line Python script:

.gitignore

.env

h2o.conf

uwsgi.ini

create_db.sql

manifest.json

browserconfig.xml

robots.txt

requirements.txt

/static

/static/icons

/static/icons/*various_icons.png

/media

/venv

/log

Basically as much as I can possibly create to launch a new project and have it ready to deploy on my main web server.  I’m adding more items as well, including fonts, CSS and JS templates.

There are so many little things that are needed to deploy. I wish Django had it built in, but yes my configuration is pretty custom, with H20 instead of Nginx or Apache.

Is this something of interest for Django directly?

-bobby

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFcS-hD4RFMt%2BwQCCkt6TB5P3rEjarD9Bc9WcArvdaKBzxiYEw%40mail.gmail.com.

[Carlton Gibson]

There are plenty of project templates out there that offer more fully-featured starting points. It's generally agreed that's out-of-scope for the provided startproject template.

But could we make it easier to use environment variables in the default template? That seems like a reasonable thing to discuss.

Is it now time that Django brings this into the core? (In principle +1, yes)

So the PR adds parsing of environment variable into the correct types, but we already have packages in the community doing that.

The one that comes to mind is django-configurations (part of Jazzband) — and there's discussion there of moving the env parsing bit to depend on django-environ.

Can we bring in the battletest code there, clearing up known niggles, rather that re-writing something from scratch?

(I'm hoping folks from django-configurations might input here.)

Kind Regards,

Carlton

[Javier Buzzi]

I looked at the libs that do what we want:

django-configurations - it looks like they use environment variables / either via loading them from the environ or a key/value pair file. Having classes inside the settings.py might be weird to people.. at the least very different.
confucius - very simplistic, only supports environ and is classed based, similar to django-configurations.
django-environ - supports env file and environ, non-class based.
dynaconf - supports all kinds of loading options (toml, json, ini, environ, .env +) non-class based.

In my opinion, django-environ and dynaconf would be the easiest to sell to the community, it would require the least changes/paradigm shifts from how everyone is already using django.

Personally, i would really like to see something like this inside my settings.py:

from django.conf import from_env  # using standard os.environ

DEBUG = from_env.bool("DEBUG", default=False)

DATABASES = {
    "default":  from_env.db("DATABASE_URL")  # crash if it cant find it
}

...

for more complex examples:

from django.conf import EnvFileLoader

from_env = EnvFileLoader("path/to/.secret")

...

We can have how ever many loaders we want: toml, json, ini .. 

This is both borrowing heavily from dynaconf and django-environ, making the fewest changes to how people are accustomed to doing things.

.. what do you guys think?

- Buzzi

[Kit La Touche]

Carlton—thanks very much for the feedback. Javier—likewise. In particular, the imagined API you describe above is very appealing to me: start with `from_env` and then if you learn more about this and want to, add in some `EnvFileLoader`.

I want to make clear my motivation and agenda here: I have recently had some conversations with newer devs about their experiences with deployment of apps they're working on, and with a friend at Heroku about his informal research into the problems people have with the same. One recurring friction point (and this is not just on Heroku at all, to be clear) is that there are a number of things that people don't know they need to configure for a working deployment.

There are four settings that are recurring particular gotchas that people miss: the secret key, debug, static files, and databases. Static files seems big and out of scope, databases seems adequately handled by dj-database-url for most cases, and if your case is more complex, you'll learn it, but the other two (secret key and debug) seemed easy enough to flag as "you probably need to configure these!" with this sort of change to settings. This would be a first step towards shortening the distance from `startproject` to a working deployment.

Newer devs in particular have, based on my conversations and this friend's research, been unlikely to (a) know that there are different `startproject` templates, and (b) feel equipped to choose one, if they do know.

My hope is to make the smallest possible change to just start us moving towards more clearly flagging, especially for newer devs, "these are things that will need additional configuration in order to move from 'works on my machine' to 'deployed'."

Towards that end, I thought that adding a "you might want to get this from the env" helper would be a clear indication to a new dev that this is a matter to even consider. Adding other configuration-getting options like different secret-store file backends seems like a good next step.

Thanks,

--Kit

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/41cea044-ffe1-445d-ab7a-a65f69e6d85ao%40googlegroups.com.

[Dan Davis]

 tMost of the world is not as seamless as heroku.  My DevOps won't give me any more than a handful of environment variables.  I wanted something like DATABASE_URL, but all I have is DJANGO_LOG_DIR and DJANGO_SETTINGS_MODULE, and so I need many, many settings files. I think that happens a lot, and maybe a common pattern.

From a 12factor perspective, I would like to get it down to local settings (development) and production settings - yet for a lot of users, DevOps is not really supporting a full PaaS-like experience any way.

So - all of this has to be optional, which seems to rule out making it part of the starting project template.  For sure, I've got my personal template, and work has an on-premise template and a Cloud template as well - but the department of developers doesn't always use these.  I find databases containing the tables for other projects, long after the models and migrations are gone, indicating a start by copy mode.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFcS-hDn0AFW4dj_-secrMGq-iWYz-TWKG-vw9%3DVmZ3L6j04aA%40mail.gmail.com.

[Kit La Touche]

Those are valid concerns, and I want to especially come back to "all of this has to be optional, which seems to rule out making it part of the starting project template".

But first, I want to reiterate that what's vital to me (and this may be only my concern, of course!) is that newer developers, especially folks without an ops team and without ops experience, have a clearer path to deployment. Partly this is selfish: I want Django to grow and flourish, and I think that giving new folks a good experience with it their first time will facilitate that. Folks with an ops team can already be assumed to have people to talk with and work with to negotiate and understand their particular deployment context.

So, in light of that, coming back to "has to be optional", I agree—I very much agree. However, I think it also has to be visible. That was my motivation for, in my proposed PR, including it in the default `settings.py` template, but making sure defaults were set to the values they would be without the addition of `from_env`. I'm not sure at all that that's the best way, but that was my thought process! "Optional" means a few things, and I think that there's a difference between "setting env vars should be optional" and "using `from_env` should be optional" and "having `from_env` ever be in your `settings.py` should be optional", and I'd love to clarify which part of that spectrum most closely aligns to your intent.

Cheers,

--Kit

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFzonYZc%3DqjMM%2BeQ3QTQ4XJfQwAkJzxU-to8SVxiuDJJjiHy_A%40mail.gmail.com.

[Javier Buzzi]

This makes sense, I have a project that has a lot of settings files that get activated depending on the value of DJANGO_SETTINGS_MODULE. The solution i outlined above takes your reservations under consideration, if you want to use it, great, if not also great -- its a supplement not a requirement.

- Buzzi

On Wednesday, June 24, 2020 at 5:24:35 PM UTC-4, Dan Davis wrote:

 tMost of the world is not as seamless as heroku.  My DevOps won't give me any more than a handful of environment variables.  I wanted something like DATABASE_URL, but all I have is DJANGO_LOG_DIR and DJANGO_SETTINGS_MODULE, and so I need many, many settings files. I think that happens a lot, and maybe a common pattern.

From a 12factor perspective, I would like to get it down to local settings (development) and production settings - yet for a lot of users, DevOps is not really supporting a full PaaS-like experience any way.

So - all of this has to be optional, which seems to rule out making it part of the starting project template.  For sure, I've got my personal template, and work has an on-premise template and a Cloud template as well - but the department of developers doesn't always use these.  I find databases containing the tables for other projects, long after the models and migrations are gone, indicating a start by copy mode.

On Wed, Jun 24, 2020 at 1:35 PM Kit La Touche <kit...@gmail.com> wrote:

Carlton—thanks very much for the feedback. Javier—likewise. In particular, the imagined API you describe above is very appealing to me: start with `from_env` and then if you learn more about this and want to, add in some `EnvFileLoader`.

I want to make clear my motivation and agenda here: I have recently had some conversations with newer devs about their experiences with deployment of apps they're working on, and with a friend at Heroku about his informal research into the problems people have with the same. One recurring friction point (and this is not just on Heroku at all, to be clear) is that there are a number of things that people don't know they need to configure for a working deployment.

There are four settings that are recurring particular gotchas that people miss: the secret key, debug, static files, and databases. Static files seems big and out of scope, databases seems adequately handled by dj-database-url for most cases, and if your case is more complex, you'll learn it, but the other two (secret key and debug) seemed easy enough to flag as "you probably need to configure these!" with this sort of change to settings. This would be a first step towards shortening the distance from `startproject` to a working deployment.

Newer devs in particular have, based on my conversations and this friend's research, been unlikely to (a) know that there are different `startproject` templates, and (b) feel equipped to choose one, if they do know.

My hope is to make the smallest possible change to just start us moving towards more clearly flagging, especially for newer devs, "these are things that will need additional configuration in order to move from 'works on my machine' to 'deployed'."

Towards that end, I thought that adding a "you might want to get this from the env" helper would be a clear indication to a new dev that this is a matter to even consider. Adding other configuration-getting options like different secret-store file backends seems like a good next step.

Thanks,

--Kit

On Wed, Jun 24, 2020 at 11:13 AM Javier Buzzi <buzzi...@gmail.com> wrote:

I looked at the libs that do what we want:

django-configurations - it looks like they use environment variables / either via loading them from the environ or a key/value pair file. Having classes inside the settings.py might be weird to people.. at the least very different.
confucius - very simplistic, only supports environ and is classed based, similar to django-configurations.
django-environ - supports env file and environ, non-class based.
dynaconf - supports all kinds of loading options (toml, json, ini, environ, .env +) non-class based.

In my opinion, django-environ and dynaconf would be the easiest to sell to the community, it would require the least changes/paradigm shifts from how everyone is already using django.

Personally, i would really like to see something like this inside my settings.py:

from django.conf import from_env  # using standard os.environ

DEBUG = from_env.bool("DEBUG", default=False)

DATABASES = {
    "default":  from_env.db("DATABASE_URL")  # crash if it cant find it
}

...

for more complex examples:

from django.conf import EnvFileLoader

from_env = EnvFileLoader("path/to/.secret")

...

We can have how ever many loaders we want: toml, json, ini .. 

This is both borrowing heavily from dynaconf and django-environ, making the fewest changes to how people are accustomed to doing things.

.. what do you guys think?

- Buzzi

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/41cea044-ffe1-445d-ab7a-a65f69e6d85ao%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

[Tom Carrick]

Javier, I just wanted to point out another option for configuration: pydantic - it offers a very slick and intuitive interface for settings management across environments, seamless handing of environment variables by using type hints, and so on. I wouldn't recommend it for anything other than large sites with complex configurations, but it does work well for those, once you grapple with how to integrate it with django's settings so they're all exposed as `settings.FOO`, and so on.

I don't think I would want to integrate anything like this into Django proper, but it might deserve a mention in the documentation.

Tom

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7ad2c8c1-c829-42ca-8292-46d0850c0e4co%40googlegroups.com.

[Javier Buzzi]

Hey Tom, cool project haven't heard of it, looks to me more inline with validating and converting user input from an api/form. I could really see myself using this in my personal projects, however this looks like we'd be going back to the class based configuration that im trying to avoid. Nonetheless thank you for the share!

- Buzzi

On Thursday, June 25, 2020 at 4:34:11 AM UTC-4, Tom Carrick wrote:

Javier, I just wanted to point out another option for configuration: pydantic - it offers a very slick and intuitive interface for settings management across environments, seamless handing of environment variables by using type hints, and so on. I wouldn't recommend it for anything other than large sites with complex configurations, but it does work well for those, once you grapple with how to integrate it with django's settings so they're all exposed as `settings.FOO`, and so on.

I don't think I would want to integrate anything like this into Django proper, but it might deserve a mention in the documentation.

Tom

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7ad2c8c1-c829-42ca-8292-46d0850c0e4co%40googlegroups.com.

[Bobby Mozumder]

There’s also python-decouple that I use that I haven’t seen mentioned in this thread. It lets you set specific environment variables in a separate .env file or INI file: https://github.com/henriquebastos/python-decouple

-bobby

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/afd2c408-f32f-497e-814d-7c0bf3806cabo%40googlegroups.com.

[Javier Buzzi]

Hi Bobby, yes, thank you, this looks around the line of what i would like us to implement in Django. 

Side note: i saw this config('DEBUG', default=False, cast=bool) and thought "there is NO WAY that works", that led me to from distutils.util import strtobool, absolute mind blown! Thanks!

-Buzzi

On Thursday, June 25, 2020 at 1:03:19 PM UTC-4, Bobby Mozumder wrote:

There’s also python-decouple that I use that I haven’t seen mentioned in this thread. It lets you set specific environment variables in a separate .env file or INI file: https://github.com/henriquebastos/python-decouple

-bobby

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/afd2c408-f32f-497e-814d-7c0bf3806cabo%40googlegroups.com.

[Kit La Touche]

Wow, `distutils.util.strtobool` is great to know about!

So, can we refocus this conversation? This is starting to look like previous conversations on this topic, which pull in a lot of possibilities but don't lead to a change. How do we go about generating a DEP or other consensus-building tool on what we want here?

It seems to me this conversation has historically gotten stuck by trying to bite off a bigger bite. Therefore, I would recommend a minimal change that gestures towards the direction we want to explore.

Personally, I think that at minimum providing Django-builtin "get from env"  helpers would be great; beyond that, I'd love to have them be included around `DEBUG` and `SECRET_KEY` with the current values as defaults, so they're optional. Once we see how this gets used, we can see about passing it a file instead of `os.environ`, or borrowing other ideas from any of the various supporting projects that have been suggested.

It's clear that different people have different use-cases and different needs, but regardless, I think that it's clear also that including values like DEBUG and SECRET_KEY as hard coded values in settings by default does not point people towards good practices. What "good practices" are is likely to differ in each person's case, but I think that suggesting one option (again, my vote is "look in the environment") will at least help newer devs understand that this is a topic they should learn more about.

Thanks,

--Kit

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/bbf6e1f8-3405-42fe-bc7d-7e8695b275d0o%40googlegroups.com.

[Tom Forbes]

A small incremental change with some Django helpers and showing their use in the default settings.py is a great step forward, I would be +1 on that.

Tom

On 25 Jun 2020, at 18:52, Kit La Touche <kit....@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFcS-hDOiZBrNevUFwdpR0kvN0u74pTPyLHyH7aCM5dPuXb9Xg%40mail.gmail.com.

[Bobby Mozumder]

If you at least don’t separate all variables that are dependent on the environment from the settings.py, then you’re going to have to edit your settings.py file anyways, defeating the purpose of this. Database and Cache connection settings are clearly dependent on the environment.

-bobby

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFcS-hDOiZBrNevUFwdpR0kvN0u74pTPyLHyH7aCM5dPuXb9Xg%40mail.gmail.com.

[Javier Buzzi]

Think of it like this, you make a simple settings/base.py where you define something like this:

from django.conf import from_env

MEDIA_PATH = '/mnt/media'  # hardcoded to show you don't have to use `from_env`

DEBUG = from_env.bool('DEBUG')

SECRET_KEY = from_env('SECRET_KEY')

DATABASES = {

    'default': from_env.db('POSTGRES_URL'),  # "postgres://user:pa...@db.server:4321/db_name?params"

    'extra': from_env.db('SQLITE_URL', default='sqlite:////tmp/my-tmp-sqlite.db')

}

CACHES = {

    'default': from_env.cache('MEMCACHE_URL'),

    'redis': from_env.cache('REDIS_URL')

}

..

and you can still have an other settings/dev.py or settings/test.py that all inherit from settings/base.py and still can customize it some more in those files. I have a project that because of ops, environment variables are an issue and i have to rely on having multiple environment files, i dont like it, but i live with it, such is life. I much rather have a base.py that ALL environments use, and then have a tests.py that inherits from base.py and will add mocks to things like s3 and other things needed to ensure tests pass. But i would really recommend to keep the amount of environment files to an absolute minimum -- nevertheless this doesnt rob you of being able to accomplish what you're used to doing.

- Buzzi

On Thursday, June 25, 2020 at 4:12:26 PM UTC-4, Bobby Mozumder wrote:

If you at least don’t separate all variables that are dependent on the environment from the settings.py, then you’re going to have to edit your settings.py file anyways, defeating the purpose of this. Database and Cache connection settings are clearly dependent on the environment.

-bobby

On Jun 25, 2020, at 1:51 PM, Kit La Touche <kit...@gmail.com> wrote:

Wow, `distutils.util.strtobool` is great to know about!

So, can we refocus this conversation? This is starting to look like previous conversations on this topic, which pull in a lot of possibilities but don't lead to a change. How do we go about generating a DEP or other consensus-building tool on what we want here?

It seems to me this conversation has historically gotten stuck by trying to bite off a bigger bite. Therefore, I would recommend a minimal change that gestures towards the direction we want to explore.

Personally, I think that at minimum providing Django-builtin "get from env"  helpers would be great; beyond that, I'd love to have them be included around `DEBUG` and `SECRET_KEY` with the current values as defaults, so they're optional. Once we see how this gets used, we can see about passing it a file instead of `os.environ`, or borrowing other ideas from any of the various supporting projects that have been suggested.

It's clear that different people have different use-cases and different needs, but regardless, I think that it's clear also that including values like DEBUG and SECRET_KEY as hard coded values in settings by default does not point people towards good practices. What "good practices" are is likely to differ in each person's case, but I think that suggesting one option (again, my vote is "look in the environment") will at least help newer devs understand that this is a topic they should learn more about.

Thanks,

--Kit

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/bbf6e1f8-3405-42fe-bc7d-7e8695b275d0o%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFcS-hDOiZBrNevUFwdpR0kvN0u74pTPyLHyH7aCM5dPuXb9Xg%40mail.gmail.com.

[Carsten Fuchs]

Hello,

Am 25.06.20 um 19:51 schrieb Kit La Touche:
> […] Once we see how this gets used, we can see about passing it a file instead of `os.environ`, […]

This is probably a stupid question, but what is the advantage of this (and env vars) over this:

In project_dir/project_dir:

settings.py
localconfig.example
localconfig.py

settings.py is the normal settings file.
localconfig.example is an example file for a local configuration, kept in version control.
localconfig.py is a customized copy of localconfig.example, not kept in version control.

localconfig.example and localconfig.py would contain e.g.:

DEBUG = False
SECRET_KEY = '1234'
# ...

and in settings.py:

from project_dir import localconfig

DEBUG = localconfig.DEBUG
SECRET_KEY = localconfig.SECRET_KEY

?

Best regards,
Carsten

[Javier Buzzi]

Hi Carsten, great question! The idea as i see it is: having the least amount of moving parts the better. The trick here is unification, keep a single settings file with 99% of the configuration needed using environment variables/secrets/zookeeper/etc to swap out environment specific options (activate debug mode? database connections, cache connections, domain_allowlist (formally whitelist), etc.)

This has happened a couple of times in my lifetime, stop me if you've heard it: You have N environments, they're all slightly different, you're working on local and some kind of qa environment, everything is going great, but you forgot to do that last minute change you did on the higher environments to the prod settings -- you deployed; deployment failed. Sound familiar? Having a single / or at least keeping the amount of environment setting files to a minimum will greatly benefit you in the long run, its also easier to test: remember flat is better than nested.

With the method i'm suggesting, your example would not need a localconfig.py since settings.py would just import them from the environment variables.

- Buzzi

[Florian Apolloner]

Hi there,

On Thursday, June 25, 2020 at 7:52:31 PM UTC+2 kit....@gmail.com wrote:

Personally, I think that at minimum providing Django-builtin "get from env"  helpers would be great; beyond that, I'd love to have them be included around `DEBUG` and `SECRET_KEY` with the current values as defaults, so they're optional. Once we see how this gets used, we can see about passing it a file instead of `os.environ`, or borrowing other ideas from any of the various supporting projects that have been suggested.

I am all for minimal variants, but I do not think this would could it. your "get from env" helpers would basically be os.environ.get("SECRET_KEY", "secret_default") which does provide much usefulness on it's own imo. So even if we started out with a minimal implementation we'd at least have to have a good plan on what to do in the future.

And there are plenty more things to consider; for instance I do not agree that it makes sense to have "SECRET_KEY" default to a value when missing in the env. It is way to easy to type "SECRT_KEY" and never realize that. So if "SECRET_KEY" is taken from the environment it should fail loudly if it is not present. "DEBUG" is in a similar category there but could default to False to be safe.

I personally rather have no solution in Django itself before forcing a half-baked one down everyone. Also please note that the bar to add this to Django is very high since it can (at least for things like django-environ) easily live outside of Django with no realy downside.

Cheers,

Florian

[Javier Buzzi]

Hi Florian, thank for your input, i dont belive in adding default names to environment variables, they're up to the user to define. Nothing will be given by default, you need the SECRET_KEY? `from_env` has no idea what that means, its just another name, so you tell it what it will do with this such as `from_env.str("SECRET_KEY")` or simply `from_env("SECRET_KEY")` if its a string. But if "SECRET_KEY" is taken by some other process and you need to add another one to be your SECRET_KEY for django and you call it "FOO" then `FOO` would be the name you pass to `from_env` and assign it to your SECRET_KEY = from_env('FOO'). Trying to really nail the point home: there are NO default environment names (other than DJANGO_SETTINGS_MODULE but that has nothing to do with this).

- Buzzi

[Adam Johnson]

i dont belive in adding default names to environment variables, they're up to the user to define

Javier, I think you missed what Florian was talking about. He was suggesting there shouldn't be default *values* for some settings.

 

I personally rather have no solution in Django itself before forcing a half-baked one down everyone. Also please note that the bar to add this to Django is very high since it can (at least for things like django-environ) easily live outside of Django with no realy downside.

I agree with this sentiment. The proliferation of libraries can be a bit confusing, but I've not yet felt one solution was "the winner".

 

Using environment variables isn't even suitable for all situations. They make sense for managed platforms like Heroku, or single process servers. But on shared servers, they can be a security risk because other users can potentially read them from the process: https://www.diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/ . This is why Docker has a system for providing managed secrets through a filesystem mount: https://docs.docker.com/engine/swarm/secrets/ .

On Fri, 26 Jun 2020 at 09:55, Javier Buzzi <buzzi....@gmail.com> wrote:

Hi Florian, thank for your input, i dont belive in adding default names to environment variables, they're up to the user to define. Nothing will be given by default, you need the SECRET_KEY? `from_env` has no idea what that means, its just another name, so you tell it what it will do with this such as `from_env.str("SECRET_KEY")` or simply `from_env("SECRET_KEY")` if its a string. But if "SECRET_KEY" is taken by some other process and you need to add another one to be your SECRET_KEY for django and you call it "FOO" then `FOO` would be the name you pass to `from_env` and assign it to your SECRET_KEY = from_env('FOO'). Trying to really nail the point home: there are NO default environment names (other than DJANGO_SETTINGS_MODULE but that has nothing to do with this).

- Buzzi

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d6cf479b-4e37-411b-b999-73a5944995a3o%40googlegroups.com.

--

Adam

[René Fleschenberg]

Hi Flo,

> And there are plenty more things to consider; for instance I do not
> agree that it makes sense to have "SECRET_KEY" default to a value when
> missing in the env. It is way to easy to type "SECRT_KEY" and never
> realize that. So if "SECRET_KEY" is taken from the environment it should
> fail loudly if it is not present. "DEBUG" is in a similar category there
> but could default to False to be safe.

There is a trade-off between security and development convenience here.
At the moment, the settings generated by manage.py startproject are
geared towards development (random default value for SECRET_KEY, DEBUG =
False). Personally, I like to keep that convenience, so I do

SECRET_KEY = os.environ.get(SECRET_KEY, original_default_value)

There is also manage.py check --deploy. This catches DEBUG = True, but
not SECRET_KEY. Would it be a good idea to prefix the default SECRET_KEY
with something like "insecure-" and check for that in manage.py check
--deploy?

> I personally rather have no solution in Django itself before forcing a
> half-baked one down everyone. Also please note that the bar to add this
> to Django is very high since it can (at least for things like
> django-environ) easily live outside of Django with no realy downside.

+1

--
René Fleschenberg

[René Fleschenberg]

Hi,

On 6/26/20 12:11 PM, René Fleschenberg wrote:
> geared towards development (random default value for SECRET_KEY, DEBUG =
> False).

I meant DEBUG = True here, of course :)

--
René Fleschenberg

[Shai Berger]

Hello,

On Fri, 26 Jun 2020 00:46:02 -0700 (PDT)
Florian Apolloner <f.apo...@gmail.com> wrote:

> On Thursday, June 25, 2020 at 7:52:31 PM UTC+2 kit....@gmail.com
> wrote:
>

> > Personally, I think that *at minimum* providing Django-builtin "get

> > from env" helpers would be great; beyond that, I'd love to have
> > them be included around `DEBUG` and `SECRET_KEY` with the current
> > values as defaults, so they're optional. Once we see how this gets
> > used, we can see about passing it a file instead of `os.environ`,
> > or borrowing other ideas from any of the various supporting
> > projects that have been suggested.
>
> I am all for minimal variants, but I do not think this would could

> it. [...]

> And there are plenty more things to consider; for instance I do not
> agree that it makes sense to have "SECRET_KEY" default to a value
> when missing in the env. It is way to easy to type "SECRT_KEY" and
> never realize that. So if "SECRET_KEY" is taken from the environment
> it should fail loudly if it is not present.
>

> [...] please note that the bar to add

> this to Django is very high since it can (at least for things like
> django-environ) easily live outside of Django with no realy downside.
>

Before this, when explaining the motivation for this, Kit said:

> My hope is to make the smallest possible change to just start us
> moving towards more clearly flagging, especially for newer devs,
> "these are things that will need additional configuration in order to
> move from 'works on my machine' to 'deployed'."

We already have a tool designed for this: the "check --deploy"
management command[1]. We can improve it a little by helping it detect
that the SECRET_KEY is carelessly kept hard-coded from the initial
project creation, e.g. by including somewhere a marker class:

class HardCoded(str): pass

and then in the default template

SECRET_KEY = HardCoded('generated_value_like_today')

Then the check could easily detect it and tell the user they need to
change it.

I think that a settings-only solution cannot be appropriate here, for
the reasons Florian noted -- Kit is basically asking the default
template to educate the user about deployment, but there is a tension
between this and having things Just Work, which is important for
beginners, and Just Work Securely (which is why the default SECRET_KEY
needs to be a generated, random secret). To resolve this tension, we
need a tool which is aware of context -- a tool which differentiates
between the beginner on their laptop, and the novice doing their first
deployment. The settings, in general, cannot make this distinction; a
management command can.

We have a whole bunch of documents about deployment[2] - maybe a little
too much, and this probably encourages shorter "just do this" type
deployment tutorials. Maybe we should make the checklist[3] more
prominent, maybe we should make "check --deploy" even more prominent.
But I don't think a single settings file can be all things for all
users.

My 2 cents,
Shai.


[1] https://docs.djangoproject.com/en/3.0/ref/django-admin/#cmdoption-check-deploy
[2] https://docs.djangoproject.com/en/3.0/howto/deployment/
[3] https://docs.djangoproject.com/en/3.0/howto/deployment/checklist/

[Tom Carrick]

I do have a use-case where having a default SECRET_KEY makes things much easier - docker.

Normally you can't run management commands in a Dockerfile if there's no secret key (and often other things) set, and usually it's best to run collectstatic as a build step.

So your options end up being:

1. Providing a default - works well as long as you remember to change it for production.

2. Adding ARG SECRET_KEY=x to the line before - works ok, but when you start needing to add more env vars for various other settings, it gets really annoying.

3. Having a specific settings file just for building the image - also somewhat annoying.

What I usually do (perhaps unwisely) is this:

from django.core.management.utils import get_random_secret_key

SECRET_KEY = os.getenv("SECRET_KEY", get_random_secret_key())

This makes it pretty obvious (to me) if I've forgotten to set a value in an environment because I'll keep getting logged out of the site.

That is probably a terrible experience for new users, though, I can imagine it's pretty confusing if you don't know what the problem is.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20200626131942.553c2b54.shai%40platonix.com.

[Florian Apolloner]

On Friday, June 26, 2020 at 12:11:34 PM UTC+2 re...@fleschenberg.net wrote:

There is a trade-off between security and development convenience here.
At the moment, the settings generated by manage.py startproject are
geared towards development (random default value for SECRET_KEY, DEBUG =
False).

Yes, but the situation is somewhat different in that case; for instance we advice people often to have different settings files ala dev.py and prod.py. Now dev.py can happily define `SECRET_KEY=123`, not much of an issue there. prod.py would simply use `SECRET_KEY = os.environ["SCRET_KEY"]` without a default since you absolutely need to set that. It becomes dangerous when you go full 12-factor and merge those two config files. Now it might become (as you suggested) `SECRET_KEY = os.environ.get("SECRET_KEY", "123")` which I find rather dangerous. The better solution here might be to use a random default (ie every process start) but that might make development tedious.

So all in all I think the chance of misstyping `SECRET_KEY` is higher than setting `DJANGO_SETTINGS_MODULE`  to dev in production… FWIW this is exactly the reason why I also still have dev & prod settings. They both star-import stuff from settings.common, but otherwise define whatever they need. This also allows me to keep settings files free from if/else conditionals, since a production deployment usually is vastly different from a dev variant (error logging to sentry, x-forwarded-for handling, secure cookies everyhwere).

 

There is also manage.py check --deploy. This catches DEBUG = True, but
not SECRET_KEY. Would it be a good idea to prefix the default SECRET_KEY
with something like "insecure-" and check for that in manage.py check
--deploy?

Oh wow, this is good. I'd even go further and call it "dj::insecure::" so you can really easily grep for it and maybe even add it to git hooks.

Cheers,

Florian

[Florian Apolloner]

Hi Tom,

On Friday, June 26, 2020 at 12:43:19 PM UTC+2 t...@carrick.eu wrote:

I do have a use-case where having a default SECRET_KEY makes things much easier - docker.

Normally you can't run management commands in a Dockerfile if there's no secret key (and often other things) set, and usually it's best to run collectstatic as a build step.

 I do not see that as an argument for a default SECRET_KEY. Maybe we should just defer the check for SECRET_KEY till it is accessed? There is no reason Django should complain a missing secret key when it does not need one at all.

Cheers,

Florian

[Adam Johnson]

There is no reason Django should complain a missing secret key when it does not need one at all.

I agree with this. It is an extra step in making really basic apps that don't use any encryption functionality. IIRC, the "Django needs a secret key" check has always been a custom check rather than a system check mostly because it predates the system check framework. I think we could hoist it to a system check that is triggered only when sessions or other features that depend on it are installed (although I don't know the exhaustive list).

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/83630d2b-f51b-4ec3-b92a-838feb452fa1n%40googlegroups.com.

--

Adam

[Tim Graham]

Here's an unsuccessful attempt to defer the SECRET_KEY check: https://code.djangoproject.com/ticket/29324

It seems like it could be insecure to move that to a system check as "For performance reasons, checks are not run as part of the WSGI stack that is used in deployment." (Also, it seems impossible to write a system check that determines whether or not a project will consult SECRET_KEY.)

[Florian Apolloner]

On Sunday, June 28, 2020 at 1:23:25 AM UTC+2 timog...@gmail.com wrote:

It seems like it could be insecure to move that to a system check as "For performance reasons, checks are not run as part of the WSGI stack that is used in deployment." (Also, it seems impossible to write a system check that determines whether or not a project will consult SECRET_KEY.)

The check if it is empty can be done on access in LazySettings without any real overhead. We are even calling url validators when access MEDIA_URL/STATIC_URL (https://github.com/django/django/blob/62d85a283500e9abb0e1c9ec53c59be468f056a0/django/conf/__init__.py#L152-L158) -- so we really don't have to talk about overhead here :D

I will try to remove those properties and move the checks into `__getattr__`, this should result in a (small) one time overhead. PR(s) to follow.

I think a deployment system check for non-empty SECRET_KEY might also make sense.

Cheers,

Florian

[Florian Apolloner]

As promised the PR: https://github.com/django/django/pull/13120 I also had to touch TokenResetGenerator since it stores the secret in a class level variable and that would prevent the server startup either way…

Let's see what the full test suite says, but I cannot really imagine many issues.

[Carlton Gibson]

Thanks for the discussion all, and PRs. Lots of different perspectives. 

* Florian's draft PR looks more or less right, so we should be able to eliminate the need to specify a SECRET_KEY if you're not going to use it.

* I opened #31757 to capture the idea to add a prefix (say `dj::insecure`) to the generated SECREY_KEY, and a deployment system check to guard against using that. 

Then (summarising): 

* There was a similar discussion to here, and PR, on #20081. The same kind of points against favouring env vars were raised by Django Security Team members as have been raised on this ticket. (It's an approach but not the only one.) 

* There is a difficulty for beginners coming to deployment. (Not just beginners, let's be honest 🙂). Most likely way forward is better guides to different options, either in the Django docs, or in blog posts, or beginning in one and moving to the other. The current Deployment Docs are maybe not as good as we'd like: but it's hard, and hard to keep up to date if they become too specific, and there is a steady stream of material out there that covers this — Maybe the highest ROI is a group working there. 

* There are great third-party packages out there addressing the problems here. That there's no clear winner is probably in part because there are multiple ways of doing it, that are all good in some circumstance. We shouldn't try to narrow that artificially. 

I suspect this will come up again. 🙂

Kind Regards,

Carlton

[1337 Shadow Hacker]

Do we really need DJANGO_ prefix on env vars ? In my first years of practicing 12-factor I used such prefix, but the last 5-6 years I let it go, because I just ended up with a list full of DJANGO_ variables in a containerized where only Django is running.

[Aaron C. de Bruyn]

Not everyone runs containerized.

I think some settings shouldn't be prefixed--i.e. DATABASE_URL is a pretty common one.

-A

On Tue, Jul 7, 2020 at 12:40 AM '1337 Shadow Hacker' via Django developers (Contributions to Django itself) <django-d...@googlegroups.com> wrote:

Do we really need DJANGO_ prefix on env vars ? In my first years of practicing 12-factor I used such prefix, but the last 5-6 years I let it go, because I just ended up with a list full of DJANGO_ variables in a containerized where only Django is running.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3mulSta40m2DoZMPITSVrn3XE4xm_9NBbL1CMxtE8MqCnFqR-KT_RAOpxqfhH7Gn_6he9QkZ0z0HN0O2y2GAz8ifdQFghXcjfOMQF_wsUgs%3D%40protonmail.com.

[Carles Pina i Estany]

Hi,

I haven't read all this thread in detail and I might go off-topic. Sorry
about that.

On Jul/07/2020, '1337 Shadow Hacker' via Django developers (Contributions to Django itself) wrote:
> Do we really need DJANGO_ prefix on env vars ? In my first years of

I know when having a variable named "DEBUG" instead of "DJANGO_DEBUG"
affected me... due to a bug in Thunderbird and how, in that machine,
variables for the Django project were set.

Thunderbird on Debian didn't run at all because the launcher has a bug
(https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960230 ). If DEBUG is
set it doesn't run. If the bug wasn't there, well, it would have ran in
DEBUG mode probably or if Django named it DJANGO_DEBUG it would have not
affected Thunderbird. I thought of prefixing variables when possible to
avoid these interactions.

Cheers,

--
Carles Pina i Estany
https://carles.pina.cat

[Divyesh Khamele]

Hi Charles,

Divyesh here,have 4+ years of experience to work as a Django developer. I  can help you on this.

You need to hire me.

Hourly rate: 8$/hr.

Best,

D

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20200707213034.GA2308%40pina.cat.

[Kye Russell]

Lol.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAEV-ETTAG-c7UbUqUtowWRKM-kYNPZ3SzaqzL8O-h58sKiHeTQ%40mail.gmail.com.

[Hadisur Rahman]

LOL

>> email to django-develop...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/

>> msgid/django-developers/20200707213034.GA2308%40pina.cat.

>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-develop...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAEV-ETTAG-c7UbUqUtowWRKM-kYNPZ3SzaqzL8O-h58sKiHeTQ%40mail.gmail.com

> <https://groups.google.com/d/msgid/django-developers/CAEV-ETTAG-c7UbUqUtowWRKM-kYNPZ3SzaqzL8O-h58sKiHeTQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .

>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-develop...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-developers/CANK-ykkM%2BRrztEmEVTBuDxuasXao%2BZMQXGLhQ_ANV6aaJSwE%2Bg%40mail.gmail.com.
>

[1337 Shadow Hacker]

All right, thank you for your feedback.

May I throw in the idea of using DJ_ instead of DJANGO_ as prefix ?

[Sci Mithilesh]

ok whatsapp +918709440658

On Fri, 10 Jul 2020, 2:54 pm '1337 Shadow Hacker' via Django developers (Contributions to Django itself), <django-d...@googlegroups.com> wrote:

All right, thank you for your feedback.

May I throw in the idea of using DJ_ instead of DJANGO_ as prefix ?

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/PBiCVsKC2UAJFHuD-tEkUh2063D7DgV1DcAxdlcFI2tSqpPxK-8XVWSPbSnyvG5spVf_1QYHZeyNN2dimPNPk7NrS0eh9zq-PhAQxSHAsMk%3D%40protonmail.com.

[Florian Apolloner]

Comments like these are inappropriate for this mailing list. Stop spamming your whatsapp number all over the place.