DJANGO_SETTINGS_FILE

[James Pic]

Hi all!

Life with DJANGO_SETTINGS_MODULE has been great. However, it requires the settings to be an importable module. In most cases, this is a straight forward "simple" python script. 

Sometimes it looks like it could make sense to add an environment variable that could use a file that is not importable as a python module, ie.:

DJANGO_SETTINGS_FILE=/etc/yourapp/production.py

DJANGO_SETTINGS_FILE=~/yourapp_production.py

DJANGO_SETTINGS_FILE=settings.py  # relative to pwd

Then, in this file, we could override the default project settings there without having to put the new file in an importable module, ie. in /etc/yourapp/production.py:

from yourapp.settings import *

DATABASES=...

CACHES=...

SECRET_KEY=...

I'm sure this has been discussed before but didn't find anything by searching DJANGO_SETTINGS_FILE in the ML.

What's your opinion on this ? Is it time to upgrade this ?

Best,

[Adam Johnson]

You could do this already by putting a file on PYTHONPATH that just reads the contents of the file specified by DJANGO_SETTINGS_FILE and puts it in its own module dict, e.g.

$ cat ~/tmp/test.py

foobar = 12

$ TEST_SETTINGS_FILE=~/tmp/test.py ipython

...

In [1]: import imp, os

In [2]: test = imp.load_source('test', os.environ['TEST_SETTINGS_FILE'])

In [3]: globals().update(test.__dict__)

In [4]: foobar

Out[4]: 12

You could make this a third party package too, and just point django to it.

I don't see why Django needs to change to make this somewhat esoteric usecase possible.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op19-0JeL8Jdcq70-pvwz%2Bi2hfKtopEvihca_rGfH1DrPww%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Adam

[Tobias McNulty]

I don't see Django making a change like this. If it's really impossible to get your production settings file on an already-importable Python path, you could modify your PYTHONPATH environment variable to add the directory containing your Django settings file. That being said, I think you're better off keeping all your settings files in version control where they belong, and changing the things you need to change via environment variables.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op19-0JeL8Jdcq70-pvwz%2Bi2hfKtopEvihca_rGfH1DrPww%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Tobias McNulty
Chief Executive Officer

tob...@caktusgroup.com
www.caktusgroup.com

[Chris Church]

Look through https://code.djangoproject.com/wiki/SplitSettings for many different approaches to loading settings from files that aren't importable as Python modules.

My personal favorite is django-split-settings, which would let you accomplish exactly what you're trying to do.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMGFDKThapJkEykuhxFEuTF%2BZNjHNr7t6p%3Dssenb0qM85DVXyQ%40mail.gmail.com.

[James Pic]

There are indeed a nice handful of apps out there that help doing this
in different manners. While we're at it, my own way is to make every
project a python package with a setup.py. So the settings can always
be imported. Then I let docker swarm or kubernetes handle environment
variables which are (surprise) environment-specific (the cherry on the
top being the entry point that points to manage.py).

I'm sure everybody here has a favorite way, which works, even if it's
not as pretty as my favourite way which is the prettiest in my opinion
of course ;) Let alone aesthetics,
DJANGO_SETTINGS_MODULE=production_settings
PYTHONPATH=/etc/yourapp:$PYTHONPATH is basically a lot more
error-prone than DJANGO_SETTINGS_FILE=/etc/yourapp/production.py, is
it even necessary to demonstrate this ? This somewhat reminds my
favorite talk this year, "Don't fight the controls", by Daniele
Procida.

Is there a problem to fix for any of us ? Probably not, we've been
deploying django for more than a decade with DJANGO_SETTINGS_MODULE.
We know are way in there, so nothing is going to change if our needs
don't change or if we don't take a step back to look at the bigger
picture.

Do I need this to deploy my projects ? No of course, because I use the
prettiest way hhihihi ;) I'm more than happy to win a debate
aesthetics of automated deployment any time with gitlab ci and docker
swarm or kubernetes of course, but actually that's not why that thread
was even about, sorry if I misguided you, this thread is neither about
you neither about me actually.

Do I need this in projects which I'm not the only one to deploy ? Well
now we're getting somewhere. You see, when someone makes settings.py
parse a json file, or read an environment file (yes, django-environ
actually has the feature to parse an environment file), it reminds of
E. W. Dijkstra's quote "The computing scientist’s main challenge is
not to get confused by the complexities of his own making".

Do you think this kind of situation would still have happened if
Django supported DJANGO_SETTINGS_FILE ? If so, would it at least be
less likely ? They wouldn't need to mess with the settings file that
we all share and that should not do anything more complicated than
reading the existing environment according to 12factor (and my taste
:)), they could just extend it and pass the path.

Do I ear about projects where they invent a new per-environment
configuration files ? Yeah, a lot, and you Adam, you actually have
never heard about it to the point where you call it "esoteric" ? How
is defining a configuration file esoteric ? What's esotheric here, is
PYTHONPATH=/etc/yourapp DJANGO_SETTINGS_MODULE=production (or should
it be production_settings ? doesn't matter, it'll smell a mile away
anyway). DJANGO_SETTINGS_FILE=/etc/yourapp/production.py is logical
and explicit. How me, kubernetes or docker swarm do
env-specific-file-less configuration, can be called esoteric, but it's
certainly not the other way around.

I'm sure many of you maintain open source django projects which are to
be installed by non technical people. I mean, I assume most of us are
either making big money in big corporations either dreaming of it, but
I also know most of us also dedicate a huge amount of time to
contribute to NGOs with Django, or just teach django for the pleasure
to share a passion. In any of this case, the project developer (us)
faces a bunch of people who are really willing to learn, python,
django, linux, bash, and so on, but have no experience. And this is
how the drama begins, I can imagine them in front of Tobias.

"How do I configure this project ?", they ask.

"Just **prepend** the configuration **directory** to PYTHONPATH, and
use the file name without the .py extension as
DJANGO_SETTINGS_MODULE", Adam says.

The user adds a file in /etc, called "memopol.py", to store their
settings. Then after they do what Adam told them, itsnotworking again.
"ModuleNotFoundError: No module named 'memopol.wsgi' 'memopol' is not
a package". I'm sure it won't take long before Tobias begins a
technical explanation (instead of working on the project itself):

"Well django couldn't import WSGI anymore because it was in
memopol/wsgi.py in the package, and python imported memopol from /etc
now that it's first in the PYTHONPATH".

"....", (it doesn't matter, the user is lost, let's talk him into a
solution to cheer them up).

"So what you can do for example, is to prepend something unique to
your configuration file, such as _settings"

"Same error"

"What ??"

It won't take long before Tobias figures that this project also
depends on a module named memopol_settings.

And that's not even the worst case scenario.

Once you've been through this for a while, sharing your django
projects with non technical people, you start thinking "I won't forget
to ask django-dev if my friends can haz DJANGO_SETTINGS_FILE next
time". And here I am today, at their service, at your service, thanks
for reading.

[James Bennett]

On Thu, Apr 6, 2017 at 5:22 PM, James Pic <jp...@yourlabs.org> wrote:

Do I need this to deploy my projects ? No of course, because I use the
prettiest way hhihihi ;) I'm more than happy to win a debate

Please don't do this. This does not make you look like someone who I could constructively engage with to try to figure out what problem is being reported and what a good solution would be.

 

And this is
how the drama begins, I can imagine them in front of Tobias.

Please also don't do this. You are perfectly well aware that people will give one type of explanation to a new user they're helping, and quite another to a mailing list of experienced developers discussing the merits of different approaches. And regardless of the context-insensitivity of what you're doing here, putting words in someone's mouth to try to make them look bad just makes you look bad.

Meanwhile, Django has a way to avoid all the turmoil you've proposed in your reply -- run 'startproject', get package with settings file inside, run 'manage.py <command>'. Several tools for deploying Django into production even make use of this, providing custom management commands which can be invoked to run/deploy/etc., and doing so requires no deep dives into Python imports, Python packages, or environment variables. Of course, a third-party package which develops its own independent configuration mechanism will need to teach all of its users how to make use of and deploy that configuration mechanism, but that's the responsibility a third-party project takes on when it invents its own method.

[James Pic]

I'm sorry you've seized the opportunity to use my effort to put a little joy and humor in my message to push me on the slope, sorry that it's been misinterpreted and that I just don't understand at all your technical explanation. I guess there is always that risk.

Best

James

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAL13Cg_JBcc6mYhH7un3LKgGFR22pZz%3D_kQyc0sB5dh4uq8g-g%40mail.gmail.com.

[James Pic]

Tobias, Adam, I have "imagined" you this dialogue where you're helping people deploy an open source citizen lobbying tool in Django in an NGO was not to make you look bad of course, the intention was rather the inverse of that. This situation, and exact dialogue, I have actually had, see the source code if you don't believe me: there is a module called memopol_settings and I have actually been bitten by this. Obviously I forgot humility when I thought you would have said the same to the users as you told me on the list - because I actually have - but it's true that I was biased when I thought you would have done the same mistake as me, so, sorry for making you look like you would have yourself :)

Anyway, people from over the world have been **fighting** this situation rather than **fixing it upstream** for **ages**, see the demand:

https://djangopackages.org/grids/g/configuration/

https://code.djangoproject.com/wiki/SplitSettings

On one hand, I ear about all the efforts our community is doing to be inclusive, and not being just elitist trolls (like me unfortunately but working on it!!), on the other hand, we don't even provide a native and simple way for user to put their environment specific configuration in a file, despite the obvious demand (see links above, any NGO's workshop), just because we, linux experts, are experienced enough to deal with it as is.

"You can't expect everybody to be a guru like you James" is what people have telling me over and over again, and what I'm working on (because my name is also James). And that's exactly why I'm trying to understand where they struggle and what's the most efficient way to fix it **upstream**, because that's my philosophy, don't fight the controls, fix them.

Now, you say I can invent my own settings file system, or reuse any of the existing system. While that is perfectly true, the problem with this, is that I don't need any myself: I'm happy with env vars and a settings module, because I deploy containers and set environment varibales at container creation. At the risk of looking like a broken record (in addition to everything I already look like, "le ridicule ne tue pas" :D): it's not about me, it's not about you, it's about different users.

I'm ending up with a different implementation (that i don't need) in every project I contribute to, because everyone creates their own, when after all the need is the same: they just want a configuration file outside the project with their environment specific thing. This is how I ended up with a project in my hands, that requires writing an undocumented JSON configuration file or something as warry as that. Currently I have to learn each and every environment-specific settings file app there is out there just in case somebody adds it to their project because of the lack of DJANGO_SETTINGS_FILE.

After all, isn't the framework about proposing design decisions ?

Doesn't it make sense for a framework to at least provide projects built on it with a way to pass a config file by path ?

And really, if the django project **should** be a python module so that the settings should **always** be importable, then shouldn't startproject create a setup.py ?

As you said, using django project shouldn't even **require** configuring DJANGO_SETTINGS_MODULE even at all because it's hard coded in wsgi.py and manage.py from the moment you run django-admin startproject. However, DJANGO_SETTINGS_MODULE was never designed for deployment: environment specific configuration. 

So really, all we need users to do, is to call `DJANGO_SETTINGS_FILE=/etc/theirconfig.py ./manage.py` (or `DJANGO_SETTINGS_FILE=/etc/memopol/production.py memopol`) to keep environment specific, non-defaults, where it belongs: outside the **public project**. Keep that in swarm, a json blob in etcd (ie. kubernetes) or in a repository with ansible playbooks, chef recipes, saltstack roles, for example it's your choice, but environment specific variables have nothing to do in the public repository anyway.

About my joke on deployment debates, I recon I am in a currently mad love story that drives me a bit crazy (in a good way) with gitlab-ci and docker swarm, because it lets me treat server as cattle put a django project under continuous deployment with like ~100 SLOCs of YAML and ~5 commands on a server in NGOs that have no money to buy services such as Divio cloud or else. I love it so much, it makes me euphoric, guilty as charged, I don't feel like writing about it in a boring way where we can't indicate a joke with "<some ridiculously and obviously subjective and un-technical statement> hihihihihi ;)". Because if I were going to write on this in a boring way, I'd say perhaps making Django more comfortable to deploy for non-technical users would not be good for business. But then again, I'm not talking about commercial projects at all, rather, citizen lobbying tools and the like, software for people who strive to change the world, people without technical knowledge for example, but who have already changed how politics work at some levels with a simple camera, do we need these people ? I say yes, go diversity, lower the barrier.

But my question is, do we want only people who treat servers as cattle, or do we also want newbies to be comfortable deploying Django projects which don't have invented any config file system because their maintainer doesn't need it ?

Thanks for reading, and sorry in advance for all my mistakes, I'm trying not to offend anybody but obviously I'm still very young and childish, feel free to quote each of them and push me on the sloppy slope if you feel like it. But please, read the whole message, again, I'm not speaking up for myself, I'm speaking up for normal Humans, who just want to be actors of the internet, not for profit, rather than just consumers of technical people's services.

[James Pic]

tl;dr

I'm not debating deployment here: my automated deployments are fine
the way it is today: I love linux and env vars, shells and subshells
and so on.

I'm /reporting/ that users have been fighting the lack for
DJANGO_SETTINGS_FILE for as long as I can remember.

In many case, they loose the fight and make something that's unusable
for an actual linux or django guru like you.

Have I really been the only witness of all kinds of horrors in
settings.py just because they wanted to have their deployment settings
parsed in it from another file path ?

I suggest we show people who do not abide by 12factor that they are welcome too.

I recommend we show people who do not have the technical background to
deal with env vars that we can think of them too, and not just about
ourselves hackers.

So, next time I have a Django deployment workshop with newbies, they
won't tell me: "You can't expect everybody to be a guru James".

Thanks for reading
Best
James

[Daniele Procida]

On Sat, Apr 8, 2017, James Pic <jp...@yourlabs.org> wrote:

>I'm sorry you've seized the opportunity to use my effort to put a little
>joy and humor in my message to push me on the slope, sorry that it's been
>misinterpreted and that I just don't understand at all your technical
>explanation. I guess there is always that risk.

I think in these discussions the greatest risk isn't failure to understand something technical, but failure to understand the other person.

Given differences in timezones, language, cultural expectations etc with great opportunity for misunderstandings to escalate, it seems like a good idea, when there is any doubt, to assume that what another person is saying is friendlier than it appears, and that no-one is trying to win arguments by making others look bad.

Daniele

[Josh Smeaton]

FWIW - I wasn't completely sure what the problem was you were relating or the solution you were proposing as a means to fix it. I came to this thread a bit late, and was probably overwhelmed by lots of text without reading and understand your first message in the thread well enough. The big replies further down thread don't help your case - they hurt it, because it's now much harder to understand the actual problem and the proposed solution.

Anyway, let me summarise, and you can let me know if I'm off.

- The requirement of having all settings defined within the project space (by default) is a poor user experience; because

- Good security practise is to define specific settings (SECRET_KEY, various passwords, db connection strings) outside the normal settings.py file so not to commit to source control

- Most *nix people are used to putting settings in files such as `/etc/<program>/<program>.conf`

- So Django should support such a thing

I'll admit that a number of times, including very recently, I've tried to pass `settings=settings/local.py` rather than `settings.local`. As you've also pointed out, there are quite a number of third party apps that deal with settings, and a number of strategies to making production settings available to the program.

So I think there are a few questions to go over.

1. What are the more successful strategies that work in the wild? (files in /etc/, PYTHONPATH, env vars, local_settings.py)

2. Are any of the above clearly superior?

3. Is this a serious problem that people are having?

4. If yes, does it need to be solved in core?

I think a better solution for managing production settings is probably overdue. I know we've spoken about things like autoloading DJANGO_ environment variables, allowing .ini files, and other various strategies. I kind of like the idea of just reading a python file from the file system. But I also recognise that deployment is so varied that it's going to be nearly impossible to get some kind of consensus on the "one true way". Aiming for a "best default experience" is probably the bar we should be aiming for though.

[James Pic]

Thanks friends, of course Daniele for the backup, and also James for
the advice of course now that my first emotional reaction is over -
thanks for your tolerance.

Thanks for reading pretty much everything I have on this subject,
looking forward to read feedback from all django-dev contributors of
the world !

Best
James

[Luke Plant]

On 09/04/17 16:16, Josh Smeaton wrote:

So I think there are a few questions to go over.

1. What are the more successful strategies that work in the wild? (files in /etc/, PYTHONPATH, env vars, local_settings.py)

2. Are any of the above clearly superior?

3. Is this a serious problem that people are having?

4. If yes, does it need to be solved in core?

To put a word in on number 2 regarding environment variables, we should take into account articles like:

 https://diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/


The argument for solving in core would be if there are security issues we can genuinely fix by putting something in core rather than allowing custom mechanisms, and if that mechanism will really work in most cases. We also need to consider shared hosting where /etc/ is not an option.

Luke

[Adam Johnson]

I implemented this based upon the code snippet I showed in my first reply. Anyone who wishes to use DJANGO_SETTINGS_FILE can now pip install django-settings-file and follow its setup instructions: https://pypi.python.org/pypi/django-settings-file

I've added some caveats I thought of during development in the project README. I'll copy those here that are relevant to the idea in itself, not just my implementation:

• If the Python file defined by DJANGO_SETTINGS_FILE tries to do any imports, the directory the file is in will not be on PYTHONPATH, so the author of the settings file might get some surprises.
• Additionally, you might experience other problems from loading a file whilst it’s not on PYTHONPATH.
• If you want your settings file to extend another one, it will probably find it easiest to import the base one from a location on PYTHONPATH, otherwise it too will have to do use the same import ‘hacks’ to load the default settings.
• File paths are not portable between operating systems, so you may need logic to support both Unix and Windows at once.
• File paths are not portable between .py and .pyc files. In most cases this means a .pyc file will not be used for settings since it can't be guaranteed to be there, slightly slowing down import time.
  

I think Django core should still stick with DJANGO_SETTINGS_MODULE. Python's import machinery does a lot for us.

[Jamesie Pic]

If this is about making deployment easier, then it might take long
before anybody contributes windows support.

After inheriting from the projects settings "from myproject.settings
import *", I have not experienced the caveats.

Something as simple as overriding a settings file as never been
easier, as it should be.

I recommend that you try Adam's package even if you never use
configuration files per deployment.