The unsettings project

[James Farrington]

Hello everyone,

Here are some thoughts on the unsettings issue. If you haven't heard about unsettings, it is an attempt to move away from using the settings global. There was a discussion at djangocon (which I wasn't there for, but I was told about it) which led to some of my colleagues and I working on it.

Our github repository:
https://github.com/SlashRoot/django

The remaining problems in removing settings seem to fall into one of three categories:

-Settings that are too heavily entwined with Basesettings, Lazysettings, getattr and setattr... etc. DEBUG is an obvious example. I'm not sure if we can solve this with the decorator, or if we're gonna need a different solution or methodology.

-Peculiar pieces of code. I'll include some examples below. This includes functional programming, lambda statements, optimizations, and code that clever. I think we're gonna need to consult people with more experience with this code on a case by case basis.

-Code that is not well tested. There's not too much we can do about this code. We're just going to have to deal with it when it presents itself.

Some examples:

https://github.com/SlashRoot/django/blob/master/django/db/utils.py#L135

https://github.com/SlashRoot/django/blob/master/django/utils/translation/trans_null.py#L25

[Russell Keith-Magee]

On Tue, Feb 11, 2014 at 10:27 AM, James Farrington <jamestfa...@gmail.com> wrote:

Hello everyone,

Here are some thoughts on the unsettings issue. If you haven't heard about unsettings, it is an attempt to move away from using the settings global. There was a discussion at djangocon (which I wasn't there for, but I was told about it) which led to some of my colleagues and I working on it.

Our github repository:
https://github.com/SlashRoot/django

Hi James,

You might get a better response to your message if you give a 1000 ft description of the approach you're taking. You won't get much disagreement that the global settings object is bad, but that doesn't mean that any arbitrary solution is better. 

 

I'm sure I could probably work out your approach if I worked through a diff of your repository against master, but a high level explanation of what I'm likely to find when I get there would be very helpful (and would help me work out if it's worth the time to actually *do* a full analysis of the repository against master).

Yours,

Russ Magee %-)

[Zach Borboa]

Diff https://github.com/SlashRoot/django/compare/django:master...master

[Aymeric Augustin]

On 11 févr. 2014, at 03:27, James Farrington <jamestfa...@gmail.com> wrote:

If you haven't heard about unsettings, it is an attempt to move away from using the settings global. There was a discussion at djangocon (which I wasn't there for, but I was told about it) which led to some of my colleagues and I working on it.

There was only a handful of people involved in that discussion. I overheard it and asked for details but couldn’t get an explanation of the design. I assume that 99,9% of the subscribers to this mailing-list are hearing the codename “unsettings” for the first time and 99,99% including most of the committers have no idea what it means.

I see that the patch moves global state from the “settings” object into a “uses_settings” decorator and change the signature of a bunch of functions. It isn’t clear to me how this is superior to the current code, unless we have evidence that we can get rid of the “uses_settings” decorator, which I haven’t found. “uses_settings" is a branded as a transitory utility but it seems firmly entrenched in the code after the patch. What’s the plan for removing it?

Our github repository:
https://github.com/SlashRoot/django

You’ve already put a lot of work on this, 76 commits and over 1300 lines changed. I appreciate the work you’ve been doing, however, I’d like to remind you kindly that Django is developed by a community that values communication and consensus highly. The path is unlikely to be merged without a healthy discussion on this mailing list of its goals, its design, the alternatives that were considered, and why you’re proposing that solution.

The remaining problems in removing settings seem to fall into one of three categories:

Regardless of the classification, what are these problems? Do they prevent the unsettings effort from succeeding? Do they merely delay it? Do they involve backwards incompatibilities? Do they require a design decision?

Thank you,

-- 

Aymeric.

[Florian Apolloner]

On Tuesday, February 11, 2014 8:33:09 AM UTC+1, Aymeric Augustin wrote:

On 11 févr. 2014, at 03:27, James Farrington <jamestfa...@gmail.com> wrote:

If you haven't heard about unsettings, it is an attempt to move away from using the settings global. There was a discussion at djangocon (which I wasn't there for, but I was told about it) which led to some of my colleagues and I working on it.

There was only a handful of people involved in that discussion. I overheard it and asked for details but couldn’t get an explanation of the design. I assume that 99,9% of the subscribers to this mailing-list are hearing the codename “unsettings” for the first time and 99,99% including most of the committers have no idea what it means.

++, for me the current patch is nice to see, but such a feature needs discussion on the mailing list __first__. So the next (first) step would be: Explain (in __detail__) what you are after and how you want to achieve this goal -- "it has been discussed at DjangoCon" doesn't count (hell, we even switched to Jinja there ;))

Cheers,
Florian

[Schuyler Duveen]

Hi folks,

Let me make a stab at the 1000ft view for this.

TLDR: Django modules should work as libraries (e.g. ORM, mail, etc).  "from django.conf import settings" bootstrap undermines this.  unsetting begins a path to support the legacy structure, but still allows for the 'librarification' of Django modules.  Then, we can move on with more options to whether we want to refactor things or not and how.

For the TLDR code entry point check out:

https://github.com/SlashRoot/django/blob/33c49245032115cf3fd6534d5a55313435419ffb/django/utils/unsetting.py#L37

On to the 'too long' version:

This work started at the Chicago DjangoCon sprint this past September.  Justin Holmes and I were talking to Adrian (and then roped in Jacob).  After discussing the problem and working out a possible way to move forward, it seemed like they both liked the idea.  Even more promising, during the sprint, Justin and I found that it was *working* as we started attacking different sections.

So, what is unsetting?  Basically, we see it as a problem that a large amount of Django code is not structured in 'library' form.  In order for most django libraries to be imported, settings need to be bootstrapped.

Jacob's use-case was "I'd like to import django.core.mail and use it even if I'm not running Django"

My use-case is Django's awesome (yes, I know opinions differ), simple-to-use ORM.

Stipulating that Django should work as a library without bootstrapping with settings, how do we get there?  We have some different options:

1. Refactor all the libraries to be more functional, and somehow pass settings through either the request object or other 'world-state' passing

2. A different refactor project that would move around all the code, so all the 'good' code is functional, and then all the public interfaces get the settings and call the 'good' code functionally.  Outside importers would then import the 'good' modules directly.

3. Unsetting: decorate all the methods/classes that use setting so they map to newly functional arguments.  If setting is there, and the functional arguments aren't sent in, then the setting bootstrap isn't necessary.

Attacking #1 and #2.  First, #1 would be controversial at every step, and for every module.  However world-state was passed around, it would be very difficult to make it backwards compatible.  #2 would mostly result in a documentation issue -- where to import from would be different for non-django importers than those in django.  This is not even to mention the settings references in several __init__.py files.

The general idea with #3 is trying to separate out the setting dependency *first*, and then opportunistically over time, we can start looking for opportunities to pass settings in a more functional way through our call chain.

django.utils.unsetting has the following example of how to migrate older code with some 'magical' decorators:

    The typical use case is a an old function/method looks like:
        def foobar(a, b, c=None):
            if getattr(settings, 'FOO_ENABLE_C', False):
                do_something(c)
            ...
    This decorator makes it easy to remove the need for a django settings import.
    You would modify this function above to something like:
        @uses_settings('FOO_ENABLE_C', 'enable_c')
        def foobar(a, b, c=None, enable_c=False):
            if enable_c:
                do_something(c)
            ...

For the first trial at this approach, we tried out utils.timezone:

https://github.com/SlashRoot/django/blob/33c49245032115cf3fd6534d5a55313435419ffb/django/utils/timezone.py#L302

and it seemed to have worked, so we moved on to django.core.mail, and other modules.

Actually, I, myself, have mostly been sitting on my butt the last few months.  However Justin and James have been working through other modules proving that this pretty light approach can work with no backwards incompatibility.

So, what do y'all think?  And let us know if we can answer any other questions about the approach.

cheers,

sky

[Florian Apolloner]

Hi,

On Friday, February 14, 2014 3:18:08 PM UTC+1, Schuyler Duveen wrote:

This work started at the Chicago DjangoCon sprint this past September.  Justin Holmes and I were talking to Adrian (and then roped in Jacob).  After discussing the problem and working out a possible way to move forward, it seemed like they both liked the idea.

Even if two core devs like it, our group is a little bit bigger than just those two.
 

Jacob's use-case was "I'd like to import django.core.mail and use it even if I'm not running Django"

Ok, that's something we can agree on, I am currently using translations in a non-Django project.
 

The general idea with #3 is trying to separate out the setting dependency *first*, and then opportunistically over time, we can start looking for opportunities to pass settings in a more functional way through our call chain.

To me that basically reads as never ;) I am also not convinced that passing the settings individually to each function is such a good interface; some APIs (translations) maybe could do well with a Translation object where ugettext etc then are methods on it…

So, what do y'all think?  And let us know if we can answer any other questions about the approach.

To be honest, I am currently far from convinced and I think it would be best if you stopped converting all of Django for now and wait for the outcome of a serious discussion here.

Regards,
Florian

[Schuyler Duveen]

On Friday, February 14, 2014 10:53:11 AM UTC-5, Florian Apolloner wrote:

Hi,

On Friday, February 14, 2014 3:18:08 PM UTC+1, Schuyler Duveen wrote:

This work started at the Chicago DjangoCon sprint this past September.  Justin Holmes and I were talking to Adrian (and then roped in Jacob).  After discussing the problem and working out a possible way to move forward, it seemed like they both liked the idea.

Even if two core devs like it, our group is a little bit bigger than just those two.

Of course.  I meant this just in the sense that it gave us encouragement to try it out.

 

 

Jacob's use-case was "I'd like to import django.core.mail and use it even if I'm not running Django"

Ok, that's something we can agree on, I am currently using translations in a non-Django project.
 

The general idea with #3 is trying to separate out the setting dependency *first*, and then opportunistically over time, we can start looking for opportunities to pass settings in a more functional way through our call chain.

To me that basically reads as never ;) I am also not convinced that passing the settings individually to each function is such a good interface; some APIs (translations) maybe could do well with a Translation object where ugettext etc then are methods on it…

Well, even if it turns out to be 'never', The @uses_settings decorator basically means that Django can continue operating as-is, and outside projects can still call in to methods using arguments instead of bootstrapping settings.  I should be able to name-names, but I know I've occasionally run into wanting to use some of these methods passing in a custom variable, even in a django project that's not in settings, and it's not always available as an option.  This puts a pattern behind it.

Also, whereas truly purging all settings from methods might be never, I can imagine a few easy/minor refactorings (even backwards compatible ones) where, for example, I send/put my DB settings into, e.g. database manager or the model, and then I don't have to keep on putting in db='' somewhere else.  This is not an actual proposal -- just an example of where organization might make a particular module easier to use as a library without continuing to send a setting at every call, with no harm (and possibly even some benefit) to Django's use of it.

 

So, what do y'all think?  And let us know if we can answer any other questions about the approach.

To be honest, I am currently far from convinced and I think it would be best if you stopped converting all of Django for now and wait for the outcome of a serious discussion here.

I think the main thing that converting other Django modules achieves is seeing how far our approach can go in all the contexts settings is used.  The bare branch of just the utils.unsetting library is available here:

https://github.com/schuyler1d/django/tree/unsetting_base

and if it was desirable, I imagine it wouldn't be pulled in as all one patch, but evaluated/discussed per-module.  But as you said, let's discuss the approach here first.

cheers,

sky

Regards,
Florian

[Aymeric Augustin]

Hi Schuyler,

On 14 févr. 2014, at 15:18, Schuyler Duveen <schuy...@gmail.com> wrote:

TLDR: Django modules should work as libraries (e.g. ORM, mail, etc).  "from django.conf import settings" bootstrap undermines this.

My use-case is Django's awesome (yes, I know opinions differ), simple-to-use ORM.

For the ORM, settings aren't the primary concern. The biggest problem is setting up relations between models. This needs to be done at some point before you start using models.

Before Django 1.7, the app cache took care of that at some ill-specified and project-dependant point. During the app-loading refactor, we recognized that such an important step couldn’t be left to chance and we introduced an explicit bootstrap, `django.setup()`.

Given the current implementation of `django.setup()`, it seems possible to inject the settings there in another form that a Python module, if that’s what you’re after. See https://github.com/django/django/blob/master/django/__init__.py#L11-L21.

It seems to me that you haven’t attacked the right problem for what you’re describing as your primary use case. (That’s why I’ve been asking for a mailing-list discussion since September.)

Besides, you need a plan to replace `django.setup()`.

For the first trial at this approach, we tried out utils.timezone:

https://github.com/SlashRoot/django/blob/33c49245032115cf3fd6534d5a55313435419ffb/django/utils/timezone.py#L302

and it seemed to have worked, so we moved on to django.core.mail, and other modules.

All the discourse around unsettings is based on the assumption that it’s an incremental improvement that may provide some other benefits in the future.

However, the current results aren’t looking so good to me:

- The new APIs are weird: functions end up with additional keywords arguments purely based on their implementation. This isn’t a good way to design and API. (I assume that the goal of unsettings is to make these APIs public.)

- It makes the code more complex: every contributor to Django will have to learn a new way to inject settings into functions. In order to keep the barrier to contributing to Django low, I’m not fond of such idiosyncrasies.

Also, benefits still look quite hypothetical, if not theoretical. I’m worried about beginning a path without a convincing explanation of why it isn’t a dead-end. In the past, we’ve hit dead-ends on projects much better planned that this one, eg. mitsuhiko’s “template compilation” GSoC.

That makes me -1 on the concept for now. I don’t believe it beats the status quo. To change my vote, I would need:

- a description of how you plan to deal with django.setup() — it seems more complicated than dealing with settings;

- an explanation of what comes after we replace every “settings.FOO” with “@unsettings(FOO=…)”;

- some thoughts of why we’re comfortable maintaining the resulting public APIs in the long term.

I have much more to say but I’ve tried to summarize my thoughts in this email. I hope this helps.

-- 

Aymeric.

[Waldemar Kornewald]

Hi,

I'd like to describe how we've solved the per-package settings issue at our company.

Here's a little source code example:

https://gist.github.com/wkornewald/9109270

Every Python package defines a module with a Config instance and then sets default settings on it.

Instead of defining settings as global variables in a module you import all packages' config instances and set custom settings.

Settings which require imports can be configured by using set_lazy() and passing lambdas/functions which do those imports on-demand. The lazy function's return value is cached.

There's also a sandbox mechanism where you can do:

with config:

    config.set(debug=True)

    # all code will now think we're in debug mode

# and here we're back to the previous setting

Sandboxing can be useful for unit tests.

Anyway, please take a look at the code example in the link above.

If you need more fine-grained control, you can of course also attach config instances to individual objects instead of packages.

So, what do you think?

Greetings,

Waldemar Kornewald

[Jonathan Slenders]

Not sure whether this has already been covered. Maybe it is.

I'd like to specify the settings as a class instead of a module. Mostly because I don't like the "from settings_local import *" pattern to extend some base settings with machine specific settings. By having the possibility to define it in a class, I could use inheritance.

from project.base_settings import BaseSettings

class MySettings(BaseSettings):

    INSTALLED_APPS = BaseSettings.INSTALLED_APPS + [...]

    ...

It's also a solution for "lazy" settings, like said above. Create a property in the settings class, etc..

This should easily be backwards compatible.