Setting dictionaries (was Re: integrating django-secure)

[Collin Anderson]

I'm trying to think outside the box on settings.

If we want to logically group settings together, what if we supported syntax like this?

MIDDLEWARE_CLASSES = (

    'django.middleware.clickjacking.XFrameOptionsMiddleware',

    SecurityMiddleware(content_type_nosniff=True, xss_filter=True),

)

It could be a more pythonic way for middleware to have their own settings.

[Carl Meyer]

Proposals like this generally run afoul of circular import problems. As
soon as you have people importing parts of Django or their project in
their settings file, those modules likely import other modules which in
turn import django.conf.settings, and boom!

Carl

[Collin Anderson]

yeah... I thought of circular imports... you would need to lazy-load most imports. not fun.

[Michael Manfre]

This is likely to run afoul of circular imports. It would need to do something like the following to provide init kwargs.

MIDDLEWARE_CLASSES = (

    'django.middleware.clickjacking.XFrameOptionsMiddleware',

    ('django.middleware.security.SecurityMiddleware', {'content_type_nosniff': True, 'xss_filter': True}),

)

Regards,

Michael Manfre

[Russell Keith-Magee]

Since I'm sitting at the DjangoCon US sprints, and I'm feeling a little optimistic and crazy, I thought I'd throw this idea out there...

The other way to resolve this would be to rethink the way Django starts up. 

Settings like MIDDLEWARE_CLASSES have always struck me as being a little odd - they're not really settings, they're a way to configure the way code is executed without actually writing and executing code. The same is true of things like ROOT_URLCONF, a few of the template loaders and path specifications, and so on.

If we were starting green-fields, it seems to me that we wouldn't do it this way; the Pyramid/Flask approach of doing this sort of thing by actually defining code would make a lot more sense. Django currently has a well defined, but opaque startup sequence where settings are loaded and then the middlware, urlconfs etc are set up. The alternative would be an explicit startup sequence that constructs the desired middlewares, installs them into a root urlconf, configures loaders and so on, and then sets the environment running.

This should also avoids the circular dependency problem, because anything that needs to do an import in the settings file would be part of the configuration process.

To be clear, I know this would be a huge change. If we went down this path, we'd need to maintain the old/current way of deploying Django for a long time. In the interests of getting started quickly, it might even be desirable to maintain both approaches long term. But having an explicit startup sequence would allow for complex middleware configuration like this thread has proposed. I'm just throwing this idea out on the porch to see if the cat licks it up.

Yours,

Russ Magee %-)

[Tim Graham]

In the meantime, shall we abandon the idea of organizing settings in dictionaries for "purity"? May we add new settings as described here: https://code.djangoproject.com/ticket/22734#comment:22

[Carl Meyer]

On 09/05/2014 10:46 AM, Tim Graham wrote:
> In the meantime, shall we abandon the idea of organizing settings in
> dictionaries for "purity"? May we add new settings as described here:
> https://code.djangoproject.com/ticket/22734#comment:22

IMO, yes.

When we already have a subsystem, like email, which is configured via
global settings, and it really needs a new configuration knob (we should
continue to be generally conservative in making that assessment), I see
very little downside to a new setting. If there's to be a new knob, best
to put it in the same place as the existing knobs for that component.

Carl

[Russell Keith-Magee]

+1. If we actually need a new email setting for timeouts, I have no objection to adding EMAIL_TIMEOUT.

Russ %-) 

[Ryan Hiebert]

​I'd like to see the email settings get moved into a dictionary so that I can use a single environment variable to configure it very easily, ala dj-database-url. There's dj-email-url right now, but it requires 7 lines to configure unless I want to restore to using vars() to update them all at once. There should be a way to set this one logical setting (based on the environment variable)​ without being implicit (using var().update) or spending 7 lines on something this stupidly simple.

[Carl Meyer]

Hi Russ,

I think this is generally down a similar line as the proposals that have
been floated in the past (e.g. in a talk that Alex Gaynor gave at
DjangoCon(?) a few years ago) to move Django away from its reliance on
process-global configuration.

Currently in Django a "project" is (implicitly) just "a settings
module". The way I could see your proposal happening would be to
introduce a Project (or Config?) class (in any other framework it would
be App, but in Django that name is taken) which exposes APIs for
imperative configuration of things like URLconf, middleware, etc, and
then is itself (or can provide) a WSGI application callable.

Then any script which wants to "start" Django would have two options:
call django.setup(), which implements the current "look for a settings
module and configure an implicit Project based on those settings"
startup, or instantiate their own Project instead.

The knottiest problem with moving away from process-global config is
that it allows for our current "simple" APIs (e.g. import a model class
and query on it, implicitly relying on global DATABASES config; import
"render_to_string" and render a template, implicitly relying on global
TEMPLATE_* config; import "send_mail" and send an email, implicitly
relying on global EMAIL_* config).

It would be possible to bite off the "imperative configuration" piece
without the "kill global config" piece, though - we'd just need to
provide an API (an alternative to django.setup(), or perhaps just an
optional argument to it) that lets you install your own Project/Config
instance as "the implicit global Project/Config", and then go ahead and
use all the relying-on-global-config APIs just as you do now.

Interesting to think about, but also a big chunk of very hypothetical
work :-)

Carl

[Carl Meyer]

Hi Ryan,

On 09/05/2014 12:02 PM, Ryan Hiebert wrote:
> ​I'd like to see the email settings get moved into a dictionary so that
> I can use a single environment variable to configure it very easily, ala
> dj-database-url. There's dj-email-url right now, but it requires 7 lines
> to configure unless I want to restore to using vars() to update them all
> at once. There should be a way to set this one logical setting (based on
> the environment variable)​ without being implicit (using var().update)
> or spending 7 lines on something this stupidly simple.

This is the best reason I've seen mentioned for moving the email
settings to a dictionary. It's true that email settings really are
various aspects of configuring one "thing", like DATABASES or CACHES.

I'm still not sure it's worth the backwards-compatibility break, though,
especially since it is possible for a package like dj-email-url to do
what you want with no boilerplate in your settings file, with some code
in its AppConfig.ready() that looks for a custom dictionary-formatted
setting and sets the actual individual settings based on it.

Carl

[Marc Tamlyn]

I like the idea Russ - from a brief peruse of the code base it's apparently to me how disparate the handling of various "setup" settings is throughout the codebase. For example:

INSTALLED_APPS is handled by django.setup() and Apps.populate() (cached attribute)

MIDDLEWARE_CLASSES is handled by BaseHandler.setup_middleware() (done once and cached)

ROOT_URLCONF is handled by BaseHandler.get_response() (done every request)

DATABASES is handled by ConnectionHandler (cached property)

DATABASE_ROUTERS is handled by ConnectionRouter (cached attribute) 

TEMPLATE_LOADERS is handled by find_template() (saved in a global)

CACHES is handled by CacheHandler (saved in a threading.local())

etc.

Perhaps there is an interesting possibility here to move some of these disparate "inspect settings, import or configure some stuff and store it for the duration of the process" onto a "global" object - something like ProjectConfig. It could have methods (cached properties?) like .get_middleware_classes() which would return the imported, configured middleware classes. We then (dun dun duuuun) introduce a setting PROJECT_CONFIG = 'django.project.ProjectConfig' which you can then set yourself if you want to do a more procedural approach.

In a sense, this does nothing. It doesn't remove global state (still lives in settings/project config), and it doesn't really change anything for most users. What it does do is start the "centralize" the configuration and use of settings, giving us 1) a way of changing how a setting is used, not just defined and 2) the option of not setting that setting at all and just doing it directly. This may (or may not!) interact quite nicely with override_settings() which simply does not work with some settings which are cached forever.

What is also to me interesting about it is that we may be able to do quite a lot of the work without breaking any APIs at all - BaseHandler.setup_middleware() can stay, it just calls something else to do the work.

</throwing ideas>

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/540A1B4B.8080009%40oddbird.net.

For more options, visit https://groups.google.com/d/optout.

[Carl Meyer]

On 09/05/2014 02:54 PM, Marc Tamlyn wrote:
> I like the idea Russ - from a brief peruse of the code base it's
> apparently to me how disparate the handling of various "setup" settings
> is throughout the codebase. For example:
>
> INSTALLED_APPS is handled by django.setup() and Apps.populate() (cached
> attribute)
> MIDDLEWARE_CLASSES is handled by BaseHandler.setup_middleware() (done
> once and cached)
> ROOT_URLCONF is handled by BaseHandler.get_response() (done every request)
> DATABASES is handled by ConnectionHandler (cached property)
> DATABASE_ROUTERS is handled by ConnectionRouter (cached attribute)
> TEMPLATE_LOADERS is handled by find_template() (saved in a global)
> CACHES is handled by CacheHandler (saved in a threading.local())
>
> etc.
>
> Perhaps there is an interesting possibility here to move some of these
> disparate "inspect settings, import or configure some stuff and store it
> for the duration of the process" onto a "global" object - something like
> ProjectConfig. It could have methods (cached properties?) like
> .get_middleware_classes() which would return the imported, configured
> middleware classes.

I like all this.

> We then (dun dun duuuun) introduce a setting
> PROJECT_CONFIG = 'django.project.ProjectConfig' which you can then set
> yourself if you want to do a more procedural approach.

But I think this may be backwards; I prefer the idea of creating a
ProjectConfig yourself (via any method you like, you may not need to use
a settings module at all; you could totally get rid of
DJANGO_SETTINGS_MODULE if you want) and passing it as an optional
argument to django.setup().

Carl

[Marc Tamlyn]

Yup, that works. The issue is that most users never explicitly call django.setup(), so they need to know how to modify manage.py and wsgi.py to do

conf = MyProjectConfig()

django.setup(conf)

<something here>

In the case of wsgi.py this isn't too obtuse, but django.setup() is here - https://github.com/django/django/blob/master/django/core/management/__init__.py#L310 for management commands. Not the easiest place to customise!

Carl

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/540A2456.80601%40oddbird.net.

[Carl Meyer]

On 09/05/2014 03:11 PM, Marc Tamlyn wrote:
> Yup, that works. The issue is that most users never explicitly call
> django.setup(), so they need to know how to modify manage.py and wsgi.py
> to do
>
> conf = MyProjectConfig()
> django.setup(conf)
> <something here>
>
> In the case of wsgi.py this isn't too obtuse, but django.setup() is here
> - https://github.com/django/django/blob/master/django/core/management/__init__.py#L310
> for management commands. Not the easiest place to customise!

Yes, that approach would definitely require some thinking (which I
haven't fully done yet) about how best to get a custom ProjectConfig
inserted into both the wsgi.py and manage.py entry-points.

Carl

[Jannis Leidel]

Just a quick drive-by comment: ZOMG YES PLEASE.

I also could see a default project class that uses a global settings module that we’d provide for backwards compatibility, but that could eventually be replaced by something that follows other configuration styles.

As to how to normalise the API between wsgi.py and manage.py I think environment variables have worked well for not having to reach too far into the WSGI and management code. So I suggest to introduce a new env var called DJANGO_PROJECT with the dotted path to the Django project object (or to something that quacks like one at least) that defaults to ‘django.default_project’ or similar that does the DJANGO_SETTINGS_MODULE inspection for django.conf.settings.

But if a different config style is needed, we’d override it:

DJANGO_PROJECT = ‘mysite.project.foo’

Default style:

import django

class FooProject(django.SettingsProject):
DEBUG = False

bar = FooProject()

Flask style config:

import django
import foo

bar = foo.ConfigProject()
bar.config.update(
DEBUG=True,
SECRET_KEY=‘...',
)

Or ini/yaml style config:

import django
import foo

bar = foo.YamlProject('/path/to/config.yaml')

Whether we’d want to ship the latter two project style classes as part of Django is a different question, I just think that offering a common API for project configuration (via a “Project” base classes) may be a sane way to slowly move away from the Django settings file via duck typing.

Alternatively we could have the project itself be the wsgi app but I’m reluctant to go down that road given the past refactoring around wsgi.py that basically resulted in not making Django projects WSGI apps themselves. I like the fact that we separate configuration from the WSGI gateway, but maybe projects like Flask and Pyramid have shown that to be a non-issue after all. I’m on the fence.

So far my quick brain dump.. :)

Jannis

[Carl Meyer]

On 09/06/2014 03:10 AM, Jannis Leidel wrote:
>> Yes, that approach would definitely require some thinking (which I
>> haven't fully done yet) about how best to get a custom
>> ProjectConfig inserted into both the wsgi.py and manage.py
>> entry-points.
>
> Just a quick drive-by comment: ZOMG YES PLEASE.
>
> I also could see a default project class that uses a global settings
> module that we’d provide for backwards compatibility, but that could
> eventually be replaced by something that follows other configuration
> styles.
>
> As to how to normalise the API between wsgi.py and manage.py I think
> environment variables have worked well for not having to reach too
> far into the WSGI and management code. So I suggest to introduce a
> new env var called DJANGO_PROJECT with the dotted path to the Django
> project object (or to something that quacks like one at least) that
> defaults to ‘django.default_project’ or similar that does the
> DJANGO_SETTINGS_MODULE inspection for django.conf.settings.

I think we should avoid requiring a new environment variable, if
possible, otherwise we'd just be repeating the unfortunate "you can't
import this module because you haven't set DJANGO_SETTINGS_MODULE" type
of error, and adding a new instance of "look up some class by dotted
string", which is the sort of not-quite-Pythonic stuff that this
proposal is aiming to avoid.

Instead, I'd envision just providing an alternative new "run management
command" API that did not call django.setup() for you, but expected you
to have already called it. Then the stock manage.py could be updated to
use that new API after calling django.setup() explicitly, and then it
would be possible to pass in a custom ProjectConfig as an argument to
django.setup() just by modifying your manage.py.

At this point, wsgi.py and manage.py would not be special in any way;
they would just be typical examples of the general rule that "if you
want to write a script that uses Django, you should call django.setup()
before doing anything else."

All of this wouldn't really fix the circular-import issues we already
have with importing stuff in settings, though. If you want to configure
your ProjectConfig in code you'd need to import some classes (e.g.
middleware), and you'd still need to be careful that that doesn't
involve importing a module that has an import-time dependency on global
config. So it would remain valuable to reduce import-time dependencies
on the global config.

In a sense, what a proposal like this would achieve is to merge the two
current separate questions "are settings configured?" and "is the app
cache ready?" into a single "is Django setup?". This is conceptually
simpler, but I think it remains an open question how much practical
improvement it achieves.

Carl

[Marc Tamlyn]

It does slight more than that - settings configured and settings used are two very different things at present - to use our canonical example MIDDLEWARE_CLASSES is *configured* by checking that it's a tuple of strings and is there. It's used/setup much later when those classes get imported and initialised. It opens up more potential to mess around with this second step - freeing MIDDLEWARE_CLASSES from being a tuple of dotted paths to being, well, a set of classes. Circular imports are an issue, but IMO it wouldn't be *that* awful to have the ProjectConfig file recommended that you just put the imports in the customised methods all the time to be on the safe side.

e.g.

MyProjectConfig(ProjectConfig):

    def get_middleware_classes(self):

        from thirdparty.middleware import ConfigurableMiddleware

        classes = super().get_middleware_classes()

        config = {...}

        classes.append(ConfigurableMiddleware(**config)

        return classes

Obviously you could not call the super at all (which inspects the setting), do all the importing and initialisation of the default middleware yourself and BOOM no settings.

[There are some technical issues to solve here to do with how you actually access the built middleware classes from code, especially regarding potential for changing them at runtime in tests]

Hypthetical: What we may find in this refactoring is that some of the places Django currently dies if the settings are not configured can be relaxed somewhat to not-at-import-time, so we may be able to avoid some of this.