declarative settings

[Christian González]

Hello,

I recently "finished" my first really working version of GDAPS, my
Generic Django Application Plugin System. It's noway perfect, but does
what it should: providing pluggable apps for an app framework, including
a more or less flexible frontend with each django app.

I had much struggle with it, and one of the lessons I learned was
Django's setup system, and how it deals with loading apps. Unfortunately
Django can't load/unload apps on the fly, so it is necessary to restart
Django whenever a new GDAPS app is installed via pip.

But: I want to resurrect an old theme again which would, in a way,
improve some of the loading problems I encountered. Django's settings
are code. Which is, in fact, a very good thing, as it makes it extremely
flexible and adaptable to different setups. But, as discussed with the
SECRET_KEY here, some of the settings _have_ to be coded very
complicated, and it makes some things like per-app-settings extremely
uncomfortable.

What if - and please don't kill me instantly - yes, I am a newcomer, and
not a good programmer maybe - but some things are viewed better from
"outside" - what if Django settings could be "declarative"?

So instead of Python code like

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes'
]

This would be in an e.g. JSON file

{

    "INSTALLED_APPS": [
        "django.contrib.admin",
        "django.contrib.auth",
        "django.contrib.contenttypes"
    ] ,
    ROOT_URLCONF: "fooproject.urls"
}

Django's settings.py would look different: It would load that
settings.json file and set the appropriate values into local code - so
this wouldn't make much difference.

Except 2 things:

1. Apps could have (default) settings, and they could be merged MUCH
easier. Things like namespaced classes that are overwriting values like
DRF/graphene does, would be completely unnecessary. The main
settings.json file could be the "last word" in the process of settings,
so anything an app would suggest could be overrided in the main file.

2. Installed apps could be managed much more comfortable. Adding an app
could be done by a script (JSON editing is easy. Editing code
(=settings.py) is error prone and uncomfortable). I have a Django
command script ATM for that, but just because I add a line into
settings.py to add some additional apps to the list.

This even could be done with backwards compatibility, because Django
would keep it's settings.py file optionally:

* read json settings (if they exist), use them
* load settings.py which allows to override them again (using some
special code tricks like dynamic loading, environments etc.)

Please tell me what you think about that.

Christian


--
Dr. Christian González
https://nerdocs.at

[Matthew Pava]

It's an interesting idea, and I'm not opposed to it myself; however, keeping the settings as Python code is not an abnormal practice compared to other software.

I've been working with some Drupal stuff lately, and it is written in PHP. Drupal is a content management system that can be extended with various modules written in PHP. It's settings file is also just a code file of PHP. Drupal does take advantage of a package manger called composer, which would be similar to our pipenv. Both of those managers use JSON files for their appropriate settings. But package managers are not frameworks.

Perhaps we could set up a hybrid in which we have a declarative settings file that can be utilized by a coded settings file, but I feel that might make the whole system a bit too complex to maintain.

Just my thoughts.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3047a7b6-3fa0-d574-4bb6-7842b7aed44a%40nerdocs.at.

[Jacob Rief]

You have hit a salient point in Django. It indeed is a mess how every third party

application must add its own configuration classes in order to make their own

default settings configurable through a settings.py. This results in settings directives

which can have any attribute name and do not follow any namespacing.

Some external apps, such as easy-thumbnails add their own configuration framework

with many different setting attributes, while others such as DRF use one Python dictionary

to keep all configurations in one closed namespace. The latter in my opinion is the

better approach but has its own issues. Having a consistent naming convention of setting

attributes, and a reconfiguration framework would certainly be beneficial in Django.

In my opinion, just switching to JSON does not resolve these naming convention issues,

but adds a bunch of other problems:

Some configuration settings must be lazy.

For instance, ugettext_lazy is used very often in the setting.py. This can't be handled by JSON.

Writing JSON by hand is harder than writing Python.

An alternative would be the usage of yaml or toml.

Some settings must be taken from the environment.

Especially database passwords and the SECRET_KEY shall be injected through the environment.

Typically here one would use os.getenv('DJANGO_SECRET_KEY') or similar. This can't be

handled by JSON either.

On Monday, December 30, 2019 at 11:46:03 PM UTC+1, Christian González wrote:

* read json settings (if they exist), use them
* load settings.py which allows to override them again (using some
special code tricks like dynamic loading, environments etc.)

If implemented, wouldn't it make more sense to use a JSON file to override a settings.py

rather than doing it vice versa? 

– just my two cents,

Jacob

[Tom Forbes]

My two cents: JSON isn’t great as a configuration language - It’s annoyingly picky about some things. YAML or TOML are “better” (for some definition of better) choices for this domain, in my option. However, Django is historically quite hesitant about including third party packages and I think including a TOML or YAML parser even as an optional dependency might not be the way forward.

So rather than just “allowing people to use JSON files for settings” it would be very interesting to explore what a pluggable settings backend would look like. It seems that work in this area would be best spent on a general abstraction rather than a specific one. There was some discussion around this idea in the GSOC thread and I’m sure it’s come up before. Personally I think configuration management is an area of Django that is quite underdeveloped compared to other frameworks.

On 30 Dec 2019, at 22:46, Christian González <christian...@nerdocs.at> wrote:

Hello,

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3047a7b6-3fa0-d574-4bb6-7842b7aed44a%40nerdocs.at.

<pEpkey.asc>

[Adam Johnson]

When using settings for my third party packages I use:

* A class to read django.conf.settings and add defaults/other logic through properties - for example https://github.com/adamchainz/django-cors-headers/blob/31b9c2ef8a333a40f18081ffc1f1cba9fb34574d/src/corsheaders/conf.py . This has the benefit of easily providing defaults, complex logic such as deriving one value from multiple settings, and being read from settings at runtime rather than import time.

* A bunch of system checks to validate them - for example https://github.com/adamchainz/django-cors-headers/blob/31b9c2ef8a333a40f18081ffc1f1cba9fb34574d/src/corsheaders/checks.py . For example, if a setting is *required*, you can add a check error; or if a setting defaults to something unsuitable for production, you can add a deploy check. System checks make it easy for users to upgrade your package too, since many users don't read changelogs even for breaking version changes.

I am in favour of most libraries adding a single setting that controls them, like DRF has REST_FRAMEWORK. I know django-cors-headers doesn't implement the pattern but this is for legacy reasons since I took it over.

I think this pattern solves your first concern Christian. As for your second concern, I think checks make manual installation of apps easier. If a user adds your app to INSTALLED_APPS, the system checks will run and can highlight any missing/misconfigured settings from following the installation docs. Most libraries take 3 minutes to do the initial install so I'm not sure of the value of automating that step compared to adding system checks.

Jacob you're right it's a bit of an inconvenience to write a class to wrap django.conf.settings but I'm not sure there's any general purpose wrapper that could be provided. There are too many cases for evolution - a setting changes name, one value combines multiple settings, etc.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/D7000C61-69C0-4E59-9DB8-4BB1413B0C98%40tomforb.es.

--

Adam

[Christian González]

Hi all,

thanks for your fast response - it seems that this is not a "ah, forget it, boooring." theme.
I'll try to answer all (for me) relevant things at once here - sorry to write such a huge mail.

@Adam Johnson:

* A class to read django.conf.settings and add defaults/other logic through properties - for example https://github.com/adamchainz/django-cors-headers/blob/31b9c2ef8a333a40f18081ffc1f1cba9fb34574d/src/corsheaders/conf.py . This has the benefit of easily providing defaults, complex logic such as deriving one value from multiple settings, and being read from settings at runtime rather than import time.

This is a good approach too, but it's not namespaced automatically. And app settings can't provide and override settings from other apps. This seems a weird suggest, but when apps depend on another app it should be able to override settings for that dependency too (because it "uses" that app anyway).

I am in favour of most libraries adding a single setting that controls them, like DRF has REST_FRAMEWORK. I know django-cors-headers doesn't implement the pattern but this is for legacy reasons since I took it over.

Yes, this basically means namespacing. Depending on the size of the library, this could be one big bunch of a setting.

I think this pattern solves your first concern Christian.

No it doesn't completely. I used this pattern myself in GDAPS, copied from graphene (which copied from DRF). But this does _not_ solve the problem: This pattern  creates a per-app settings object which checks at the time of using (which is anywhere after main settings.py) if there is a global override of the local app's variable - and takes the right one. In this way it works. but this is not very convenient - you always have to ask the right settings object - it maybe would be more convenient, simpler and easier to code to just ask for a namespaced setting, e.g. in a dotted path style.

like settings("django.ROOT_URLCONF") or settings("foo_app.SETTINGA")

This is not a really good approach as it implements strings.

As for your second concern, I think checks make manual installation of apps easier. If a user adds your app to INSTALLED_APPS, the system checks will run and can highlight any missing/misconfigured settings from following the installation docs. Most libraries take 3 minutes to do the initial install so I'm not sure of the value of automating that step compared to adding system checks.

Perfect.

@Matthew Pava

I have worked with Drupal for a long time before Django, and am maintaining a few sites with Drupal, one of it rather big, with many domains & shared content, 40+ modules, API to ther web services, some modules custom coded. I learned a lot of it, but PHP is really ugly. Django does much better, but, compared with Drupal, it has a few drawbacks. I know, Drupal is a CMS, Django is a framework (which you can build a CMS with) - BUT: Drupal is able (like Wordpress) to add modules/plugins per web download and on-the-fly - it installs modules into place, has a very good hooks system and a plugin manager. It also was a help when I designed GDAPS and was one of my blueprints. Drupal however keeps ALL of it's settings in the database which creates many problems on it's own (development/deployment problems how to sync settings from disk to database and vice versa). I wouldn't do that. Settings in code/conf files is a good thing IMHO.

The package manager in Drupal is composer, yes, and together with GDAPS, you could use GDAPS Django packages exactly the sameway. Just install it using pip/pipenv, and Dajngo/GDAPS finds them automatically.

But this can only be achieved by adding a line to settings.py (call the plugin manager), like I did:
https://gdaps.readthedocs.io/en/latest/Installation.html#installation

These are just hacks because Django doesn't have declarative settings.
I suggest that if changing to declarative, it MUST be backwards compatible, yes. So there could be an _internal_ settings.py which loads environment variables, the bespoken config file, and like I said, **if** there is a settings.py in the project, Django could use it - to override settings in a way just code could do it.

@Jacob Rief and @Tom Forbes

Sure, JSON was just one idea. It's "xml 2.0", and not suited to be written by humans easily. yaml or toml would be better, in fact, in a first approach of GDAPS I used toml as meta config data for my Django plugins. I decided (because there IS no common framework in Django, and I didn't want to add another dependency) to let it go and am using an attribute of AppConfig ATM - code again.

> Some settings must be taken from the environment.

Sure. DJANGO_SECRET_KEY etc. But this could be handled by the "internal" settings.py file.

> So rather than just “allowing people to use JSON files for settings” it would be very interesting to explore what a pluggable settings backend would look like.

Now that's the main point IMHO.

I know Drupal a bit here, and They struggled until v7. You can read about "Configuration management" here:
https://www.drupal.org/docs/8/configuration-management/managing-your-sites-configuration

By default, the "active" configuration is stored in the database ("config" table). This is for performance and security reasons. This is the complete configuration for the entire site at that moment. Configuration can be exported and imported as YAML files, either in its entirety, or a single piece of configuration, using Drush and/or Drupal Console config commands or the Configuration Manager. (See below for more details.)

Performance is a good reason here. getting settings from database is really fast. But I didn't like it. If I would design a settings backend, I would even make it pluggable... so that the user doesn't have to know there the settings come from.

from django.conf import settings
if settings.FOO.FOO_SETTING:
    pass

the settings object could transparently check if any apps implement a "FOO" named object, and ask that object for "FOO_SETTING". This way the user always could ask the main settings object, but settings would be keeped namespaced.

This is just like **i** would do it - again - just my 2 cents.

BTW - and this is combined with it - Django IMHO needs a hooks system, like Drupal has. each point in code which needs "hooks" has it's own pattern of checking apps (search for modules, attributes in modules etc). Signals are not suitable enough and add complexity to the code structure. The GDAPS approach is an idea on how this "could" look like - even to use per-app settings:

# Dream- and pseudo code (GDAPS integrated in Django ;-) ):

# get interface for settings from Django (now GDAPS)
from drupal.interfaces import ISettings

for settings_object in ISettings:
    global_settings.merge(settings_object)

So all plugins implementing ISettings are automatically found and can be merged in a way which needs to be done yet ;-)

Yes, maybe I'm dreaming a bit. Dynamic app loading won't be applied any time soon in Django I suppose, btu settings could be done really better.

If a can help in any way here, maybe even with code, please tell me.

Have a Happy New Year everyone!

Greetings from Salzburg,
Christian

[Adam Johnson]

This seems a weird suggest, but when apps depend on another app it should be able to override settings for that dependency too (because it "uses" that app anyway).

It does seem weird to suggest. I haven't seen such a use case myself. Can you describe in more detail?

For me, I can only imagine it would violate the "principle of least astonishment" that if I had app A installed for some time, then installed app B on top, app B would change the default or manually configured settings of app A. I would prefer if app B complained (via system check) that I had an incompatible app A configuration.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/06f4e659-70f2-cda1-a59e-35d7d79f3984%40nerdocs.at.

--

Adam

[Alexei Znamensky]

Hi Christian,

Definitely not a boring theme, but as pointed by others, there are many aspects to be taken into consideration.

Having said that, I recently stumbled on this package django-configurations by chance, while working on something else. It looks quite promising, as one can use (multiple) class inheritance to merge and combine different settings. As far as I can tell, it lacks the dynamic aspect of your GDAPS idea, so I suppose it would not solve the problem of loading plugins in runtime. But, for the part in which you want to load settings from json/yaml/toml/database/wherever, it might be simpler to piggyback on that project and have Configuration classes that load markup, and a config class that loads from DB (although this would probably be trickier than it looks).

Happy New Year to you as well, greetings from NZ

Alex

[Christian González]

Am 02.01.20 um 08:06 schrieb Alexei Znamensky:

Having said that, I recently stumbled on this package django-configurations by chance, while working on something else. It looks quite promising, as one can use (multiple) class inheritance to merge and combine different settings. As far as I can tell, it lacks the dynamic aspect of your GDAPS idea, so I suppose it would not solve the problem of loading plugins in runtime. But, for the part in which you want to load settings from json/yaml/toml/database/wherever, it might be simpler to piggyback on that project and have Configuration classes that load markup, and a config class that loads from DB (although this would probably be trickier than it looks).

yes, I already had stumbled upon django-configurations too. But apart from not solving the dynamic loading "problem" of Django (which is not my main concern: reloading the django server is ok for me ATM), it drags in a bunch of own config necessities and changes that have to be done to make it work: The "quick start" replaces the whole settings.py with a class, and you have to change the manage.py file too. No plug'n'play. Sure, you can make this class load a config file then too.

But for me it reminds me of the joke:

    Waiter: What would you like to eat?
    Guest: I'd like to have a burger with fries, but instead of the fries please bring me rice,
        and instead of the burger I'd like a soup.

If I replace everything the config does, it can do something else. But it still does not solve my problem at all... which is easily *changing* the installed apps programmatically - so that, after a restart - new apps are available. In GDAPS, I only add one line, if you install GDAPS apps per pip, they are found automatically and work after a restart. But I think this is Django's "fault" - it could do better.

What would be really a game changer: in django.conf there are LazySettings and Settings classes, and they currently load from a module (https://github.com/django/django/blob/c574bec0929cd2527268c96a492d25223a9fd576/django/conf/__init__.py#L171), after checking the ENVIRONMENT_VARIABLE.

A quick proposal would be to

    1. load the environment variable to get the settings name
    2. NEW: look for a declarative settings file and load that
    3. if that doesn't exist (or additionally), load the settings from project's settings.py as usual

This way everything would be completely unchanged and backwards compatible, while adding declarative settings functionality.

This doesn't solve per-app settings - but while loading the declarative config, installed apps would be defined already - and their paths should be possible to determine by then. So *their* declarative settings could be found and merged as well.

[Christian González]

Am 01.01.20 um 15:04 schrieb Adam Johnson:

This seems a weird suggest, but when apps depend on another app it should be able to override settings for that dependency too (because it "uses" that app anyway).

It does seem weird to suggest. I haven't seen such a use case myself. Can you describe in more detail?

For me, I can only imagine it would violate the "principle of least astonishment" that if I had app A installed for some time, then installed app B on top, app B would change the default or manually configured settings of app A. I would prefer if app B complained (via system check) that I had an incompatible app A configuration.

Maybe you're right. I was about to answer how the system should behave when not thinking in "apps" - which are in a flat hierarchy. In GDAPS I try to think in modules/plugins of ONE application - that's what I try to achieve. Let's explain shortly: My main aim is to create an Open Source Electronic Medical Record - meaning I want to create a base system where maybe 3rd party companies could add their own plugins to the system. Everything based upon django as backend (with a REST/Graphene web API), and variable frontends, let's say a web frontend with Vue.js.

But even when a base module has some "default settings", and I install a module B which overrides that defaults because system behaviour is different when using this module, it doesn't make sense. Such a case should be done in code (module B circumvents the default settings etc.) - but settings merging / hierarchy maybe isn't the best way to go. I also strictly believe in "namespaces are a good thing."

So forget overriding settings, there only remains the easy settings access via one object, which would be great.

[Adrian Turjak]

Declarative settings, and a lack of a good settings file parsing system,
led me into some rather interesting directions not that long ago.

I maintain an OpenStack project called Adjutant, and I built it on
Django, but ended up using yaml as my config file and having settings.py
read it and pull in certain values. But I ended up doing some rather
weird nested stuff, and kept introducing small sections of code that
needed some config value. But because I didn't have a way to declare
those configs somewhere, it essentially turned into a nightmare to know
exactly what values were configurable, and where in the code they were used.

OpenStack uses a config library system called oslo.config which allows
declaring config, and then consuming it. Sadly for me that didn't handle
nesting. dynamic config registration... or too much weird complexity.
Plus it was based off INI files, and didn't support yaml or toml. So I
ended up writing my own variant which did.

The library I wrote is:
https://gitlab.com/catalyst-cloud/confspirator

And the way I use it:
https://github.com/openstack/adjutant/tree/master/adjutant/config
https://github.com/openstack/adjutant/blob/master/adjutant/settings.py

Which this being how I handle my plugins, and their config:
https://github.com/openstack/adjutant/blob/master/adjutant/feature_set.py

The whole thing is maybe a little over the top, but it allows dynamic
config registration via plugins, declarative values, so you can't ever
use a config without defining it, and it can read those config values
from envvars or you can pass it a dict structure to parse (which you can
load in from json, yaml, or toml).

And a way to then use that config tree to generate example config:
https://github.com/openstack/adjutant/blob/master/adjutant/commands/management/commands/exampleconfig.py

This may not be what you need, but given my random pain in what I think
is a similar area, I thought I may as well share. I do intent to
maintain CONFspirator, and will be adding native support for
parsing/loading yaml/toml, as well as utils for example config
generation soon.

[Robert Marsanyi]

I have to say, having settings as Python code is one of the features of Django for me.  Being able to calculate settings values, take advantage of Python’s module/class parsing, and so on all make settings more than just a set of dictionary values.  I think we would lose a bit of functionality if we chose to reduce them to something that YAML or its ilk knows how to express.

[Alexei Znamensky]

Howdy,

You guys ever seen or used this:

https://pypi.org/project/django-configurations/

Reusable settings. That would be something I would like to see in Django. 

On Fri, 28 Feb 2020 at 5:49 AM, Robert Marsanyi <r...@whidbey.com> wrote:

I have to say, having settings as Python code is one of the features of Django for me.  Being able to calculate settings values, take advantage of Python’s module/class parsing, and so on all make settings more than just a set of dictionary values.  I think we would lose a bit of functionality if we chose to reduce them to something that YAML or its ilk knows how to express.

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/eDdUYM3YN1U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3de22278-7bad-4409-a2d1-185792a4375c%40googlegroups.com.

--

Alexei Znamensky | russoz gmail com
http://github.com/russoz