Integrate dj-database-url into Django

[Kenneth Reitz]

dj-database-url gives Django developers two main things:

1. the ability to represent their database settings via a string (a-la sqlalchemy); very useful for environmnent variables and 12factor apps.
2. it will automatically use the DATABASE_URL environment variable, if present.

I'm not sure if #2 is appropriate for Django Core (but it would be nice, as Gunicorn supports both PORT and WEB_CONCURRENCY), but I know #1 is perfect.

This plan has previously been discussed (at a conference, DjangoCon EU in Zurich, long ago) and approved by JKM, before his role at Heroku, if that is helpful information.

I think this change would vastly improve the usability of Django, and would be an excellent and simple move for the project.

Many thanks for your consideration. <3

--
Kenneth Reitz

https://code.djangoproject.com/ticket/28236

[Florian Apolloner]

Hi Kenneth,

I think there are a few things to consider before we can merge/reuse dj-database-url.

 * How are third party backends going to fit into this scheme?
 * What about other options like CACHES which very much suffers from the same problem.

Cheers,
Florian

[Kenneth Reitz]

I don't think the code will be able to be re-used as much as the idea/ux/functionality that it provides.

I can imagine any backend being able to convert from a url string with some methods (e.g. `detect_url` (bool return) and `parse_url` (settings dict return type).)

Perhaps the same scheme could be applied to caches as well.

[Kenneth Reitz]

Ah, I see what you're saying about third-party backends.

I imagine that you'd have to be able to merge a url string with a settings dict (one overrides the other). This is how usage of that functionality is manually provided in dj-database-url.

[Tom Forbes]

My two cents: connection strings/database URI's are a feature I've sorely missed in Django.

Built-in functionality to convert environment variables like DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant DATABASE setting would make some deployment situations a lot simpler. Currently, unless you use dj-database-uri you have to define a bunch of ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary together yourself.

How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?

To help support third part backends: perhaps the scheme portion of the URI could be either a relative import from django.db.backends or an absolute import to a third party library? It seems URI schemes can have dots and underscores in them, so they can be python package paths.

I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but sqlserver_ado://xyz would resolve to the third party django-mssql engine via an absolute import.

Not sure how to handle arbitrary extra settings though.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/057ced98-4471-4939-960b-900ec39f54b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout

[Aymeric Augustin]

Hello,

I'm wondering what the exact definition of the URL format is. Is it specified somewhere? Or is it just:

[engine]://[username]:[password]@[host]:[port]/[name]

where we create arbitrary [engine] values in an ad-hoc fashion?

On 24 May 2017, at 21:21, Tom Forbes <t...@tomforb.es> wrote:

My two cents: connection strings/database URI's are a feature I've sorely missed in Django.

Built-in functionality to convert environment variables like DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant DATABASE setting would make some deployment situations a lot simpler. Currently, unless you use dj-database-uri you have to define a bunch of ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary together yourself.

Fully agreed. While relatively minor, it's an annoyance.

How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?

I don't think it's reasonable to cram them in a URL.

dj-database-url allows passing options as extra keyword arguments. Other values should be explicitly added in the settings module, by updating the dict generated from the URL.

To help support third part backends: perhaps the scheme portion of the URI could be either a relative import from django.db.backends or an absolute import to a third party library? It seems URI schemes can have dots and underscores in them, so they can be python package paths.

I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but sqlserver_ado://xyz would resolve to the third party django-mssql engine via an absolute import.

I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

I'd rather specify the database engine explicitly when calling dj-database-url if it's a third-party engine. There's an open question about what to do with the [engine] part of the URL in that case. Ignoring it entirely is the easiest.

Best regards,

-- 

Aymeric.

[Florian Apolloner]

On Thursday, May 25, 2017 at 9:46:56 AM UTC+2, Aymeric Augustin wrote:

I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

One possibility would be to use entrypoints in setuptools, this way 3rd party backends could specify a name which then has a fixed & verified import path.

[Tom Forbes]

> I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

Good point, it's not wise to enable this even without a clear threat model. Django does import the settings based on an environment variable, but it's relative and if you can use that to do anything nasty then you're most likely already through the airtight hatch (so to speak). However importing potentially global modules could be bad news.

Ignoring it is always an option, but could we not specify that the third party database provider has to be in the `INSTALLED_APPS`? That could provide some form of whitelisting so not any old module is imported. Not sure about any issues that may arise from this though.

> One possibility would be to use entrypoints in setuptools, this way 3rd party backends could specify a name which then has a fixed & verified import path.

This seems like it could get complex, and be quite unlike anything else in Django.

Perhaps just supporting this for first-party database backends is easiest?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

[Tom Forbes]

Edit: DJANGO_SETTINGS_MODULE isn't relative, it will import any arbitrary module you give it. If we accept that then I think we are accepting the risk of imports via an attacker controlling environment variables whilst Django starts up?

On Sat, May 27, 2017 at 8:49 PM, Tom Forbes <t...@tomforb.es> wrote:

> I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

Good point, it's not wise to enable this even without a clear threat model. Django does import the settings based on an environment variable, but it's relative and if you can use that to do anything nasty then you're most likely already through the airtight hatch (so to speak). However importing potentially global modules could be bad news.

Ignoring it is always an option, but could we not specify that the third party database provider has to be in the `INSTALLED_APPS`? That could provide some form of whitelisting so not any old module is imported. Not sure about any issues that may arise from this though.

> One possibility would be to use entrypoints in setuptools, this way 3rd party backends could specify a name which then has a fixed & verified import path.

This seems like it could get complex, and be quite unlike anything else in Django.

Perhaps just supporting this for first-party database backends is easiest?

[Tim Allen]

I've recently been introduced to `django-environ`, a similar library that has additional features to DB connect URLs that we may want to consider: https://github.com/joke2k/django-environ

It has the same issue with third party DB engines; for example, I recently issued a PR to include `pyodbc` as an option. It has some nice newcomer friendly features, such as a cleaner way of determining BASE_URL / SITE_ROOT. Have a look at the URL for a good "before" and "after" settings file example.

Having full support for .env files would make Django projects easier to bring in line with 12-factor.

Regards,

Tim

On Saturday, May 27, 2017 at 3:52:23 PM UTC-4, Tom Forbes wrote:

Edit: DJANGO_SETTINGS_MODULE isn't relative, it will import any arbitrary module you give it. If we accept that then I think we are accepting the risk of imports via an attacker controlling environment variables whilst Django starts up?

On Sat, May 27, 2017 at 8:49 PM, Tom Forbes <t...@tomforb.es> wrote:

> I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

Good point, it's not wise to enable this even without a clear threat model. Django does import the settings based on an environment variable, but it's relative and if you can use that to do anything nasty then you're most likely already through the airtight hatch (so to speak). However importing potentially global modules could be bad news.

Ignoring it is always an option, but could we not specify that the third party database provider has to be in the `INSTALLED_APPS`? That could provide some form of whitelisting so not any old module is imported. Not sure about any issues that may arise from this though.

> One possibility would be to use entrypoints in setuptools, this way 3rd party backends could specify a name which then has a fixed & verified import path.

This seems like it could get complex, and be quite unlike anything else in Django.

Perhaps just supporting this for first-party database backends is easiest?

On Thu, May 25, 2017 at 8:46 AM, Aymeric Augustin <aymeric....@polytechnique.org> wrote:

Hello,

I'm wondering what the exact definition of the URL format is. Is it specified somewhere? Or is it just:

[engine]://[username]:[password]@[host]:[port]/[name]

where we create arbitrary [engine] values in an ad-hoc fashion?

On 24 May 2017, at 21:21, Tom Forbes <t...@tomforb.es> wrote:

My two cents: connection strings/database URI's are a feature I've sorely missed in Django.

Built-in functionality to convert environment variables like DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant DATABASE setting would make some deployment situations a lot simpler. Currently, unless you use dj-database-uri you have to define a bunch of ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary together yourself.

Fully agreed. While relatively minor, it's an annoyance.

How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?

I don't think it's reasonable to cram them in a URL.

dj-database-url allows passing options as extra keyword arguments. Other values should be explicitly added in the settings module, by updating the dict generated from the URL.

To help support third part backends: perhaps the scheme portion of the URI could be either a relative import from django.db.backends or an absolute import to a third party library? It seems URI schemes can have dots and underscores in them, so they can be python package paths.

I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but sqlserver_ado://xyz would resolve to the third party django-mssql engine via an absolute import.

I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.

I'd rather specify the database engine explicitly when calling dj-database-url if it's a third-party engine. There's an open question about what to do with the [engine] part of the URL in that case. Ignoring it entirely is the easiest.

Best regards,

-- 

Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

[li...@lew21.net]

There's also https://github.com/doismellburning/django12factor - and it simplifies all the boilerplate, that is still required in django-environ, into:

import django12factor
globals().update(django12factor.factorise())

It would be quite nice if Django supported 12factor configuration natively, or at least with a single function call like this one.

Regards,
Linus

[Curtis Maloney]

My django-classy-settings also supports env-based settings, in a
declarative manner with defaults.

What I've yet to crack (though the discussion is underway) is good
formats for structured settings like caches and databases.

Generally I use dj-database-url for my own projects :)

I like the idea of having it scan INSTALLED_APPS to allow registering
custom DB backend protocols... though I think the modern approach is
AppConfig.ready ?

--
Curtis

> <https://groups.google.com/group/django-developers>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org

> <https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org?utm_medium=email&utm_source=footer>.

> For more options, visit
> https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com

> <mailto:django-develop...@googlegroups.com>.

> To post to this group, send email to django-d...@googlegroups.com

> <mailto:django-d...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-developers/69f0a647-42cf-4ef6-9ad7-bbe5e8fa779b%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/69f0a647-42cf-4ef6-9ad7-bbe5e8fa779b%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Shai Berger]

If I understand things correctly, registration of new url-schemas to enable
something like dj-database-url to support 3rd parties should be a trivial
matter; just add an entry to some dictionary, with the schema as key, and the
dotted-path of the backend as value.

IMO, scanning INSTALLED_APPS, or relying on ready(), is a lot of over-
engineering for this. Just do the registration from the settings file, before
invoking the url-parsing mechanism:

from django.database_url import register_backend, configure_db

register_backend('pyodbc-azure', 'sql_server.pyodbc')

DATABASES = {'default' : configure_db() }

The only barrier I see is namespacing -- the functions mentioned above cannot
live in django.conf (would cause a circular import if imported from the
settings module) nor django.db (which shouldn't be imported too early), and
that's why I wrote `django.database_url` above.

What am I missing?

[Kenneth Reitz]

My thoughts are the same as Shai's below. 

[Tom Forbes]

I spent some time working on this today and fixed up my branch that implements this. I think I’ve got a nice API that passes all of the dj-database-url tests.

As there is a bit of database-specific shenanigans we have to perform on URLs I added a config_from_url static method to database adapters that takes the URL and returns a valid django configuration dictionary. As it’s a method inside the adapter itself we can override it to apply any db-specific fixups and third party backends can hook into the process as well via the same mechanism.

It expands on dj-database-url by also allowing the configuration of caches using the same method, which I think is pretty neat.

It uses a registration process described below, users just call register_db_backend or register_cache_backend with a path to their third party engine in the settings to link a scheme to a backend dotted path.

I had one question that I’m not sure the answer to: for databases I added the config_from_url to the .base.DatabaseWrapper class inside each built in adapter. I’ve kind of assumed each third party database would have the same structure, how much of a safe assumption is this?

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/065601c3-9488-42aa-8b23-c7f906c26b5a%40googlegroups.com.

[Michael Manfre]

On Sun, Feb 4, 2018 at 12:59 PM Tom Forbes <t...@tomforb.es> wrote:

I had one question that I’m not sure the answer to: for databases I added the config_from_url to the .base.DatabaseWrapper class inside each built in adapter. I’ve kind of assumed each third party database would have the same structure, how much of a safe assumption is this?

Every 3rd party database backend adheres to the internal API and must provide its own or an inherited DatabaseWrapper. Adding config_from_url seems like the best place to me.

Regards,

Michael Manfre

[gw...@fusionbox.com]

I'd like to approach this as 'support database urls in django', rather than 'copy/paste dj-database-url into django'. For postgres (I'm not sure about other backends), a database url can be passed directly to psycopg2. The postgres connection string format actually supports more features than is possible with django's HOST/USER/PORT... style settings. For example, you can pass multiple hosts and psycopg2 will attempt to connect to one of them: https://paquier.xyz/postgresql-2/postgres-10-multi-host-connstr/. Any attempt to parse the url in django will result in a loss of those features.

The only problem I see is that we have to parse the database backend out of the url, and you can't pass a url like 'postgis://....' to psyscopg2. I'd like to be able to do something like:

DATABASES = {

    'default': {

        'DSN': 'postgres://....',

        'ENGINE': 'django.contrib.gis.db.backends.postgis',

    },

}

And let psycopg2 handle the DSN.

[Luke Murphy]

I'd love to see native dj-database-url support in Django. I suppose it
would have to support all the back-ends, no? It might lead to confusion
otherwise.

> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [1]django-develop...@googlegroups.com.

> To post to this group, send email to

> [2]django-d...@googlegroups.com.
> Visit this group at [3]https://groups.google.com/group/django-developers.

> To view this discussion on the web visit

> [4]https://groups.google.com/d/msgid/django-developers/38553bb5-59b0-4772-a17a-282f795f1548%40googlegroups.com.
> For more options, visit [5]https://groups.google.com/d/optout.
>
>References
>
> Visible links
> 1. mailto:django-develop...@googlegroups.com
> 2. mailto:django-d...@googlegroups.com
> 3. https://groups.google.com/group/django-developers
> 4. https://groups.google.com/d/msgid/django-developers/38553bb5-59b0-4772-a17a-282f795f1548%40googlegroups.com?utm_medium=email&utm_source=footer
> 5. https://groups.google.com/d/optout

[Maciej Urbański]

I would agree that DSN support seems like a nicer alternative to just copying dj-database-url, because it not only focuses on 12factor configuration in enviroment variables, but also enables some additional flexibility for the database connection option passing.

As for 12factor, I think https://gist.github.com/telent/9742059 is a good read as to why exposing as enviroment variables maybe not the best motivation. Having to accommodate settings, like database connection information, just so they can be fitted into single string put conveyable by enviroment variable is a case in point. We likely can do the same for Cache addresses as mentioned previously, although we may end up inventing new URI schemes do to that.., but django overall has multitude of other options that are not as easy to stringify.

[Tom Forbes]

So in the PR I proposed I only bits I took verbatim from dj-database-url are the tests. The rest is re-implemented. I think it's a pretty good POC but I haven't touched it in a while.

In any case we have to implement our own parsing for backends that do not support passing in a URL to the connection library. 

Making postgres skip our parsing step and passing it in directly is an implementation detail and there are much more important questions around the API design to answer before this has any chance of being included.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1a55cc1c-ba9c-4950-ab94-50da8eec7d06%40googlegroups.com.

[Raffaele Salmaso]

Hi all,

I'm working on https://github.com/django/django/pull/10786 (which is a port of https://pypi.org/project/django-service-urls/ , which is a 'fork/rewrite' of Tom PR).

I need to (re)read all these emails to find ideas to improve the PR/package.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJMa1xSzBeGUYygG7nxfCVz8jUPNEiSEJhbAqLDTwga9BQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

| Raffaele Salmaso
| https://salmaso.org
| https://bitbucket.org/rsalmaso
| https://github.com/rsalmaso

[Raphael G]

Alright, I tried to revive Tom Forbe's work on this in  https://github.com/django/django/pull/16331

My honest feeling here is that if Django existed just for me, I would really just want to get this pulled in as an option, and trying to increase the scope beyond "given a URL, get a dictionary of settings" is a tarpit that is not worth venturing into in a first merge. Though it's definitely a tarpit worth venturing into as a future step! 

So my strategy here was to try and get this as close to mergeable as possible, with as few points of discussion needed as possible. I tried to document what Tom did, more or less disregarded the idea of providing DSN support in settings, and for memcache avoided blessing an "official" backend. The main thing should be that if people google Django DATABASE_URL there should be a good answer in the docs, IMO.

As someone who's dealt with custom backends in various parts of Django, and with libs, I think Tom's simple registry pattern for custom protocols is more than sufficient for library writers.

Documentation is minimal, but exists. While I basically expect people writing custom DB backends or cache backends to directly look at the source, at the very least we are pointing to the existence of functions for registration in the docs in this PR.

I'm sure we all have these simple projects that really don't need to do much, we all have been pulling in dj_database_url on the side to get this and feeling a bit off about it, so let's try and get that use case handled in a way that doesn't prevent further niceness in the future!

Raphael

[Carlton Gibson]

Hi Raphael, thanks for picking this up. 

Looking at the history here, particularly the discussion here https://groups.google.com/g/django-developers/c/UQjpzB39JN0/m/XGqdV8nbBwAJ but there are others, the reason this hasn't already happened is that if we're going to merge this into core it would be nice (required?) to handle extra cases than (to paraphrase) "just what Heroku expects".

It doesn't look like anyone has sat down to spec out exactly what that means, but there seem to be plenty of thoughts in the discussion.  ... 🤔

JKM's comment on the thread there is representative of the general thought: 

> I suspect this is a "good enough is good enough" situation. Something like what

Raffaele is talking about, or dsnparse, or whatever would probably be ideal. And
for something to be merged into core, I think it'd need to be a more full
solution than just dj-database-url.

The trouble with merging a suboptimal solution is that once it's in, it's very hard to change. We have the whole API stability policy to contend with (and rightly so). 

We bring in a half-baked solution; likely "recommend" it (for some value of that) and then face a long stream of difficult adjustments whilst we try and accommodate the necessary adjustments. 

Given that dj-database-url is stable, a single line install, and (most importantly) not being developed to try and accommodate the issues that stopped it just being merged into core already, I think it should continue to live as a third-party package. 

The place for the development to happen is in dj-database-url or a successor package. (The first step would be a Roadmap drawn from the historical discussion.)

Once (if) that gets to the point where it clearly addresses the 90% of needs, then there's a case for adding it to core. I don't think we can just push forward given that nothing changed in the last few years.

Kind Regards,

Carlton

 

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20a991ee-1601-4969-b12d-272cd9a1556an%40googlegroups.com.

[Carlton Gibson]

I see you posted afresh in the other thread, addressing some of the points here. C.

[Tobias McNulty]

Hi Carlton,

Responses inline below:

On Mon, Nov 28, 2022, 2:40 AM Carlton Gibson <carlton...@gmail.com> wrote:

Hi Raphael, thanks for picking this up. 

Looking at the history here, particularly the discussion here https://groups.google.com/g/django-developers/c/UQjpzB39JN0/m/XGqdV8nbBwAJ but there are others, the reason this hasn't already happened is that if we're going to merge this into core it would be nice (required?) to handle extra cases than (to paraphrase) "just what Heroku expects".

It doesn't look like anyone has sat down to spec out exactly what that means, but there seem to be plenty of thoughts in the discussion.  ... 🤔

JKM's comment on the thread there is representative of the general thought: 

> I suspect this is a "good enough is good enough" situation. Something like what

Raffaele is talking about, or dsnparse, or whatever would probably be ideal. And
for something to be merged into core, I think it'd need to be a more full
solution than just dj-database-url.

Isn't this already a more full solution insofar as it supports setting CACHES as well? This may or may not be correct, but I read the history a little differently, more that "life happens" and no one quite found the energy to push the current solution over the finish line (I empathize completely!).

The trouble with merging a suboptimal solution is that once it's in, it's very hard to change. We have the whole API stability policy to contend with (and rightly so). 

We bring in a half-baked solution; likely "recommend" it (for some value of that) and then face a long stream of difficult adjustments whilst we try and accommodate the necessary adjustments.

I agree 100%, and I feel like we can and will take this into consideration sufficiently as we refine the current solution, if we continue down this path. 

 Given that dj-database-url is stable, a single line install, and (most importantly) not being developed to try and accommodate the issues that stopped it just being merged into core already, I think it should continue to live as a third-party package. 

The place for the development to happen is in dj-database-url or a successor package. (The first step would be a Roadmap drawn from the historical discussion.)

Once (if) that gets to the point where it clearly addresses the 90% of needs, then there's a case for adding it to core. I don't think we can just push forward given that nothing changed in the last few years.

Could you (or others) provide some examples of the needs this solution doesn't address (not necessarily complete, just to give an idea)?

Many thanks,

Tobias

[Carlton Gibson]

Hey Tobias. 

> Could you (or others) provide some examples of the needs this solution doesn't address (not necessarily complete, just to give an idea)?

I hadn't seen the other thread this morning when replying to this one. It looks like some of the issues are indeed addressed but I was only able to give the historical discussion a quick read — I can't if it's Job Done™ — I kind of feel that review falls to the proposal.

From my glance, all positive points: 

* The register functions for a backend look quite flexible. 

* The cache case was covered. 

* The need to wire-up the configure functions looked to resolve somes concerns too. 

Given that it's a single import I might still lean towards seeing it as an external package, at least for a cycle, so unknowns that come up can be resolved, and folks on an older LTS can opt-in early, etc. 

(But that's not a point of religion.) 

Kind Regards,

Carlton

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMGFDKQg8ikr%2BVh9ZXm2UXJidbz%2BpUr-GvZa%2BKoGz%2BBfRyZggg%40mail.gmail.com.

[James Bennett]

It feels like this needs to be a broader conversation about not just
changing DATABASES to accept a URL, or integrating a particular
database-settings project, but to re-think how Django does
settings-from-environment as a whole.

When I'm setting up new Django-based projects, django-environ
(https://pypi.org/project/django-environ/) is now one of the first
packages I reach for, and it has a much more comprehensive set of
helpers for configuring Django from environment variables and doing
appropriate parsing and type casts.

There also is Pydantic's settings model
(https://pydantic-docs.helpmanual.io/usage/settings/) which uses type
annotations to drive parsing/validation/type-casting and defaults to
reading from environment variables.

I'd rather see some sort of comprehensive "make settings-from-env
easier" in Django than one-off pieces like a database URL.

[Curtis Maloney]

I recently did a bit of an audit of "settings types" whilst preparing for an overhaul of django-classy-settings.

I found you could cover a majority of the common config types if you could somehow parse from an environ string the following (which closely aligns with django-environ):

+ str

+ int

+ bool

+ list/tuple<str>

+ list/tuple<int>

+ DB Config

+ Cache Config

The complicated ones being DB/Cache configs, and mostly because:

- whilst a basic URL suffices, there are lots of optional and sometimes nested options

- sqlite in-memory syntax is "special".

Alternatively, I've been telling myself for a while now I'd add django-compatible types to confucius, which uses a type annotations approach to env-driven settings.

Both support providing __dir__ and __getattr__ so you can plug them "transparently" into your settings.py whilst still benefiting from class features.

-- 

You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAL13Cg-cqJi1JanmhN0S3rf1MYME9qH4XJsCEjLdB5wFXEQeAA%40mail.gmail.com.

--

C

[Carlton Gibson]

This must be do-able now. 

There was some discussion of something along these lines a couple of years ago, with the idea of doing it for GSoC. 

There was much back-and-forth but no consensus. 

It has then come up again with Peter's suggestions about making it easier to get to a deployment ready state. 

The discussion with Florian there touches on this exact point.

It looks like there's two sides to it: 

* A user-facing API, perhaps like ...get("key" default) 🤔

* A pluggable backend API for that, so folks can use things like Vault and temporary secrets files as well as the environment. 

If I followed correctly, it would be OK to ship an environment backend with Django (perhaps with some simpler ones for development/testing) leaving more advanced options open. 

I think Florian has ideas here. 

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAL13Cg-cqJi1JanmhN0S3rf1MYME9qH4XJsCEjLdB5wFXEQeAA%40mail.gmail.com.

[Florian Apolloner]

The thread that Carlton refers to is https://groups.google.com/g/django-developers/c/q20_Cxske88 -- don't be deterred by the title of it, it really talks about handling settings as well :)

I kinda agree with James. I'd like to have a proper answer for handling settings rather then go setting by setting. The main difficulty I see is to decide whether we really go low-level in the sense of providing a simple API that can get settings from a variety of sources (the `get("key", default)`) or if we go further and provide a newer opinionated high level API like pydantic etc… Both approaches have their ups and downs. But before repeating everything from the other thread here, I'd like to ask that we decide on one spot to continue this discussion and then stick to that one (the settings one, as noted by James I am also rather -0 on adding DATABASE_URL support now). Titlewise neither of the two threads seem to fit very well so we can just as well open a topic on the forum ;)

Cheers,

Florian

[Tobias McNulty]

I started a forum thread on the settings refactor here: https://forum.djangoproject.com/t/settings-refactor/17352

Regarding the original topic of this thread (https://code.djangoproject.com/ticket/28236), are we ready to close this as wontfix? I'm okay not proceeding with it (dj-database-url, django-environ, etc. are more than fine as stop-gaps), but it would be nice to preclude further effort/work on this issue if we're never going to move forward with it. While I realize it was never Accepted, that nuance may be lost on new visitors given the prior work history on the issue...

Cheers,

Tobias