I can imagine any backend being able to convert from a url string with some methods (e.g. `detect_url` (bool return) and `parse_url` (settings dict return type).)
Perhaps the same scheme could be applied to caches as well.
I imagine that you'd have to be able to merge a url string with a settings dict (one overrides the other). This is how usage of that functionality is manually provided in dj-database-url.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/057ced98-4471-4939-960b-900ec39f54b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout
On 24 May 2017, at 21:21, Tom Forbes <t...@tomforb.es> wrote:My two cents: connection strings/database URI's are a feature I've sorely missed in Django.Built-in functionality to convert environment variables like DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant DATABASE setting would make some deployment situations a lot simpler. Currently, unless you use dj-database-uri you have to define a bunch of ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary together yourself.
How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?
To help support third part backends: perhaps the scheme portion of the URI could be either a relative import from django.db.backends or an absolute import to a third party library? It seems URI schemes can have dots and underscores in them, so they can be python package paths.I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but sqlserver_ado://xyz would resolve to the third party django-mssql engine via an absolute import.
I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/A0870D90-75DE-496B-A838-AC7D7488A67D%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.
Good point, it's not wise to enable this even without a clear threat model. Django does import the settings based on an environment variable, but it's relative and if you can use that to do anything nasty then you're most likely already through the airtight hatch (so to speak). However importing potentially global modules could be bad news.> I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.Ignoring it is always an option, but could we not specify that the third party database provider has to be in the `INSTALLED_APPS`? That could provide some form of whitelisting so not any old module is imported. Not sure about any issues that may arise from this though.
> One possibility would be to use entrypoints in setuptools, this way 3rd party backends could specify a name which then has a fixed & verified import path.This seems like it could get complex, and be quite unlike anything else in Django.Perhaps just supporting this for first-party database backends is easiest?
Edit: DJANGO_SETTINGS_MODULE isn't relative, it will import any arbitrary module you give it. If we accept that then I think we are accepting the risk of imports via an attacker controlling environment variables whilst Django starts up?
On Sat, May 27, 2017 at 8:49 PM, Tom Forbes <t...@tomforb.es> wrote:
Good point, it's not wise to enable this even without a clear threat model. Django does import the settings based on an environment variable, but it's relative and if you can use that to do anything nasty then you're most likely already through the airtight hatch (so to speak). However importing potentially global modules could be bad news.> I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.Ignoring it is always an option, but could we not specify that the third party database provider has to be in the `INSTALLED_APPS`? That could provide some form of whitelisting so not any old module is imported. Not sure about any issues that may arise from this though.
> One possibility would be to use entrypoints in setuptools, this way 3rd party backends could specify a name which then has a fixed & verified import path.This seems like it could get complex, and be quite unlike anything else in Django.Perhaps just supporting this for first-party database backends is easiest?
On Thu, May 25, 2017 at 8:46 AM, Aymeric Augustin <aymeric....@polytechnique.org> wrote:
Hello,I'm wondering what the exact definition of the URL format is. Is it specified somewhere? Or is it just:[engine]://[username]:[password]@[host]:[port]/[name]where we create arbitrary [engine] values in an ad-hoc fashion?On 24 May 2017, at 21:21, Tom Forbes <t...@tomforb.es> wrote:My two cents: connection strings/database URI's are a feature I've sorely missed in Django.Built-in functionality to convert environment variables like DJANGO_DB_DEFAULT (or more generally DJANGO_DB_*key*) into the relevant DATABASE setting would make some deployment situations a lot simpler. Currently, unless you use dj-database-uri you have to define a bunch of ad-hoc DB_USER/DB_PASSWORD etc env variables and price the dictionary together yourself.Fully agreed. While relatively minor, it's an annoyance.How does this library complex keys like OPTIONS, TEST or DEPENDENCIES?I don't think it's reasonable to cram them in a URL.dj-database-url allows passing options as extra keyword arguments. Other values should be explicitly added in the settings module, by updating the dict generated from the URL.To help support third part backends: perhaps the scheme portion of the URI could be either a relative import from django.db.backends or an absolute import to a third party library? It seems URI schemes can have dots and underscores in them, so they can be python package paths.I.e sqlite3://xyz would resolve go django.db.backends.sqlite3, but sqlserver_ado://xyz would resolve to the third party django-mssql engine via an absolute import.I'm wary of possible security ramifications: if we do this, changing a configuration value will import an arbitrary module, which could make it easier to run arbitrary code in some scenarios. I don't have a clear threat model in mind here, though.I'd rather specify the database engine explicitly when calling dj-database-url if it's a third-party engine. There's an open question about what to do with the [engine] part of the URL in that case. Ignoring it entirely is the easiest.Best regards,--Aymeric.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
import django12factor globals().update(django12factor.factorise())
I spent some time working on this today and fixed up my branch that implements this. I think I’ve got a nice API that passes all of the dj-database-url tests.
As there is a bit of database-specific shenanigans we have to perform on URLs I added a config_from_url
static method to database adapters that takes the URL and returns a valid django configuration dictionary. As it’s a method inside the adapter itself we can override it to apply any db-specific fixups and third party backends can hook into the process as well via the same mechanism.
It expands on dj-database-url by also allowing the configuration of caches using the same method, which I think is pretty neat.
It uses a registration process described below, users just call register_db_backend
or register_cache_backend
with a path to their third party engine in the settings to link a scheme to a backend dotted path.
I had one question that I’m not sure the answer to: for databases I added the config_from_url
to the .base.DatabaseWrapper
class inside each built in adapter. I’ve kind of assumed each third party database would have the same structure, how much of a safe assumption is this?
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/065601c3-9488-42aa-8b23-c7f906c26b5a%40googlegroups.com.
I had one question that I’m not sure the answer to: for databases I added the
config_from_url
to the.base.DatabaseWrapper
class inside each built in adapter. I’ve kind of assumed each third party database would have the same structure, how much of a safe assumption is this?
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1a55cc1c-ba9c-4950-ab94-50da8eec7d06%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFNZOJMa1xSzBeGUYygG7nxfCVz8jUPNEiSEJhbAqLDTwga9BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20a991ee-1601-4969-b12d-272cd9a1556an%40googlegroups.com.
Hi Raphael, thanks for picking this up.Looking at the history here, particularly the discussion here https://groups.google.com/g/django-developers/c/UQjpzB39JN0/m/XGqdV8nbBwAJ but there are others, the reason this hasn't already happened is that if we're going to merge this into core it would be nice (required?) to handle extra cases than (to paraphrase) "just what Heroku expects".It doesn't look like anyone has sat down to spec out exactly what that means, but there seem to be plenty of thoughts in the discussion. ... 🤔JKM's comment on the thread there is representative of the general thought:> I suspect this is a "good enough is good enough" situation. Something like whatRaffaele is talking about, or dsnparse, or whatever would probably be ideal. And
for something to be merged into core, I think it'd need to be a more full
solution than just dj-database-url.
The trouble with merging a suboptimal solution is that once it's in, it's very hard to change. We have the whole API stability policy to contend with (and rightly so).We bring in a half-baked solution; likely "recommend" it (for some value of that) and then face a long stream of difficult adjustments whilst we try and accommodate the necessary adjustments.
Given that dj-database-url is stable, a single line install, and (most importantly) not being developed to try and accommodate the issues that stopped it just being merged into core already, I think it should continue to live as a third-party package.The place for the development to happen is in dj-database-url or a successor package. (The first step would be a Roadmap drawn from the historical discussion.)Once (if) that gets to the point where it clearly addresses the 90% of needs, then there's a case for adding it to core. I don't think we can just push forward given that nothing changed in the last few years.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMGFDKQg8ikr%2BVh9ZXm2UXJidbz%2BpUr-GvZa%2BKoGz%2BBfRyZggg%40mail.gmail.com.
--You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAL13Cg-cqJi1JanmhN0S3rf1MYME9qH4XJsCEjLdB5wFXEQeAA%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAL13Cg-cqJi1JanmhN0S3rf1MYME9qH4XJsCEjLdB5wFXEQeAA%40mail.gmail.com.