[Django] #26029: Provide an API to configure arbitrary file storage backends
Django
------------------------------------------------+------------------------
Reporter: aaugustin | Owner: nobody
Type: New feature | Status: new
Component: File uploads/storage | Version: master
Severity: Normal | Keywords:
Triage Stage: Accepted | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
------------------------------------------------+------------------------
Currently Django has two hardcoded file storage backends, "default" (for
media files) and "static" (for static files). Idioms for configuring file
storage backends in pluggable apps are cumbersome.
It would be nice to be able to configure arbitrary file storage backends,
like caches and databases e.g.:
{{{
FILE_STORAGES = {
'media': {
'BACKEND': settings.DEFAULT_FILE_STORAGE,
'OPTIONS': {
'location': settings.MEDIA_ROOT,
'base_url': settings.MEDIA_URL,
# possible override of settings.FILE_CHARSET
},
},
'static': {
'BACKEND': settings.STATICFILES_STORAGE,
'OPTIONS': {
'location': settings.STATIC_ROOT,
'base_url': settings.STATIC_URL,
# replacement for STATICFILES_FINDERS and STATICFILES_DIRS
that would look a lot like template loaders
# possible override of settings.FILE_CHARSET
},
}
}}}
This was discussed on django-developers: https://groups.google.com/d/msg
/django-developers/gEbFApLLuzg/IW1LDUwmEgAJ
There were some concerns about introducing another large dict in settings.
The general ideas was uncontroversial.
--
Ticket URL: <https://code.djangoproject.com/ticket/26029>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: aaugustin | Owner: nobody
Type: New feature | Status: new
Component: File uploads/storage | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by kezabelle):
For what it's worth, bmispelon and I discussed something akin to this
recently, out of which I started implementing something similar
[https://github.com/kezabelle/django-storagecellar entirely as an
experiment]. One thing I liked about where I went with that was that a
named storage key in the dictionary was a dotted path, and behaved
somewhat like a logging config in that the nearest 'parent' match such
that each model field which uses a storage could be addressed separately,
like so:
{{{
class Test(Model):
a = FileField(storage='myapp.test.a')
b = ImageField(storage='myapp.test.b')
}}}
and then if `myapp.test.a` or `myapp.test.b` were in the configuration
dictionary, they would be used, but if not, and `myapp.test` was, that
would be used, and so on walking backwards up the dotted path (falling
back to ostensibly `default` [`''`] in the end)
The notion there was third-party apps could namespacing storages per-
field, but at a project level one only has to opt-in as much as one cares
(ie: `myapp`, or `myapp.test`, or `default` etc.) to have finegrained
control.
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:1>
Django
Reporter: aaugustin | Owner: nobody
Type: New feature | Status: new
Component: File uploads/storage | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by kezabelle):
* cc: django@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:2>
Django
Reporter: aaugustin | Owner: nobody
Type: New feature | Status: new
Component: File uploads/storage | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by sasha0):
Configurable file storage backends was already proposed to introduce in
terms of task #23251.
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:3>
Django
Reporter: aaugustin | Owner: sasha0
Type: New feature | Status: assigned
Component: File uploads/storage | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by sasha0):
* owner: nobody => sasha0
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:4>
Django
Reporter: aaugustin | Owner: sasha0
Type: New feature | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by sasha0):
Replying to [ticket:26029 aaugustin]:
> {{{
> FILE_STORAGES = {
> 'media': {
> 'BACKEND': settings.DEFAULT_FILE_STORAGE,
> 'OPTIONS': {
> 'location': settings.MEDIA_ROOT,
> 'base_url': settings.MEDIA_URL,
> # possible override of settings.FILE_CHARSET
> },
> ...
> }
> }}}
>
Media backend will have `media` key in settings, not `default`, to keep it
consistent with the current settings?
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:5>
Django
Reporter: aaugustin | Owner: sasha0
Type: New feature | Status: assigned
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by sasha0):
As proposed Aymeric Augustin in the original PR, I started to compose DEP,
so now it's in progress and I hope to finish it sometime soon.
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:6>
Django
Reporter: Aymeric Augustin | Owner: (none)
Type: New feature | Status: new
Component: File uploads/storage | Version: master
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Sasha Gaevsky):
* owner: Sasha Gaevsky => (none)
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:7>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: assigned
uploads/storage |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Jarosław Wygoda):
* owner: (none) => Jarosław Wygoda
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:8>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: assigned
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Jarosław Wygoda):
I'd like to introduce a file storage registry similar to
BaseConnectionHandler (django/utils/connection.py) and EngineHandler
(django/template/utils.py).
Example settings.py snippet:
{{{
STORAGES = { # rename to FILE_STORAGES to make it more explictit?
'example': {
'BACKEND': 'django.core.files.storage.FileSystemStorage',
'OPTIONS': {
'location': '/example',
'base_url': '/example/',
},
},
}
}}}
Changes introduced by this pr are backward compatible. Users can still use
existing settings to configure static and media storages.
Currently storages can be retrieved from the following objects:
django/core/files/storage.py:
* get_storage_class
* DefaultStorage
* default_storage
django/contrib/staticfiles/storage.py:
* ConfiguredStorage
* staticfiles_storage
What do you think about deprecating them?
I'll write tests and docs if this approach is acceptable.
https://github.com/django/django/pull/15610
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:9>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: assigned
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz):
* has_patch: 0 => 1
* needs_tests: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:10>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: assigned
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:11>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: assigned
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"9e7cb27a5b7363239d1db02d29fe12efdf25b899" 9e7cb27]:
{{{
#!CommitTicketReference repository=""
revision="9e7cb27a5b7363239d1db02d29fe12efdf25b899"
Refs #26029 -- Doc'd django.core.files.storage.default_storage.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:12>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: assigned
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 1
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"04ec8bf92a0c3f3d89adbc38a187f972b8424941" 04ec8bf9]:
{{{
#!CommitTicketReference repository=""
revision="04ec8bf92a0c3f3d89adbc38a187f972b8424941"
[4.1.x] Refs #26029 -- Doc'd django.core.files.storage.default_storage.
Backport of 9e7cb27a5b7363239d1db02d29fe12efdf25b899 from main
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:13>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: assigned
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* needs_better_patch: 1 => 0
* needs_tests: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:14>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: assigned
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* needs_better_patch: 0 => 1
Comment:
PR looks promising, but have comments outstanding ref adding the new
`STORAGES` defaults.
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:15>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: assigned
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* needs_better_patch: 1 => 0
Comment:
I think the PR looks close. (I suggested a few docs tweaks)
Main remaining point (for me) is being sure about the signal handling with
`override_settings`, and the usual `Settings`/`UserSettingsHolder`
complexities.
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:16>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: assigned
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"d16079dd90320141c2024809ba0fce43f0f2cb02" d16079d]:
{{{
#!CommitTicketReference repository=""
revision="d16079dd90320141c2024809ba0fce43f0f2cb02"
Refs #26029 -- Added LazySettings._show_deprecation_warning() hook.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:17>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Component: File | Version: dev
uploads/storage |
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"1ec3f0961fedbe01f174b78ef2805a9d4f3844b1" 1ec3f09]:
{{{
#!CommitTicketReference repository=""
revision="1ec3f0961fedbe01f174b78ef2805a9d4f3844b1"
Fixed #26029 -- Allowed configuring custom file storage backends.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:18>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: closed
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution: fixed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"32940d390a00a30a6409282d314d617667892841" 32940d3]:
{{{
#!CommitTicketReference repository=""
revision="32940d390a00a30a6409282d314d617667892841"
Refs #26029 -- Deprecated DEFAULT_FILE_STORAGE and STATICFILES_STORAGE
settings.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:19>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: closed
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution: fixed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"f72f420f17c1bf6aea4022ecdb9b5f53a46597cc" f72f420]:
{{{
#!CommitTicketReference repository=""
revision="f72f420f17c1bf6aea4022ecdb9b5f53a46597cc"
Refs #26029 -- Removed DEFAULT_FILE_STORAGE and STATICFILES_STORAGE
settings.
This also removes django.core.files.storage.get_storage_class().
Per deprecation timeline.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:20>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: closed
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution: fixed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by GitHub <noreply@…>):
In [changeset:"fa6e6f31137ee4025adfb27f88c07eebd488d446" fa6e6f31]:
{{{
#!CommitTicketReference repository=""
revision="fa6e6f31137ee4025adfb27f88c07eebd488d446"
[5.0.x] Refs #26029 -- Improved get_storage_class() deprecation warning
with stacklevel=2.
Addition of the `stacklevel` argument shows the source of the
deprecated call, making updating the client code simpler.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:21>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: closed
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution: fixed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"0cbc92bc3adb219265bd7f1bc00af1f526e1278a" 0cbc92bc]:
{{{
#!CommitTicketReference repository=""
revision="0cbc92bc3adb219265bd7f1bc00af1f526e1278a"
[4.2.x] Refs #26029 -- Improved get_storage_class() deprecation warning
with stacklevel=2.
Addition of the `stacklevel` argument shows the source of the
deprecated call, making updating the client code simpler.
Backport of fa6e6f31137ee4025adfb27f88c07eebd488d446 from stable/5.0.x.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:22>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: closed
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution: fixed
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by GitHub <noreply@…>):
{{{#!CommitTicketReference repository=""
revision="6dcab75d5d8c2ef18de15323930057e6fa9ec00f"
Refs #26029 -- Extended docs for the StorageHandler default instance.
Third-party packages that provide storages need to rely on the
StorageHandler API in order to allow users to use the `storages` module
instance to override defaults. Minimally documenting these methods
allows package authors to rely on them.
Co-authored-by: Sarah Boyce <42296566+...@users.noreply.github.com>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:23>
Django
Reporter: Aymeric Augustin | Owner: Jarosław
| Wygoda
Type: New feature | Status: closed
Component: File | Version: dev
uploads/storage |
Severity: Normal | Resolution: fixed
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Natalia <124304+nessita@…>):
In [changeset:"56ad530ade53e30596ca95c85169c187b1c33ced" 56ad530a]:
{{{#!CommitTicketReference repository=""
revision="56ad530ade53e30596ca95c85169c187b1c33ced"
[5.1.x] Refs #26029 -- Extended docs for the StorageHandler default
Third-party packages that provide storages need to rely on the
StorageHandler API in order to allow users to use the `storages` module
instance to override defaults. Minimally documenting these methods
allows package authors to rely on them.
Co-authored-by: Sarah Boyce <42296566+...@users.noreply.github.com>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/26029#comment:24>