Storage engine aliases?

[Jeremy Dunck]

Right now, I think that static/media handling is fairly confused in the documentation, and a bit confused in the code itself.

We have a few special-cases floating around:

   default_storage (needed for legacy before storage backends)

   staticfiles_storage (needed for collectstatic/handling)

   {% static %} handles mapping relative URLs to absolute URLs, but does not allow using a storage engine other than {% STATIC_URL %} or staticfiles_storage.

I was surprised to find that 

   django.contrib.staticfiles.templatetags

   and

   django.templatetags.static

both have implementations of {% static %} with slightly different semantics (one based on STATIC_URL, one based on staticfiles_storage).

It seems to me that it might be useful to introduce aliases (as in CACHES and DATABASES) in order to allow referring to storage engines in a less-coupled way.

{% static %} could then take a storage engine alias, and the special-case of repo-static file handling and user-uploaded file handling could mostly go away.

MEDIA_URL/MEDIA_ROOT and STATIC_URL/STATIC_ROOT as special cases could (after a deprecation process) go away in favor of these storage aliases.

The syntax I'll hazard as a proposal could be:

FILE_STORES = {

   'static': <Storage instance>,

   'dynamic': <Storage instance>,

   ....

   'alias1': 'alias2' # possibly map aliases to satisfy app-required aliases.

}

{% file relative-url storage-alias %}

Would do people think of this idea?

[Florian Apolloner]

Hi Jeremy,

To be honest I don't really see the gains (yet). How are those FILE_STORES supposed to handle the *_URL bits of those settings? Instances in settings are imo not going to fly, we should stay with strings to classes/instances there for now. So how would the 'static' or 'dynamic' instance know where to serve files from and how to generate the url?

Regarding the two {% static %} tags, the distinction is quite clear: Django has to be useable without staticfiles installed, hence the need for a tag in core.

Cheers,
Florian

[Aymeric Augustin]

Hi Jeremy,

That could be useful for any website that gets some of its assets from the code (JS, CSS), others from a CDN (eg. product photos), others from another CDN (eg. tutorial illustrations), etc. However we’d have to make sure it beats the common solution of having a model instance for each file and a method that generates the URL.

Even though the documentation has improved a lot, static files still tend to confuse beginners. On one hand, making them less of a special case could help. On the other hand changing the APIs will create confusion again.

Finally, static and media (user-uploaded) file have different requirements, especially in terms of security. I think it’s useful to keep the concepts separate, even if they ultimately depend on the same APIs — basically the Storage base class that defines the usual file APIs plus an URL.

-- 

Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAM0i3f6sMwgFxAVC4P6U9Z97zAd%2B%3DmDV837KbYZmBx3ycuu%2BMg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Jeremy Dunck]

Aymeric, 

  That's an interesting idea I hadn't considered -- perhaps storages could also then be marked trusted and untrusted, and processing/display of those files could take it into consideration.  I agree the security requirements are different.

Florian,

  I agree that changing APIs would cause confusion.  It may be possible to provide an upgrade path as we did for DATABASES and CACHES, though.

  The benefit I'm shooting for is to make media handling more modular and clear, rather than seeming a special case.

  As for instances, yes, I was using that as a shorthand -- even so, it could be a dict with common interface (**kwargs) handed to the storage backend.  FileSystemStorage takes location=None, base_url=None, s3boto takes acl=None, bucket=None, **settings.  I doubt there will be much common interface, but to me this supports the argument that aliases would be useful -- right now it is hard to construct a storage backend dynamically without coding to a specific store.  connections['default'] is a powerful abstraction.

This could be a migration path:

FILE_STORAGES = {} -> expands to 

  'static': {

    'class': 'django.contrib.staticfiles.storage.StaticFilesStorage',

    'trusted': True, # based on Aymeric's feedback

    'OPTIONS': {

       'location': settings.STATIC_ROOT,

       'base_url': settings.STATIC_URL

    },

  },

  'media': {

    'class': 'django.core.files.storage.FileSystemStorage',

    'trusted': False,

    'OPTIONS': {

      'location': settings.MEDIA_ROOT,

      'base_url': settings.MEDIA_URL

    }

  }

}

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6C594E01-F946-4806-B569-2221832205A3%40polytechnique.org.

[Jannis Leidel]

On 29 Sep 2014, at 22:46, Jeremy Dunck <jdu...@gmail.com> wrote:

> Right now, I think that static/media handling is fairly confused in the documentation, and a bit confused in the code itself.

Please make sure to open tickets about this, I’ve long lost the ability to see at this code from the beginners perspective. Any specific problems with the docs would be appreciated. I understand that we can only do so much of course by fixing the docs.

> We have a few special-cases floating around:
>
> default_storage (needed for legacy before storage backends)

Yeah, the naming is unfortunate, but really only a legacy thing when storage only meant “upload API”. We could rename it into “upload_storage”?

> staticfiles_storage (needed for collectstatic/handling)
> {% static %} handles mapping relative URLs to absolute URLs, but does not allow using a storage engine other than {% STATIC_URL %} or staticfiles_storage.
>
> I was surprised to find that
> django.contrib.staticfiles.templatetags
> and
> django.templatetags.static
> both have implementations of {% static %} with slightly different semantics (one based on STATIC_URL, one based on staticfiles_storage).

The historic reason for this is simply the requirement for being able to disable the staticfiles contrib app. It was decided when I worked on this to be a blocker for a merge.

So I implemented the static tag twice and made the core tag as simple as possible. Using urljoin with STATIC_URL was the recommended thing to do at the time after all. staticfiles couldn’t go that route directly though as it needed some better API and especially something that can be extended. The storage system was there so I used it.

In hindsight I regret not having pushed harder for integration of staticfiles into core instead, lots of confusion could have been prevented, IMO.

> It seems to me that it might be useful to introduce aliases (as in CACHES and DATABASES) in order to allow referring to storage engines in a less-coupled way.
>
> {% static %} could then take a storage engine alias, and the special-case of repo-static file handling and user-uploaded file handling could mostly go away.
>
> MEDIA_URL/MEDIA_ROOT and STATIC_URL/STATIC_ROOT as special cases could (after a deprecation process) go away in favor of these storage aliases.
>
> The syntax I'll hazard as a proposal could be:
>
> FILE_STORES = {
> 'static': <Storage instance>,
> 'dynamic': <Storage instance>,
> ....
> 'alias1': 'alias2' # possibly map aliases to satisfy app-required aliases.
> }
>
> {% file relative-url storage-alias %}
>
> Would do people think of this idea?

That implies that we stop referring to uploaded files directly via the FieldField instance attribtue, e.g. {{ my_model.uploaded_file.url }}. I’m okay with that, if your proposed tag would be able to take file field objects as well as relative paths.

On a related note, could we simply rename the MEDIA_* settings to UPLOAD_* and just fix this ambiguity in other places as well?

Jannis

[Collin Anderson]

In hindsight I regret not having pushed harder for integration of staticfiles into core instead, lots of confusion could have been prevented, IMO.

Is it too late to move to core?

[Jannis Leidel]

On 01 Oct 2014, at 14:33, Collin Anderson <cmawe...@gmail.com> wrote:

> In hindsight I regret not having pushed harder for integration of staticfiles into core instead, lots of confusion could have been prevented, IMO.
>
> Is it too late to move to core?

Good question, if we’d left the public API in django.contrib.staticfiles following the deprecation policy intact we may be able to move it into django.core.files I guess.

If something like that were be tried I would also recommend to fix the ambiguity with regard to “media” and “static” by renaming the first to “uploads”.

What do others think about that idea? Too much effort for little gain? Too late?

Jannis

[Curtis Maloney]

On 2 October 2014 16:46, Jannis Leidel <lei...@gmail.com> wrote:

On 01 Oct 2014, at 14:33, Collin Anderson <cmawe...@gmail.com> wrote:

> In hindsight I regret not having pushed harder for integration of staticfiles into core instead, lots of confusion could have been prevented, IMO.
>
> Is it too late to move to core?

Good question, if we’d left the public API in django.contrib.staticfiles following the deprecation policy intact we may be able to move it into django.core.files I guess.

If something like that were be tried I would also recommend to fix the ambiguity with regard to “media” and “static” by renaming the first to “uploads”.

That would be misleading, as it's not just uploads.  For instance, if your site generated report PDFs, I'd expect them to be stored in MEDIA.

I tend to characterise them as :

static -- What is an integral part of your site. What you keep in source control, and is required for the site to function.

media -- what happens as a result of your site functioning.

--

Curtis

[Raphael Michel]

Hi,

Am Thu, 2 Oct 2014 08:46:26 +0200
schrieb Jannis Leidel <lei...@gmail.com>:

> If something like that were be tried I would also recommend to fix
> the ambiguity with regard to “media” and “static” by renaming the
> first to “uploads”.
>
> What do others think about that idea? Too much effort for little
> gain? Too late?

No, I can tell this IS a big source of confusion not only for
beginners, so it is worth a change to me.

Cheers,
Raphael

[Aymeric Augustin]

On 2 oct. 2014, at 09:36, Curtis Maloney <cur...@acommoncreative.com> wrote:

I tend to characterise them as :

static -- What is an integral part of your site. What you keep in source control, and is required for the site to function.

media -- what happens as a result of your site functioning.

I have two shorter descriptions to suggest: 

1) static is code, media is data.

2) static is version controlled, media is backed up.

-- 

Aymeric.

[Wim Feijen]

Hi Jannis, although much work, I would find that brilliant.

On Thursday, 2 October 2014 08:47:13 UTC+2, Jannis Leidel wrote:

"If something like that were be tried I would also recommend to fix the ambiguity with regard to “media” and “static” by renaming the first to “uploads”." 

Jannis