Management of static assets

[David Smith]

Hi All, 

I hope you are all well. 

I've been thinking about static assets over the past week or so following my email on widget media. I collated the past 6 year's worth of discussions into a single source, with relevant extracts and links to sources. I've also set out some options and given my thoughts (FWIW). I've not yet done any detailed planning as I think we need to agree general principles first. Sorry this is a bit long, may need to go get a cup of tea first :-)

I hope this is helpful to aid discussion on this topic 🤞 

Summary

The below sets out this historical context for ticket #22298 (What to do with Widget Media) and the wider discussions this generated on how Django should manage assets. Whilst there are some detailed points below, these are to help set context and to aid discussion. This is a wide and deep topic and therefore I suggest at this stage we discuss the topic with an aim to agree the overall direction. The principle I suggest we discuss is:

In alignment with it's "batteries included" philosophy, should Django have an aim to improve the management of static assets? 

In addition to seeking agreement to this broad principle I'd also like to encourage discussion on the topics which fall under this category. Here are some questions to aid a discussion, although I fully expect the conversation to be wider than these initial suggestions. Some items may be easier to seek agreement on than others; however, all will take time and effort of the community to agree the correct design and implementation. 

1. Should Django have an improved way of managing assets within apps (something akin to #29586)
2. Should Django have the ability to compile and compress assets?
3. Should Django have Webpack support?
4. Should Django have improved support for JS frameworks?
5. Should Django have support for NPM?

Based on evidence below my view is that there is enough support to improve Django's support in this area generally. Assuming this is agreed I think the a range of options are:

• Provide improve awareness of static assets with Django core to help 3rd party packages
• In addition, integrate an asset pipeline
• In addition, integration with webpack, and support JS frameworks.

My personal view is that option one is fairly cautious, there are already 3rd party packages which enable this functionality, and that the third one is too ambitious. I suspect most people using JS frameworks are doing it anyway irrespective of what is part of Django code. I think there the most value can be added is for sites with mostly HTML / CSS and, but would like some help compiling CSS and "sprinkling" a bit of JS here and there. 

I'd therefore suggest we start by revisiting work previously done by Claude to understand if this is the right foundation. Once this is complete this will enable phase two to integrate an asset pipeline / compressor into Django. I would suggest that this is via a separate package to start with under the Django umbrella. I suspect the pace of iteration will be too fast for Django core, and we would like it to be more stable before merging it. Would it be possible to use some of the existing work that the wider community has already developed and explore building on (or 'just' use) an existing 3rd party package? (I don't know how acceptable this is, or not)

My personal recommendation is also that ticket #22298 is closed or marked as someday / maybe. Widget Media may be useful in the future, we don't know yet. I would not support the deprecation of this functionality without  something better to replace it. I also feel that a rename would have little benefit but would cause disruption for those using it and numbers may be high given it's been in place for many years. If agreement can be sought on this then it will help other Media related tickets to be progressed (example : #21987).

Below is an overview of various comments, on tickets, mailing list and pull requests over the past 6 years. 

The ticket in question was on Widget Media, So what does Widget Media do?

It is used to load CSS or JS to widgets / forms (e.g. add <script> tags). This is useful when a certain widget can be enhanced by some extra JS. A calendar would be a classic example of this. By using the media class the JS for the calendar is only loaded on pages where the widget is present. 

How long has been around? 

Media classes have been in Django for a long time. They are included from at least version 1.0.

And why may we want to rename to static, what's the history here?

In [Django 1.3] `django.contrib.staticfiles` was added "to handle static files in a generic way".

In [Django 1.4](https://docs.djangoproject.com/en/3.0/releases/1.4/#django-contrib-admin) "Django's admin's static files also followed this convention by removing `ADMIN_MEDIA_PREFIX` and replaced it with `STATIC_URL`". 

So whilst most of the project moved away from using the phase 'media'; Widgets continues to use this phrase. This is inconsistent and _could_ be fixed. 

Now you've mentioned it, do we even need it?

Whilst the ticket originally suggested a name chance should be considered the discussion on the ticket is much deeper. This is where the complexity of this ticket grows significantly with and a wide range of options is available. Some of the options mentioned are:

- Let's deprecate Widget Media, it's not best practice
- Tell people to use django-compressor / pipeline
- Should these tools be integrated?
- Should there be better support for front end JS frameworks
- Should we have support for webpack
- We should support NPM

All of these items I think come under the question of "How should Django work with assets?"

Django ecosystem

Currently Django has a number of 3rd party packages which help to manage assets. A couple of options which have been mentioned in previous discussions on this topic are django-compressor (>2k github stars, >250k downloads p/month) and django-pipeline (>1k GitHub stars, >100k downloads p/month). These both augment the standard Django functionality to "Compresses linked and inline JavaScript or CSS into a single cached file." 

In addition there is also Django webpack loader (2k GitHub Stars, >300k downloads p/month) "Use webpack to generate your static bundles without django's staticfiles or opaque wrappers."


Some of these packages were also mentioned in the discussion on the django forum under the "top 5 packages" thread [4], but they did not feature time and time again. So whilst they have reasonable use, they don't seem to feature at the top of a good proportion of people's lists. 

What do other frameworks do? 

Previous comments have been that rails includes the following from Carlton [1]

 

The static files story is a little different. It seems to me we don't tell the best story there. 
Rails has two things which we could be envious of, even if we didn't want to copy exactly:
* The frontend framework integration that's already been mentioned. 
* The very easy "Ajax your form", with controllers (i.e. for us "generic views") automatically handling ajax form submissions. 
Both these features get users further quicker in these aspects than we are able to offer. 

For a more in depth overview of rails, this link was previously provided to the mailing list [5]. Further a technical write up was provided [1a]

Previous work 

Whilst I'm only seeking discussion on principles at this stage (should Django have these features). I think it's useful to draw reference to previous discussions and attempts to make progress in this area, as I feel it is useful context. 

It came up again as another GSOC. However, it was felt that the proposal itself wasn't feasible and that it was to big and complex to achieve through GSOC. [3]

It also got a mention on a separate GSOC proposal again in 2018, not particularly aligned to this topic but generated some relevant topics. . This one was a bit more complex, but did gain some comments on what may be useful for Django. Tom's post from this thread is particularly helpful [6]. 

"I'm perhaps a bit biased but I would be very interested in anything that can make JavaScript a real first class citizen in Django"

Claude has also attempted to add a base structure to Django. Ticket was closed as it was felt that developing this within Django itself was the wrong approach. Florian:

I am just a little bit worried about adding this without any concrete plan on how 3rd party apps are going to use it. What speaks against trying this outside of core like channels? (I'll happily put it under the django umbrella, but core seems a little bit fast tracked to me).

(https://code.djangoproject.com/ticket/29586)
(https://github.com/django/django/pull/10218)
(https://groups.google.com/forum/#!topic/django-developers/KYmNnvwXDUI)


So does any one use Widget Media it anyway?

Opinions in on the ticket are mixed, some comments of having never used it vs some which think it is quite common. 

Examples : 

A ticket where a site used it extensively enough to encounter performance issues [#30563](https://code.djangoproject.com/ticket/30563). The comment suggested 194,016 media objects were being loaded but only 13 unique items. 

Wagtail makes use of it. [Wagtail sourcecode](https://github.com/wagtail/wagtail/search?l=Python&p=1&q=.media)

References
[1] https://groups.google.com/d/msg/django-developers/KVAZkRCq9KU/AwnM5UbCGQAJ
[1a] https://blog.yourlabs.org/posts/2019-02-26-django-js-research-report/
[3] https://groups.google.com/forum/#!searchin/django-developers/media$20class%7Csort:date/django-developers/XF7F__mETrM/ECa4JpF-wRgJ
[4]https://forum.djangoproject.com/t/top-5-3rd-party-packages/391
[5]https://medium.com/@hpux/rails-5-1-loves-javascript-a1d84d5318b
[6] https://groups.google.com/d/msg/django-developers/ldhY8sAQc-0/h2RngIeZCQAJ

[Carlton Gibson]

Hi David.

Wowser. Good work! 

(A cup of tea indeed — I should have replied to the first question better 😝)

(Inter alia) You raised five questions. These were the last three:

- Should there be better support for front end JS frameworks
- Should we have support for webpack
- We should support NPM

I think the key point here is not to bind to a particular package, webpack say. It's already Parcel.js no? — and it'll be something else next week. 🙂

django-compressor, beyond concatenating and compressing files has this neat ability to define `COMPRESS_PRECOMPILERS`

Here you can map files to any CLI tool (or write a bit of Python if it gets too clever — but TBH I never have to do that).

An example from a project:

COMPRESS_PRECOMPILERS = (
    ("text/x-tailwind", TAILWIND_COMMAND),     # Compile tailwind via PostCSS, with tree-shaking, FTW 💃
    # ("text/x-scss", SASS_COMMAND),              # Would compile Sass. Oops didn't delete it yet.
    ("text/x-elm", ELM_MAKE_COMMAND),        # Run elm make.
)

You can just as easily put in webpack or parcel or babel (or a chain of any or all of them...)

It's that plus the offline compress tool that means I've been happy with compressor for a long time — I just never had to move on.

(There's a point where you hand off to a frontend specialist but ...)

So, particularly given how fast the tools landscape moves here, and how slow Django moves in comparison, some kind of wrapper where you shell out to A.N.Tool — it doesn't have to be JavaScript. It might be make, or pyinvoke — rather than something tied to webpack, as the example, would be a requirement for me.

Not sure how much of this we need to pull into Django itself? compressor, say, does the whole combine and compress thing well. If we pull that in are we going to do the same for image optimization? Or pull in Whitenoise? Or...? For a user just with contib.staticfiles, what's the very next thing we could do that would make life easier? (i.e. is "I can't run webpack" top of their list?) — What can we do better in core vs a third-party app? — Is that just awareness, which is not a nothing?

Note the "not sure" — I think about this a lot, it's a big issue, but I don't have a single answer: I've got what I do, and that works for me, but I see lots of people doing different with success.

Kind Regards,

Carlton

[Claude Paroz]

Le mercredi 22 avril 2020 17:22:02 UTC+2, Carlton Gibson a écrit :

...

Not sure how much of this we need to pull into Django itself? compressor, say, does the whole combine and compress thing well. If we pull that in are we going to do the same for image optimization? Or pull in Whitenoise? Or...? For a user just with contib.staticfiles, what's the very next thing we could do that would make life easier? (i.e. is "I can't run webpack" top of their list?) — What can we do better in core vs a third-party app? — Is that just awareness, which is not a nothing?

Note the "not sure" — I think about this a lot, it's a big issue, but I don't have a single answer: I've got what I do, and that works for me, but I see lots of people doing different with success.

I fully understand that it's not necessarily the role of Django to provide a full backed solution to management of assets, which is indeed changing quickly in some aspects.

However, I think that the problem now is that Django isn't aware of global project or app assets (they are just lines in templates), which means that third-party packages must provide their own different implementation of asset "objects". It does also mean that Django is helpless with regards to any management of static assets, typically in a case like subresource integrity.

My opinion is still the same, that we should add basic blocks of asset objects in Django core, giving ability to Django to offer basic services that are useful for most projects, while letting third party apps to build upon those "blocks" to give specific functionalities that need a quicker development pace than Django itself.

Claude

[Aymeric Augustin]

Thanks David for investigating the topic thoroughly! I wasn't expecting all that when I filed a one-line ticket six years ago :-) So, here's a bunch of opinions.

Before I start, I'd like to quote the intro to the Media class:

Rendering an attractive and easy-to-use Web form requires more than just HTML - it also requires CSS stylesheets, and if you want to use fancy « Web2.0 » widgets, you may also need to include some JavaScript on each page. The exact combination of CSS and JavaScript that is required for any given page will depend upon the widgets that are in use on that page.

That was the reasonable thing to do before single-page apps, asset pipelines, bundlers and code splitting. It's still a fairly reasonable thing to do on a website that relies on good old HTML forms but would benefit from "fancy « Web2.0 » widgets".

1. Django shouldn't do anything (besides what django.contrib.staticfiles already does) for projects that use webpack (or any other bundler)

The correct approach with a bundler is to bundle all JavaScript code and, if needed, to optimize with code splitting. Attempting to include only what's needed on each page, like Media does, will usually be counterproductive, because it will general different bundle for different pages and defeat caching by the browser.

Then I know two techniques for integrating the frontend code with Django:

- the single page app (website and API on separate domains) — this is clearly out of scope of this discussion as Django only provides an API in this scenario

- what I call the hybrid app (website and API on the same domain) — here django.contrib.staticfiles helps; it comes after the JavaScript bundler in the deployment pipeline (and I prefer plain django.contrib.staticfiles over django-webpack-loader, but that's another story)

(NB: while the blog posts I just linked to focus on create-react-app, the concepts apply to any modern JavaScript toolchain.)

Regardless, Django already provides more than we need. For example, both webpack and django.contrib.staticfiles add hashes to file names to ensure cache invalidation.

So my answer to questions 3, 4 and 5 is "no, except maybe documentation".

Now, let's leave this brave new world and remember the jQuery era.

2. Media makes sense

Although pluggable apps are fantastic, the concept doesn't work well for templates, which is where <link rel="stylesheet"> and <script> tags are written. As a consequence, there's no good way for a pluggable app to add CSS and JS for rendering a rich field, that is, a better <input>.

Pluggable apps could provide custom template tags for rich fields — it's fairly easy with inclusion_tag — but that would require writing a specific template for each form. That wouldn't support rendering all fields in a loop like the admin does. This is where Form Media fills the gap.

Form Media is a data structure representing a list of <link rel="stylesheet"> tags and a list of <script> tags, nothing more, nothing less. It knows how to render itself to HTML. When merging two instances (or extending one), it deduplicates tags while maintaining their order.

Form Media is the only place where Django renders <link> or <script> tags in Python. Everywhere else, these tags are written in Django templates. This is consistent with my observation that Django templates don't provide a convenient solution for pluggable rich fields.

For websites that haven't adopted a site-wide asset pipeline, this is useful, self-contained, well-defined functionality. It doesn't have to fit into a bigger design.

3. Areas for improvement

The name is wrong — it should be Static — which is why I originally filed the ticket. This is a backwards-incompatible change. I haven't thought about how a deprecation path could work.

The docs describe a fallback to MEDIA_URL which, as far as I can tell, no longer exists. This paragraph should be removed.

The API would be simpler if `css` was a list rather than a dict of lists keyed by media. Media-specific styles could be written inside a media query. Accepting lists in addition to the more verbose, dict-based format would be convenient and backwards-compatible.

4. Could Django do more?

I don't believe Django should have the ability to compile and compress assets. It's a messy problem; we don't have the resources to tackle it. I'd rather leave it to third-party apps like we currently do.

To me, the key question is — is there something useful that Django can do beyond what it currently does, short of providing an asset pipeline?

The discussion often revolves about listing which assets to include in a page:

- Media is basically a list of assets

- Claude's work was about letting apps describe a list of assets

- django-compressor extracts lists of assets by parsing HTML inside {% compress %} tags, if memory serves

- django-pipeline asks the user to include a list of assets in a setting

Perhaps Django could standardize a way to accumulate a list of CSS and JS assets to include in a page, which could then be rendered in HTML, perhaps after optimizations (provided by third party apps).

Essentially this would solve the same problem as django-sekizai and could provide a better integration point for django-compressor.

To move forwards, we'd need a better description of the problem to be solved and of the expected benefits of the proposed solution. This sounds a lot like a DEP ;-)

If we did that, we'd still need the Media (or Static) class on forms, except it would feed its data into this new system instead of rendering HTML. Since it's here to stay, we should proceed with the renaming.

I hope this helps!

-- 

Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/bbfaffce-f6e2-4fea-b21c-34db7886d153%40googlegroups.com.

[1337 Shadow Hacker]

Le samedi, mai 9, 2020 10:39 PM, Aymeric Augustin <aymeric....@polytechnique.org> a écrit :

Perhaps Django could standardize a way to accumulate a list of CSS and JS assets to include in a page, which could then be rendered in HTML, perhaps after optimizations (provided by third party apps).

The Cubic web framework has something like response.scripts and response.styles where you introspect and attach or remove stuff along the way.

Typically, if two different apps ship their own jquery.js then things are not going to work, one jquery will override the other and plugins will not be registered anymore.

So in Cubic at least your lib can check if there's a jquery js prior to adding your own. It's no silver bullet though.

[David Smith]

Hi All,

Thank you all for your time taken to read and respond to this topic. 

Based on the conversation I'll try and summarise to try and gain wider approval. 

-  There is a valid use case for form media so it should not be deprecated

-  The name is wrong so we should proceed with the rename

-  There are some areas where the current feature can be improved and we should progress with these (some items noted on this thread + other open tickets on trac)

-  If we wish to 'do more' it requires much more thought and a DEP. A DEP should most likely focus on building blocks rather than a full solution. 

The key thing here in the short term is the name change. If there are no objections to this I am happy to look at an implementation. Could then come back to the mailing list before it's merged, especially as it will be a breaking change?

Kind Regards

David

-- 

Aymeric.

To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

[1337 Shadow Hacker]

Web components are now standard HTML without JS frameworks, so that could be supported by Django. In which case, even StencilJS tsx components would work out of the box.

Prior to rendering, a middleware could scan the response and add the registered scripts/styles for the custom HTML tags it finds.

This would be a lot more convenient than form media, more portable and cover the full spectrum of use cases.