Django-Pipeline and WhiteNoise

[Paul McLanahan]

In the new spirit of information sharing and "Best Practices[tm]" I
wanted to take a moment to introduce those of you who've not heard of
these packages and are using Django (Playdoh).

## Django-Pipeline [0] ##

It's a JS/CSS aggregator, compiler, and minifier in the vein of
django-compressor. It's more like jingo-minify in that you create your
media bundle definitions in your settings file instead of the templates.
This means no dependence on lxml and cumbersome parsing of HTML with
python, as well as being one central spot to easily see how your media
is laid out so you can spot redundancies or slight variations that can
cause large un-cached downloads for browsers. It also centralizes the
templates for your media tags, which are customizable globally or on a
per-bundle basis. It supports compilation with all of the major CSS and
JS pre-processors, as well as making it easy to add your own. It even
supports easy inclusion of JS templates and various JS templating
systems (though a severe lack of Nunjucks). And of course it supports
both Django Templates and Jinja2.

Perhaps the best part is that it seamlessly integrates with Django's
staticfiles framework (thanks Jezdez!). The whole thing runs as a hook,
so the only command that needs to be run is "./manage.py collectstatic",
and it automatically understands any changes made by any staticfiles
backend mixins.

## WhiteNoise [1] ##

This is a WSGI middleware (so not Django specific, though does have some
Django hotness built in) that serves your static media via your WSGI
process(es). I know, that sounds daft, but hear me out.

You should be using a CDN. Seriously. So how fast (or not) the files are
served from your app-server is mostly irrelevant (though it uses the
sendfile system calls to serve the files so it isn't slow). The
advantages you get, however, are many.

1. When used with Django and the staticfiles backends that add a hash to
the names of your files, it understands that naming convention and adds
very long expires headers automatically to those files, along with a
configurable default expire to the others (1 hour by default).
2. Provides a backported ManifestStorage backend from Django 1.7 that
uses a JSON manifest file to store and discover your hashed names
instead of on-the-fly calculation or using the cache.
3. Provides a gzip version of the backend from #2 that will pre-gzip all
of your appropriate files, then serve those if the request has the
correct Accept-Encoding header and adds the proper Vary header to the
response.
4. Allows very easy local running of your full production site by
relegating Nginx (or Apache) to mostly just an SSL terminator and proxy.
5. Allows you to set a folder that will act as a place for files that
should be served from the root if requested (e.g. robots.txt or
favicon.ico).

More is in the docs of course. I've submitted an issue to Playdoh [2] so
we can discuss this stack as a possible option along-site or replacement
for django-compressor and jingo-minify.

[0] http://django-pipeline.readthedocs.org/
[1] http://whitenoise.evans.io/
[2] https://github.com/mozilla/playdoh/issues/173

This stack is currently running quite successfully on https://scrumbu.gs
if you want to see it in action. I'm very interested to hear what you
think either here or in the issue [2].

Thanks,

--
Paul [pmac] McLanahan
Web Developer
Mozilla Web Productions

[Wil Clouser]

These both sound interesting to me. We're pushing for simplicity in setting up
the Marketplace for the first time and even though we're pushing for a
pre-configured VM, something like WhiteNoise could make that simpler by not
having to pre-fill nginx rules.

Since WhiteNoise enables gzip, I'll ask - is BREACH still a concern by default?
Last year django recommended disabling gzip by default[1] to avoid the attack.
We've only recently re-enabled it on the Marketplace API (because private data
isn't exchanged there). Is gzip something we're encouraging by default (read:
in Playdoh) again?

Cheers,

Wil


[1] https://www.djangoproject.com/weblog/2013/aug/06/breach-and-django/

> _______________________________________________
> dev-webdev mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-webdev

[Peter Bengtsson]

Paul McLanahan wrote:
> In the new spirit of information sharing and "Best Practices[tm]" I
> wanted to take a moment to introduce those of you who've not heard of
> these packages and are using Django (Playdoh).
>
> ## Django-Pipeline [0] ##
>
> It's a JS/CSS aggregator, compiler, and minifier in the vein of
> django-compressor. It's more like jingo-minify in that you create your
> media bundle definitions in your settings file instead of the templates.
> This means no dependence on lxml and cumbersome parsing of HTML with
> python, as well as being one central spot to easily see how your media
> is laid out so you can spot redundancies or slight variations that can
> cause large un-cached downloads for browsers. It also centralizes the
> templates for your media tags, which are customizable globally or on a
> per-bundle basis. It supports compilation with all of the major CSS and
> JS pre-processors, as well as making it easy to add your own. It even
> supports easy inclusion of JS templates and various JS templating
> systems (though a severe lack of Nunjucks). And of course it supports
> both Django Templates and Jinja2.
>
> Perhaps the best part is that it seamlessly integrates with Django's
> staticfiles framework (thanks Jezdez!). The whole thing runs as a hook,
> so the only command that needs to be run is "./manage.py collectstatic",
> and it automatically understands any changes made by any staticfiles
> backend mixins.
>

I skimmed through the documentation.

* It looks fresh and solid!

* I actually really like deciding which scripts and css to use when I'm
in the context of the template but I admit to sometimes losing control
over which jquery-ui or select2 versions I use where.

* Has anybody tried django-pipeline with jingo?

* I don't get why you'd want to specify the `output_filename`. Because,
who cares?! The output filename should be a md5 hash of the combined
content + .js|.css.

* If you don't use .less, a nice thing about django-compressor was that
you didn't need node since it would use python packages jsmin and cssmin.

* Reading the docs, it seems really easy to write your own compiler so
Nunjucks wouldn't be a problem for very long.

> ## WhiteNoise [1] ##
>
> This is a WSGI middleware (so not Django specific, though does have some
> Django hotness built in) that serves your static media via your WSGI
> process(es). I know, that sounds daft, but hear me out.
>
> You should be using a CDN. Seriously. So how fast (or not) the files are
> served from your app-server is mostly irrelevant (though it uses the
> sendfile system calls to serve the files so it isn't slow). The
> advantages you get, however, are many.
>
> 1. When used with Django and the staticfiles backends that add a hash to
> the names of your files, it understands that naming convention and adds
> very long expires headers automatically to those files, along with a
> configurable default expire to the others (1 hour by default).
> 2. Provides a backported ManifestStorage backend from Django 1.7 that
> uses a JSON manifest file to store and discover your hashed names
> instead of on-the-fly calculation or using the cache.
> 3. Provides a gzip version of the backend from #2 that will pre-gzip all
> of your appropriate files, then serve those if the request has the
> correct Accept-Encoding header and adds the proper Vary header to the
> response.
> 4. Allows very easy local running of your full production site by
> relegating Nginx (or Apache) to mostly just an SSL terminator and proxy.
> 5. Allows you to set a folder that will act as a place for files that
> should be served from the root if requested (e.g. robots.txt or
> favicon.ico).
>

* Do we have a CDN we can use easily? I know bedrock uses one and AMO too?

* I've argued before that a highly tuned nginx or apache is "obsolete"
when you have a smart CDN.
http://www.peterbe.com/plog/is-nginx-obsolete-amazon-cloudfront
WhiteNoise fits that perfectly.

* Careful with putting your CSS on a CDN though! The CDN might be fast
and all but it can cause another DNS lookup which is slow when your
browser is ready to start rendering the HTML and if it gets blocked on
some .css it'll wait with displaying anything on the screen.

* I really like the idea of taking over the static files configuration
business stuff instead of relying on filing IT bugs to have certain
rewrites and cache-control headers messed with. Will WhiteNoise solve that?

[Peter Bengtsson]

I wrote a followup re django-pipeline here:
https://github.com/mozilla/playdoh/issues/173

Let's continue the discussion there shall we?

[Paul McLanahan]

Great response Peter! Thanks! Responses below.

Peter Bengtsson wrote:
> * Has anybody tried django-pipeline with jingo?

Yes. Works great. I do this in Scrumbugs [0].

[0] https://github.com/mozilla/scrumbugz/blob/master/settings/base.py#L88

> * I don't get why you'd want to specify the `output_filename`. Because,
> who cares?! The output filename should be a md5 hash of the combined
> content + .js|.css.

I'm with ya. I'm all for configurability, but a sane default with the
ability to override would be great here. Perhaps that's something to
contribute.

> * If you don't use .less, a nice thing about django-compressor was that
> you didn't need node since it would use python packages jsmin and cssmin.

You can configure any of the many available backends, some of which are
python (jsmin and slimit off the top of my head).

> * Reading the docs, it seems really easy to write your own compiler so
> Nunjucks wouldn't be a problem for very long.

Agreed. Another useful potential contribution upstream.

>> ## WhiteNoise [1] ##

>
> * Do we have a CDN we can use easily? I know bedrock uses one and AMO too?

Yes. We have relationships with a few. As far as I know you just need to
ask IT to hook you up. But if you're on Mozilla infra your prod instance
should be behind Zeus, which would help you w/ the good cache headers
regardless.

> * I've argued before that a highly tuned nginx or apache is "obsolete"
> when you have a smart CDN.
> http://www.peterbe.com/plog/is-nginx-obsolete-amazon-cloudfront
> WhiteNoise fits that perfectly.

Yes! This is it exactly. Nice work.

> * Careful with putting your CSS on a CDN though! The CDN might be fast
> and all but it can cause another DNS lookup which is slow when your
> browser is ready to start rendering the HTML and if it gets blocked on
> some .css it'll wait with displaying anything on the screen.

That is true, but you gain some of that back because the browser won't
send all of the cookies and whatnot to the alternate domain, as well as
it being only a single extra DNS lookup if you use that CDN for all of
your statics. You are indeed hosed if it goes down, but changing the
STATIC_URL setting to be local again is pretty quick and easy if you
have to.

> * I really like the idea of taking over the static files configuration
> business stuff instead of relying on filing IT bugs to have certain
> rewrites and cache-control headers messed with. Will WhiteNoise solve that?

Depends what you mean exactly, but I think it will solve much of that, yes.

/pmac