RFC732 (a.k.a. brotli) support

[Johannes Hoppe]

Hi there,

we all know the the wonderful GZip middleware that allows response compression. Which is great especially if you are serving large responses.

Now gzip is only one of many supported encoding types supported by many browsers. The coolest kid on the block seems to be brotli right now.

It is both faster and smaller in most cases then gzip and has been standardized in https://www.ietf.org/rfc/rfc7932.txt

Of course there is already a package that adds brotli support to Django:

https://pypi.org/project/django-brotli/

But I would propose to refactor the GZip middleware into something like "CompressionMiddlware" and add support for both GZip and Brotli

in the same middleware.

I have the feeling something basing like response encoding should be build into Django. Of course I would not add a hard dependency to brotli,

but add all methods to `Accept-Encoding` that are actually installed.

What do you guys think? Is this worth a little refactoring?

Best

-Joe

[Adam Johnson]

Sounds like a good idea to me.

Would you be looking at merging any code from django-brotli, and have you been in contact with its creator and license holder Vasek Dohnal?

And what kind of methods around Accept-Encoding are you imagining?

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/29cf5142-fa82-4d74-9d8f-5fb0c8be34d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Adam

[Aymeric Augustin]

Hello,

I would like to be sure that Brotli is actually faster. In my experience, while decompression is fast, compression can be extremely slow.

This is fine for static assets compressed by a build pipeline or at least when the compressed version is cached. It would be a problem for on-the-fly compression of dynamic content, which is what we're talking about here.

I expect there's a setting for trading off compression time vs. compression ratio. It should be set appropriately. It isn't worth spending a lot more CPU to save a tiny bit of data transfer. The result could be slower, especially for users with fast connections.

I would like to see a quick performance benchmark of gzip as currently implemented in Django vs. brotli with a setting you recommend, to see the effect on compression time (CPU use) and compression ratio.

Best regards,

-- 

Aymeric.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/29cf5142-fa82-4d74-9d8f-5fb0c8be34d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Aymeric.

[Curtis Maloney]

On 09/14/2018 08:24 PM, Aymeric Augustin wrote:
> Hello,
>
> I would like to be sure that Brotli is actually faster. In my
> experience, while decompression is fast, compression can be extremely slow.

https://blogs.akamai.com/2016/02/understanding-brotlis-potential.html

It's primarily tuned for fast decompression, making it especially good
for offline compression (i.e. compression of static assets on deploy)

At lower levels it can get as good compression as gzip, but much faster,
or similar speed for better results. IIRC the general "wisdom" is that
brotli at level 4 is typically faster than gzip, and gives better results.

I like the general idea of an "extensible" or plugin driven
Content-Encoding handler, though it would require at least some
preferencing engine so you can influence which algorithm is chosen by
(a) browser supported, (b) local preference, and (c) content type.

--
Curtis

[Johannes Hoppe]

Hi Adam,

no I would not take any of his code. I would certainly contact him as a reviewer or co-author of a possible patch.

In general I would like to consolidate the two middlewares into a single response compression middleware that support multiple algorithms. All except gzip would require users to install additional dependencies. But I believe this to be acceptable. Same as you need to install Pillow if you want Image support in Django.

Best

-Joe

[Johannes Hoppe]

Good Point Aymeric,

it will probably not always be a good idea. I would require some kind of configuration to define which response content-type should be encoded by which algorithm.

It might also be interesting if you could use pre-compressed full page caching. Which I think does currently not work. This could however be a good way to reduce CPU time.

Curtis also makes a good point on what kind of configraution should be supported by such a middlware.

Best

-Joe

[Johannes Hoppe]

Hi Curtis,

very good remarks. I would make sense to provide the best possible preset for such a middleware. That could even be content-type sensitive.

I would however add any settings to overwrite the preset. I believe if someone want to mingle with those things, one should inherit and override.

But even with a preset, I would follow content negotiation tactics, where the request can define preference over the content encoding.

https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3

Best

-Joe

[Adam Johnson]

I would require some kind of configuration to define which response content-type should be encoded by which algorithm.

I think there's room there to support custom tactics, for example content length may also be useful to determine which algorithms are applicable, or perhaps compression could be disabled for certain URL's. GZipMiddleware doesn't support this but if we're going to add flexibility we should make it easy to customize, at least by subclassing.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3673361a-dc59-4d27-8050-1ab2cae5d11c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam

[Václav Dohnal]

Hello everyone and sorry for the late response. I'm the author of django-brotli package. Creating a single middleware that can handle multiple algorithms sounds like a great idea to me. If you need any kind of assistance and/or code from django-brotli package, I'll be happy to help you.

Vašek Dohnal

Dne pátek 14. září 2018 10:01:00 UTC+2 Johannes Hoppe napsal(a):