Intent to Prototype: AVIF Decode

[Jai Krishnan]

Title

AVIF Decode

Contact emails

ja...@google.com, dalec...@google.com, w...@google.com

Specification

https://aomediacodec.github.io/av1-avif/

TAG review

https://github.com/w3ctag/design-reviews/issues/495

Chromium bug

https://bugs.chromium.org/p/chromium/issues/detail?id=960620

Summary

Enable native AVIF decode support (with the existing AV1 video decoder). 

Motivation

AVIF offers two major benefits for images served on the web:

1. AVIF offers significant file size reduction for images compared with JPEG or WebP. Prior to full optimization, Netflix published results on their test set showing ~50% savings vs JPEG across use cases, going past 60% for 4:4:4 content. This reduction helps load pages faster and reduce overall data consumption: To quote from Essential Image Optimization by Addy Osman: “Images are still the number one cause of bloat on the web. Images take up massive amounts of internet bandwidth because they often have large file sizes. According to the HTTP Archive, 60% of the data transferred to fetch a web page is images composed of JPEGs, PNGs and GIFs. As of July 2017, images accounted for 1.7MB of the content loaded for the 3.0MB average site.” Smaller file sizes will benefit users, developers, and CDN / pipeline services. 

2. AVIF is a path to HDR image support for the web. JPEG is limited in practice to 8-bit color depth. With displays increasingly capable of higher brightness, color bit depth, and color gamuts, web stakeholders are increasingly interested in preserving image data that is lost with JPEG.

Why AVIF instead of other competing formats? 

• Part of the Alliance for Open Media’s effort to offer AV1 for free, a key tenet of the open web and a reason JPEG has succeeded where successor formats have failed to gain traction. 

• Widespread decoder availability. AVIF decoders can leverage the already growing AV1 video decode ecosystem, including sharing the same decoder binary in Chrome. 

• Perhaps most importantly, there is significant interest from content serving sites.  Facebook, Netflix, YouTube and others have expressed interest in AVIF and said they use a decoder available natively on the web. Alternatives like a JS / WASM implementation would largely undermine the efficiency benefits of the format. 

Risks

Interoperability and Compatibility

Edge: Public support

Firefox:In development

Safari:No signals

Web / Framework developers: Active interest on crbug 

Ergonomics

N/A

  

Activation

Active library work related to this feature already exists - AVIF decode implementations through libavif are working with multiple AV1 video decoders, and there are multiple related new open and closed source projects in development.

Debuggability

N/A

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes, but not immediately on Android. Support will be gated on a DFM for the AV1 decoder, with default native support across desktop platforms (Windows, Mac, Linux, ChromeOS). If AV1 gets default support on Android via increased demand outweighing binary size and expected performance limitations on smart phones (battery drain considerations, etc), the support would extend to AVIF on Android as well. 

Is this feature fully tested by web-platform-tests?

No, will complete before requesting to ship

Link to entry on the feature dashboard

No dashboard entry planned until requesting approval to ship.

Requesting approval to ship?

Not at this time, will prioritize testing with interested partner sites first.

[PhistucK]

Just to summary this for others because it was not immediately obvious -

This intent is for prototyping support for a new image format (like PNG, for example) that is based on the AV1 video codec (much like WebP is based on the VP8 codec and HEIF is based on the n HEVC/H.265 codec).

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKQBk4w8naY_mRdAAm9xUNy2e5%3D4V65yueccOZeEa7i3U8OGnQ%40mail.gmail.com.

[Christian Biesinger]

Out of curiosity, what is the file extension and mime type for such images?

Christian

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_JqmgJa5RCae2cZNfZXn6rkxjG%3DGGhnU1tpPnBwrSaUWw%40mail.gmail.com.

[Dale Curtis]

https://aomediacodec.github.io/av1-avif/#avif-mime-definition 

image/avif (potentially also image/avif-sequence) and an extension of .avif.

- dale 

[Yoav Weiss]

Thanks for working on this. Exciting times! :)

On Wed, Apr 8, 2020 at 11:58 PM Jai Krishnan <ja...@chromium.org> wrote:

Title

AVIF Decode

Contact emails

ja...@google.com, dalec...@google.com, w...@google.com

Specification

https://aomediacodec.github.io/av1-avif/

TAG review

https://github.com/w3ctag/design-reviews/issues/495

Chromium bug

https://bugs.chromium.org/p/chromium/issues/detail?id=960620

Summary

Enable native AVIF decode support (with the existing AV1 video decoder). 

Motivation

AVIF offers two major benefits for images served on the web:

1. AVIF offers significant file size reduction for images compared with JPEG or WebP. Prior to full optimization, Netflix published results on their test set showing ~50% savings vs JPEG across use cases, going past 60% for 4:4:4 content. This reduction helps load pages faster and reduce overall data consumption: To quote from Essential Image Optimization by Addy Osman: “Images are still the number one cause of bloat on the web. Images take up massive amounts of internet bandwidth because they often have large file sizes. According to the HTTP Archive, 60% of the data transferred to fetch a web page is images composed of JPEGs, PNGs and GIFs. As of July 2017, images accounted for 1.7MB of the content loaded for the 3.0MB average site.” Smaller file sizes will benefit users, developers, and CDN / pipeline services. 

2. AVIF is a path to HDR image support for the web. JPEG is limited in practice to 8-bit color depth. With displays increasingly capable of higher brightness, color bit depth, and color gamuts, web stakeholders are increasingly interested in preserving image data that is lost with JPEG.

Why AVIF instead of other competing formats? 

• Part of the Alliance for Open Media’s effort to offer AV1 for free, a key tenet of the open web and a reason JPEG has succeeded where successor formats have failed to gain traction. 

• Widespread decoder availability. AVIF decoders can leverage the already growing AV1 video decode ecosystem, including sharing the same decoder binary in Chrome. 

• Perhaps most importantly, there is significant interest from content serving sites.  Facebook, Netflix, YouTube and others have expressed interest in AVIF and said they use a decoder available natively on the web. Alternatives like a JS / WASM implementation would largely undermine the efficiency benefits of the format. 

Risks

Interoperability and Compatibility

Edge: Public support

This link represents (rumors of) support from Windows in general, but would be good to confirm if Edge is planning to support the format as a web exposed one.

Firefox:In development

While encouraging, the issue seems to be a fairly recent one.

Might be worthwhile to file an issue on the Mozilla standards positions repo for their official opinion. 

Safari:No signals

Have we asked? 

Web / Framework developers: Active interest on crbug 

Ergonomics

N/A

  

Activation

Active library work related to this feature already exists - AVIF decode implementations through libavif are working with multiple AV1 video decoders, and there are multiple related new open and closed source projects in development.

Debuggability

N/A

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes, but not immediately on Android. Support will be gated on a DFM for the AV1 decoder, with default native support across desktop platforms (Windows, Mac, Linux, ChromeOS). If AV1 gets default support on Android via increased demand outweighing binary size and expected performance limitations on smart phones (battery drain considerations, etc), the support would extend to AVIF on Android as well. 

Is this feature fully tested by web-platform-tests?

No, will complete before requesting to ship

Link to entry on the feature dashboard

No dashboard entry planned until requesting approval to ship.

Requesting approval to ship?

Not at this time, will prioritize testing with interested partner sites first.

[Yoav Weiss]

On Thu, Apr 9, 2020 at 1:48 AM Dale Curtis <dalec...@chromium.org> wrote:

https://aomediacodec.github.io/av1-avif/#avif-mime-definition 

image/avif (potentially also image/avif-sequence) and an extension of .avif.

Would be good to verify, as part of the implementation, that:

* Our `Accept` headers for image destinations reflect that MIME type.

* Support for `type` as part of `<picture>` does the same.

Are we planning to ship the full set of features in one go? Or does AVIF have different "levels" of support?

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPUDrwfDT4k9msGRf%3Dt4yMsbHZh-0m60UVWkSFF9MFX7hMdwOg%40mail.gmail.com.

[Dale Curtis]

On Thu, Apr 9, 2020 at 12:37 AM Yoav Weiss <yo...@yoav.ws> wrote:

On Thu, Apr 9, 2020 at 1:48 AM Dale Curtis <dalec...@chromium.org> wrote:

https://aomediacodec.github.io/av1-avif/#avif-mime-definition 

image/avif (potentially also image/avif-sequence) and an extension of .avif.

Would be good to verify, as part of the implementation, that:

* Our `Accept` headers for image destinations reflect that MIME type.

* Support for `type` as part of `<picture>` does the same.

Are we planning to ship the full set of features in one go? Or does AVIF have different "levels" of support?

We plan to implement full support from the get go, but some pixel formats and color spaces may be a bit slow to display until we optimize the transforms.

- dale 

[Daniel Bratell]

You compare the new format to jpeg. How does it compare to WebP?

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACj%3DBEg_6WOArLYTYbY6Jt9o1mMi%2BfowAggeOAMrWjnPfcoU7Q%40mail.gmail.com.

[Dale Curtis]

On Thu, Apr 9, 2020 at 11:51 AM Daniel Bratell <brat...@gmail.com> wrote:

You compare the new format to jpeg. How does it compare to WebP?

/Daniel 

https://compare.rokka.io/_compare/ has some informal results. Since WebP is ~VP8, AV1 should outperform it significantly. We do expect lower overhead from the webp container than iso-bmff though.

A formal analysis will be done prior to ship; there are still some encoding aspects (4:4:4) which need improvement first.

- dale  

[Daniel Bratell]

On 2020-04-09 21:23, Dale Curtis wrote:

On Thu, Apr 9, 2020 at 11:51 AM Daniel Bratell <brat...@gmail.com> wrote:

You compare the new format to jpeg. How does it compare to WebP?

/Daniel 

https://compare.rokka.io/_compare/ has some informal results. Since WebP is ~VP8, AV1 should outperform it significantly. We do expect lower overhead from the webp container than iso-bmff though.

I find it hard to see any relevant difference between webp and av1 on the default image side when setting parameters to make the same size image for jpeg, webp and av1. Someone with pickier eyes than I need to make that call. Jpeg is clearly worse though.

/Daniel

[Peter Kasting]

I have pickier eyes :).  The hair on the back side of the head, the brownish plant on the far right edge of the image (vertical middle), the bottom edge of the bright white top of the railing, and the jeans on the thigh fronts all look better in the AVIF version.

PK

[Wan-Teh Chang]

On Thu, Apr 9, 2020 at 12:37 AM Yoav Weiss <yo...@yoav.ws> wrote:
>

> On Thu, Apr 9, 2020 at 1:48 AM Dale Curtis <dalec...@chromium.org> wrote:
>>
>> https://aomediacodec.github.io/av1-avif/#avif-mime-definition
>>
>> image/avif (potentially also image/avif-sequence) and an extension of .avif.
>
> Would be good to verify, as part of the implementation, that:
> * Our `Accept` headers for image destinations reflect that MIME type.

I can do this. Thanks for the suggestion. Do you know which source
file I need to modify? Using cs.chromium.org, I found three files that
have a kImageAcceptHeader or kFrameAcceptHeaderValue string:

src/services/network/public/cpp/constants.cc
src/third_party/blink/renderer/platform/loader/fetch/url_loader/request_conversion.cc
src/third_party/blink/renderer/core/loader/alternate_signed_exchange_resource_info.cc

Should I modify all of them?

> * Support for `type` as part of `<picture>` does the same.

I don't understand this. Could you elaborate?

Thanks,
Wan-Teh

[Dale Curtis]

In addition to all that, the entire webp image has blocking -- it looks like it's on a grid.

- dale

 

[Joe Medley]

Actually, you do need a Chrome Status entry, especially if prototyping includes an origin trial or a period behind a flag. If you just use the Chrome Status form to create your intent, you get the entry for free. 

Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195

If an API's not documented it doesn't exist.

[Yoav Weiss]

On Thu, Apr 9, 2020 at 8:05 PM Wan-Teh Chang <w...@google.com> wrote:

On Thu, Apr 9, 2020 at 12:37 AM Yoav Weiss <yo...@yoav.ws> wrote:
>
> On Thu, Apr 9, 2020 at 1:48 AM Dale Curtis <dalec...@chromium.org> wrote:
>>
>> https://aomediacodec.github.io/av1-avif/#avif-mime-definition
>>
>> image/avif (potentially also image/avif-sequence) and an extension of .avif.
>
> Would be good to verify, as part of the implementation, that:
> * Our `Accept` headers for image destinations reflect that MIME type.

I can do this. Thanks for the suggestion. Do you know which source
file I need to modify? Using cs.chromium.org, I found three files that
have a kImageAcceptHeader or kFrameAcceptHeaderValue string:

src/services/network/public/cpp/constants.cc

This string would need to be changed for navigation requests - indicating the server that AVIF is supported when the user browses directly to an image.

 

src/third_party/blink/renderer/platform/loader/fetch/url_loader/request_conversion.cc

This string would need to be changed for subresource requests for image destinations.

  

src/third_party/blink/renderer/core/loader/alternate_signed_exchange_resource_info.cc

I believe this is needed to make sure prefetched images match their WebPackage equivalents.

+Hayato Ito +Kinuko Yasuda to confirm.

  

Should I modify all of them?

I believe so. Would also be good to add tests that verify that requests are sent with the right values for these 3 cases.

 

> * Support for `type` as part of `<picture>` does the same.

I don't understand this. Could you elaborate?

It seems like AVIF already adds itself to the supported MIME types when the flag is on, so that seems good. Would be good to add tests that make sure e.g. `<picture><source srcset="foo" type="image/avif"><img></picture>` actually load a resource.

 

Thanks,
Wan-Teh

[Anne van Kesteren]

On Sun, Apr 12, 2020 at 7:25 PM Yoav Weiss <yo...@yoav.ws> wrote:
> This string would need to be changed for navigation requests - indicating the server that AVIF is supported when the user browses directly to an image.

Note that this practice has lead to issues in the recent past:
https://github.com/whatwg/fetch/issues/274#issuecomment-545825881.
Apart from that issue it's not clear to me that it's necessarily good
to keep expanding the set of bytes transmitted for each time the
browser navigates (or requests an image, for that matter).

I think it would be good to standardize what the expectations are around this.

[Yoav Weiss]

On Tue, Apr 14, 2020 at 10:45 AM Anne van Kesteren <ann...@annevk.nl> wrote:

On Sun, Apr 12, 2020 at 7:25 PM Yoav Weiss <yo...@yoav.ws> wrote:
> This string would need to be changed for navigation requests - indicating the server that AVIF is supported when the user browses directly to an image.

Note that this practice has lead to issues in the recent past:
https://github.com/whatwg/fetch/issues/274#issuecomment-545825881.

That issue seems to just be content negotiation WAI: if you don't advertize support for a format, servers will not serve it. 

The alternative to that is UA sniffing, which is less than ideal.

Apart from that issue it's not clear to me that it's necessarily good
to keep expanding the set of bytes transmitted for each time the
browser navigates (or requests an image, for that matter).

Might be interesting to come up with an alternative proposal (based on Client Hints?), where servers only ask for the formats they care about.

I'd be supportive of something like that.

 

I think it would be good to standardize what the expectations are around this.

I agree. 

[Christian Biesinger]

On Tue, Apr 14, 2020 at 4:55 AM Yoav Weiss <yo...@yoav.ws> wrote:
>
>
>
> On Tue, Apr 14, 2020 at 10:45 AM Anne van Kesteren <ann...@annevk.nl> wrote:
>>
>> On Sun, Apr 12, 2020 at 7:25 PM Yoav Weiss <yo...@yoav.ws> wrote:
>> > This string would need to be changed for navigation requests - indicating the server that AVIF is supported when the user browses directly to an image.
>>
>> Note that this practice has lead to issues in the recent past:
>> https://github.com/whatwg/fetch/issues/274#issuecomment-545825881.
>
>
> That issue seems to just be content negotiation WAI: if you don't advertize support for a format, servers will not serve it.
> The alternative to that is UA sniffing, which is less than ideal.

From reading the bug, the issue seems a bit more complicated: servers
seem to use the presence of image/webp in the *navigation* request to
determine whether to put an <img src="foo.webp"> in the HTML they
generate. It's not entirely clear to me that this is how content
negotiation is supposed to work. I don't know why servers don't use
content negotiation for the image requests themselves instead....

Christian

[Yoav Weiss]

That can still be a legitimate use case (e.g. in cases where the HTML serving is dynamic, but the images are not, and for some reason adding `<picture>` tags with the multiple image format variants is more complex).

 Although I agree we're sending the image format in those cases to adapt image documents, not HTML ones.

[PhistucK]

Forget about its complexity, <picture> uses considerably more bytes than "image/avif; q=0.9" (or whatever it is) does.

Right, it might be irrelevant to most pages (so those are 17 wasteful bytes, or even less with header compression), but the pages that do care about this will save hundreds of bytes (or thousands, if there are a lot of those).

☆PhistucK

[Ian Clelland]

On Thu, Apr 16, 2020 at 10:21 AM PhistucK <phis...@gmail.com> wrote:

Forget about its complexity, <picture> uses considerably more bytes than "image/avif; q=0.9" (or whatever it is) does.

Right, it might be irrelevant to most pages (so those are 17 wasteful bytes, or even less with header compression), but the pages that do care about this will save hundreds of bytes (or thousands, if there are a lot of those).

The calculus here though, is that by accepting this as a legitimate form of content negotiation, the browser needs to send details of every format that it can handle in *any context* with every navigation request, because any navigation *could* be to an HTML resource that will make use of that data. This might significantly reduce egress data costs for a server which:

  - Generates HTML dynamically after inspecting headers

  - Includes many image tags on every page

  - Cannot do similar content-negotiation on image resources (say, linking to /foo rather than /foo.webp, and switching out the returned content based on *those* headers)

  - Can't compress their response bodies, through TLS or even Transport-Encoding:gizp, so every extra <picture> counts

But this is at the cost of adding those header bytes to every other navigation on the Internet, the vast majority of which are to pages which are not in this situation. (And don't forget that we are *also* sending the extra header bytes on the request to /foo.wepb for the hypothetical server above, even though it's already learned that information on the initial navigation request)

☆PhistucK

On Thu, Apr 16, 2020 at 4:09 PM Yoav Weiss <yo...@yoav.ws> wrote:

On Tue, Apr 14, 2020 at 7:46 PM Christian Biesinger <cbies...@chromium.org> wrote:

On Tue, Apr 14, 2020 at 4:55 AM Yoav Weiss <yo...@yoav.ws> wrote:
>
>
>
> On Tue, Apr 14, 2020 at 10:45 AM Anne van Kesteren <ann...@annevk.nl> wrote:
>>
>> On Sun, Apr 12, 2020 at 7:25 PM Yoav Weiss <yo...@yoav.ws> wrote:
>> > This string would need to be changed for navigation requests - indicating the server that AVIF is supported when the user browses directly to an image.
>>
>> Note that this practice has lead to issues in the recent past:
>> https://github.com/whatwg/fetch/issues/274#issuecomment-545825881.
>
>
> That issue seems to just be content negotiation WAI: if you don't advertize support for a format, servers will not serve it.
> The alternative to that is UA sniffing, which is less than ideal.

From reading the bug, the issue seems a bit more complicated: servers
seem to use the presence of image/webp in the *navigation* request to
determine whether to put an <img src="foo.webp"> in the HTML they
generate. It's not entirely clear to me that this is how content
negotiation is supposed to work. I don't know why servers don't use
content negotiation for the image requests themselves instead....

That can still be a legitimate use case (e.g. in cases where the HTML serving is dynamic, but the images are not, and for some reason adding `<picture>` tags with the multiple image format variants is more complex).

 Although I agree we're sending the image format in those cases to adapt image documents, not HTML ones.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_KRbyBVg5jnmrijYaXxqTkTYE7yyhTyEYp%2BVbKq%3DEZgqA%40mail.gmail.com.

[Yoav Weiss]

That sounds like an argument in favor of better content negotiation... <cough>client hints</cough>

Just need to solve the first view problem

[Artur Janc]

Does this intent introduce a new file format parser, or is all the heavy lifting going to be done by the existing AV1 decoder?

If not, are you planning to check in a fuzzer for it (https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/README.md)?

[Wan-Teh Chang]

On Thu, Apr 16, 2020 at 8:17 AM Artur Janc <a...@google.com> wrote:
>
> Does this intent introduce a new file format parser, or is all the heavy
> lifting going to be done by the existing AV1 decoder?

This intent introduces a new file format parser. The AVIF file format
puts AV1 images in the HEIF image file format.

The HEIF image file format is parsed by third_party/libavif,
specifically third_party/libavif/src/src/read.c. AV1 images are parsed
by the existing AV1 decoder in third_party/dav1d.

> If not, are you planning to check in a fuzzer for it (https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/README.md)?

Both libavif and dav1d are tested by oss-fuzz:

https://github.com/google/oss-fuzz/tree/master/projects/libavif
https://github.com/google/oss-fuzz/tree/master/projects/dav1d

Wan-Teh

[Artur Janc]

That's great to hear -- thank you, Wan-Teh!