VP9 Color Space

[fnordware]

How are we to use the color space encoder parameter? On the surface, it seems like it would just put a label in the bitstream, but then I notice if you set it to sRGB, write_bitdepth_colorspace_sampling() expects you use use profile 1, or 3, presumably with 4:4:4 sampling. Is it also expected to be RGB instead of YUV?

So what's the spec?

Brendan

[Ronald S. Bultje]

Hi,

On Tue, Nov 10, 2015 at 10:38 PM, fnordware <fnor...@gmail.com> wrote:

How are we to use the color space encoder parameter? On the surface, it seems like it would just put a label in the bitstream, but then I notice if you set it to sRGB, write_bitdepth_colorspace_sampling() expects you use use profile 1, or 3, presumably with 4:4:4 sampling. Is it also expected to be RGB instead of YUV?

The YUV types are indeed just labels. The sRGB one is special as you point out, and the answer to each of the above questions is yes.

In FFmpeg, we have assumed that the plane ordering of RGB input is GBR (i.e. "similar to" YCbCr), which is identical to how this (RGB colorspace substitution) works for other modern codecs. See also https://bugs.chromium.org/p/webm/issues/detail?id=1001

Ronald

[Kagami Hiiragi]

Btw, while we're talking about color spaces, could you please explain,
what does this issue mean:
https://bugs.chromium.org/p/webm/issues/detail?id=1087 ?

I'm able to correctly tag VP9 profile 0 videos, and read colorspace info
back. Is this about some other/edge case?

[Ronald S. Bultje]

Hi,

It's missing in profile 0 intraonly frames. I don't think libvpx emits these by default (it uses normal keyframes instead).

Ronald

[fnordware]

On Wednesday, November 11, 2015 at 3:37:27 AM UTC-8, rbultje wrote:

The YUV types are indeed just labels. The sRGB one is special as you point out, and the answer to each of the above questions is yes.

In FFmpeg, we have assumed that the plane ordering of RGB input is GBR (i.e. "similar to" YCbCr), which is identical to how this (RGB colorspace substitution) works for other modern codecs. See also https://bugs.chromium.org/p/webm/issues/detail?id=1001

Aha, thanks for the info.  So I guess the next question is: for sRGB color space, would you expect VPX_CR_FULL_RANGE to always be used?  The comments for vpx_color_range_t seem to imply this.

For the RGB order, shouldn't we be looking to img->fmt? The type vpx_img_fmt_t has some RGB types, but none are GBR.

Are any browsers playing back RGB yet?  Do any support VPX_CR_FULL_RANGE for YUV?

[Ronald S. Bultje]

Hi,

On Wed, Nov 11, 2015 at 2:21 PM, fnordware <fnor...@gmail.com> wrote:

On Wednesday, November 11, 2015 at 3:37:27 AM UTC-8, rbultje wrote:

The YUV types are indeed just labels. The sRGB one is special as you point out, and the answer to each of the above questions is yes.

In FFmpeg, we have assumed that the plane ordering of RGB input is GBR (i.e. "similar to" YCbCr), which is identical to how this (RGB colorspace substitution) works for other modern codecs. See also https://bugs.chromium.org/p/webm/issues/detail?id=1001

Aha, thanks for the info.  So I guess the next question is: for sRGB color space, would you expect VPX_CR_FULL_RANGE to always be used?  The comments for vpx_color_range_t seem to imply this.

Yes, the flag is not coded in the bitstream with sRGB and should always be 1.

For the RGB order, shouldn't we be looking to img->fmt? The type vpx_img_fmt_t has some RGB types, but none are GBR.

I don't think these mean the same thing. The encoder interface may certainly support multiple input formats, which are possibly converted before being handed to the encoder (or maybe it just gives an error). However, the codec internally only supports a small subset of image formats as listed in vpx_img_fmt_t. Don't forget nobody has ever encoded RGB content with vpxenc, so I'd really restrict myself to vpxdec or ffmpeg (with either the libvpx-vp9 or internal decoder). ffmpeg (with either decoder as backend) supports sRGB and returns it to the user with AV_PIX_FMT_GBRP (planar GBR).

Are any browsers playing back RGB yet?  Do any support VPX_CR_FULL_RANGE for YUV?

I unfortunately don't know...

Ronald

[fnordware]

On Wednesday, November 11, 2015 at 11:31:21 AM UTC-8, rbultje wrote:

I don't think these mean the same thing. The encoder interface may certainly support multiple input formats, which are possibly converted before being handed to the encoder (or maybe it just gives an error). However, the codec internally only supports a small subset of image formats as listed in vpx_img_fmt_t. Don't forget nobody has ever encoded RGB content with vpxenc, so I'd really restrict myself to vpxdec or ffmpeg (with either the libvpx-vp9 or internal decoder). ffmpeg (with either decoder as backend) supports sRGB and returns it to the user with AV_PIX_FMT_GBRP (planar GBR).

Right, validate_img() in vp9_cx_iface.c does not support any of the RGB formats, so I guess I won't try passing in any of those. But I'd think if RGB was going to be supported for real, you'd want to pass in a vpx_image_t with an RGB format and get one back on the other end, no?

Would love to hear an official word on this.

[Johann Koenig]

On Wed, Nov 11, 2015 at 11:44 AM, fnordware <fnor...@gmail.com> wrote:
> Right, validate_img() in vp9_cx_iface.c does not support any of the RGB
> formats, so I guess I won't try passing in any of those. But I'd think if
> RGB was going to be supported for real, you'd want to pass in a vpx_image_t
> with an RGB format and get one back on the other end, no?
>
> Would love to hear an official word on this.

I am not aware of any efforts to use RGB internally in the codec or
provide automatic conversion. What's the benefit of doing such
conversions? Within the codec my understanding is that separating
chroma and luma from the Y plane is very beneficial for motion search
in particular.

[fnordware]

On Wednesday, November 11, 2015 at 12:13:48 PM UTC-8, Johann wrote:

I am not aware of any efforts to use RGB internally in the codec or
provide automatic conversion. What's the benefit of doing such
conversions? Within the codec my understanding is that separating
chroma and luma from the Y plane is very beneficial for motion search
in particular.

Then why is there an sRGB color space option?

There are reasons one would want to stick with RGB, such as when using VP9's lossless option, to avoid loss during the RGB->YUV conversion. But maybe that's outside the scope of WebM.

[Johann Koenig]

On Wed, Nov 11, 2015 at 1:44 PM, fnordware <fnor...@gmail.com> wrote:
> On Wednesday, November 11, 2015 at 12:13:48 PM UTC-8, Johann wrote:
>> I am not aware of any efforts to use RGB internally in the codec or
>> provide automatic conversion. What's the benefit of doing such
>> conversions? Within the codec my understanding is that separating
>> chroma and luma from the Y plane is very beneficial for motion search
>> in particular.
>
> Then why is there an sRGB color space option?

I was mistaken about what sRGB entailed. After discussing this with
some people it sounds like it's just using 4:4:4 and putting G in the
Y plane, B in the U plane and R in the V plane. Since we do have 4:4:4
support working it should just be an issue of arranging the data.

> There are reasons one would want to stick with RGB, such as when using VP9's
> lossless option, to avoid loss during the RGB->YUV conversion. But maybe
> that's outside the scope of WebM.

That sounds interesting. In theory all that should be involved is
wiring up the existing components. If anyone wants to submit patches
that would be wonderful.