sRGBA as RGBA

[Rik Cabanier]

The Oculus browser is working on a transition from an internal API to OpenXR.

One of the changes is that it's supposed to use an sRGB swapchain in order to get the right gamma applied to the output pixels.

Unfortunately, WebXR is defined as exposing RGBA textures by default so we had to register the XR textures as RGB but with an internal format of sRGB. In addition, we use FRAMEBUFFER_SRGB_EXT to disable the automatic gamma conversion.

Basically, we want the sRGBA swapchain to behave as it's RGBA.

My questions are:

- Is it OK to register a texture with a different internal profile?

- I see that chromium also uses FRAMEBUFFER_SRGB_EXT sometimes. Why is that?

- How does chromium handle exporting RGB to a display. Is the gamma conversion skipped?

Thanks!

[Ken Russell]

CC'ing Brandon (WebXR expert), Chris (colorspace expert),  Vasiliy (Android graphics and SharedImage expert), Peng (Android graphics expert)

Hi Rik,

This behavior likely depends on a few variables:

  - How exactly the textures are being allocated (e.g. are they AHardwareBuffers wrapped into OpenGL textures?)

  - How exactly the resulting texture is being put on screen (e.g. is that texture being displayed via OpenGL or is the AHardwareBuffer being scanned out directly?)

  - Whether Vulkan is being used by the browser for compositing (although I suspect this won't affect the handling of these OpenXR-provided textures)

I don't know definitive answers to any of these, but hope that the CC'd experts can help answer them and your questions.

Note: I don't see any real references to GL_FRAMEBUFFER_SRGB_EXT in Chromium:

https://source.chromium.org/search?q=FRAMEBUFFER_SRGB_EXT%20-third_party%2Fapitrace%20-glext.h%20-gl2ext.h%20-third_party%2Fmesa%20-third_party%2Fwebgl%20-virglrenderer

All of the references are in autogenerated code, or third-party libraries like Mesa which aren't actually used in Chromium's graphics stack. I don't think any of Chromium's internal code, including Skia, is enabling or disabling this capability bit.

One more wrinkle: we are working on moving Chromium on Android to be hosted on top of ANGLE, and that will soon introduce another interposition layer. However, this one will also confer some significant advantages, like being able to transparently re-host the WebGL implementation on top of ANGLE's Vulkan backend.

-Ken

[Rik Cabanier]

Thanks for the reply Ken!

On Thu, Mar 17, 2022 at 6:25 PM Ken Russell <k...@chromium.org> wrote:

CC'ing Brandon (WebXR expert), Chris (colorspace expert),  Vasiliy (Android graphics and SharedImage expert), Peng (Android graphics expert)

Hi Rik,

This behavior likely depends on a few variables:

  - How exactly the textures are being allocated (e.g. are they AHardwareBuffers wrapped into OpenGL textures?)

They are regular texture that come from a swapchain (= they were allocated in the system compositor and transferred to the browser)

 

  - How exactly the resulting texture is being put on screen (e.g. is that texture being displayed via OpenGL or is the AHardwareBuffer being scanned out directly?)

When the texture is released at the end of the Raf call, it is picked up by the system compositor which display it on the headset (after doing things like timewarp and color correction)

 

  - Whether Vulkan is being used by the browser for compositing (although I suspect this won't affect the handling of these OpenXR-provided textures)

No. The browser does use that texture to draw to screen.

 

I don't know definitive answers to any of these, but hope that the CC'd experts can help answer them and your questions.

Note: I don't see any real references to GL_FRAMEBUFFER_SRGB_EXT in Chromium:

https://source.chromium.org/search?q=FRAMEBUFFER_SRGB_EXT%20-third_party%2Fapitrace%20-glext.h%20-gl2ext.h%20-third_party%2Fmesa%20-third_party%2Fwebgl%20-virglrenderer

True. I saw that there were checks for EXT_framebuffer_sRGB which is the extension that enables this flag.

 

All of the references are in autogenerated code, or third-party libraries like Mesa which aren't actually used in Chromium's graphics stack. I don't think any of Chromium's internal code, including Skia, is enabling or disabling this capability bit.

One more wrinkle: we are working on moving Chromium on Android to be hosted on top of ANGLE, and that will soon introduce another interposition layer. However, this one will also confer some significant advantages, like being able to transparently re-host the WebGL implementation on top of ANGLE's Vulkan backend.

That would be very cool! I assume ANGLE doesn't support this flag so we'd have to add it.

FWIW I really don't like this change that OpenXR is pushing on us and I'm trying to make our compositor consume RGBA textures as if they were sRGBA like before.

It seems that other implementations would run into the same issue. Maybe they didn't notice that the color on screen and the headset are different?

[Rik Cabanier]

Sorry, typos :-)

On Thu, Mar 17, 2022 at 8:25 PM Rik Cabanier <caba...@gmail.com> wrote:

Thanks for the reply Ken!

On Thu, Mar 17, 2022 at 6:25 PM Ken Russell <k...@chromium.org> wrote:

CC'ing Brandon (WebXR expert), Chris (colorspace expert),  Vasiliy (Android graphics and SharedImage expert), Peng (Android graphics expert)

Hi Rik,

This behavior likely depends on a few variables:

  - How exactly the textures are being allocated (e.g. are they AHardwareBuffers wrapped into OpenGL textures?)

They are regular texture that come from a swapchain (= they were allocated in the system compositor and transferred to the browser)

 

  - How exactly the resulting texture is being put on screen (e.g. is that texture being displayed via OpenGL or is the AHardwareBuffer being scanned out directly?)

When the texture is released at the end of the Raf call, it is picked up by the system compositor which display it on the headset (after doing things like timewarp and color correction)

 

  - Whether Vulkan is being used by the browser for compositing (although I suspect this won't affect the handling of these OpenXR-provided textures)

No. The browser does use that texture to draw to screen.

does NOT use it to draw it to the screen

[Ken Russell]

On Thu, Mar 17, 2022 at 8:25 PM Rik Cabanier <caba...@gmail.com> wrote:

Thanks for the reply Ken!

On Thu, Mar 17, 2022 at 6:25 PM Ken Russell <k...@chromium.org> wrote:

CC'ing Brandon (WebXR expert), Chris (colorspace expert),  Vasiliy (Android graphics and SharedImage expert), Peng (Android graphics expert)

Hi Rik,

This behavior likely depends on a few variables:

  - How exactly the textures are being allocated (e.g. are they AHardwareBuffers wrapped into OpenGL textures?)

They are regular texture that come from a swapchain (= they were allocated in the system compositor and transferred to the browser)

OK - I don't know how this works in OpenXR if that's the underlying API being used. Would hope others could comment.

 

  - How exactly the resulting texture is being put on screen (e.g. is that texture being displayed via OpenGL or is the AHardwareBuffer being scanned out directly?)

When the texture is released at the end of the Raf call, it is picked up by the system compositor which display it on the headset (after doing things like timewarp and color correction)

 

  - Whether Vulkan is being used by the browser for compositing (although I suspect this won't affect the handling of these OpenXR-provided textures)

No. The browser does not use that texture to draw to screen.

OK. (Corrected the typo here as you pointed out in your follow-up email) 

 

I don't know definitive answers to any of these, but hope that the CC'd experts can help answer them and your questions.

Note: I don't see any real references to GL_FRAMEBUFFER_SRGB_EXT in Chromium:

https://source.chromium.org/search?q=FRAMEBUFFER_SRGB_EXT%20-third_party%2Fapitrace%20-glext.h%20-gl2ext.h%20-third_party%2Fmesa%20-third_party%2Fwebgl%20-virglrenderer

True. I saw that there were checks for EXT_framebuffer_sRGB which is the extension that enables this flag.

 

All of the references are in autogenerated code, or third-party libraries like Mesa which aren't actually used in Chromium's graphics stack. I don't think any of Chromium's internal code, including Skia, is enabling or disabling this capability bit.

One more wrinkle: we are working on moving Chromium on Android to be hosted on top of ANGLE, and that will soon introduce another interposition layer. However, this one will also confer some significant advantages, like being able to transparently re-host the WebGL implementation on top of ANGLE's Vulkan backend.

That would be very cool! I assume ANGLE doesn't support this flag so we'd have to add it.

Actually ANGLE already has support for the GL_EXT_sRGB_write_control extension:

https://source.chromium.org/search?q=GL_EXT_sRGB_write_control%20f:third_party%2Fangle

and it looks like ANGLE's Vulkan backend supports it:

https://source.chromium.org/chromium/chromium/src/+/main:third_party/angle/src/libANGLE/renderer/vulkan/vk_caps_utils.cpp;l=489;drc=d74c336647586eccbd98975a0956bda1cf5de25f;bpv=1;bpt=1

FWIW I really don't like this change that OpenXR is pushing on us and I'm trying to make our compositor consume RGBA textures as if they were sRGBA like before.

Hope that you'll find a way to make this happen.

 

It seems that other implementations would run into the same issue. Maybe they didn't notice that the color on screen and the headset are different?

There are probably relatively few implementations of WebXR, and Oculus' is one of them, so it doesn't surprise me that you're the first to encounter this.

Also hoping that other folks on this list with more experience than me can offer their suggestions for what to do.

-Ken

[Vasiliy Telezhnikov]

Unfortunately I don't know much about how WebXR really works, but what part of chrome draws to those textures? From chrome/gpu perspective, the display compositor should be able to draw to any (reasonable) color space requested.

I tried to look in code, but there seem to be few VR related paths and I'm not sure which one is taken in this place.

-Vasiliy

[Peng Huang]

+klausw@ and +alcooper@

I remember I was told the chrome browser process is using the platform gl to draw textures. I am not sure of the details. kklausw@ or alcooper@, could you please give us some details about the graphics pipeline of OpenXR?

[Alex Cooper]

Our current OpenXR implementation is Windows only, where it predominantly uses DirectX and I believe ANGLE to composite (Currently the code extends the compositor_base file (XRCompositorCommon class) under https://source.chromium.org/chromium/chromium/src/+/main:device/vr/windows/). I don't know that we currently have a "generic" compositor on Android; I think a lot of the existing Android VR compositing is tied together with GVR pretty tightly in GVRSchedulerDelegate. AR implements a Viz CompositorFrameSink and prepares the textures via glFoo style calls in ArCoreGl.

I have no problem extending the platforms that support OpenXR, but we lacked anything to use to test it, and Microsoft contributed a lot of the OpenXR implementation.

Once we get outside of Chrome I'm not as familiar with what requirements OpenXR has; either Brandon may know, or I'd need to check with/add the Microsoft folks that did the large portion of the implementation.

Thanks,

Alex

[Rik Cabanier]

On Wed, Mar 23, 2022 at 6:11 AM Vasiliy Telezhnikov <vas...@chromium.org> wrote:

Unfortunately I don't know much about how WebXR really works, but what part of chrome draws to those textures? From chrome/gpu perspective, the display compositor should be able to draw to any (reasonable) color space requested.

We have a custom code path to register the swapchain textures but apart from that, we use the regular Chromium WebGL code.

The swapchain texture isn't picked up and drawn to the display compositor. Instead it is passed to the system compositor. 

Because it performs a gamma conversion on RGBA textures in the OpenXR path, our content becomes too bright. Submitting an sRGB texture works around it.