GL_RGBA16 missing, not easily emulated

[Alecazam]

This appears to have been omitted from the ES2/3/3.1/WebGL 2 spec.   GL_RGBA16 was filterable, has high precision, and fed in nicely to floating point samplers.   

I found RGBA16ui in the ES3/WebGL2 spec, but none of these integer formats are filterable.  The whole point of a 16-bit format is to provide higher precision than 16f, but do filtering in as few ops as possible.  Desktop has had this since GL 1.1.  If you have to do 8 ops for trilinear mips, then the integer formats seem rather costly and unusable.  I know that mobile devices like Tegra had a 16f pixel shader, but surely the newer hardware has enough precision to handle GL_RGBA16.  There are plenty of image formats where higher precision is desirable (RAW, PNG, PSD, others).

Also R32f was commonly used with desktop, but fails in WebGL 1.   I'm encouraged to see EXT_SRGB support everywhere now, but it feels like exposing some of these desktop formats (an extension per format?) is warranted.

[Kenneth Russell]

On Fri, Jan 30, 2015 at 12:48 AM, Alecazam <al...@figma.com> wrote:
> This appears to have been omitted from the ES2/3/3.1/WebGL 2 spec.
> GL_RGBA16 was filterable, has high precision, and fed in nicely to floating
> point samplers.
>
> I found RGBA16ui in the ES3/WebGL2 spec, but none of these integer formats
> are filterable. The whole point of a 16-bit format is to provide higher
> precision than 16f, but do filtering in as few ops as possible. Desktop has
> had this since GL 1.1. If you have to do 8 ops for trilinear mips, then the
> integer formats seem rather costly and unusable. I know that mobile devices
> like Tegra had a 16f pixel shader, but surely the newer hardware has enough
> precision to handle GL_RGBA16. There are plenty of image formats where
> higher precision is desirable (RAW, PNG, PSD, others).

Thanks for raising this issue. After a little searching I found this
OpenGL ES extension:
https://www.khronos.org/registry/gles/extensions/EXT/EXT_texture_norm16.txt

It's recent, and written against the OpenGL ES 3.1 spec. I don't know
whether devices that support e.g. ES 3.0 might be able or allowed to
expose this extension.

There must be some reason this isn't in the core spec, but I don't
know what it might be. We could consider exposing it as an extension
to WebGL. It would be good to first gather information about what
devices support it. Since it's so recent, it's unlikely any shipping
mobile devices do. The downside to exposing this as an extension is
that if your app is written assuming it's present, it won't be
portable.

> Also R32f was commonly used with desktop, but fails in WebGL 1.

This format will come along with many others in WebGL 2.0. Please use
RGBA/FLOAT textures for now in conjunction with WebGL's exposure of
the OES_texture_float extension.

> I'm
> encouraged to see EXT_SRGB support everywhere now, but it feels like
> exposing some of these desktop formats (an extension per format?) is
> warranted.

Please check the OpenGL ES extension registry at
https://www.khronos.org/registry/gles/ and see if there are other
extensions providing similar functionality that might be older and
more widely supported.

Please feel free to propose exposing specific extensions on
public_webgl -- or submit pull requests to
https://github.com/KhronosGroup/WebGL adding proposals.

-Ken


> --
> You received this message because you are subscribed to the Google Groups
> "WebGL Dev List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to webgl-dev-lis...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Alecazam]

> After a little searching I found this OpenGL ES extension: https://www.khronos.org/registry/gles/extensions/EXT/EXT_texture_norm16.txt 

Thanks Ken.  That is indeed the extension to the feature (but really a core feature in GL).  As a guess, this was likely left out of the ES2 spec since fragment shaders were only fp16The precision of the format couldn't be guaranteed after interpolation.  On a similar note, GL 1.1 doesn't have a GL_RGBA32 format because fp24/fp32 (with 24-bit mantissa) were/are used to implement the pixel shaders.

> This format will come along with many others in WebGL 2.0. Please use RGBA/FLOAT textures

I'm just wondering if there is a fp32 rendering format (with OES_texture_float or the renderbuffer extension).   This is important, since WebGL 1 doesn't expose the extensions that allow mixed depth/color sizes, and also mixed bit-depths for MRT.  If you want to render depth + RGBA8un, you'd have to use a depth texture, but that has less precision and range (24i vs. 32f).

[Alecazam]

The downside to exposing this as an extension is that if your app is written assuming it's present, it won't be portable.

This is the risk of all texture formats.  That's why you have to test to see if they work (since GL and ES sadly don't have caps bits).  EXT_SRGB is now everywhere on desktop, but still not on iOS, but that doesn't mean it's not worth adding or using.  You'll need a fallback for that for mobile, or just not use it.  

I really think all texture formats that a platform supports should be exposed, since this is what makes the GPU run fast and makes porting the simplest to web.  I could do without all the other extensions, but I need formats.   Keeping to the l.c.d. of the mobile world isn't great for desktop apps, and mobile is catching up (X1 is OpenGL 4.5).  You've always had this limitation of formats on platforms (PVRTC vs. DXT vs. ETC vs. ATIC vs ATSC, GL_RGBA16, 32f rendering, 16f rendering), but people adapt to the dominant systems.  There's obviously the work of translating to DX11(Angle), and moving WebGL to core 3.2, but those have a much larger set of constants to pick from. 

Thanks for all the help.  Just in the last week EXT_SRGB on Chrome, and possibly EXT_shader_texture_lod on Firefox are all good additions to the browsers.

[Kenneth Russell]

On Fri, Jan 30, 2015 at 10:56 AM, Alecazam <al...@figma.com> wrote:
>> After a little searching I found this OpenGL ES extension:
>> https://www.khronos.org/registry/gles/extensions/EXT/EXT_texture_norm16.txt
>
> Thanks Ken. That is indeed the extension to the feature (but really a core
> feature in GL). As a guess, this was likely left out of the ES2 spec since
> fragment shaders were only fp16The precision of the format couldn't be
> guaranteed after interpolation. On a similar note, GL 1.1 doesn't have a
> GL_RGBA32 format because fp24/fp32 (with 24-bit mantissa) were/are used to
> implement the pixel shaders.
>
>> This format will come along with many others in WebGL 2.0. Please use
>> RGBA/FLOAT textures
>
> I'm just wondering if there is a fp32 rendering format (with
> OES_texture_float or the renderbuffer extension).

Yes, you can certainly allocate FP32 RGBA textures in WebGL 1.0 if you
enable the OES_texture_float extension.

> This is important, since
> WebGL 1 doesn't expose the extensions that allow mixed depth/color sizes,
> and also mixed bit-depths for MRT.

I don't follow.
https://www.khronos.org/registry/webgl/extensions/WEBGL_draw_buffers/
should allow you to attach multiple color buffers to an FBO. Some of
those could use 8-bit textures and some could use FP textures.
Typically you'll use a packed depth/stencil depth buffer in that
configuration.

> If you want to render depth + RGBA8un,
> you'd have to use a depth texture, but that has less precision and range
> (24i vs. 32f).

See above.

[Alecazam]

I brought this up in the "WebGL 1 Extensions" thread.   ARB_framebuffer_object isn't exposed by WebGL.  WebGL currently requires matching dimension FBO targets, so you can't easily mix depth/color targets of different sizes on desktop.  WebGL throws it's own errors out about mismatched FBO depth/color texture dimensions (even though all desktop hardware exposes ARB_framebuffer_object).   Mark Callow mentioned that ES3 is also ARB_framebuffer_object compliant, but ES2 often wasn't.

The other part of ARB_framebuffer_object is that it allows you to mix different bit-depths for MRT.  Without it, the sizes and bit-depths must match.  So WebGL should really enforce that R32F is paired with RGBA8UN targets, or it should allow mixed size FBO's textures and MRT's by testing for the extension.  I think it's doing the latter for MRT, but enforcing matching for FBO creation which is incorrect.

> Yes, you can certainly allocate FP32 RGBA textures in WebGL 1.0 if you enable the OES_texture_float extension. 

I didn't think the L32F formats were renderable, so that's why I was suggesting that R32f be exposed on WebGL 1.  I know RGBA32F is, but the older Nvidia vertex texture units only accepted R32F and RGBA32F.  Having to resort to RGBA32F for a depth-only target is also significantly bigger.  D10+ expanded the vertex texturing to handle all formats.

[Kenneth Russell]

On Fri, Jan 30, 2015 at 11:21 AM, Alecazam <al...@figma.com> wrote:
> I brought this up in the "WebGL 1 Extensions" thread.
> ARB_framebuffer_object isn't exposed by WebGL. WebGL currently requires
> matching dimension FBO targets, so you can't easily mix depth/color targets
> of different sizes on desktop. WebGL throws it's own errors out about
> mismatched FBO depth/color texture dimensions (even though all desktop
> hardware exposes ARB_framebuffer_object). Mark Callow mentioned that ES3
> is also ARB_framebuffer_object compliant, but ES2 often wasn't.

WebGL exposes consistent behavior between desktop and mobile hardware
as much as possible. It would be a tragedy if a substantial amount of
WebGL content used FBO attachments of different sizes and this caused
the content to fail to run on mobile devices.

> The other part of ARB_framebuffer_object is that it allows you to mix
> different bit-depths for MRT. Without it, the sizes and bit-depths must
> match. So WebGL should really enforce that R32F is paired with RGBA8UN
> targets, or it should allow mixed size FBO's textures and MRT's by testing
> for the extension. I think it's doing the latter for MRT, but enforcing
> matching for FBO creation which is incorrect.

I had to refresh my memory about this but as you point out it's
referred to in the "Issues" section of
https://www.khronos.org/registry/gles/extensions/EXT/EXT_draw_buffers.txt
:

"""5) What are the requirements on the color attachment sizes and formats?

RESOLVED: ES 2.0 requires that all color buffers attached to application-
created framebuffer objects must have the same number of bitplanes
(Chapter 4 overview p91)."""

I agree, this is unfortunate. However, if there's no ES 2.0 extension
for relaxing this requirement, the working group is unlikely to
support lifting this restriction. Instead we will focus on delivering
WebGL 2.0 based on ES 3.0, because it removes many restrictions, this
one included.

>> Yes, you can certainly allocate FP32 RGBA textures in WebGL 1.0 if you
>> enable the OES_texture_float extension.
>
> I didn't think the L32F formats were renderable, so that's why I was
> suggesting that R32f be exposed on WebGL 1. I know RGBA32F is, but the
> older Nvidia vertex texture units only accepted R32F and RGBA32F. Having to
> resort to RGBA32F for a depth-only target is also significantly bigger.
> D10+ expanded the vertex texturing to handle all formats.

Exposing the single-channel floating-point textures as a WebGL 1.0
extension is a large chunk of work that is better invested in getting
WebGL 2.0 done.

[Alecazam]

> WebGL exposes consistent behavior between desktop and mobile hardware as much as possible. It would be a tragedy if a substantial amount of 
> WebGL content used FBO attachments of different sizes and this caused the content to fail to run on mobile devices. 

Yes, if you didn't provide a fallback, but that's the case with any unsupported extension, format, etc.   It just feels like having the option to fully utilize the desktop with WebGL 1 extensions would be a good thing, but I can wait for WebGL 2 to address this.  I realize there are limited resources.   

I was just running one of the subdivision surface demos the other day on my iPhone 6, and while the app loaded, it was barely performant with modest triangle counts.  Not the fault of the WebGL, just a reality that desktop still holds a good lead over mobile.  I'd just like that desktop experience to be as close to native as possible, but some of these mobile limitations require non-performant workarounds.

[Kenneth Russell]

Keep in mind that mobile devices frequently have one-tenth the memory
bandwidth, fill rate, etc. compared to desktop devices. Applications
need to be written to keep that in mind and scale their workloads down
in order to achieve their performance goals on mobile.

[Alecazam]

GPU code has always had to scale, but native apps could at least get at what the card was, memory limits, and vram limits.  WebGL and the browsers are getting better about releasing the renderer info, but could still reveal more in this regard to help with scaling.  

There is a common denominator of GPU functionality that Apple is putting in place on iOS/Mac (SSO, texture-barrier/framebuffer-fetch, and others), that WebGL 2 doesn't address or encapsulate.  These affect shader writers.  Maintaining the shader variants for SSO, UBO, fetch, lod, derivatives gets quite complex.  Plus you have all of the WebGL 1 extensions and fallbacks.  These native and WebGL 2 shaders look nothing like WebGL 1 shaders at some point.   Not an easy task, but appreciate your feedback.