Issue 8566 in angleproject: Performance degradation when using Multi Draw on MacOS

[e… via monorail]

Status: Untriaged
Owner: ----
Priority: Medium
Type: Defect

New issue 8566 by e...@cloudzeta.com: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566

Hello!

In our application, we are using the WEBGL_multi_draw extension to batch draw commands. On Windows, this works quite well and we see a very noticeable performance boost.
On MacOS, however, if the number of multi-draw calls exceeds a certain limit (about 110), we see a massive performance degradation, with single frame starting to take 100s of ms to render. This does not happen if we use normal draw commands.
It feels like some internal buffer overflows, which triggers a very slow path.
If we flush the context after each draw call, the problem goes away.
I don't see any error messages in the Console.

Any suggestion will be very helpful as we would like to keep this optimization enabled.

Attachments:
about-gpu-2024-03-01T18-37-19-293Z.txt 49.2 KB

--
You received this message because:
1. The project was configured to send all issue notifications to this address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[syous… via monorail]

Updates:
Cc: geoff...@chromium.org lehoan...@chromium.org
Labels: OS-Mac Renderer-Metal

Comment #1 on issue 8566 by syou...@chromium.org: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c1

Thank you for the report. The best way for us to debug this is if you can provide a minimal example to reproduce it.

Do you reproduce the slowdown if you run Chrome with `--use-angle=gl`? I'm asking to make sure this is specific to ANGLE's metal backend or not.

[e… via monorail]

Comment #2 on issue 8566 by e...@cloudzeta.com: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c2

> Do you reproduce the slowdown if you run Chrome with `--use-angle=gl`? I'm asking to make sure this is specific to ANGLE's metal backend or not.

With `--use-angle=gl` the slowdown does not reproduce. It runs really well

GPU0 : VENDOR= 0x106b [Google Inc. (Apple)], DEVICE=0x0000 [ANGLE (Apple, Apple M1 Max, OpenGL 4.1 Metal - 83)], DRIVER_VENDOR=Apple, DRIVER_VERSION=83 *ACTIVE*

> The best way for us to debug this is if you can provide a minimal example to reproduce it.

Will it work if we host the problematic web page so that you can open it? I tried to create a minimal reproducer, but it looks like there may be some other factors besides just using the multi draw

[e… via monorail]

Comment #3 on issue 8566 by e...@cloudzeta.com: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c3

Can you please try this page that reproduces the problem?
https://gozeta-prod--multi-draw-test-x2xcs2nd.web.app/player/lr9wbcmptvigxdgl

You can try to open it on Windows or on Mac in GL mode - it will work without problems. But on Mac with Metal it will be extremely slow.

[syous… via monorail]

Updates:
Cc: d...@apple.com kpidd...@apple.com

Comment #4 on issue 8566 by syou...@chromium.org: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c4

Thank you. I'm not a metal developer myself, but CC some other folks who may be interested.

[lehoa… via monorail]

Updates:
Cc: k...@chromium.org

Comment #5 on issue 8566 by lehoan...@chromium.org: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c5

It seems the performance problem is caused by the flat shading which is not supported that well on Metal backend. The default provoking vertex is the last vertex in a primitive but Metal doesn't natively support that so we have to do several workarounds/conversions by the internal ProvokingVertexHelper. This severely slowed down the draws especially in multi draw commands. ProvokingVertexHelper also allocated many intermediate buffers within a single MultiDrawn command. When opening the provided test page, I saw the GPU process reached 20GB of memory on my local macbook.

egor@ could you try to use the WEBGL_provoking_vertex extension to tell WebGL context to use FIRST_VERTEX_CONVENTION? that should speed up the draw calls that use flat shading significantly.

Internally, I think we should revamp the way Metal backend handles the default provoking vertex. At least we should try to reuse the intermediate buffers instead of creating new one everytime.

[e… via monorail]

Comment #6 on issue 8566 by e...@cloudzeta.com: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c6

lehoangquyen@ thanks a lot! Setting provoking vertex to FIRST_VERTEX_CONVENTION fixed the problem completely. The scene I shared now runs almost twice as fast on the CPU side when using multi-draw.

OpenGL keeps biting me time and time again. TBH, I did not even know that this convention was different between OpenGL and all other graphics APIs as whenever we pass a flat attribute, the value is always the same for all vertices, so whether it is the first or the last vertex, the result is the same.
In this specific example we pass gl_DrawID from vertex to fragment shader and since it is an int attribute, it has to be flat.

The performance hit is too high here, so I think this pitfall should at least be mentioned somewhere (e.g. here https://emscripten.org/docs/optimizing/Optimizing-WebGL.html)

Again, thanks a lot for helping with this issue!

[syous… via monorail]

Comment #7 on issue 8566 by syou...@chromium.org: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c7

FYI, in the Vulkan backend we extensively use `ANGLE_PERF_WARNING` (or a variant of it) to notify the application when we detect they've hit such bad paths. Those are caught by Chrome and reported somewhere. I've seen them show up in the bottom of chrome://gpu, but hopefully somewhere more directly linked with the page too. I highly suggest the metal backend adds such warnings too to warn unsuspecting developers.

[syous… via monorail]

Comment #8 on issue 8566 by syou...@chromium.org: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c8

(Thanks a lot for triaging this by the way Quyen!)

[e… via monorail]

Comment #9 on issue 8566 by e...@cloudzeta.com: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c9

I am wondering how this issue is handled on Windows with Direct3D backend, which also does not have a way to change the provoking vertex.

[lehoa… via monorail]

Comment #10 on issue 8566 by lehoan...@chromium.org: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c10

IIRC D3D backend uses geometry shader to rotate the vertices' order. Geometry shader is quite convenient since it is just extra stage which could be inserted between user's vertex shader and pixel shader. Metal doesn't have geometry shader, so that would require a more complicated approach to handle this. The closest stage to geometry shader in Metal is mesh shader, and using it would have to completely change the user's pipeline (vertex shader will need to be replaced, vertex attributes, indices would have to be manually parsed, etc).

[lehoa… via monorail]

Comment #11 on issue 8566 by lehoan...@chromium.org: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c11

Currently in Metal backend, we just generate a new index buffer to rotate the vertices' order. And it has higher overhead than D3D's geometry shader approach.

[e… via monorail]

Comment #12 on issue 8566 by e...@cloudzeta.com: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c12

It would've been very helpful if the browser printed out a warning message when it detects a program that uses flat attibutes when provoking vertex is set to LAST, something similar to this.

zetaWasm.js:1 WebGL: this extension has very low support on mobile devices; do not rely on it for rendering effects: WEBGL_polygon_mode

With WEBGL_polygon_mode, the problem would be obvious to spot: lines or points wouldn't render. However, with flat attributes and the provoking vertex, it is very unclear that the browser has to do expensive workarounds.

[k… via monorail]

Updates:
Status: Available

Comment #13 on issue 8566 by k...@chromium.org: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c13

Thanks Quyen for triaging this and Shabi for the suggestion about adding a performance warning.

Submitter, thanks for the test case. We'll add a performance warning to ANGLE's Metal backend. Additionally I've just uploaded https://github.com/mdn/content/pull/32569 adding a note about WEBGL_provoking_vertex to Mozilla's WebGL Best Practices. If someone wouldn't mind adding similar guidance to Emscripten's docs, would appreciate it.

[e… via monorail]

Comment #14 on issue 8566 by e...@cloudzeta.com: Performance degradation when using Multi Draw on MacOS
https://bugs.chromium.org/p/angleproject/issues/detail?id=8566#c14

Thanks kbr@

I would suggest always printing this warning message because this will ensure that the site behavior is consistent across browsers. We tested our site on Windows and were very confused when we found out it did not work on Mac.
On the other hand, workaround with Geometry Shader is unnecessary and can be avoided. I believe, for most developers the situation is similar to ours: they either unaware of this extension (when coming from Direct3D for example), or don't realize how expensive it may be.