Issue 5780 in angleproject: ANGLE FPS of T-Rex got 17% drop on Nevis

[chery… via monorail]

Status: Accepted
Owner: ----
OS: Android
Priority: Medium
Renderer: Vulkan
Type: Defect

New issue 5780 by chery...@arm.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780

T-Rex FPS:101->83

Regression point:
commit 867411a4d36398835bb5c3a78361d292cc0f6c32
Author: Charlie Lao <cc...@google.com>
Date: Mon Mar 8 19:08:14 2021 -0800

Vulkan: Track images used for both attachment and sampler

This CL adds a new layout ImageLayout::ColorAttachmentAndShaderRead. We
detect that the same ImageHelper object is used for both sampler and
attachment and will pick this new layout instead of ColorAttachment.

Bug: b/175584609
Change-Id: I2089f59d4dc1ad4de7edab0e067797adcfa9b020
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2749479
Commit-Queue: Charlie Lao <cc...@google.com>
Reviewed-by: Shahbaz Youssefi <syou...@chromium.org>
Reviewed-by: Tim Van Patten <ti...@google.com>
Reviewed-by: Jamie Madill <jma...@chromium.org>

--
You received this message because:
1. The project was configured to send all issue notifications to this address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[chery… via monorail]

Updates:
Cc: cc...@google.com

Comment #1 on issue 5780 by chery...@arm.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c1

(No comment was entered for this change.)

[cc… via monorail]

Comment #2 on issue 5780 by cc...@google.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c2

Hi Cheryl,
I just checked on our ARM device, I am not seeing any meaningful performance difference with gl_trex_off. Note that there are a series of CLs associated with this. The very last CL is 81432c24b.

Before:
03-22 13:43:20.566 17718 17845 I _logcat_: "fps": 222.15508201965628,
03-22 13:43:20.566 17718 17845 I _logcat_: "frame_count": 12500,
03-22 13:43:20.566 17718 17845 I _logcat_: "graphics_version": "OpenGL ES 3.1.0 (ANGLE 2.1.15223 git hash: 0a80f2e15f07)",
03-22 13:43:20.566 17718 17845 I _logcat_: "renderer": "ANGLE (ARM, Vulkan 1.1.161 (Mali-G78 (0x92020003)), Mali-G78-29.0.0)",
03-22 13:43:20.566 17718 17845 I _logcat_: "surface_height": 1080,
03-22 13:43:20.566 17718 17845 I _logcat_: "surface_width": 1920,
03-22 13:43:20.566 17718 17845 I _logcat_: "result_id": "gl_trex_off",
03-22 13:43:20.566 17718 17845 I _logcat_: "score": 12440.6845703125,

After:
03-22 13:37:59.895 17067 17217 I _logcat_: "fps": 219.85563417843045,
03-22 13:37:59.895 17067 17217 I _logcat_: "frame_count": 12366,
03-22 13:37:59.895 17067 17217 I _logcat_: "graphics_version": "OpenGL ES 3.1.0 (ANGLE 2.1.15236 git hash: 570970f90757)",
03-22 13:37:59.895 17067 17217 I _logcat_: "renderer": "ANGLE (ARM, Vulkan 1.1.161 (Mali-G78 (0x92020003)), Mali-G78-29.0.0)",
03-22 13:37:59.895 17067 17217 I _logcat_: "surface_height": 1080,
03-22 13:37:59.895 17067 17217 I _logcat_: "surface_width": 1920,
03-22 13:37:59.895 17067 17217 I _logcat_: "result_id": "gl_trex_off",
03-22 13:37:59.895 17067 17217 I _logcat_: "score": 12311.9150390625,

Also I put the following line in CommandBufferHelper::finalizeColorImageLayout() where if(usedByCurrentRenderPassAsAttachmentAndSampler()) returns true
ERR()<<"ColorAttachmentAndShaderRead used\n";
And I am not hitting this (which is expected).

I don't know what Nevis is, but if you could try with latest ANGLE to see if perf regression still reproduced.

[cc… via monorail]

Updates:
Owner: chery...@arm.com

Comment #3 on issue 5780 by cc...@google.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c3

(No comment was entered for this change.)

[cc… via monorail]

Comment #4 on issue 5780 by cc...@google.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c4

Assign to Cheryl to respond

[chery… via monorail]

Comment #5 on issue 5780 by chery...@arm.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c5

Hi Charlie,
This regression still exists in latest ANGLE.Reverting this commit could make it disappear.

without 867411a:
paretrace64: retrace_egl.cpp,586: Renderer: ANGLE (ARM, Vulkan 1.1.154 (Mali-G77 (0x90800011)), Mali-G77-580.248.2494)
paretrace64: retrace_egl.cpp,587: Version: OpenGL ES 3.1.0 (ANGLE 2.1.15230 git hash: 7e9b497c984d)
paretrace64: retrace_egl.cpp,603: GL_MAX_ATOMIC_COUNTER_BUFFER_BINDINGS = 8
paretrace64: retrace_egl.cpp,603: GL_MAX_SHADER_STORAGE_BUFFER_BINDINGS = 64
paretrace64: retrace_egl.cpp,603: GL_MAX_TRANSFORM_FEEDBACK_SEPARATE_ATTRIBS = 4
paretrace64: retrace_egl.cpp,603: GL_MAX_UNIFORM_BLOCK_SIZE = 65536
paretrace64: retrace_egl.cpp,603: GL_MAX_UNIFORM_BUFFER_BINDINGS = 72
paretrace64: retrace_egl.cpp,603: GL_SHADER_STORAGE_BUFFER_OFFSET_ALIGNMENT = 64
paretrace64: retrace_egl.cpp,603: GL_UNIFORM_BUFFER_OFFSET_ALIGNMENT = 16
paretrace64: retracer.cpp,2239: Duration = 5.488463
paretrace64: retracer.cpp,2240: Frame cnt = 556, FPS = 101.303406

with 867411a:
retrace_egl.cpp,586: Renderer: ANGLE (ARM, Vulkan 1.1.154 (Mali-G77 (0x90800011)), Mali-G77-580.248.2494)
paretrace64: retrace_egl.cpp,587: Version: OpenGL ES 3.1.0 (ANGLE 2.1.15230 git hash: 7e9b497c984d)
paretrace64: retrace_egl.cpp,603: GL_MAX_ATOMIC_COUNTER_BUFFER_BINDINGS = 8
paretrace64: retrace_egl.cpp,603: GL_MAX_SHADER_STORAGE_BUFFER_BINDINGS = 64
paretrace64: retrace_egl.cpp,603: GL_MAX_TRANSFORM_FEEDBACK_SEPARATE_ATTRIBS = 4
paretrace64: retrace_egl.cpp,603: GL_MAX_UNIFORM_BLOCK_SIZE = 65536
paretrace64: retrace_egl.cpp,603: GL_MAX_UNIFORM_BUFFER_BINDINGS = 72
paretrace64: retrace_egl.cpp,603: GL_SHADER_STORAGE_BUFFER_OFFSET_ALIGNMENT = 64
paretrace64: retrace_egl.cpp,603: GL_UNIFORM_BUFFER_OFFSET_ALIGNMENT = 16
paretrace64: retracer.cpp,2239: Duration = 6.676550
paretrace64: retracer.cpp,2240: Frame cnt = 556, FPS = 83.276535

T-Rex params:{"file":"1_trex_onscreen_msaa0_10fps_1920x1080.pat","frames":"5-563","offscreen":true,"preload":true,"singlewindow":true}

[cc… via monorail]

Comment #6 on issue 5780 by cc...@google.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c6

Thanks Cheryl. The problem is that we can not reproduce it here, and we do not know what Nevis is. I just looked at the code a bit more and also not seeing how this could trigger 17% perf drop. The added layout should not been used by TRex. The only cost introduced should be an if() statement on every texture bind, which is not free but can't be 17% drop. If possible, could you dig in a bit by apply the attached diff and build with ASSERT enabled to see if you ever hit an assertion.

Attachments:
assertion.diff 1.5 KB

[chery… via monorail]

Updates:
Cc: jun...@arm.com

Comment #7 on issue 5780 by chery...@arm.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c7

(No comment was entered for this change.)

[chery… via monorail]

Comment #8 on issue 5780 by chery...@arm.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c8

Hi Charlie,Exynos 990 is called Nevis,and there are two assertions by applying above diff:
paretrace64: retracer.cpp,2103: ================== Start timer (Frame: 5) ==================
03-16 20:39:37.669 10898 10920 E libEGL : ANGLE Error:ContextVk.cpp:1496 (handleDirtyTexturesImpl): ! Assert failed in handleDirtyTexturesImpl (../../third_party/angle/src/libANGLE/renderer/vulkan/ContextVk.cpp:1496): false

paretrace64: retracer.cpp,2103: ================== Start timer (Frame: 5) ==================
03-16 20:44:30.524 12738 12760 E libEGL : ANGLE Error:vk_helpers.cpp:1243 (finalizeColorImageLayout): ! Assert failed in finalizeColorImageLayout (../../third_party/angle/src/libANGLE/renderer/vulkan/vk_helpers.cpp:1243): false

[cc… via monorail]

Comment #9 on issue 5780 by cc...@google.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c9

I have a S20+ which has Exynos 990, but a little bit older driver (ARM, Vulkan 1.1.131 Mali-G77 (0x90800011), Mali-G77-26.0.0). I still can not reproduce it.

I also tried to disable XFB extension to see if I can reproduce, and still no.

I also tried 7e9b497c984d and revert 867411a, and still no repro on both S20 and the other device that has G78.

I also applied the assertion patch and run though the bots and no test hit this. (see https://chromium-review.googlesource.com/c/angle/angle/+/2523683)

Bellow are a few random notes to see how we can move forward on this:

The gfxbench I am using is corporate version gfxbench-5.0.0-1030+corporate.arm64-v8a.armeabi-v7a.x86.x86_64.apk. I launch the gfxbench and then in terminal, run the adb command to launch the test:
adb shell am broadcast -a net.kishonti.testfw.ACTION_RUN_TESTS -n net.kishonti.gfxbench.v50000.corporate/net.kishonti.benchui.corporate.CommandLineSession -e test_ids "gl_trex_off"
And mean while in another terminal I have adb logcat running and watch for result.

Could you also check your gfxbench version and command you are running so that I can try the exact steps here as well. If possible could you also run it on other devices and driver combinations to see if it is specific to device/driver?

Could you please also try with the trex trace to see if it also reproduces? If yes, that will be much easier to debug.

And as the last resort, we may need help from you guys to debug it. The assertion I added you should only hit if we have picked ImageLayout::ColorAttachmentAndShaderRead for the texture. This means we detect the texture is been attached to renderpass as color attachment AND it is also used for texture sampling. TRex does not use color textures that way, and that is why you are not expected to hit assertion.

[syous… via monorail]

Comment #10 on issue 5780 by syou...@chromium.org: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c10

Cheryl, is it possible that you are using an older version of gfxbench? A texture simultaneously bound to color attachment and used as sampler in T-Rex was brought up in issue 3453 in the past. That bug was never really fixed, T-Rex just stopped doing that (I am presuming due to an update). The symptoms here strongly hint at the same issue with whatever version of T-Rex was used to report issue 3453. TL;DR of that issue is that T-Rex has statically-used but effectively-unused samplers that confuses ANGLE into thinking the sample is active. A texture is simultaneously bound to that sampler and the framebuffer.

[chery… via monorail]

Comment #11 on issue 5780 by chery...@arm.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c11

Yes,What we are using is GLBenchmark30/corporate_3.0.6/1920x1080/trex_10fps/trex_10fps.orig.std.etc1.gles2.pat,and I checked that GFXBench40/gold/1920x1080/trex_10fps_onscreen/trex_10fps_onscreen.pat could also produce about 5% drop.

[cc… via monorail]

Comment #12 on issue 5780 by cc...@google.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c12

The latest gfxbench is 5.0, and that is what we are using here.

There are two possibilties that you see perf regression on gfxbench 4.0
1) It was using the wrong layout and now it is using the correct layout, but correct layout is slower (because you have to turn off AFBC etc.
2) Due to heavy barrier I added. I created CL to add ImageLayout::ColorAttachmentAndFragmentShaderRead, Could you try that? I have added you to the CC.

[chery… via monorail]

Comment #13 on issue 5780 by chery...@arm.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c13

https://chromium-review.googlesource.com/c/angle/angle/+/2795269 could reduce the decline by 8% with GLBenchmark30 on nevis(17%->9%), and we currently doesn't support GFXBench50/pc2/1920x1080/trex_10fps_onscreen_1080p/trex_10fps_onscreen_1080p.orig.pat.Could you please upload gfxbench-5.0.0-1030+corporate.arm64-v8a.armeabi-v7a.x86.x86_64.apk for me to check it?

[cc… via monorail]

Comment #14 on issue 5780 by cc...@google.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c14

The gfxbench we are using is corporate licensed so I don't think I am able to upload. Maybe check with other folks in ARM? It is also available on play store as well.

[Git Watcher via monorail]

Comment #15 on issue 5780 by Git Watcher: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c15

The following revision refers to this bug:
https://chromium.googlesource.com/angle/angle/+/fd3b8fafca5ea252cead6e5176c08814a9877997

commit fd3b8fafca5ea252cead6e5176c08814a9877997
Author: Charlie Lao <cc...@google.com>
Date: Tue Mar 30 18:55:49 2021

Vulkan: Split fragment shader from ColorAttachmentAndAllShadersRead

Even though it is not common usage case that the same texture is used as
attachment and texture, but aztec ruins is using it. And earlier version
of gfxbench's T-Rex also running into this. So performance is still
important. This CL splits the texture from fragment shader usage case
out so we can have lighter barrier.

Bug: angleproject:5780
Change-Id: Ifdcbfe24488fdac62826c5af6ecfbb05f87c0499
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2795269

Commit-Queue: Charlie Lao <cc...@google.com>
Reviewed-by: Shahbaz Youssefi <syou...@chromium.org>
Reviewed-by: Tim Van Patten <ti...@google.com>

[modify] https://crrev.com/fd3b8fafca5ea252cead6e5176c08814a9877997/src/libANGLE/renderer/vulkan/ContextVk.cpp
[modify] https://crrev.com/fd3b8fafca5ea252cead6e5176c08814a9877997/src/libANGLE/renderer/vulkan/vk_helpers.cpp
[modify] https://crrev.com/fd3b8fafca5ea252cead6e5176c08814a9877997/src/libANGLE/renderer/vulkan/vk_helpers.h

[chery… via monorail]

Comment #16 on issue 5780 by chery...@arm.com: ANGLE FPS of T-Rex got 17% drop on Nevis
https://bugs.chromium.org/p/angleproject/issues/detail?id=5780#c16

Hi Charlie,
Thanks for the reply,since this issue is hard to reproduce on your side,then,we will set out to do it.