glgeneratemipmap randomly crashes on 64bit qualcomm

[Rik Cabanier]

We recently finally move the Quest browser to 64 bit.

Almost immediately, we saw a large uptick in crashes in the `glgeneratemipmap` OpenGL call. We are unable to come up with reproduction steps so we're wondering if anyone else has seen this problem.

If so, did you find a workaround?

Thanks!

[Zhenyao Mo]

Not I am aware of. I also checked our driver bug work around list and
I saw nothing mentioning crashes in the glGenerateMipmap.

[Rik Cabanier]

Thanks for checking!

[Ken Russell]

For what it's worth, here's the currently-shipping code which invokes glGenerateMipmap in the GPU process on behalf of WebGL:

https://source.chromium.org/chromium/chromium/src/+/main:gpu/command_buffer/service/gles2_cmd_decoder.cc;l=6939?q=DoGenerateMipmap

For reference, here's ANGLE's implementation of similar code, which isn't yet shipping on Android:

https://source.chromium.org/chromium/chromium/src/+/main:third_party/angle/src/libANGLE/renderer/gl/TextureGL.cpp;l=1351?q=TextureGL::generateMipmap

I don't see any Android-specific driver bug workarounds, and nothing related to crashes.

Do you have crash stacks? Is this coming from the browser's rasterization, or WebGL / WebXR content?

-Ken

[Rik Cabanier]

Thanks Ken!

At this point, we are unsure where it is coming from and we have no reproduction steps; just random crashes.

The crash only happens on Snapdragon 835 (Quest 1). The identical code works fine on Snapdragon XR2 (Quest 2) 

Here is a stack trace:

- (libGLESv2_adreno.so [offset: 0x18d8e4, 0AB8D3CC3F63EA659B32B33C2EBD013D0])

- (libGLESv2_adreno.so [offset: 0x18d840, 0AB8D3CC3F63EA659B32B33C2EBD013D0])

- gpu::gles2::GLES2DecoderImpl::DoGenerateMipmap(unsigned int) (../../gpu/command_buffer/service/gles2_cmd_decoder.cc:0)

- gpu::gles2::GLES2DecoderImpl::HandleGenerateMipmap(unsigned int, void const volatile*) (../../gpu/command_buffer/service/gles2_cmd_decoder_autogen.h:1143)

- gpu::error::Error gpu::gles2::GLES2DecoderImpl::DoCommandsImpl<false>(unsigned int, void const volatile*, int, int*) (../../gpu/command_buffer/service/gles2_cmd_decoder.cc:6145)

- gpu::CommandBufferService::Flush(int, gpu::AsyncAPIInterface*) (../../gpu/command_buffer/service/command_buffer_service.cc:70)

- gpu::CommandBufferStub::ExecuteDeferredRequest(gpu::mojom::DeferredCommandBufferRequestParams&) (../../gpu/ipc/service/command_buffer_stub.cc:501)

- gpu::GpuChannel::ExecuteDeferredRequest(mojo::StructPtr<gpu::mojom::DeferredRequestParams>) (../../gpu/ipc/service/gpu_channel.cc:670)

- base::internal::Invoker<base::internal::BindState<void (gpu::GpuChannel::*)(mojo::StructPtr<gpu::mojom::DeferredRequestParams>), base::WeakPtr<gpu::GpuChannel>, mojo::StructPtr<gpu::mojom::DeferredRequestParams> >, void ()>::RunOnce(base::internal::BindStateBase*) (../../base/bind_internal.h:542)

- gpu::Scheduler::RunNextTask() (../../base/callback.h:143)

- base::TaskAnnotator::RunTaskImpl(base::PendingTask&) (../../base/callback.h:143)

- base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() (../../base/task/common/task_annotator.h:74)

- base::MessagePumpDefault::Run(base::MessagePump::Delegate*) (../../base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:0)

- base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(bool, base::TimeDelta) (../../base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:498)

- base::RunLoop::Run(base::Location const&) (../../base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:0)

- content::GpuMain(content::MainFunctionParams) (../../content/gpu/gpu_main.cc:408)

- content::ContentMainRunnerImpl::Run() (../../content/app/content_main_runner_impl.cc:682)

- content::RunContentProcess(content::ContentMainParams, content::ContentMainRunner*) (../../content/app/content_main.cc:407)

- Java_J_N_M1Y_1XVCN (../../content/app/android/content_main_android.cc:51)

- (libart.so [offset: 0x143b50, 62901D7EB26E8F461D0F51806DEEBC3C0])

- (base.vdex [offset: 0x5fc4cf, module unmapped])

- (base.vdex [offset: 0x5fc4cf, module unmapped])

- (base.vdex [offset: 0x5fc4cf, module unmapped])

- art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*) (art/runtime/art_method.cc:363)

- art::ClassLinker::FindResolvedMethod(art::ObjPtr<art::mirror::Class>, art::ObjPtr<art::mirror::DexCache>, art::ObjPtr<art::mirror::ClassLoader>, unsigned int) (art/runtime/class_linker.cc:8515)

- (base.vdex [offset: 0x8b828, module unmapped])

- art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*) (art/runtime/interpreter/interpreter_common.cc:619)

- MterpAddHotnessBatch (art/runtime/jit/jit-inl.h:0)

- bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*) (art/runtime/common_dex_operations.h:70)

- MterpInvokeStatic (art/runtime/interpreter/interpreter_common.h:304)

- (base.vdex [offset: 0x17cbc6, module unmapped])

- (base.vdex [offset: 0x17cbc6, module unmapped])

- art::interpreter::MterpHandleIntrinsic(art::ShadowFrame*, art::ArtMethod*, art::Instruction const*, unsigned short, art::JValue*) (art/runtime/interpreter/interpreter_intrinsics.cc:283)

- MterpInvokeInterface (art/runtime/interpreter/interpreter_common.h:289)

- (base.vdex [offset: 0x17cbc6, module unmapped])

- (base.vdex [offset: 0x17cbc4, module unmapped])

- (base.vdex [offset: 0x14ff66, module unmapped])

- (base.vdex [offset: 0x17cbc4, module unmapped])

- (base.vdex [offset: 0x14ff66, module unmapped])

- (base.vdex [offset: 0x617411, module unmapped])

- art::interpreter::MterpHandleIntrinsic(art::ShadowFrame*, art::ArtMethod*, art::Instruction const*, unsigned short, art::JValue*) (art/runtime/interpreter/interpreter_intrinsics.cc:283)

- art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (art/runtime/interpreter/interpreter.cc:341)

- art::interpreter::EnterInterpreterFromEntryPoint(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*) (art/runtime/jit/jit-inl.h:0)

- (base.vdex [offset: 0x617411, module unmapped])

- artQuickToInterpreterBridge (art/runtime/entrypoints/quick/quick_trampoline_entrypoints.cc:791)

- (base.vdex [offset: 0x617411, module unmapped])

- (base.vdex [offset: 0x14ff66, module unmapped])

- (base.vdex [offset: 0x14fda6, module unmapped])

- (base.vdex [offset: 0x14fda6, module unmapped])

- (base.vdex [offset: 0x617411, module unmapped])

- (core-oj.jar [offset: 0x35c7ce, module unmapped])

- (boot.art [offset: 0x25bf4c, module unmapped])

- asinf (bionic/libm/upstream-freebsd/lib/msun/src/e_asinf.c:0)

- art::JniMethodEndWithReferenceHandleResult(_jobject*, unsigned int, art::Thread*) (art/runtime/entrypoints/quick/quick_jni_entrypoints.cc:0)

- (core-oj.jar [offset: 0x35c7ce, module unmapped])

- (core-oj.jar [offset: 0x35c7ce, module unmapped])

- (boot.oat [offset: 0x1a3088, C19844B709A8D813A7B11D130D2234B70])

- (boot.art [offset: 0x1ea45c, module unmapped])

- (core-oj.jar [offset: 0x35c7ce, module unmapped])

- (boot.art [offset: 0x1ea45c, module unmapped])

- art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*) (art/runtime/art_method.cc:361)

- art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*) (art/runtime/reflection.cc:458)

- (boot.art [offset: 0x1ea45c, module unmapped])

- (core-oj.jar [offset: 0x35c7ce, module unmapped])

- je_arena_malloc_hard (external/jemalloc_new/include/jemalloc/internal/mutex.h:216)

- je_malloc (external/jemalloc_new/include/jemalloc/internal/arena_inlines_b.h:105)

- je_free (external/jemalloc_new/include/jemalloc/internal/arena_inlines_b.h:182)

- art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue const*) (art/runtime/reflection.cc:608)

- (core-oj.jar [offset: 0x35c7ce, module unmapped])

- (boot.art [offset: 0x1ea45c, module unmapped])

- (core-oj.jar [offset: 0x35c7ce, module unmapped])

- art::Thread::CreateCallback(void*) (art/runtime/thread.cc:641)

- (boot.art [offset: 0x1ea45c, module unmapped])

- __pthread_start(void*) (bionic/libc/bionic/pthread_create.cpp:338)

- pthread_create (bionic/libc/bionic/pthread_create.cpp:0)

- __start_thread (bionic/libc/bionic/clone.cpp:53)

- art::Thread::InitAfterFork() (art/runtime/thread.cc:581)

[Zhenyao Mo]

Is it possible this is an Out-of-Memory situation that the driver
fails to handle gracefully?

[Rik Cabanier]

That was one of the first things we looked into. Unfortunately, there is no memory shortage when this call fails.

Also, we never saw this crash on 32 bit and there are definitely instances there where we ran out of memory.

[Ken Russell]

+penghuang and geofflang in case they remember any similar issues on 64-bit Adreno while working on the switch to Chrome's passthrough command decoder + ANGLE.

Rik, can you reproduce this at all? This command's coming from a renderer process, so it must be coming either from WebGL or 2D canvas. Core HTML rasterization's done in the GPU process so the call stack would be different if it were coming from there.

-Ken

[Rik Cabanier]

Interesting!

Can we call into the GL driver from a renderer?

[Ken Russell]

On Fri, Jul 22, 2022 at 2:47 PM Rik Cabanier <caba...@gmail.com> wrote:

Interesting!

Can we call into the GL driver from a renderer?

No. The renderer is sandboxed and is not allowed to touch the graphics driver. Here, the GPU process is executing a glGenerateMipmap call on behalf of a renderer process. If it were coming from rasterization then Skia would be on the call stack.

[Rik Cabanier]

ok, I see. I misunderstood your previous message.

We have unable to reproduce this in house...

[Rik Cabanier]

In case it helps other people down the road, we tracked the majority of cases where this crash happened and implemented a fix:

A graphics framework always asked for mips on every generated texture. 

It turns out that on Snapdragon 835 in 64 bit, if you call glgeneratemipmap on a 1x1 texture it will *sometimes* crash.

[Ken Russell]

Great diagnosis Rik! Thank you for sharing this with us. Filed https://bugs.chromium.org/p/angleproject/issues/detail?id=7555 to consider incorporating this driver bug workaround in ANGLE.