glgeneratemipmap randomly crashes on 64bit qualcomm

22 views
Skip to first unread message

Rik Cabanier

unread,
Jul 20, 2022, 1:51:28 PMJul 20
to Graphics-dev
We recently finally move the Quest browser to 64 bit.
Almost immediately, we saw a large uptick in crashes in the `glgeneratemipmap` OpenGL call. We are unable to come up with reproduction steps so we're wondering if anyone else has seen this problem.
If so, did you find a workaround?

Thanks!

Zhenyao Mo

unread,
Jul 20, 2022, 6:06:51 PMJul 20
to Rik Cabanier, Graphics-dev
Not I am aware of. I also checked our driver bug work around list and
I saw nothing mentioning crashes in the glGenerateMipmap.

Rik Cabanier

unread,
Jul 20, 2022, 10:08:56 PMJul 20
to Graphics-dev, z...@chromium.org, Graphics-dev, Rik Cabanier
Thanks for checking!

Ken Russell

unread,
Jul 21, 2022, 1:26:36 AMJul 21
to Rik Cabanier, Graphics-dev, z...@chromium.org
For what it's worth, here's the currently-shipping code which invokes glGenerateMipmap in the GPU process on behalf of WebGL:

For reference, here's ANGLE's implementation of similar code, which isn't yet shipping on Android:

I don't see any Android-specific driver bug workarounds, and nothing related to crashes.

Do you have crash stacks? Is this coming from the browser's rasterization, or WebGL / WebXR content?

-Ken


Rik Cabanier

unread,
Jul 21, 2022, 2:39:13 AMJul 21
to Graphics-dev, Kenneth Russell, Graphics-dev, z...@chromium.org, Rik Cabanier
Thanks Ken!

At this point, we are unsure where it is coming from and we have no reproduction steps; just random crashes.
The crash only happens on Snapdragon 835 (Quest 1). The identical code works fine on Snapdragon XR2 (Quest 2) 
Here is a stack trace:

- (libGLESv2_adreno.so [offset: 0x18d8e4, 0AB8D3CC3F63EA659B32B33C2EBD013D0])
- (libGLESv2_adreno.so [offset: 0x18d840, 0AB8D3CC3F63EA659B32B33C2EBD013D0])
- gpu::gles2::GLES2DecoderImpl::DoGenerateMipmap(unsigned int) (../../gpu/command_buffer/service/gles2_cmd_decoder.cc:0)
- gpu::gles2::GLES2DecoderImpl::HandleGenerateMipmap(unsigned int, void const volatile*) (../../gpu/command_buffer/service/gles2_cmd_decoder_autogen.h:1143)
- gpu::error::Error gpu::gles2::GLES2DecoderImpl::DoCommandsImpl<false>(unsigned int, void const volatile*, int, int*) (../../gpu/command_buffer/service/gles2_cmd_decoder.cc:6145)
- gpu::CommandBufferService::Flush(int, gpu::AsyncAPIInterface*) (../../gpu/command_buffer/service/command_buffer_service.cc:70)
- gpu::CommandBufferStub::ExecuteDeferredRequest(gpu::mojom::DeferredCommandBufferRequestParams&) (../../gpu/ipc/service/command_buffer_stub.cc:501)
- gpu::GpuChannel::ExecuteDeferredRequest(mojo::StructPtr<gpu::mojom::DeferredRequestParams>) (../../gpu/ipc/service/gpu_channel.cc:670)
- base::internal::Invoker<base::internal::BindState<void (gpu::GpuChannel::*)(mojo::StructPtr<gpu::mojom::DeferredRequestParams>), base::WeakPtr<gpu::GpuChannel>, mojo::StructPtr<gpu::mojom::DeferredRequestParams> >, void ()>::RunOnce(base::internal::BindStateBase*) (../../base/bind_internal.h:542)
- gpu::Scheduler::RunNextTask() (../../base/callback.h:143)
- base::TaskAnnotator::RunTaskImpl(base::PendingTask&) (../../base/callback.h:143)
- base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() (../../base/task/common/task_annotator.h:74)
- base::MessagePumpDefault::Run(base::MessagePump::Delegate*) (../../base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:0)
- base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(bool, base::TimeDelta) (../../base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:498)
- base::RunLoop::Run(base::Location const&) (../../base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:0)
- content::GpuMain(content::MainFunctionParams) (../../content/gpu/gpu_main.cc:408)
- content::ContentMainRunnerImpl::Run() (../../content/app/content_main_runner_impl.cc:682)
- content::RunContentProcess(content::ContentMainParams, content::ContentMainRunner*) (../../content/app/content_main.cc:407)
- Java_J_N_M1Y_1XVCN (../../content/app/android/content_main_android.cc:51)
- (libart.so [offset: 0x143b50, 62901D7EB26E8F461D0F51806DEEBC3C0])
- (base.vdex [offset: 0x5fc4cf, module unmapped])
- (base.vdex [offset: 0x5fc4cf, module unmapped])
- (base.vdex [offset: 0x5fc4cf, module unmapped])
- art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*) (art/runtime/art_method.cc:363)
- art::ClassLinker::FindResolvedMethod(art::ObjPtr<art::mirror::Class>, art::ObjPtr<art::mirror::DexCache>, art::ObjPtr<art::mirror::ClassLoader>, unsigned int) (art/runtime/class_linker.cc:8515)
- (base.vdex [offset: 0x8b828, module unmapped])
- art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*) (art/runtime/interpreter/interpreter_common.cc:619)
- MterpAddHotnessBatch (art/runtime/jit/jit-inl.h:0)
- bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*) (art/runtime/common_dex_operations.h:70)
- MterpInvokeStatic (art/runtime/interpreter/interpreter_common.h:304)
- (base.vdex [offset: 0x17cbc6, module unmapped])
- (base.vdex [offset: 0x17cbc6, module unmapped])
- art::interpreter::MterpHandleIntrinsic(art::ShadowFrame*, art::ArtMethod*, art::Instruction const*, unsigned short, art::JValue*) (art/runtime/interpreter/interpreter_intrinsics.cc:283)
- MterpInvokeInterface (art/runtime/interpreter/interpreter_common.h:289)
- (base.vdex [offset: 0x17cbc6, module unmapped])
- (base.vdex [offset: 0x17cbc4, module unmapped])
- (base.vdex [offset: 0x14ff66, module unmapped])
- (base.vdex [offset: 0x17cbc4, module unmapped])
- (base.vdex [offset: 0x14ff66, module unmapped])
- (base.vdex [offset: 0x617411, module unmapped])
- art::interpreter::MterpHandleIntrinsic(art::ShadowFrame*, art::ArtMethod*, art::Instruction const*, unsigned short, art::JValue*) (art/runtime/interpreter/interpreter_intrinsics.cc:283)
- art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (art/runtime/interpreter/interpreter.cc:341)
- art::interpreter::EnterInterpreterFromEntryPoint(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*) (art/runtime/jit/jit-inl.h:0)
- (base.vdex [offset: 0x617411, module unmapped])
- artQuickToInterpreterBridge (art/runtime/entrypoints/quick/quick_trampoline_entrypoints.cc:791)
- (base.vdex [offset: 0x617411, module unmapped])
- (base.vdex [offset: 0x14ff66, module unmapped])
- (base.vdex [offset: 0x14fda6, module unmapped])
- (base.vdex [offset: 0x14fda6, module unmapped])
- (base.vdex [offset: 0x617411, module unmapped])
- (core-oj.jar [offset: 0x35c7ce, module unmapped])
- (boot.art [offset: 0x25bf4c, module unmapped])
- asinf (bionic/libm/upstream-freebsd/lib/msun/src/e_asinf.c:0)
- art::JniMethodEndWithReferenceHandleResult(_jobject*, unsigned int, art::Thread*) (art/runtime/entrypoints/quick/quick_jni_entrypoints.cc:0)
- (core-oj.jar [offset: 0x35c7ce, module unmapped])
- (core-oj.jar [offset: 0x35c7ce, module unmapped])
- (boot.oat [offset: 0x1a3088, C19844B709A8D813A7B11D130D2234B70])
- (boot.art [offset: 0x1ea45c, module unmapped])
- (core-oj.jar [offset: 0x35c7ce, module unmapped])
- (boot.art [offset: 0x1ea45c, module unmapped])
- art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*) (art/runtime/art_method.cc:361)
- art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*) (art/runtime/reflection.cc:458)
- (boot.art [offset: 0x1ea45c, module unmapped])
- (core-oj.jar [offset: 0x35c7ce, module unmapped])
- je_arena_malloc_hard (external/jemalloc_new/include/jemalloc/internal/mutex.h:216)
- je_malloc (external/jemalloc_new/include/jemalloc/internal/arena_inlines_b.h:105)
- je_free (external/jemalloc_new/include/jemalloc/internal/arena_inlines_b.h:182)
- art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue const*) (art/runtime/reflection.cc:608)
- (core-oj.jar [offset: 0x35c7ce, module unmapped])
- (boot.art [offset: 0x1ea45c, module unmapped])
- (core-oj.jar [offset: 0x35c7ce, module unmapped])
- art::Thread::CreateCallback(void*) (art/runtime/thread.cc:641)
- (boot.art [offset: 0x1ea45c, module unmapped])
- __pthread_start(void*) (bionic/libc/bionic/pthread_create.cpp:338)
- pthread_create (bionic/libc/bionic/pthread_create.cpp:0)
- __start_thread (bionic/libc/bionic/clone.cpp:53)
- art::Thread::InitAfterFork() (art/runtime/thread.cc:581)

Zhenyao Mo

unread,
Jul 21, 2022, 11:49:23 AMJul 21
to Rik Cabanier, Graphics-dev, Kenneth Russell
Is it possible this is an Out-of-Memory situation that the driver
fails to handle gracefully?

Rik Cabanier

unread,
Jul 21, 2022, 12:15:39 PMJul 21
to Graphics-dev, z...@chromium.org, Graphics-dev, Kenneth Russell, Rik Cabanier
That was one of the first things we looked into. Unfortunately, there is no memory shortage when this call fails.
Also, we never saw this crash on 32 bit and there are definitely instances there where we ran out of memory.

Ken Russell

unread,
Jul 22, 2022, 5:43:25 PMJul 22
to Rik Cabanier, Graphics-dev, z...@chromium.org, peng...@chromium.org, Geoff Lang
+penghuang and geofflang in case they remember any similar issues on 64-bit Adreno while working on the switch to Chrome's passthrough command decoder + ANGLE.

Rik, can you reproduce this at all? This command's coming from a renderer process, so it must be coming either from WebGL or 2D canvas. Core HTML rasterization's done in the GPU process so the call stack would be different if it were coming from there.

-Ken


Rik Cabanier

unread,
Jul 22, 2022, 5:47:31 PMJul 22
to Graphics-dev, Kenneth Russell, Graphics-dev, z...@chromium.org, peng...@chromium.org, Geoff Lang, Rik Cabanier
Interesting!
Can we call into the GL driver from a renderer?

Ken Russell

unread,
Jul 22, 2022, 6:36:34 PMJul 22
to Rik Cabanier, Graphics-dev, z...@chromium.org, peng...@chromium.org, Geoff Lang
On Fri, Jul 22, 2022 at 2:47 PM Rik Cabanier <caba...@gmail.com> wrote:
Interesting!
Can we call into the GL driver from a renderer?

No. The renderer is sandboxed and is not allowed to touch the graphics driver. Here, the GPU process is executing a glGenerateMipmap call on behalf of a renderer process. If it were coming from rasterization then Skia would be on the call stack.

Rik Cabanier

unread,
Jul 22, 2022, 7:54:23 PMJul 22
to Graphics-dev, Kenneth Russell, Graphics-dev, z...@chromium.org, peng...@chromium.org, Geoff Lang, Rik Cabanier
ok, I see. I misunderstood your previous message.
We have unable to reproduce this in house...

Rik Cabanier

unread,
Aug 5, 2022, 2:06:38 PMAug 5
to Graphics-dev, Rik Cabanier, Kenneth Russell, Graphics-dev, z...@chromium.org, peng...@chromium.org, Geoff Lang
In case it helps other people down the road, we tracked the majority of cases where this crash happened and implemented a fix:
A graphics framework always asked for mips on every generated texture. 
It turns out that on Snapdragon 835 in 64 bit, if you call glgeneratemipmap on a 1x1 texture it will *sometimes* crash.

Ken Russell

unread,
Aug 5, 2022, 2:17:37 PMAug 5
to Rik Cabanier, Graphics-dev, z...@chromium.org, peng...@chromium.org, Geoff Lang
Great diagnosis Rik! Thank you for sharing this with us. Filed https://bugs.chromium.org/p/angleproject/issues/detail?id=7555 to consider incorporating this driver bug workaround in ANGLE.

Reply all
Reply to author
Forward
0 new messages