Issue 1425198 in chromium: Step "ash_unittests on Ubuntu-20.04" failing on builder "chromium/ci/Linux ChromiumOS MSan Tests"

23 views
Skip to first unread message

afakhry via monorail

unread,
Mar 16, 2023, 6:21:37 PM3/16/23
to graphi...@chromium.org
Updates:
Cc: graphi...@chromium.org k...@chromium.org sun...@chromium.org kyle...@chromium.org
Components: -UI>Shell Internals>GPU>Internals
Owner: geof...@chromium.org

Comment #1 on issue 1425198 by afa...@chromium.org: Step "ash_unittests on Ubuntu-20.04" failing on builder "chromium/ci/Linux ChromiumOS MSan Tests"
https://bugs.chromium.org/p/chromium/issues/detail?id=1425198#c1

Doesn't seem related to capture mode. Many other tests fail as well.

It seems to be crashing at this line: https://source.chromium.org/chromium/chromium/src/+/main:ui/gl/gl_context_egl.cc;drc=c13d0b19698dacedfa4d2d71d8c4e18be0b2ee01;l=224

Adding some folks who may help triage further.


[ RUN ] CaptureModeDemoToolsTest.CaptureBoundsChangeTest
VERBOSE1 ash_unittests[40818:40818]: [power_event_observer.cc(250)] PowerEventObserver::PowerEventObserver lock=0, can_lock=0
VERBOSE1 ash_unittests[40818:40818]: [power_event_observer.cc(382)] PowerEventObserver::OnLoginStatusChanged
Received signal 11 SI_KERNEL000000000000
Possibly a General Protection Fault, can be due to a non-canonical address dereference. See "Intel 64 and IA-32 Architectures Software Developer?s Manual", Volume 1, Section 3.3.7.1.
#0 0x56310d3ebb58 in backtrace /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/msan/../sanitizer_common/sanitizer_common_interceptors.inc:4410:13
#1 0x56311c465524 in base::debug::CollectStackTrace(void**, unsigned long) ./../../base/debug/stack_trace_posix.cc:979:7
#2 0x56311c42a71a in StackTrace ./../../base/debug/stack_trace.cc:221:12
#3 0x56311c42a71a in base::debug::StackTrace::StackTrace() ./../../base/debug/stack_trace.cc:218:28
#4 0x56311c463f40 in base::debug::(anonymous namespace)::StackDumpSignalHandler(int, siginfo_t*, void*) ./../../base/debug/stack_trace_posix.cc:456:3
#5 0x56310d419178 in SignalAction(int, void*, void*) /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/msan/msan_interceptors.cpp:1052:3
#6 0x7fde9d47c420 in __funlockfile ??:?
#7 0x7fde9c234bc0 in __nss_database_lookup ??:?
#8 0x56310d42614e in __msan::SetShadow(void const*, unsigned long, unsigned char) /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/msan/msan_poisoning.cpp:215:9
#9 0x56310d3f21a7 in __msan_unpoison /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/msan/msan_interceptors.cpp:1608:3
#10 0x56310d3f21a7 in __tls_get_addr /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/msan/../sanitizer_common/sanitizer_common_interceptors.inc:5479:5
#9 0x7fde96a43633 <unknown>
#10 0x7fde969a03a2 <unknown>
#11 0x56311f36c370 in gl::GLContextEGL::Initialize(gl::GLSurface*, gl::GLContextAttribs const&) ./../../ui/gl/gl_context_egl.cc:224:8
#12 0x56311f27fe2f in gl::InitializeGLContext(scoped_refptr<gl::GLContext>, gl::GLSurface*, gl::GLContextAttribs const&) ./../../ui/gl/gl_context.cc:525:17
#13 0x56311470b11a in ui::GLOzoneEGL::CreateGLContext(gl::GLShareGroup*, gl::GLSurface*, gl::GLContextAttribs const&) ./../../ui/ozone/common/gl_ozone_egl.cc:87:10
#14 0x56311f8bf5ca in gl::init::CreateGLContext(gl::GLShareGroup*, gl::GLSurface*, gl::GLContextAttribs const&) ./../../ui/gl/init/gl_factory_ozone.cc:41:26
#15 0x563120e66e48 in gpu::GpuChannelManager::GetSharedContextState(gpu::ContextResult*) ./../../gpu/ipc/service/gpu_channel_manager.cc:906:9
#16 0x5631225745cd in viz::GpuServiceImpl::GetContextState() ./../../components/viz/service/gl/gpu_service_impl.cc:644:32
#17 0x5631221c0d2a in gpu::InProcessCommandBuffer::InitializeOnGpuThread(gpu::InProcessCommandBuffer::InitializeOnGpuThreadParams const&) ./../../gpu/ipc/in_process_command_buffer.cc:276:36
#18 0x5631221d08fe in Run ./../../base/functional/callback.h:152:12
#19 0x5631221d08fe in operator() ./../../gpu/ipc/in_process_command_buffer.cc:81:31
#20 0x5631221d08fe in Invoke<(lambda at ../../gpu/ipc/in_process_command_buffer.cc:79:18), base::OnceCallback<gpu::ContextResult ()>, gpu::ContextResult *, base::WaitableEvent *> ./../../base/functional/bind_internal.h:619:12
#21 0x5631221d08fe in MakeItSo<(lambda at ../../gpu/ipc/in_process_command_buffer.cc:79:18), std::Cr::tuple<base::OnceCallback<gpu::ContextResult ()>, base::internal::UnretainedWrapper<gpu::ContextResult, base::unretained_traits::MayNotDangle, (base::RawPtrTraits)0>, base::internal::UnretainedWrapper<base::WaitableEvent, base::unretained_traits::MayNotDangle, (base::RawPtrTraits)0> > > ./../../base/functional/bind_internal.h:923:12
#22 0x5631221d08fe in RunImpl<(lambda at ../../gpu/ipc/in_process_command_buffer.cc:79:18), std::Cr::tuple<base::OnceCallback<gpu::ContextResult ()>, base::internal::UnretainedWrapper<gpu::ContextResult, base::unretained_traits::MayNotDangle, (base::RawPtrTraits)0>, base::internal::UnretainedWrapper<base::WaitableEvent, base::unretained_traits::MayNotDangle, (base::RawPtrTraits)0> >, 0UL, 1UL, 2UL> ./../../base/functional/bind_internal.h:1018:12
#23 0x5631221d08fe in base::internal::Invoker<base::internal::BindState<base::OnceCallback<void ()> gpu::(anonymous namespace)::WrapTaskWithResult<gpu::ContextResult>(base::OnceCallback<gpu::ContextResult ()>, gpu::ContextResult*, base::WaitableEvent*)::'lambda'(base::OnceCallback<gpu::ContextResult ()>, gpu::ContextResult*, base::WaitableEvent*), base::OnceCallback<gpu::ContextResult ()>, base::internal::UnretainedWrapper<gpu::ContextResult, base::unretained_traits::MayNotDangle, (base::RawPtrTraits)0>, base::internal::UnretainedWrapper<base::WaitableEvent, base::unretained_traits::MayNotDangle, (base::RawPtrTraits)0>>, void ()>::RunOnce(base::internal::BindStateBase*) ./../../base/functional/bind_internal.h:969:12
#24 0x5631205ef6b5 in Run ./../../base/functional/callback.h:152:12
#25 0x5631205ef6b5 in gpu::Scheduler::RunNextTask() ./../../gpu/command_buffer/service/scheduler.cc:789:26
#26 0x56311c2ddb8d in Run ./../../base/functional/callback.h:152:12
#27 0x56311c2ddb8d in base::TaskAnnotator::RunTaskImpl(base::PendingTask&) ./../../base/task/common/task_annotator.cc:178:34
#28 0x56311c33d8e1 in RunTask<(lambda at ../../base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:478:11)> ./../../base/task/common/task_annotator.h:89:5
#29 0x56311c33d8e1 in base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl(base::LazyNow*) ./../../base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:476:23
#30 0x56311c33c58f in base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() ./../../base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:341:41
#31 0x56311c33f185 in non-virtual thunk to base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() ./../../base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:0:0
#32 0x56311c478563 in base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) ./../../base/message_loop/message_pump_libevent.cc:290:55
#33 0x56311c33fedf in base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(bool, base::TimeDelta) ./../../base/task/sequence_manager/thread_controller_with_message_pump_impl.cc:641:12
#34 0x56311c262b60 in base::RunLoop::Run(base::Location const&) ./../../base/run_loop.cc:134:14
#35 0x56311c3c5d2f in base::Thread::Run(base::RunLoop*) ./../../base/threading/thread.cc:345:13
#36 0x56311c3c6390 in base::Thread::ThreadMain() ./../../base/threading/thread.cc:417:3
#37 0x56311c41136b in base::(anonymous namespace)::ThreadFunc(void*) ./../../base/threading/platform_thread_posix.cc:101:13
#38 0x7fde9d470609 in start_thread ??:0:0
#39 0x7fde9c1c8133 in __clone ??:0:0
r8: 00007030000e9010 r9: 00007030000e9010 r10: 0000000000000022 r11: 0000000000000246
r12: 0000500000000000 r13: ff129eafff006000 r14: ff12eeafff006000 r15: ff129eafff00676e
di: ff12eeafff006000 si: 0000000000000000 bp: 000000000000076e bx: 0000500000000000
dx: 000000000000076e ax: ff12eeafff006000 cx: ff12eeafff006080 sp: 00007fde854e9188
ip: 00007fde9c234bc0 efl: 0000000000010206 cgf: 002b000000000033 erf: 0000000000000000
trp: 000000000000000d msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(EXIT_FAILURE). Core file will not be generated.

--
You received this message because:
1. You were specifically CC'd on the issue

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

Reply to this email to add a comment or make updates.

kbr via monorail

unread,
Mar 16, 2023, 6:28:40 PM3/16/23
to graphi...@chromium.org

Comment #2 on issue 1425198 by k...@chromium.org: Step "ash_unittests on Ubuntu-20.04" failing on builder "chromium/ci/Linux ChromiumOS MSan Tests"
https://bugs.chromium.org/p/chromium/issues/detail?id=1425198#c2

Could someone try to bisect this? Nothing jumps out from:
https://ci.chromium.org/ui/p/chromium/builders/ci/Linux%20ChromiumOS%20MSan%20Tests/b8786460552932183009/blamelist

takemoto via monorail

unread,
Mar 17, 2023, 9:13:10 AM3/17/23
to graphi...@chromium.org
Updates:
Status: Assigned

Comment #3 on issue 1425198 by take...@google.com: Step "ash_unittests on Ubuntu-20.04" failing on builder "chromium/ci/Linux ChromiumOS MSan Tests"
https://bugs.chromium.org/p/chromium/issues/detail?id=1425198#c3

(No comment was entered for this change.)

ynovikov via monorail

unread,
Mar 17, 2023, 9:46:59 AM3/17/23
to graphi...@chromium.org
Updates:
Cc: geof...@chromium.org
Components: UI>Shell
Labels: Pri-2
Owner: ----
Status: Available

Comment #4 on issue 1425198 by ynov...@chromium.org: Step "ash_unittests on Ubuntu-20.04" failing on builder "chromium/ci/Linux ChromiumOS MSan Tests"
https://bugs.chromium.org/p/chromium/issues/detail?id=1425198#c4
This is because this build https://ci.chromium.org/ui/p/chromium/builders/ci/Linux%20ChromiumOS%20MSan%20Tests/34540/overview is not the first failure.

ash_unittests is flaky at least since https://ci.chromium.org/ui/p/chromium/builders/ci/Linux%20ChromiumOS%20MSan%20Tests/33083/overview, but it had a different stack at that time:
#11 0x5567fc9b0c7f in DoFinish ./../../gpu/command_buffer/service/gles2_cmd_decoder.cc:6055:10
#12 0x5567fc9b0c7f in gpu::gles2::GLES2DecoderImpl::HandleFinish(unsigned int, void const volatile*) ./../../gpu/command_buffer/service/gles2_cmd_decoder_autogen.h:996:3
#13 0x5567fca3b961 in gpu::error::Error gpu::gles2::GLES2DecoderImpl::DoCommandsImpl<false>(unsigned int, void const volatile*, int, int*) ./../../gpu/command_buffer/service/gles2_cmd_decoder.cc:5985:18
#14 0x5567fd0ae6ab in gpu::CommandBufferService::Flush(int, gpu::AsyncAPIInterface*) ./../../gpu/command_buffer/service/command_buffer_service.cc:226:35
#15 0x5567fe39de5d in gpu::InProcessCommandBuffer::FlushOnGpuThread(int, std::Cr::vector<gpu::SyncToken, std::Cr::allocator<gpu::SyncToken>> const&) ./../../gpu/ipc/in_process_command_buffer.cc:625:22

Then something changed around https://ci.chromium.org/ui/p/chromium/builders/ci/Linux%20ChromiumOS%20MSan%20Tests/33751/overview and the stack became:
#13 0x55ba74b7e797 in GrGLFunction<void ()>::GrGLFunction<GrGLFunction<void ()> gl::init::(anonymous namespace)::bind_impl<false, true, false, void>(void (*)(), gl::ProgressReporter*)::'lambda'()>(GrGLFunction<void ()> gl::init::(anonymous namespace)::bind_impl<false, true, false, void>(void (*)(), gl::ProgressReporter*)::'lambda'())::'lambda'(void const*)::__invoke(void const*) ./../../third_party/skia/include/gpu/gl/GrGLFunctions.h:286:17
#14 0x55ba71d0e446 in operator() ./../../third_party/skia/include/gpu/gl/GrGLFunctions.h:294:16
#15 0x55ba71d0e446 in finishOutstandingGpuWork ./../../third_party/skia/src/gpu/ganesh/gl/GrGLGpu.cpp:4270:5
#16 0x55ba71d0e446 in GrGLGpu::onSubmitToGpu(bool) ./../../third_party/skia/src/gpu/ganesh/gl/GrGLGpu.cpp:4153:15
#17 0x55ba7179e61e in GrGpu::submitToGpu(bool) ./../../third_party/skia/src/gpu/ganesh/GrGpu.cpp:757:28
#18 0x55ba71753cfb in GrDirectContext::submit(bool) ./../../third_party/skia/src/gpu/ganesh/GrDirectContext.cpp:432:18
#19 0x55ba765d129a in viz::SkiaOutputSurfaceImplOnGpu::~SkiaOutputSurfaceImplOnGpu() ./../../components/viz/service/display_embedder/skia_output_surface_impl_on_gpu.cc:379:19
#20 0x55ba765d3c6d in viz::SkiaOutputSurfaceImplOnGpu::~SkiaOutputSurfaceImplOnGpu() skia_output_surface_impl_on_gpu.cc:0:0

First occurrence of the recent stack is in https://ci.chromium.org/ui/p/chromium/builders/ci/Linux%20ChromiumOS%20MSan%20Tests/33817/overview
And the last of the previous stack is in https://ci.chromium.org/ui/p/chromium/builders/ci/Linux%20ChromiumOS%20MSan%20Tests/33798/overview

So, the blamelist one needs to look at is https://chromium.googlesource.com/chromium/src/+log/03ecc1cf2604bf7c1dc415f39b76cd67eccb87e8..5849d28a9b0cbf92a28b0e9ba1c6b22d43328da4

Doesn't look like P1 if this fails since 2023-01-07.
Also doesn't look like the frequency increased recently, 69 failures in last 200 builds
https://ci.chromium.org/p/chromium/builders/ci/Linux%20ChromiumOS%20MSan%20Tests?cursor=id%3E8786374849478078161&limit=200
and there were 93 failures 1 month ago
https://ci.chromium.org/p/chromium/builders/ci/Linux%20ChromiumOS%20MSan%20Tests?cursor=id%3E8788962629743532673&limit=200

tsergeant via monorail

unread,
Mar 19, 2023, 11:09:50 PM3/19/23
to graphi...@chromium.org
Updates:
Mergedinto: 1404443
Status: Duplicate

Comment #5 on issue 1425198 by tser...@chromium.org: Step "ash_unittests on Ubuntu-20.04" failing on builder "chromium/ci/Linux ChromiumOS MSan Tests"
https://bugs.chromium.org/p/chromium/issues/detail?id=1425198#c5


(No comment was entered for this change.)

Reply all
Reply to author
Forward
0 new messages