mojo::InterfaceEndpointClient::`scalar deleting destructor' ThreadChecker crash at exit

[Nigel Tao]

I'm mojofying the ChromeViewHostMsg_FieldTrialActivated message: https://codereview.chromium.org/2453313004

Trybots are all happy, except for win_chromium_rel_ng and win_chromium_x64_rel_ng.

https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/326489/steps/browser_tests%20%28with%20patch%29%20on%20Windows-7-SP1/logs/PhishingClassifierTest.DisableDetection

says:

PhishingClassifierTest.DisableDetection (run #1):

[ RUN      ] PhishingClassifierTest.DisableDetection

[4924:4744:1105/174813:ERROR:proxy_service_factory.cc(130)] Cannot use V8 Proxy resolver in single process mode.

[4924:4744:1105/174813:ERROR:proxy_service_factory.cc(130)] Cannot use V8 Proxy resolver in single process mode.

[4924:4844:1105/174813:WARNING:chrome_browser_main_win.cc(419)] Command line too long for RegisterApplicationRestart

[4924:2500:1105/174813:WARNING:histograms.cc(40)] Started multiple compositor clients (Browser, Renderer) in one process. Some metrics will be disabled.

[4924:4844:1105/174813:WARNING:pref_notifier_impl.cc(23)] Pref observer found at shutdown.

[4924:4844:1105/174813:WARNING:pref_notifier_impl.cc(23)] Pref observer found at shutdown.

[4924:4844:1105/174813:WARNING:pref_notifier_impl.cc(23)] Pref observer found at shutdown.

[4924:4844:1105/174813:WARNING:url_request_context_getter.cc(43)] URLRequestContextGetter leaking due to no owning thread.

[       OK ] PhishingClassifierTest.DisableDetection (254 ms)

[----------] 1 test from PhishingClassifierTest (254 ms total)

[----------] Global test environment tear-down

[==========] 1 test from 1 test case ran. (255 ms total)

[  PASSED  ] 1 test.

[4924:4844:1105/174813:FATAL:interface_endpoint_client.cc(167)] Check failed: thread_checker_.CalledOnValidThread().

Backtrace:

base::debug::StackTrace::StackTrace [0x0240CEF7+23]

logging::LogMessage::~LogMessage [0x023C2EB1+49]

mojo::InterfaceEndpointClient::~InterfaceEndpointClient [0x02EC7012+103]

mojo::InterfaceEndpointClient::`scalar deleting destructor' [0x02EC71FF+11]

mojo::internal::InterfacePtrState<ui::mojom::WindowTreeFactory,1>::~InterfacePtrState<ui::mojom::WindowTreeFactory,1> [0x0217101E+20]

ChromeContentRendererClient::~ChromeContentRendererClient [0x02735A00+140]

base::LazyInstance<extensions::WorkerThreadDispatcher,base::DefaultLazyInstanceTraits<extensions::WorkerThreadDispatcher> >::OnExit [0x0377989C+15]

base::internal::Invoker<base::internal::BindState<void (__cdecl*)(void const *),void const *>,void __cdecl(void)>::Run [0x02413E8E+14]

base::AtExitManager::ProcessCallbacksNow [0x02413BE7+471]

base::AtExitManager::~AtExitManager [0x024139D3+163]

std::default_delete<base::AtExitManager>::operator() [0x023ACC6F+17]

base::TestSuite::~TestSuite [0x02465141+44]

content::LaunchTests [0x027E2B48+585]

LaunchChromeTests [0x04AC9923+49]

main [0x04AC973C+63]

__scrt_common_main_seh [0x04A8E0DF+249] (f:\ddtools\crtщtartup\src\startup\exe_common.inl:253)

BaseThreadInitThunk [0x74AC337A+18]

RtlInitializeExceptionChain [0x771092B2+99]

RtlInitializeExceptionChain [0x77109285+54]

First up, a simple noob question: how do I tell whether that stack trace is for the browser process, renderer process or something else?

My CL does indeed introduce a ThreadChecker (https://codereview.chromium.org/2453313004/diff/40001/chrome/browser/field_trial_recorder_impl.h) but I'm not enough of a Mojo or C++ wizard to understand the stack trace, which contains the line:

    mojo::InterfaceEndpointClient::`scalar deleting destructor' [0x02EC71FF+11]

I also have no idea how to debug this. My workstation is Linux and the only failing trybots are Windows (but win_clang is happy).

I also find it weird that this fails for release builds (the "rel" in "win_chromium_rel_ng" stands for "release", right?), in that, IIUC, ThreadChecker's are no-ops in release mode.

I saw a recent chromium-mojo thread "AssociatedInterfacePtr on WorkerThread" (https://groups.google.com/a/chromium.org/d/topic/chromium-mojo/0mYM091HbLM/discussion), which also mentions thread_checker_.CalledOnValidThread, but I don't know whether or not it's the same underlying issue. The stack trace in that discussion doesn't mention AtExitManager the way mine does.

Any suggestions?

[Daniel Cheng]

Some thoughts inline:

This is in the renderer, since ChromeContentRendererClient is on the stack.

 

My CL does indeed introduce a ThreadChecker (https://codereview.chromium.org/2453313004/diff/40001/chrome/browser/field_trial_recorder_impl.h) but I'm not enough of a Mojo or C++ wizard to understand the stack trace, which contains the line:

    mojo::InterfaceEndpointClient::`scalar deleting destructor' [0x02EC71FF+11]

This is just a fancy name for ~InterfaceEndpointClient: scalar delete in this context means we're doing a normal delete (as opposed to array delete[]). In this case, the log message also points to the destructor of InterfaceEndpointClient:

[4924:4844:1105/174813:FATAL:interface_endpoint_client.cc(167)] Check failed: thread_checker_.CalledOnValidThread().

 

I also have no idea how to debug this. My workstation is Linux and the only failing trybots are Windows (but win_clang is happy).

I also find it weird that this fails for release builds (the "rel" in "win_chromium_rel_ng" stands for "release", right?), in that, IIUC, ThreadChecker's are no-ops in release mode.

Our release builders build with dcheck_always_on = 1. The test in question appears to be disabled on Linux and run in single-process mode. The only suspicious thing I see here is single-process mode: if you can repro locally, you might be able to see what thread the interface is getting requested on, and why it's getting destroyed on a separate thread.

Daniel

 

I saw a recent chromium-mojo thread "AssociatedInterfacePtr on WorkerThread" (https://groups.google.com/a/chromium.org/d/topic/chromium-mojo/0mYM091HbLM/discussion), which also mentions thread_checker_.CalledOnValidThread, but I don't know whether or not it's the same underlying issue. The stack trace in that discussion doesn't mention AtExitManager the way mine does.

Any suggestions? 

--
You received this message because you are subscribed to the Google Groups "chromium-mojo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-moj...@chromium.org.
To post to this group, send email to chromi...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-mojo/CAEdON6ZBGt3rJEupZZbEGqp57jZ50sOECc7PzP1qzHoDAuZCzQ%40mail.gmail.com.

[Ken Rockot]

Basically, look at the bottom of the stack. In this case content::LaunchTests implies the process is the main test process -- in-process browser tests run the browser in the test process. As Daniel points out this is a single-process test, which explains why we see renderer code in the stack as well.

 

My CL does indeed introduce a ThreadChecker (https://codereview.chromium.org/2453313004/diff/40001/chrome/browser/field_trial_recorder_impl.h) but I'm not enough of a Mojo or C++ wizard to understand the stack trace, which contains the line:

    mojo::InterfaceEndpointClient::`scalar deleting destructor' [0x02EC71FF+11]

I also have no idea how to debug this. My workstation is Linux and the only failing trybots are Windows (but win_clang is happy).

I also find it weird that this fails for release builds (the "rel" in "win_chromium_rel_ng" stands for "release", right?), in that, IIUC, ThreadChecker's are no-ops in release mode.

I saw a recent chromium-mojo thread "AssociatedInterfacePtr on WorkerThread" (https://groups.google.com/a/chromium.org/d/topic/chromium-mojo/0mYM091HbLM/discussion), which also mentions thread_checker_.CalledOnValidThread, but I don't know whether or not it's the same underlying issue. The stack trace in that discussion doesn't mention AtExitManager the way mine does.

Any suggestions?

My guess is, ChromeRenderThreadObserver is initialized and used primarily on the in-process RenderThreadImpl, and so that's where your FieldTrialRecorderPtr is bound.

Meanwhile, ChromeContentRendererClient isn't destroyed until shutdown, on the main thread. Just so happens that nothing else it owns is doing thread checks now. This seems like a bug and we should be able to tear down the ContentRendererClient before tearing down the RenderThread.

--
You received this message because you are subscribed to the Google Groups "chromium-mojo" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-mojo+unsubscribe@chromium.org.

[Ken Rockot]

And this would only affect single-process tests, since in multiprocess mode, the RenderThread is the main thread in the render process. 

[Nigel Tao]

On Sun, Nov 6, 2016 at 7:27 PM, Ken Rockot <roc...@chromium.org> wrote:
> This seems like a bug and we should be able to tear down the
> ContentRendererClient before tearing down the RenderThread.

I filed http://crbug.com/672646