"service_manager:connector" prevented service: content_renderer from binding interface

[Nigel Tao]

I have a CL: https://codereview.chromium.org/2453313004/

I've added my new mojo interface name to a manifest_overlay.json file (https://codereview.chromium.org/2453313004/diff/80001/chrome/browser/chrome_content_browser_manifest_overlay.json), and things seem OK on my local machine (vanilla Linux).

The commit queue says that linux_chromium_chromeos_rel_ng is unhappy: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/316786

Close to the very bottom of https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/316786/steps/mash_browser_tests%20%28with%20patch%29/logs/stdio is this message:

[21703:21724:1116/210328:ERROR:render_process_host_impl.cc(2997)] Terminating render process for bad Mojo message: Received bad user message: InterfaceProviderSpec "service_manager:connector" prevented service: content_renderer from binding interface: chrome::mojom::FieldTrialRecorder exposed by: content_browser

Indeed, "chrome::mojom::FieldTrialRecorder" is my new interface. So maybe this really is a problem with my new code. Am I missing another json file somewhere? I'm not overly familiar with the chromeos builds.

Find-on-page gives 37 hits for "received bad user", and a lot of them are for FieldTrialRecorder. But not all of them are: there's also error messages for "translate::mojom::ContentTranslateDriver" and "metrics::mojom::LeakDetector". So maybe this isn't my problem??

Furthermore, just find-on-page-ing for "InterfaceProviderSpec "service_manager:connector" prevented service: content_renderer" gives 126 hits. Sometimes it seems to be fatal, sometimes not. I don't know why.

Is this a problem with my CL or not? How do I tell? If it is, it feels like I just need to add a 1-line patch to a .json file somewhere, but which .json file??

I see stdio output:

----

[21703:21724:1116/210328:ERROR:render_process_host_impl.cc(2997)] Terminating render process for bad Mojo message: Received bad user message: InterfaceProviderSpec "service_manager:connector" prevented service: content_renderer from binding interface: chrome::mojom::FieldTrialRecorder exposed by: content_browser

InterfaceRegistry(service_manager:connector):

Owned by:

  content_browser@ run as: 7057ba96-bb29-46b2-a534-3e80bc0072c3

  Providing:

    capability: gpu containing interfaces:

      device::mojom::PowerMonitor

    capability: plugin containing interfaces:

      device::mojom::PowerMonitor

    capability: renderer containing interfaces:

      blink::mojom::BackgroundSyncService

      blink::mojom::BroadcastChannelProvider

      blink::mojom::BudgetService

      blink::mojom::Hyphenation

      blink::mojom::MimeRegistry

      blink::mojom::NotificationService

      blink::mojom::OffscreenCanvasCompositorFrameSinkProvider

      blink::mojom::OffscreenCanvasSurface

      blink::mojom::PermissionService

      blink::mojom::ShapeDetection

      blink::mojom::WebSocket

      content::mojom::MemoryCoordinatorHandle

      content::mojom::ServiceWorkerDispatcherHost

      content::mojom::StoragePartitionService

      content::mojom::URLLoaderFactory

      content::mojom::VideoCaptureHost

      device::BatteryMonitor

      device::mojom::LightSensor

      device::mojom::MotionSensor

      device::mojom::OrientationSensor

      device::mojom::PowerMonitor

      device::mojom::TimeZoneMonitor

      media::mojom::ImageCapture

      memory_coordinator::mojom::MemoryCoordinatorHandle

      payments::mojom::PaymentAppManager

    capability: service_manager:service_factory containing interfaces:

      service_manager::mojom::ServiceFactory

  Requiring:

    From: *:

      app

    From: content_browser:

      ash

    From: content_gpu:

      browser

    From: content_plugin:

      browser

    From: content_renderer:

      browser

    From: content_utility:

      browser

    From: file:

      file:filesystem

      file:leveldb

    From: media:

      media:media

    From: service_manager:

      service_manager:client_process

      service_manager:instance_name

      service_manager:user_id

Bound to:

  content_renderer@1_2 run as: 7057ba96-bb29-46b2-a534-3e80bc0072c3

  Providing:

    capability: browser containing interfaces:

      IPC::mojom::ChannelBootstrap

      content::mojom::EmbeddedWorkerInstanceClient

      content::mojom::EmbeddedWorkerSetup

      content::mojom::FrameFactory

      content::mojom::RenderWidgetWindowTreeClientFactory

      visitedlink::mojom::VisitedLinkNotificationSink

      web_cache::mojom::WebCache

    capability: service_manager:service_factory containing interfaces:

      service_manager::mojom::ServiceFactory

  Requiring:

    From: content_browser:

      renderer

    From: ui:

      ui:gpu_client

Binders registered for:

 * blink::mojom::BackgroundSyncService

 * blink::mojom::BroadcastChannelProvider

 * blink::mojom::BudgetService

 * blink::mojom::MimeRegistry

 * blink::mojom::NotificationService

 * blink::mojom::OffscreenCanvasCompositorFrameSinkProvider

 * blink::mojom::OffscreenCanvasSurface

 * blink::mojom::PermissionService

 * blink::mojom::WebSocket

 * content::mojom::StoragePartitionService

 * content::mojom::VideoCaptureHost

 * device::BatteryMonitor

 * device::mojom::LightSensor

 * device::mojom::MotionSensor

 * device::mojom::OrientationSensor

 * device::mojom::PowerMonitor

 * device::mojom::TimeZoneMonitor

 * media::mojom::ImageCapture

 * payments::mojom::PaymentAppManager

 * - denotes an interface exposed to remote per policy.

----

but I don't know what I'm meant to be looking for in that message, or whether I'm meant to be looking for the presence or something or the absence of something.

[Ben Goodger]

I've seen stuff like this before and it's almost guaranteed to be related to the wacky manifest munging the cros mash_browser_tests target does somehow failing.

You can do a chromeos build on linux by adding these lines to gn args:

target_os="chromeos"

enable_pacage_mash_services=true

and then building mash_browser_tests

then run browser_tests with --run-in-mash --service-overrides=chrome/app/mash/mash_service_overrides.json

-Ben

--
You received this message because you are subscribed to the Google Groups "chromium-mojo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-mojo+unsubscribe@chromium.org.
To post to this group, send email to chromi...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-mojo/CAEdON6Y-CAN6bfr7wxdoyam6VuTp%2BrhE5FRZn7NoF5_DGR%2BrEg%40mail.gmail.com.

[Nigel Tao]

On Thu, Nov 17, 2016 at 5:52 PM, Ben Goodger <b...@chromium.org> wrote:
> I've seen stuff like this before and it's almost guaranteed to be related to the wacky manifest munging the cros mash_browser_tests target does somehow failing.

Do you have any leads on reading up on this "wacky manifest munging"?
I'm still relatively new to large parts of the Chromium codebase.

> You can do a chromeos build on linux by adding these lines to gn args:
>
> target_os="chromeos"
> enable_pacage_mash_services=true

I assume that "pacage" should be "package".

> and then building mash_browser_tests
>
> then run browser_tests with --run-in-mash --service-overrides=chrome/app/mash/mash_service_overrides.json

Even in a clean branch, with browser_tests --run-in-mash, a new window
pops up with a "Quick Launch" box and something task-bar-like (sorry,
I don't know cros nomenclature), but all of the tests are (slowly)
timing out:

----
[20/6637] PlatformAppBrowserTest.OpenLink (TIMED OUT)
[ RUN ] PlatformAppBrowserTest.MutationEventsDisabled
[26711:26711:1118/140745:264600306456:WARNING:diagnostics_writer.cc(209)]
[FAIL] 008 PathUserData (Cannot obtain size for:
/tmp/.org.chromium.Chromium.8cy7Kj/dCkVWpB)
[26711:26711:1118/140745:264600306492:WARNING:diagnostics_writer.cc(209)]
[FAIL] 007 PathLocalState (Path not found:
/tmp/.org.chromium.Chromium.8cy7Kj/dCkVWpB/Local State)
[26711:26711:1118/140745:264600306584:WARNING:diagnostics_writer.cc(209)]
[FAIL] 013 JSONPreferences (File not found)
[26711:26711:1118/140745:264600306599:WARNING:diagnostics_writer.cc(209)]
[FAIL] 013 JSONLocalState (File not found)
[26711:26711:1118/140745:264600306618:WARNING:diagnostics_writer.cc(209)]
[FAIL] 002 SQLiteIntegrityWebData (File not found)
[26711:26711:1118/140745:264600306632:WARNING:diagnostics_writer.cc(209)]
[FAIL] 002 SQLiteIntegrityCookie (File not found)
[26711:26711:1118/140745:264600306650:WARNING:diagnostics_writer.cc(209)]
[FAIL] 002 SQLiteIntegrityHistory (File not found)
[26711:26711:1118/140745:264600307547:WARNING:diagnostics_writer.cc(209)]
Finished 18 tests.
[26711:26711:1118/140745:264600307564:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: Install type
[26711:26711:1118/140745:264600307573:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: Chrome version test
[26711:26711:1118/140745:264600307581:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: User data path
[26711:26711:1118/140745:264600307589:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: Local state path
[26711:26711:1118/140745:264600307600:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: App dictionaries directory path
[26711:26711:1118/140745:264600307609:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: Resources path
[26711:26711:1118/140745:264600307617:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: Available disk space
[26711:26711:1118/140745:264600307625:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: User preferences integrity
[26711:26711:1118/140745:264600307634:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: Local state integrity
[26711:26711:1118/140745:264600307642:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: Bookmark file
[26711:26711:1118/140745:264600307650:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: Web Data database
[26711:26711:1118/140745:264600307659:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: Cookie database
[26711:26711:1118/140745:264600307666:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: Favicons database
[26711:26711:1118/140745:264600307675:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: History database
[26711:26711:1118/140745:264600307683:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: Top Sites database
[26711:26711:1118/140745:264600307693:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: Database tracker database
[26711:26711:1118/140745:264600307701:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: NSS certificate database
[26711:26711:1118/140745:264600307709:WARNING:diagnostics_writer.cc(209)]
Finished Recovery for: NSS Key database
[26711:26711:1118/140745:264600307716:WARNING:diagnostics_writer.cc(209)]
Finished All Recovery operations.
[26711:26711:1118/140745:ERROR:zygote_host_impl_linux.cc(93)] SUID
sandbox binary is missing. Won't be able to adjust OOM scores.
[21/6637] PlatformAppBrowserTest.MutationEventsDisabled (TIMED OUT)
----

browser_tests seems to run OK (although it looks like it will take
tens of minutes to finish) without --run-in-mash.

Is there a better place than chromium-mojo to discuss problems running
"browser_tests --run-in-mash"?

Even if I can run the tests, and reproduce the failure, I suspect that
I'll just get the same error messages as what the commit queue gave
me, and I still have little idea what I'm supposed to poke at to try
and fix this, other than following the "wacky manifest munging" trail
alluded to above. Any guidance?

[Ben Goodger]

oh sorry. you also need to pass --gtest_filter=BrowserTest.Title since that's the only test run on the bots and the one you were seeing that was failing.

-Ben

[Nigel Tao]

> On Wed, Nov 16, 2016 at 10:39 PM, Nigel Tao <nige...@chromium.org> wrote:
>> I have a CL: https://codereview.chromium.org/2453313004/

BTW, if anyone else wants to try this at home, I think I have a
simpler repo: https://codereview.chromium.org/2476493002/ "Add chrome
namespace to chrome/common/resource_usage_reporter.mojom" is another
CL that is a pure rename, and no (intentional) change in
functionality, but the commit queue is refusing with errors that look
superficially similar.
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/316836/steps/mash_browser_tests%20%28with%20patch%29/logs/stdio
says

[1116/223034:ERROR:interface_registry.cc(223)] InterfaceProviderSpec
"service_manager:connector" prevented service: content_browser from
binding interface: ui::mojom::IMERegistrar exposed by: ui

amongst other errors. Note that "ui::mojom::IMERegistrar" is not
otherwise related to that CL.

[Nigel Tao]

On Fri, Nov 18, 2016 at 2:20 PM, Ben Goodger <b...@chromium.org> wrote:
> oh sorry. you also need to pass --gtest_filter=BrowserTest.Title since
> that's the only test run on the bots and the one you were seeing that was
> failing.
>

Ah, OK. The test passes on my machine (with a bunch of errors and a
stack trace, but it's still "PASSED"). Weird. Maybe I'll try a sync
and upload and re-send to the CQ...

----
[1118/145031:ERROR:interface_registry.cc(210)] Failed to locate a
binder for interface: tracing::mojom::Factory requested by:
quick_launch exposed by: tracing via InterfaceProviderSpec
"service_manager:connector".
etc
[1118/145033:ERROR:native_widget_mus.cc(1037)] Not implemented reached
in virtual void views::NativeWidgetMus::StackAbove(gfx::NativeView)
[1118/145033:ERROR:layer_tree_host_impl.cc(2174)] Forcing zero-copy
tile initialization as worker context is missing
[7418:7418:1118/145033:ERROR:zygote_host_impl_linux.cc(93)] SUID

sandbox binary is missing. Won't be able to adjust OOM scores.

[7370:7370:1118/145033:WARNING:url_request_context_getter.cc(43)]
URLRequestContextGetter leaking due to no owning thread.
[7370:7370:1118/145033:WARNING:url_request_context_getter.cc(43)]
URLRequestContextGetter leaking due to no owning thread.
[ OK ] BrowserTest.Title (2523 ms)
[----------] 1 test from BrowserTest (2523 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (2523 ms total)
[ PASSED ] 1 test.
[1/1] BrowserTest.Title (2881 ms)
SUCCESS: all tests passed.
[1118/145033:ERROR:native_widget_mus.cc(859)] Not implemented reached
in virtual void views::NativeWidgetMus::ViewRemoved(views::View *)
[1118/145033:ERROR:session.cc(20)] Restarting service: quick_launch
[1118/145034:ERROR:session.cc(20)] Restarting service: ash
Received signal 11 SEGV_MAPERR fffffffc058db0c3
#0 0x7f4e49dc01a7 base::debug::(anonymous namespace)::StackDumpSignalHandler()
#1 0x7f4e49f1f330 <unknown>
#2 0x7f4e45afebbe gpu::GpuChannelHost::Send()
#3 0x7f4e45aff09d gpu::GpuChannelHost::InternalFlush()
#4 0x7f4e45afee04 gpu::GpuChannelHost::OrderingBarrier()
#5 0x7f4e45afa7df gpu::CommandBufferProxyImpl::Flush()
#6 0x7f4e459f0bb2 gpu::CommandBufferHelper::Flush()
#7 0x7f4e43d7f47d gpu::gles2::GLES2Implementation::FlushHelper()
#8 0x7f4e43d7ce0b
gpu::gles2::GLES2Implementation::SetAggressivelyFreeResources()
etc
[7418:7418:1118/145034:ERROR:native_widget_mus.cc(1255)] Not
implemented reached in virtual bool
views::NativeWidgetMus::IsMouseEventsEnabled() const
[7418:7418:1118/145035:ERROR:native_widget_mus.cc(859)] Not
implemented reached in virtual void
views::NativeWidgetMus::ViewRemoved(views::View *)
[1:1:0100/000000:ERROR:broker_posix.cc(41)] Invalid node channel message
----

[Ben Goodger]

I've seen that stack on shutdown too. I think it's buggy/racy behavior in the mash session's auto-restarter.

If you retry are you still seeing the error on the bot?

-Ben

[Scott Violet]

Some flake seems to have been introduced recently. But is filed here: 666481.

-Scott

> --
> You received this message because you are subscribed to the Google Groups
> "chromium-mojo" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to chromium-moj...@chromium.org.

> To post to this group, send email to chromi...@chromium.org.
> To view this discussion on the web visit

> https://groups.google.com/a/chromium.org/d/msgid/chromium-mojo/CAJDgTytSdjQ9uH5%2Bdhq9zSR6mrawPJkjre-6_0K1_ytEECxHTQ%40mail.gmail.com.

[Nigel Tao]

On Fri, Nov 18, 2016 at 3:04 PM, Ben Goodger <b...@chromium.org> wrote:
> If you retry are you still seeing the error on the bot?

http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_ozone_rel_ng/builds/276466
is purple / red. "mash_browser_tests (with patch)"'s stdio at
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_ozone_rel_ng/builds/276466/steps/mash_browser_tests%20%28with%20patch%29/logs/stdio
ends with:

----
[2/2] BrowserTest.Title (1309 ms)
SUCCESS: all tests passed.


--------------------------------------------------------------------------------
started: Sat Nov 19 15:29:16 2016
ended: Sat Nov 19 17:14:01 2016
duration: 1 hrs, 44 mins, 44 secs
status: FAILURE
status reason: return code was -1.
----

I'm thoroughly confused. I guess I'll wait until crbug.com/666481 is fixed.

[Nigel Tao]

On Sun, Nov 20, 2016 at 2:38 PM, Nigel Tao <nige...@chromium.org> wrote:
> I'm thoroughly confused. I guess I'll wait until crbug.com/666481 is fixed.

That bug has been marked as fixed, but my CL is still red on the
commit queue. It might not be due to test flakiness after all.


On Fri, Nov 18, 2016 at 2:13 PM, Nigel Tao <nige...@chromium.org> wrote:

> On Thu, Nov 17, 2016 at 5:52 PM, Ben Goodger <b...@chromium.org> wrote:
>> I've seen stuff like this before and it's almost guaranteed to be related to the wacky manifest munging the cros mash_browser_tests target does somehow failing.
>
> Do you have any leads on reading up on this "wacky manifest munging"?
> I'm still relatively new to large parts of the Chromium codebase.

I'll repeat: Do you have any leads on reading up on this "wacky

[Nigel Tao]

On Thu, Dec 1, 2016 at 10:54 AM, Nigel Tao <nige...@chromium.org> wrote:
> I'll repeat: Do you have any leads on reading up on this "wacky
> manifest munging"? I'm still relatively new to large parts of the
> Chromium codebase.

BTW I have read
https://chromium.googlesource.com/chromium/src/+/master/services/service_manager/README.md#Statically-Declaring-Capabilities
and I tried cargo-cult patching
chrome/app/mash/chrome_mash_content_browser_manifest_overlay.json
as per
https://codereview.chromium.org/2453313004/diff/180001/chrome/app/mash/chrome_mash_content_browser_manifest_overlay.json
to be similar to the
chrome/browser/chrome_content_browser_manifest_overlay.json
diff in my CL:
https://codereview.chromium.org/2453313004/diff/180001/chrome/browser/chrome_content_browser_manifest_overlay.json
but I'm really just guessing and hoping.

In any case, I am still getting CQ failures for
https://codereview.chromium.org/2453313004/ as per
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/325123/steps/mash_browser_tests%20%28with%20patch%29/logs/stdio

[Colin Blundell]

Hi Nigel,

This must be frustrating :\.

I don't have any special insight on manifests here, but a couple questions that could "help others help". It looks like you answered some of these questions earlier in the thread, but it's not clear to me what the current state of affairs is.

- Have you rebased your CL to ToT and then verified that you get the same failure?

- If so: Is the failure consistent, i.e., if you run "git cl try" multiple times on the patch at ToT does it always give that same failure?

- If so: If you build your patch locally against ToT (using Ben's instructions), does it give the same failure?

If the answer to all of these is "yes", I'd be happy to try to repro locally and add another pair of debugging eyes.

Thanks,

Colin

--
You received this message because you are subscribed to the Google Groups "chromium-mojo" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-moj...@chromium.org.

To post to this group, send email to chromi...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-mojo/CAEdON6YzvaPSCEMjmUYzCU7hHqOUp62S8DpuOQLyoOZmLNiQVA%40mail.gmail.com.

[Nigel Tao]

On Fri, Dec 2, 2016 at 8:00 PM, Colin Blundell <blun...@chromium.org> wrote:
> - Have you rebased your CL to ToT and then verified that you get the same
> failure?
>
> - If so: Is the failure consistent, i.e., if you run "git cl try" multiple
> times on the patch at ToT does it always give that same failure?

Yeah, the failure has been consistent for some weeks now. I'm
relatively new to the Chromium workflow, so I haven't run "git cl try"
per se, but there's a half-dozen patch sets uploaded to
https://codereview.chromium.org/2453313004/ over the course of the
last few weeks and each one has been re-based to ToT before uploading,
with no other substantive changes. I have 3 LGTMs, so I've tried
hitting the "Commit" checkbox a half-dozen times, over two weeks, and
the chromeos trybots never seem happy.

(In addition to those LGTMs, rockot, the Mojo TL, said "As far as I
can tell the mash failures you see are purely coincidental. Nothing in
your CL is relevant to their behavior." on
https://codereview.chromium.org/2453313004/#msg63)

> - If so: If you build your patch locally against ToT (using Ben's
> instructions), does it give the same failure?

Yes? No? I really don't know. I run:

----
$ cat out/cros/args.gn
# Build arguments go here. Examples:
# is_component_build = true
# is_debug = false
# See "gn args <out_dir> --list" for available build arguments.

use_goma = true

target_os="chromeos"
enable_package_mash_services=true
----

and then running browser_tests with the args that ben suggested:

----
$ out/cros/browser_tests --run-in-mash
--service-overrides=chrome/app/mash/mash_service_overrides.json
--gtest_filter=BrowserTest.Title
[lots of output]
$ echo $?
0
----

Does browser_tests returning 0 mean that the tests passed? The "lots
of output" contains this fragment:

----
[ OK ] BrowserTest.Title (25338 ms)
[----------] 1 test from BrowserTest (25338 ms total)

[----------] Global test environment tear-down

[==========] 1 test from 1 test case ran. (25341 ms total)
[ PASSED ] 1 test.
----

But it also ends with a stack trace:

----
[1204/115342.921660:ERROR:surface_manager.cc(131)] No reference from
SurfaceId(FrameSinkId(0, 0), LocalFrameId(0,
(F76390E54B5EF3319FE4EF45B206A825)lu)) to SurfaceId(FrameSinkId(2, 1),
LocalFrameId(1, (A12AEF2F4023B0EA455DA38CEDFC2DDA)lu))
[1204/115342.979445:FATAL:surface_manager.cc(49)] Check failed:
frame_sink_source_map_.size() == 0u (1 vs. 0)
#0 0x7f4e0f397e9e base::debug::StackTrace::StackTrace()
#1 0x7f4e0f3fd89c logging::LogMessage::~LogMessage()
#2 0x7f4e048a0405 cc::SurfaceManager::~SurfaceManager()
#3 0x0000036a3af5 ui::DisplayCompositor::~DisplayCompositor()
#4 0x0000036a3b59 ui::DisplayCompositor::~DisplayCompositor()
#5 0x00000093502f std::default_delete<>::operator()()
#6 0x0000036c839c std::unique_ptr<>::reset()
#7 0x0000036c61c7 ui::GpuMain::TearDownOnCompositorThread()
#8 0x0000008e0095
_ZN4base8internal13FunctorTraitsIM25RenderViewContextMenuBaseFvvEvE6InvokeIP21RenderViewContextMenuJEEEvS4_OT_DpOT0_
#9 0x0000008dffb1
_ZN4base8internal12InvokeHelperILb0EvE8MakeItSoIRKM25RenderViewContextMenuBaseFvvEJP21RenderViewContextMenuEEEvOT_DpOT0_
#10 0x0000036c9947
_ZN4base8internal7InvokerINS0_9BindStateIMN2ui7GpuMainEFvvEJNS0_17UnretainedWrapperIS4_EEEEEFvvEE7RunImplIRKS6_RKSt5tupleIJS8_EEJLm0EEEEvOT_OT0_NS_13IndexSequenceIJXspT1_EEEE
#11 0x0000036c988c
_ZN4base8internal7InvokerINS0_9BindStateIMN2ui7GpuMainEFvvEJNS0_17UnretainedWrapperIS4_EEEEEFvvEE3RunEPNS0_13BindStateBaseE
#12 0x7f4e0f39d441
_ZNO4base8internal8RunMixinINS_8CallbackIFvvELNS0_8CopyModeE0ELNS0_10RepeatModeE0EEEE3RunEv
#13 0x7f4e0f39ce29 base::debug::TaskAnnotator::RunTask()
#14 0x7f4e0f424acf base::MessageLoop::RunTask()
#15 0x7f4e0f424d34 base::MessageLoop::DeferOrRunPendingTask()
#16 0x7f4e0f42501e base::MessageLoop::DoWork()
#17 0x7f4e0f439bb4 base::MessagePumpDefault::Run()
#18 0x7f4e0f424695 base::MessageLoop::RunHandler()
#19 0x7f4e0f4c5233 base::RunLoop::Run()
#20 0x7f4e0f563109 base::Thread::Run()
#21 0x7f4e0f5639e1 base::Thread::ThreadMain()
#22 0x7f4e0f54ca3a base::(anonymous namespace)::ThreadFunc()
#23 0x7f4e0f7b2184 start_thread
#24 0x7f4df3db937d clone

Received signal 6
#0 0x7f4e0f397e9e base::debug::StackTrace::StackTrace()
#1 0x7f4e0f3979df base::debug::(anonymous namespace)::StackDumpSignalHandler()
#2 0x7f4e0f7ba330 <unknown>
#3 0x7f4df3cf5c37 gsignal
#4 0x7f4df3cf9028 abort
#5 0x7f4e0f394476 base::debug::(anonymous namespace)::DebugBreak()
#6 0x7f4e0f394458 base::debug::BreakDebugger()
#7 0x7f4e0f3fdc3f logging::LogMessage::~LogMessage()
#8 0x7f4e048a0405 cc::SurfaceManager::~SurfaceManager()
#9 0x0000036a3af5 ui::DisplayCompositor::~DisplayCompositor()
#10 0x0000036a3b59 ui::DisplayCompositor::~DisplayCompositor()
#11 0x00000093502f std::default_delete<>::operator()()
#12 0x0000036c839c std::unique_ptr<>::reset()
#13 0x0000036c61c7 ui::GpuMain::TearDownOnCompositorThread()
#14 0x0000008e0095
_ZN4base8internal13FunctorTraitsIM25RenderViewContextMenuBaseFvvEvE6InvokeIP21RenderViewContextMenuJEEEvS4_OT_DpOT0_
#15 0x0000008dffb1
_ZN4base8internal12InvokeHelperILb0EvE8MakeItSoIRKM25RenderViewContextMenuBaseFvvEJP21RenderViewContextMenuEEEvOT_DpOT0_
#16 0x0000036c9947
_ZN4base8internal7InvokerINS0_9BindStateIMN2ui7GpuMainEFvvEJNS0_17UnretainedWrapperIS4_EEEEEFvvEE7RunImplIRKS6_RKSt5tupleIJS8_EEJLm0EEEEvOT_OT0_NS_13IndexSequenceIJXspT1_EEEE
#17 0x0000036c988c
_ZN4base8internal7InvokerINS0_9BindStateIMN2ui7GpuMainEFvvEJNS0_17UnretainedWrapperIS4_EEEEEFvvEE3RunEPNS0_13BindStateBaseE
#18 0x7f4e0f39d441
_ZNO4base8internal8RunMixinINS_8CallbackIFvvELNS0_8CopyModeE0ELNS0_10RepeatModeE0EEEE3RunEv
#19 0x7f4e0f39ce29 base::debug::TaskAnnotator::RunTask()
#20 0x7f4e0f424acf base::MessageLoop::RunTask()
#21 0x7f4e0f424d34 base::MessageLoop::DeferOrRunPendingTask()
#22 0x7f4e0f42501e base::MessageLoop::DoWork()
#23 0x7f4e0f439bb4 base::MessagePumpDefault::Run()
#24 0x7f4e0f424695 base::MessageLoop::RunHandler()
#25 0x7f4e0f4c5233 base::RunLoop::Run()
#26 0x7f4e0f563109 base::Thread::Run()
#27 0x7f4e0f5639e1 base::Thread::ThreadMain()
#28 0x7f4e0f54ca3a base::(anonymous namespace)::ThreadFunc()
#29 0x7f4e0f7b2184 start_thread
#30 0x7f4df3db937d clone
r8: fffffffffffffed8 r9: fffffffffffffec8 r10: 0000000000000008
r11: 0000000000000202
r12: 0000000000000000 r13: 0000000000000000 r14: 00007f4ded9589c0
r15: 00007f4ded958700
di: 0000000000004fad si: 0000000000004fc6 bp: 00007f4ded955060
bx: 0000000000000000
dx: 0000000000000006 ax: 0000000000000000 cx: ffffffffffffffff
sp: 00007f4ded954f28
ip: 00007f4df3cf5c37 efl: 0000000000000202 cgf: 0000000000000033
erf: 0000000000000000
trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
----

but that stack trace looks graphics-related, and therefore unrelated??

If I re-run the test (locally) and capture the stdout / stderr:

----
$ out/cros/browser_tests --run-in-mash
--service-overrides=chrome/app/mash/mash_service_overrides.json
--gtest_filter=BrowserTest.Title > /tmp/stdout 2> /tmp/stderr
----

The stdout says "SUCCESS: all tests passed." and grepping stderr for
"prevented service" gives nothing.

In comparison, on the commit queue,
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/325123/steps/mash_browser_tests%20%28with%20patch%29/logs/stdio
contains a bunch of lines that look like:

----
[30947:31003:1130/234642.899124:ERROR:interface_registry.cc(218)]

InterfaceProviderSpec "service_manager:connector" prevented service:
content_renderer from binding interface:
chrome::mojom::FieldTrialRecorder exposed by: content_browser

...
[30947:31003:1130/234642.899056:ERROR:interface_registry.cc(218)]

InterfaceProviderSpec "service_manager:connector" prevented service:
content_renderer from binding interface:

startup_metric_utils::mojom::StartupMetricHost exposed by:
content_browser
----

FieldTrialRecorder is the new mojom interface introduced by my CL.
StartupMetricHost is unrelated to my CL.

Also, on the commit queue,
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/325123/steps/mash_browser_tests%20%28with%20patch%29/logs/stdio
says

----
../../chrome/browser/ui/browser_browsertest.cc:461: Failure
Value of: browser()->GetWindowTitleForCurrentTab( true )
Actual: Chromium - about:blank
Expected: LocaleWindowCaptionFromPageTitle(test_title)
Which is: Chromium - Title Of Awesomeness
../../chrome/browser/ui/browser_browsertest.cc:464: Failure
Value of: tab_title
Actual: title2.html
Expected: test_title
Which is: Title Of Awesomeness
[31206:31206:1130/234650.067149:ERROR:native_widget_mus.cc(863)] Not

implemented reached in virtual void
views::NativeWidgetMus::ViewRemoved(views::View *)

[31206:31206:1130/234650.770454:WARNING:url_request_context_getter.cc(43)]

URLRequestContextGetter leaking due to no owning thread.

[31206:31206:1130/234650.770494:WARNING:url_request_context_getter.cc(43)]

URLRequestContextGetter leaking due to no owning thread.

[ FAILED ] BrowserTest.Title, where TypeParam = and GetParam() = (1531 ms)
[4/4] BrowserTest.Title (1832 ms)
1 test failed:
BrowserTest.Title (../../chrome/browser/ui/browser_browsertest.cc:453)
----

but as I mailed two weeks ago, an earlier CQ run
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux%2Flinux_chromium_chromeos_ozone_rel_ng%2F276466%2F%2B%2Frecipes%2Fsteps%2Fmash_browser_tests__with_patch_%2F0%2Fstdout
ended with:

----
[2/2] BrowserTest.Title (1309 ms)
SUCCESS: all tests passed.
----

yet https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_ozone_rel_ng/builds/276466
still ended with red.

> If the answer to all of these is "yes", I'd be happy to try to repro locally
> and add another pair of debugging eyes.

Well, two yesses but then one maybe??

[Colin Blundell]

Hi Nigel,

Thanks for this info. I was able to repro the failure locally. The cause of the failure is that the bot is not passing "--service-overrides=chrome/app/mash/mash_service_overrides.json" on the command line when running the tests. This is causing the renderer to be terminated when it attempts to connect to FieldTrialRecorder, after which the test unsurprisingly doesn't run properly. It looks like you just happened to be the canary in the coal mine here.

Can you test whether adding the following change to your CL makes it work (i.e., add in this change and then run "git cl try")?

diff --git a/testing/buildbot/chromium.chromiumos.json b/testing/buildbot/chromium.chromiumos.json

index d9580e6..1d060e1 100644

--- a/testing/buildbot/chromium.chromiumos.json

+++ b/testing/buildbot/chromium.chromiumos.json

@@ -64,6 +64,7 @@

         "args": [

           "--override-use-gl-with-osmesa-for-tests",

           "--run-in-mash",

+          "--service-overrides=src/chrome/app/mash/mash_service_overrides.json",

           "--test-launcher-filter-file=src/testing/buildbot/filters/mash.browser_tests.filter",

           "--use-test-config"

         ],

[Colin Blundell]

+Scott Violet BTW, Nigel wasn't kidding when he said that this test generated lots of output; it looks like thousands of line of NotImplemented messages.

[Scott Violet]

Ken knows the most about overlays.

-Scott

[Ken Rockot]

I see - sorry I was useless when looking at this before. I saw lots of assorted InterfaceRegistry spam and didn't consider the fact that this new interface would be the one that suddenly caused the test to fail.

I'm going to try to kill overlays ASAP.

[Scott Violet]

I started hitting problems because of this and am going to land the
change to incorporate the additional services test in my patch:
https://codereview.chromium.org/2539363005/.

-Scott

[Nigel Tao]

https://codereview.chromium.org/2453313004/ has finally passed the CQ.

Thanks for everyone's help.

[Colin Blundell]

Woohoo!