Debugging Android test failures = sigh.. :-(

[Gabriel Charette]

I have a CL which passes on all bots but some Android and CrOS bots.

The failures don't have callstacks on the bots :-(.

I built the Android unit_tests locally:

gn args out/Android:

  target_os = "android"

  target_cpu = "arm" 

  is_debug = true

  is_component_build = true

  is_clang = true

  symbol_level = 1

  enable_incremental_javac = true

ninja -C out/Android unit_tests_incremental

out/Android/bin/run_unit_tests -vv

I get 187 failures on 2337 tests but no failure logs...

It's always the same tests that fail but when I invoke them individually (or even in subgroups) they pass..

  out/Android/bin/run_unit_tests -vv --gtest-filter AndroidProviderBackendTest.IgnoreAboutBlank

How am I supposed to debug this...?!

PS: I also tried to symbolize crashes per online instructions:

  adb logcat -d | CHROMIUM_OUTPUT_DIR=out/Android third_party/android_platform/development/scripts/stack

but couldn't anything out of that either : "Unknown Android release, consider --[no-]packed-relocation-adjustments options".

Thanks,

Gab

[Yuly Novikov]

There are callstacks on the bots

Try to symbolize those.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[dan...@chromium.org]

Do the tests crash or just fail? You don't mention crashing earlier until here. If they fail but pass in isolation try adding more tests to the filter with some wildcards.. maybe there are bad interactions. LOG(ERROR) will show up in the logcat, i tend to add those with some distinctive string I can grep for. If you want a crash stack locally something like this is helpful

N=1000

adb logcat -v threadtime -d | tail -${N} | \

    third_party/android_platform/development/scripts/stack --arch=arm --output-directory=out_android/Debug

[Gabriel Charette]

Ok thanks, so now I have a stack (from bots), how do I symbolize it? (tried dumping to file and piping to third_party/android_platform/development/scripts/stack but that didn't work, maybe I need to pass a --chrome-symbols-dir?)

@danakj: now I know it's hitting a DCHECK (wasn't sure before), I still need the callstack to figure out what's happening in the wrong order though.

I  216.368s run_tests_on_device(0521d51109d35bb6)  [FATAL:render_process_host_impl.cc(2453)] Check failed: ::content::BrowserThread::CurrentlyOn(BrowserThread::UI). Must be called on Chrome_UIThread; actually called on Unknown Thread.

I  216.368s run_tests_on_device(0521d51109d35bb6)  #00 0x771ae86f /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x00bf686f

I  216.368s run_tests_on_device(0521d51109d35bb6)  #01 0x77f4176d /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x0198976d

I  216.368s run_tests_on_device(0521d51109d35bb6)  #02 0x77dbf331 /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x01807331

I  216.368s run_tests_on_device(0521d51109d35bb6)  #03 0x785ea735 /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x02032735

I  216.368s run_tests_on_device(0521d51109d35bb6)  #04 0x785ed11f /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x0203511f

I  216.368s run_tests_on_device(0521d51109d35bb6)  #05 0x785fa211 /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x02042211

I  216.368s run_tests_on_device(0521d51109d35bb6)  #06 0x785fa307 /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x02042307

I  216.368s run_tests_on_device(0521d51109d35bb6)  #07 0x785fa335 /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x02042335

I  216.368s run_tests_on_device(0521d51109d35bb6)  #08 0x785fa38f /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x0204238f

I  216.368s run_tests_on_device(0521d51109d35bb6)  #09 0x76b471e5 /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x0058f1e5

I  216.368s run_tests_on_device(0521d51109d35bb6)  #10 0x7841d675 /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x01e65675

I  216.369s run_tests_on_device(0521d51109d35bb6)  #11 0x7841d68f /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x01e6568f

I  216.369s run_tests_on_device(0521d51109d35bb6)  #12 0x7841d701 /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x01e65701

I  216.369s run_tests_on_device(0521d51109d35bb6)  #13 0x76a864c7 /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x004ce4c7

I  216.369s run_tests_on_device(0521d51109d35bb6)  #14 0x76a8683f /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x004ce83f

I  216.369s run_tests_on_device(0521d51109d35bb6)  #15 0x76a8684f /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x004ce84f

I  216.369s run_tests_on_device(0521d51109d35bb6)  #16 0x7846d0af /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x01eb50af

I  216.369s run_tests_on_device(0521d51109d35bb6)  #17 0x7846d15d /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x01eb515d

I  216.369s run_tests_on_device(0521d51109d35bb6)  #18 0x7846e997 /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x01eb6997

I  216.369s run_tests_on_device(0521d51109d35bb6)  #19 0x78469f7b /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x01eb1f7b

I  216.369s run_tests_on_device(0521d51109d35bb6)  #20 0x78420f6d /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x01e68f6d

I  216.369s run_tests_on_device(0521d51109d35bb6)  #21 0x77ee2abb /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x0192aabb

I  216.369s run_tests_on_device(0521d51109d35bb6)  #22 0x7842245b /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x01e6a45b

I  216.369s run_tests_on_device(0521d51109d35bb6)  #23 0x76945a1f /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x0038da1f

I  216.369s run_tests_on_device(0521d51109d35bb6)  #24 0x784196a1 /data/app-lib/org.chromium.native_test-1/lib_unit_tests__library.so+0x01e616a1

I  216.369s run_tests_on_device(0521d51109d35bb6)  #25 0x41588bcf /system/lib/libdvm.so+0x0001dbcf

I  216.369s run_tests_on_device(0521d51109d35bb6)  #26 0x415b9125 /system/lib/libdvm.so+0x0004e125

I  216.369s run_tests_on_device(0521d51109d35bb6)  #27 0x41591fe3 /system/lib/libdvm.so+0x00026fe3

I  216.369s run_tests_on_device(0521d51109d35bb6)  #28 0x41598fa3 /system/lib/libdvm.so+0x0002dfa3

I  216.369s run_tests_on_device(0521d51109d35bb6)  #29 0x4159663b /system/lib/libdvm.so+0x0002b63b

I  216.369s run_tests_on_device(0521d51109d35bb6)  #30 0x415cb863 /system/lib/libdvm.so+0x00060863

I  216.369s run_tests_on_device(0521d51109d35bb6)  #31 0x415d37c5 /system/lib/libdvm.so+0x000687c5

I  216.369s run_tests_on_device(0521d51109d35bb6)  #32 0x41591fe3 /system/lib/libdvm.so+0x00026fe3

I  216.369s run_tests_on_device(0521d51109d35bb6)  #33 0x41598fa3 /system/lib/libdvm.so+0x0002dfa3

I  216.369s run_tests_on_device(0521d51109d35bb6)  #34 0x4159663b /system/lib/libdvm.so+0x0002b63b

I  216.369s run_tests_on_device(0521d51109d35bb6)  #35 0x415cb57f /system/lib/libdvm.so+0x0006057f

I  216.369s run_tests_on_device(0521d51109d35bb6)  #36 0x415b4d0d /system/lib/libdvm.so+0x00049d0d

I  216.369s run_tests_on_device(0521d51109d35bb6)  #37 0x4022d2b3 /system/lib/libandroid_runtime.so+0x0004d2b3

I  216.369s run_tests_on_device(0521d51109d35bb6)  #38 0x4022dfd9 /system/lib/libandroid_runtime.so+0x0004dfd9

I  216.369s run_tests_on_device(0521d51109d35bb6)  #39 0x400f005d /system/bin/app_process+0x0000105d

I  216.369s run_tests_on_device(0521d51109d35bb6)  #40 0x4012734b /system/lib/libc.so+0x0000e34b

I  216.369s run_tests_on_device(0521d51109d35bb6)  #41 0x400efd7f /system/bin/app_process+0x00000d7f

I  216.369s run_tests_on_device(0521d51109d35bb6)  

[Yuly Novikov]

I'm not sure how to symbolize this.

You could try reproducing this locally using the build flags from the bot (I think the error your are getting is because you use flags like enable_incremental_javac)

dcheck_always_on = true

ffmpeg_branding = "Chrome"

is_component_build = false

is_debug = false

proprietary_codecs = true

symbol_level = 1

target_os = "android"

use_goma = true

[Gabriel Charette]

So there's no way to get symbolized stacks from the bots?

[John Budorick]

At the moment, doing so is painfully difficult. We're working on it.

I'll get back to you with a trace in a bit.

[John Budorick]

I wasn't able to symbolize the bot trace solely from the stripped .so that swarming has, but I was able to reproduce locally. I'll send you the link out-of-band.

[Torne (Richard Coles)]

Aren't the bots normally supposed to symbolise all the crashes from the log in a separate step in the build later on (after dumping the logs)? That's how I've always found stacks for crashes, but this build is missing the logcat dump and symbolise logcat steps entirely.

[John Budorick]

That's true for nonswarmed bots. We're still working on support for swarmed bots.

[Gabriel Charette]

Is there a way I can manually request a non-swarmed try job for the same configuration? (i.e. the failure is configuration specific, not swarm specific)

[John Budorick]

No, there isn't.

[Gabriel Charette]

:-(, that's a pretty sad regression on our infrastructure then (although swarming overall is definitely still a huge plus, not taking away from that). As I highlighted: setting up Android debugging is pretty convoluted and even when that's done, it still only works with a physical phone plugged in (so I'm completely blocked today because I happen to be working remotely from a Windows laptop..).

[John Budorick]

It is, and we know that. We felt that the gains we got from swarming the Android tests -- dramatically reduced cycle time, better fleet utilization, better device failure tolerance, among other things -- were worth temporarily regressing in that regard. We're not blind to the pain it causes in some cases, though.

[Gabriel Charette]

I agree that swarming is still a huge improvement overall.

Have we considered keeping a few non-swarming try bots (triggered manually only) for now for people that run into this and need stack traces?

[John Budorick]

We hadn't, but that's a good idea until we get a better solution in place. Filed https://bugs.chromium.org/p/chromium/issues/detail?id=670101

[Nico Weber]

What's the tracking bug for symbols with swarming?

[John Budorick]

https://bugs.chromium.org/p/chromium/issues/detail?id=657040 (gtests) and  https://bugs.chromium.org/p/chromium/issues/detail?id=631213 (instrumentation tests)

[Dirk Pranke]

If there are any non-swarmed android builders with devices still around, it's probably possible to change the configuration for that builder in the //tools/mb/mb_config.pyl and //testing/buildbot/*.json files to run your tests on them for try jobs.

I can provide specific instructions given an example (it's not too complicated, but also not turnkey).

-- Dirk

[Nico Weber]

Any progress on this? I just spent several hours trying to get a symbolized

stack for a swarming-bot-only crash (https://crbug.com/729059). I have one

now, but this could've been easier. (The problem for some reason didn't

repro on the non-swarming bots.)

[John Budorick]

As bpastene@ mentioned in crbug.com/657040, work on this is ongoing. Though the bug title for crbug.com/631213 only mentions instrumentation tests, the work the hzl@ is doing there affects gtests as well.