Missing symbols in backtrace?

[Ryan]

Hi all,

I'm currently trying to track down a segfault, following Simple Chrome and other docs for guidance. It seems that simplest way to go about it is to read the logs after a crash for the backtrace. After following the instructions in Simple Chrome, I've managed to get a backtrace in /var/log/ui/ui.* files when deploying a release build with the --nostrip option. However, it does not have any symbols:

../../third_party/tcmalloc/chromium/src/tcmalloc.cc:289] Attempt to free invalid pointer 0xc7a89274

Received signal 11 SEGV_MAPERR 000000000039

#0 0x0000323f40ac <unknown>

#1 0x0000323f3f98 <unknown>

#2 0x0000b50d4c40 <unknown>

#3 0x000030f0e1ec <unknown>

#4 0x000030f12ee4 <unknown>

...

[etc.]

Furthermore, when I try deploying a debug build (again with --nostrip), the DCHECK stacktraces do not have symbols either:

[21979:21979:0406/120014.686682:FATAL:pref_registry.cc(63)] Check failed: !defaults_->GetValue(path, NULL). Trying to register a previously registered pref: cryptauth.gcm_registration_id

#0 0x0000393e641e <unknown>

#1 0x0000393e5d34 <unknown>

#2 0x000039412bac <unknown>

#3 0x00003a867ce2 <unknown>

#4 0x00003a868578 <unknown>

...

[etc.]

I have the same issue with base::debug::StackTrace.

The bizarre thing about this situation is that I know for a fact that symbols are included with my deployed chrome, because I am able to get a backtrace with symbols in gdb (for the exact same crash, on the release build):

Thread 1 "chrome" received signal SIGSEGV, Segmentation fault.

0x2c0d61ec in tcmalloc::Abort() () at ../../third_party/tcmalloc/chromium/src/base/abort.cc:15

15 ../../third_party/tcmalloc/chromium/src/base/abort.cc: No such file or directory.

(gdb) bt

#0  0x2c0d61ec in tcmalloc::Abort() () at ../../third_party/tcmalloc/chromium/src/base/abort.cc:15

warning: Could not find DWO CU obj/base/allocator/tcmalloc/internal_logging.dwo(0x53d10e30960f02a8) referenced by CU at offset 0x1c656 [in module /opt/google/chrome/chrome]

#1  0x2c0daee4 in Log () at ../../third_party/tcmalloc/chromium/src/internal_logging.cc:120

warning: Could not find DWO CU obj/base/allocator/tcmalloc/debugallocation_shim.dwo(0x627392cb835b7132) referenced by CU at offset 0x22a71 [in module /opt/google/chrome/chrome]

#2  0x2c0e3284 in do_free_with_callback () at ../../third_party/tcmalloc/chromium/src/tcmalloc.cc:289

warning: Could not find DWO CU obj/components/cryptauth/ble/ble/bluetooth_low_energy_weave_client_connection.dwo(0x73b2176927ee0a4) referenced by CU at offset 0x87805ff [in module /opt/google/chrome/chrome]

#3  0x3046b964 in deallocate () at /usr/local/google/home/hansberry/chromium/.cros_cache/chrome-sdk/tarballs/veyron_minnie+9391.0.0+target_toolchain/usr/bin/../lib/gcc/armv7a-cros-linux-gnueabi/4.9.x/include/g++-v4/ext/new_allocator.h:116

#4  deallocate () at /usr/local/google/home/hansberry/chromium/.cros_cache/chrome-sdk/tarballs/veyron_minnie+9391.0.0+target_toolchain/usr/bin/../lib/gcc/armv7a-cros-linux-gnueabi/4.9.x/include/g++-v4/bits/alloc_traits.h:383

...

[etc.]

Perhaps the "Could not find DWO CU obj/components/cryptauth[...]" warnings are indicative of a larger issue, but I haven't been able to find much help for those warnings searching online.

Does anyone know what I might be doing wrong or missing?

Thanks,

Ryan

[Primiano Tucci]

which process is this? If it's the renderer self symbolization might be failing due to sandboxing.

Does it repro if you run it with --no-sandbox ?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/24ae1465-f908-4d72-b968-69686c79d195%40chromium.org.

[Steven Bennetts]

This is discussed in YAQS:

https://yaqs.googleplex.com/eng/q/5267421445488640

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

[Sunny Sachanandani]

I ran into this a while back and worked around it by manually symbolizing stack traces. See the discussion here: https://groups.google.com/a/chromium.org/d/topic/chromium-dev/jCgEkM8kZYQ/discussion. Someone on that thread was able to get automatic symbolization to work but I never got around to trying that.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CADko7PHQre8-9FNV8LT39fb3dCb0vbDbwO9q%2BZZ34ORTSPq%2BCw%40mail.gmail.com.

[Ryan]

Thank you to everyone for the suggestions; unfortunately, nothing seems to work.

Primiano: Apologies for not knowing for sure (I'm still a bit new), but I believe this code is executing from the browser process, and not the renderer process. The crash originated from src/chromeos/components/tether/. In any case, I tried running chrome with the --no-sandbox flag (I assume you meant to add it to /etc/chrome_dev.conf), and chrome did not start. /var/log/chrome/chrome had multiple backtraces like this

[7217:7217:0406/155218.705571:FATAL:sandbox_linux.cc(341)] InitializeSandbox() called with multiple threads in process gpu-process. This error can be safely ignored in VMTests.

#0 0x00002eacb0ac <unknown>

#1 0x00002eadabce <unknown>

#2 0x00002d6b70d0 <unknown>

...

[etc.]

and one other backtrace:

[7138:7138:0406/155219.340568:FATAL:gpu_process_transport_factory.cc(357)] Unable to create a UI graphics context, and cannot use software compositing on ChromeOS.

#0 0x0000331120ac <unknown>

#1 0x000033121bce <unknown>

#2 0x0000322464c2 <unknown>

...

[etc.]

It seems that Sunny also had issues with --no-sandbox given the discussion link he provided.

Steven: Unfortunately nothing helped in that discussion. The --strip-flags '-w -K "!*WebCore*"' flag did not produce a backtrace with symbols, and though gdb is a reliable way for me to get a backtrace with symbols, I would like a faster way.

Sunny: I tried the --strip-flags=-S flag that Satoru mentioned, but that did not work either. The only difference I can see between our approaches is that Satoru used a daisy but I am using a veyron_minnie. Would device model affect this issue at all? (Highly doubt it).

Best,

Ryan

[Maksim]

I'm having the same problem (as well as another Chromium engineer from our team - CC'ed isandrk@).

The approach suggested by the documentation (is_debug=true and --nostrip) doesn't work for me, even though it did work fine a couple of weeks ago.

Specifying --strip-flags=-S didn't help either.

I believe something has changed recently that broke this workflow - but no idea whether it's in Chromite or Chrome itself.

[Ryan]

Hey Maksim, that's interesting to hear -- I suspect that something broke in Chrome as opposed to Chromite since I've confirmed that the Chrome on my device has symbols via gdb. Does anyone know where in the codebase this could have possibly broken?

[Maksim]

It started working for me again: stack traces printed by CHECK or base::StackTrace display the function names now.

(I don't think I changed anything, just pulled the latest Chrome ToT.)

[Ryan Hansberry]

Hi all,

I and my team have successfully been using GDB for backtraces for the past few months after giving up on getting backtraces in logs. I'm now in a situation where backtraces in logs would be very useful, but I've been disappointed to find that this is still broken.

When a crash occurs and is logged to /var/log/ui/, it still does not have symbols:

Received signal 11 SEGV_MAPERR 000000000000

#0 0x000033e42198 <unknown>

#1 0x000033e41a3e <unknown>

[etc.]

base::debug::StackTrace is also missing symbols (I'm logging this from a tether_service.cc):

[8433:8433:0807/154824.288503:ERROR:tether_service.cc(299)] #0 0x000039f5d298 <unknown>

#1 0x000039f5cb3e <unknown>

#2 0x0000387c847e <unknown>

[etc.]

Simple Chrome still states that "If you just want crash backtraces in the logs you can deploy a release build with --nostrip. You don't need a debug build." This still does not seem to be true, so I've tried building and deploying a debug build like so (but still to no avail):

1. I've created a build config with 'is_debug = true, is_component_build = false,  is_official_build = false'.
2. I've used deploy_chrome with full symbols via --nostrip.

Has anyone had issues or success with backtraces in logs? This would be very useful for development. If I'm not the only one hitting this issue, it would be great to resolve this issue for other engineers' sake in the future.

Best,

Ryan

[Daniel Cheng]

I'm not sure why this is broken (note that we do intentionally suppress symbolization in sanitizer builds [1]; I'm not sure if there are other conditions for which this is true), but as a temporary workaround, can you just pipe it through tools/valgrind/asan/asan_symbolize.py?

Daniel

[1] https://cs.chromium.org/chromium/src/base/third_party/symbolize/symbolize.cc?rcl=17cf8f4795c83934eace66202c81cfe3940a8222&l=776 - incidentally, this is probably where I'd start looking to try to figure out why symbolization isn't working.

--

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/39a0ad46-3034-4915-ab1d-898cff9994c0%40chromium.org.

[Nico Weber]

The simplechrome build is owned by the chromeos toolchain team, and most people on this list aren't super familiar with it as it's fairly different from all of chromium's build configurations. This might be why you're not getting many responses on this list. You could try asking on the chromium os dev list too if you haven't.

--

[Ryan Hansberry]

Hi all,

I've done some digging:

Daniel, thanks for the link. I did some print debugging in that part of the code to try to figure out what's going wrong: it is failing to retrieve symbols from the object file: https://cs.chromium.org/chromium/src/base/third_party/symbolize/symbolize.cc?rcl=17cf8f4795c83934eace66202c81cfe3940a8222&l=815. (I also verified that no suppression due to a sanitizer build flag is occurring).

This reminded me of warnings that I've always seen in GDB where GDB spews out warnings about not being able to find *.dwo files, like so:

warning: Could not find DWO CU obj/ash/ash/network_icon_animation.dwo(0xa48f50474fe65c35) referenced by CU at offset 0x4ef9d70 [in module /usr/local/chrome/chrome]

I tracked down some old bugs that reference this warning (e.g. crbug.com/666278, crbug.com/404644), which led to this bug: crbug.com/603286 "gn debug build confounds gdb". I'm beginning to suspect that these missing DWO that GDB is complaining about are related to symbolizer.cc not being able to retrieve symbols from the object files. 

+jochen@ and +dpranke@, who have worked on the bug in question

+satorux@ and +phajdan.jr@, who originally added/modified the use of symbolizer.cc (https://codereview.chromium.org/545148, https://chromiumcodereview.appspot.com/11362048), and may have some insight

+thomasanderson@ who also observed this issue recently (https://bugs.chromium.org/p/chromium/issues/detail?id=404644#c29)

Best,

Ryan

On Thursday, April 6, 2017 at 11:11:19 AM UTC-7, Ryan Hansberry wrote:

[Ryan Hansberry]

Ting-Yuan: Thank you for the suggestion to disable debug fission -- unfortunately, the stack trace in the logs still has no symbols. I'm certain that the correct debug information is in Chrome, because GDB works fine.

For those who are curious: there is more discussion on this issue at chromium-dev@: https://groups.google.com/a/chromium.org/forum/#!searchin/chromium-dev/backtrace/chromium-dev/gUC7DhA5AMU/xYKGRMqRAwAJ