Issue 404644 in chromium: gdb spews "warning: Could not find DWO CU"

[chro...@googlecode.com]

Status: Untriaged
Owner: ----
Labels: Type-Bug Pri-2 Build OS-Linux

New issue 404644 by jbr...@chromium.org: gdb spews "warning: Could not
find DWO CU"
http://code.google.com/p/chromium/issues/detail?id=404644

I'm doing a fairly straightforward component build with goma. I've started
seeing annoying warnings when I use gdb, and I'm not sure what the cause is.

When I launch gdb, I see a bunch of warnings:


$ gdb --args out/Debug/chrome --single-process --no-sandbox
--disable-hang-monitor
(master)
GNU gdb (GDB) 7.7-gg7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux".
Type "show configuration" for configuration details.

<http://go/gdb-home FAQ: http://go/gdb-faq Email: gdb-team IRC: gdb>
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from out/Debug/chrome...
warning: Could not find DWO CU
obj/chrome/app/chrome_initial.chrome_exe_main_aura.dwo(0x2e0b001d737b2f0a)
referenced by CU at offset 0x0 [in module
/src/chromium/src/out/Debug/chrome]

warning: Could not find DWO CU
obj/third_party/tcmalloc/chromium/src/allocator.maybe_threads.dwo(0x6f30718808fddaee)
referenced by CU at offset 0xc00c [in module
/src/chromium/src/out/Debug/chrome]

warning: Could not find DWO CU
obj/third_party/tcmalloc/chromium/src/allocator.stacktrace.dwo(0xc08eac9c05f19e71)
referenced by CU at offset 0xc070 [in module
/src/chromium/src/out/Debug/chrome]

warning: Could not find DWO CU
obj/third_party/tcmalloc/chromium/src/base/allocator.abort.dwo(0x2ead1b976cbb8013)
referenced by CU at offset 0xc10c [in module
/src/chromium/src/out/Debug/chrome]

<snip>


And when I launch chrome from within gdb, I see more still:
(gdb) run
Starting program: /src/chromium/src/out/Debug/chrome --single-process
--no-sandbox --disable-hang-monitor
warning: Could not find DWO CU
obj/ui/base/ui_base.base_window.dwo(0x753fbc27c66a5f20) referenced by CU at
offset 0x90 [in module /src/chromium/src/out/Debug/lib/libui_base.so]
warning: Could not find DWO CU
obj/ui/base/clipboard/ui_base.custom_data_helper_linux.dwo(0x71d09565b6e45163)
referenced by CU at offset 0x174 [in module
/src/chromium/src/out/Debug/lib/libui_base.so]
warning: Could not find DWO CU
obj/ui/base/cursor/ui_base.cursor.dwo(0x87a98c07d08ea4e7) referenced by CU
at offset 0x1c8 [in module /src/chromium/src/out/Debug/lib/libui_base.so]
warning: Could not find DWO CU
obj/ui/base/cursor/ui_base.cursor_x11.dwo(0xbe634e7744ffa838) referenced by
CU at offset 0x258 [in module /src/chromium/src/out/Debug/lib/libui_base.so]
warning: Could not find DWO CU
obj/ui/base/ime/ui_base.dummy_input_method_delegate.dwo(0x5de3bfb9dcfb22ae)
referenced by CU at offset 0x4f8 [in module
/src/chromium/src/out/Debug/lib/libui_base.so]


I'm not sure whether this is a problem or just an annoyance.

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[chro...@googlecode.com]

Comment #3 on issue 404644 by dan...@chromium.org: gdb spews "warning:

Could not find DWO CU"
http://code.google.com/p/chromium/issues/detail?id=404644

Well, for some files I can, for some I can't.

[chro...@googlecode.com]

Comment #6 on issue 404644 by uk...@chromium.org: gdb spews "warning: Could
not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

I think this is fission issue.
does *.dwo file exist?
did compiler_proxy write the *.dwo file? (check /tmp/compiler_proxy.INFO)

[chro...@googlecode.com]

Updates:
Cc: yyanagis...@chromium.org

Comment #7 on issue 404644 by uk...@chromium.org: gdb spews "warning: Could

not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

(No comment was entered for this change.)

[chro...@googlecode.com]

Updates:
Cc: h...@chromium.org

Comment #8 on issue 404644 by yyanagis...@chromium.org: gdb spews "warning:

Could not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

I could reproduce the issue by "gclient sync" and "ninja -C out/Debug
chrome" with use_goma=1. (Note that I did not try using goma yet. I will
try later)

> does *.dwo file exist?

I did not check all but it seems yes.

With executing the command in #1, I got followings.
(snip)

warning: Could not find DWO CU

obj/third_party/webrtc/modules/video_coding/main/source/webrtc_video_coding.inter_frame_delay.dwo(0x36c686c4a23ce685)
referenced by CU at offset 0x951f0 [in module
/usr/local/google/ssd/yyanagisawa/chrome_src/src/out/Debug/chrome]

warning: Could not find DWO CU

obj/third_party/webrtc/modules/video_coding/utility/video_coding_utility.frame_dropper.dwo(0xddeb5e35703a1ba4)
referenced by CU at offset 0x952b0 [in module
/usr/local/google/ssd/yyanagisawa/chrome_src/src/out/Debug/chrome]

warning: Could not find DWO CU

obj/third_party/webrtc/common_audio/signal_processing/common_audio.auto_corr_to_refl_coef.dwo(0x11fa356e2b52c7a4)
referenced by CU at offset 0x96444 [in module
/usr/local/google/ssd/yyanagisawa/chrome_src/src/out/Debug/chrome]

warning: Could not find DWO CU

obj/third_party/webrtc/common_audio/signal_processing/common_audio.filter_ar_fast_q12.dwo(0x9f60ddc51e46ebb9)
referenced by CU at offset 0x96474 [in module
/usr/local/google/ssd/yyanagisawa/chrome_src/src/out/Debug/chrome]

warning: Could not find DWO CU

obj/third_party/webrtc/common_audio/signal_processing/common_audio.filter_ma_fast_q12.dwo(0x3b15ee7cd80d5020)
referenced by CU at offset 0x964a4 [in module
/usr/local/google/ssd/yyanagisawa/chrome_src/src/out/Debug/chrome]

warning: Could not find DWO CU

obj/third_party/webrtc/common_audio/signal_processing/common_audio.refl_coef_to_lpc.dwo(0xdcaa0da8d23089b5)
referenced by CU at offset 0x964d4 [in module
/usr/local/google/ssd/yyanagisawa/chrome_src/src/out/Debug/chrome]
done.
(gdb) quit

% cd out/Debug
% file
obj/third_party/webrtc/modules/video_coding/main/source/webrtc_video_coding.inter_frame_delay.dwo
obj/third_party/webrtc/modules/video_coding/main/source/webrtc_video_coding.inter_frame_delay.dwo:
ELF
64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

% file
obj/third_party/webrtc/modules/video_coding/utility/video_coding_utility.frame_dropper.dwo
obj/third_party/webrtc/modules/video_coding/utility/video_coding_utility.frame_dropper.dwo:
ELF
64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

% file
obj/third_party/webrtc/common_audio/signal_processing/common_audio.auto_corr_to_refl_coef.dwo
obj/third_party/webrtc/common_audio/signal_processing/common_audio.auto_corr_to_refl_coef.dwo:
ELF
64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

% file
obj/third_party/webrtc/common_audio/signal_processing/common_audio.filter_ar_fast_q12.dwo
obj/third_party/webrtc/common_audio/signal_processing/common_audio.filter_ar_fast_q12.dwo:
ELF
64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

% file
obj/third_party/webrtc/common_audio/signal_processing/common_audio.filter_ma_fast_q12.dwo
obj/third_party/webrtc/common_audio/signal_processing/common_audio.filter_ma_fast_q12.dwo:
ELF
64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

% file
obj/third_party/webrtc/common_audio/signal_processing/common_audio.refl_coef_to_lpc.dwo
obj/third_party/webrtc/common_audio/signal_processing/common_audio.refl_coef_to_lpc.dwo:
ELF
64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

> did compiler_proxy write the *.dwo file? (check /tmp/compiler_proxy.INFO)

I could not find anything about writing .dwo or not. However, files seems
to come from goma cache???

[chro...@googlecode.com]

Comment #9 on issue 404644 by yyanagis...@chromium.org: gdb spews "warning:

Could not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

It looks goma's issue. If I compile it without using goma, I did not see
the error.

[chro...@googlecode.com]

Comment #10 on issue 404644 by yyanagis...@chromium.org: gdb

spews "warning: Could not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

moreover, goma's cache issue. I did not see the error is I compiled it
with store only.

[chro...@googlecode.com]

Comment #11 on issue 404644 by uk...@chromium.org: gdb spews "warning: Could

not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

how different *.o and *.dwo between goma broken cached file and correct one?

[chro...@googlecode.com]

Comment #12 on issue 404644 by yyanagis...@chromium.org: gdb

spews "warning: Could not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

*.dwo looks same. *.o looks different but it looks only DW_AT_comp_dir.

[chro...@googlecode.com]

Comment #15 on issue 404644 by yyanagis...@chromium.org: gdb

spews "warning: Could not find DWO CU"

https://code.google.com/p/chromium/issues/detail?id=404644

I took a look at dwarf2read.c and identified the reason. DW_AT_comp_dir is
used to look up .dwo file.

As ukai-san mentioned, we should not omit cwd to retrieve cache.

[chro...@googlecode.com]

Comment #16 on issue 404644 by uk...@google.com: gdb spews "warning: Could

not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

we'll update goma backend soon.

for workaround, try to run gdb in out/Debug directory.
it seems gdb uses current directory to lookup *.dwo as well as
DW_AT_comp_dir in binary.

[chro...@googlecode.com]

Updates:
Status: Fixed

Comment #17 on issue 404644 by uk...@chromium.org: gdb spews "warning: Could

not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

updated goma backend.

[chro...@googlecode.com]

Comment #18 on issue 404644 by esei...@google.com: gdb spews "warning:

Could not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

Thank you so much for the work-around and for fixing this!

[chro...@googlecode.com]

Updates:
Status: Assigned
Owner: uk...@chromium.org

Comment #19 on issue 404644 by pos...@chromium.org: gdb spews "warning:

Could not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

I've started seeing lots of these errors when debugging Chrome built with
Goma on target ChromeOS devices. It started happening relatively recently,
perhaps 2 weeks ago or so... Is there anything I could do to identify the
issue?
Thanks.

[chro...@googlecode.com]

Comment #20 on issue 404644 by uk...@chromium.org: gdb spews "warning: Could

not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

which version of toolchain are you using?

I guess cros compiler wrapper adds -gsplit-dwarf so goma.frontend couldn't
know the request is using -gsplit-dwarf from the command line...
If so, I think explicitly adding -gsplit-dwarf would work

[chro...@googlecode.com]

Comment #21 on issue 404644 by pos...@chromium.org: gdb spews "warning:

Could not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

This is the current ChromeOS toolchain that comes with go/simplechrome.
It's downloaded and replaced to match the current ChromeOS one, overriding
host toolchain.

For x86:
$ x86_64-cros-linux-gnu-gcc --version
x86_64-cros-linux-gnu-gcc.real.elf (4.9.1_cos_gg_2f9796c_4.9.1-r82)
4.9.x-google 20140827 (prerelease)


For ARM:
$ armv7a-cros-linux-gnueabi-gcc --version
armv7a-cros-linux-gnueabi-gcc.real.elf (4.9.1_cos_gg_2f9796c_4.9.1-r82)
4.9.x-google 20140827 (prerelease)

By the way, these errors don't mean that I'm somehow using component build,
right?

[chro...@googlecode.com]

Comment #22 on issue 404644 by pos...@chromium.org: gdb spews "warning:

Could not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

Oh ok, looks like -gsplit-dwarf was recently enabled on simplechrome:

https://code.google.com/p/chromium/issues/detail?can=2&q=442504

What should be the correct solution here? Should we be using -gsplit-dwarf?

[chro...@googlecode.com]

Comment #23 on issue 404644 by uk...@chromium.org: gdb spews "warning: Could

not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

I couldn't access the issues (with uk...@google.com or uk...@chromium.org)

anyway, newer gcc uses fission by default (without specifying
-gsplit-dwarf).

%
usr/x86_64-pc-linux-gnu/x86_64-cros-linux-gnu/gcc-bin/4.9.x-google/x86_64-cros-linux-gnu-gcc.real
-g -c a.c -nostdinc
% readelf --debug-dump=info a.o | grep DW_AT_comp_dir
<15> DW_AT_comp_dir : (indirect string, offset: 0x49): /tmp


Is it cros gcc specific, or gcc 4.9 ?

[chro...@googlecode.com]

Comment #24 on issue 404644 by pos...@chromium.org: gdb spews "warning:

Could not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

Sorry, somehow that link was wrong, it's http://crbug.com/442504.

Removing -gsplit-dwarf "fixed" my gdb problems on target. I'm assuming it's
just simply lack of dwo files on target. I tried remote debugging from
host, and it also works, I guess it can find the dwos then. Still, the
ability to do debugging on target is very useful, so if this gets
permanently changed in future versions of gcc, it would become an issue.

But I guess we can close this bug then, since my problem is more of a
cros-specific/simplechrome issue. Thanks for the help!

[chro...@googlecode.com]

Comment #25 on issue 404644 by uk...@chromium.org: gdb spews "warning: Could

not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

ah, we should check -gsplit-dwarf not only on command lines, but also in
environment variables CFLAGS and CXXFLAGS.

[chro...@googlecode.com]

Comment #26 on issue 404644 by uk...@chromium.org: gdb spews "warning: Could

not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

well, these environment variables affects on make or gyp to construct
command line flags, so I think we don't need to change goma server side.

So, the issue posciak@ reported was that cros_chrome_sdk.py sets
-gsplit-dwarf on CFLAGS and CXXFLAGS, so gcc runs with -gsplit-dwarf, but
he didn't copy dwo files to target machine as the same directory as build
machine, so gdb couldn't find the dwo files.

[chro...@googlecode.com]

Updates:
Status: Fixed

Comment #27 on issue 404644 by uk...@chromium.org: gdb spews "warning: Could

not find DWO CU"
https://code.google.com/p/chromium/issues/detail?id=404644

(No comment was entered for this change.)