how to get symbol file for system library, like libc.so with dump_syms?

[慕冬亮]

Hi all,

when I try to dump all the symbols to convert stack trace into human-readable format, there is a problem: how to get symbol file for system library, like libc.so with dump_syms?

I read through the https://www.chromium.org/developers/decoding-crash-dumps and try to find answers. I find the following paragraph:

In order to get the symbol file for libfoo, one needs to have a copy of the exact libfoo binary from the system that generated the crash and its corresponding debugging symbols. Oftentimes, Linux distros provide libfoo and its debugging symbols as two separate packages. In the chrome build, you'll need an unstripped binary -- official builds generate these by default somewhere. After obtaining and extracting the packages, use dump_syms to extract the symbols. Assuming the library in question is /lib/libfoo.so and its debugging symbol is /usr/debug/lib/libfoo.so, run:

dump_syms /lib/libfoo.so /usr/debug/lib > /tmp/libfoo.so.sym

Then I install libc6-dbg package and find the symbol information in the /usr/lib/debug/.

~/chromium/src$ dpkg -L libc6-dbg

/.

/usr

/usr/lib

/usr/lib/debug

/usr/lib/debug/.build-id

/usr/lib/debug/.build-id/02

/usr/lib/debug/.build-id/02/4ccf84042931060e11f2e2946d903fb350e42e.debug

/usr/lib/debug/.build-id/04

/usr/lib/debug/.build-id/04/25091d8b069a12a75c5ec97279be6f5b6513ec.debug

/usr/lib/debug/.build-id/04/3278fcd845c0239471d7d7c958466bfd4194ec.debug

/usr/lib/debug/.build-id/04/70b5fd587e8bfe2b4a5a538b6e73fda96a4a20.debug

/usr/lib/debug/.build-id/04/8e54c404603bc2221c31f4af5af41a77f5f433.debug

/usr/lib/debug/.build-id/04/c69da65f8c5530d596ab7dee7e36bea13ff040.debug

......

/usr/lib/debug/.build-id/ff

/usr/lib/debug/.build-id/ff/385bb4d4394c439ab088c3e27e1e35d7e4fafd.debug

/usr/lib/debug/.build-id/ff/cff93945e56b22835d7f76b334892560c793a4.debug

/usr/lib/debug/.build-id/ff/de6de16a620257d3a1aaa157806c4ece248cbf.debug

/usr/lib/debug/.build-id/ff/fdfaa40fc97be8241150856702daf4e32cf26b.debug

/usr/share

/usr/share/doc

/usr/share/doc/libc6-dbg

/usr/share/doc/libc6-dbg/changelog.Debian.gz

/usr/share/doc/libc6-dbg/changelog.gz

/usr/share/doc/libc6-dbg/copyright

So I try to use dump_syms to dump symbol information with the following instruction:

$ out/Release/dump_syms /lib/x86_64-linux-gnu/libc-2.24.so /usr/lib/debug/ > /tmp/libcsym

Failed to write symbol file.

I don't know why. I am going to read the source code of dump_syms. If anyone finds clues to solve this problem, please let me know.

Thanks in advance.

[Lei Zhang]

What's the .gnu_debuglink in libc-2.24.so set to? If it's set to
$hash.debug, then you have to pass in the directory that contains
$hash.debug and not just /usr/lib/debug/.

If the symbols are compressed, e.g. on Fedora, then you will likely be
affected by https://bugs.chromium.org/p/google-breakpad/issues/detail?id=717
.

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
> ---
> You received this message because you are subscribed to the Google Groups
> "Chromium-dev" group.
> To view this discussion on the web visit
> https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/b96d5bf4-e5b4-49ca-9d2a-0df50f0825f8%40chromium.org.

[慕冬亮]

在 2017年8月3日星期四 UTC-4下午3:24:45，Lei Zhang写道：

What's the .gnu_debuglink in libc-2.24.so set to? If it's set to
$hash.debug, then you have to pass in the directory that contains
$hash.debug and not just /usr/lib/debug/.

$ objdump -s -j .gnu_debuglink /lib/x86_64-linux-gnu/libc-2.24.so

/lib/x86_64-linux-gnu/libc-2.24.so:     file format elf64-x86-64

Contents of section .gnu_debuglink:

 0000 34353066 36653336 32383738 36356430  450f6e36287865d0

 0010 39336561 32303962 38356132 32323230  93ea209b85a22220

 0020 39393235 66662e64 65627567 00000000  9925ff.debug....

 0030 89e47a7f                             ..z.

Then I try to make path of debug dir longer to reference $hash.debug, the result is the same with before.

[Lei Zhang]

On Thu, Aug 3, 2017 at 12:38 PM, 慕冬亮 <mudongl...@gmail.com> wrote:
>
>
> 在 2017年8月3日星期四 UTC-4下午3:24:45，Lei Zhang写道：
>>
>> What's the .gnu_debuglink in libc-2.24.so set to? If it's set to
>> $hash.debug, then you have to pass in the directory that contains
>> $hash.debug and not just /usr/lib/debug/.
>>
>
> $ objdump -s -j .gnu_debuglink /lib/x86_64-linux-gnu/libc-2.24.so
>
> /lib/x86_64-linux-gnu/libc-2.24.so: file format elf64-x86-64
>
> Contents of section .gnu_debuglink:
> 0000 34353066 36653336 32383738 36356430 450f6e36287865d0
> 0010 39336561 32303962 38356132 32323230 93ea209b85a22220
> 0020 39393235 66662e64 65627567 00000000 9925ff.debug....
> 0030 89e47a7f ..z.
>
> Then I try to make path of debug dir longer to reference $hash.debug, the
> result is the same with before.

As a sanity check, you can try running strace -e trace=file dump_syms
..., and see if dump_syms successfully opens $hash.debug.

If this is Debian 9, and your dump_syms binary has assert() enabled,
it's going to crash soon afterwards. Not sure if there's an existing
bug for this.

[慕冬亮]

在 2017年8月3日星期四 UTC-4下午4:44:26，Lei Zhang写道：

On Thu, Aug 3, 2017 at 12:38 PM, 慕冬亮 <mudongl...@gmail.com> wrote:
>
>
> 在 2017年8月3日星期四 UTC-4下午3:24:45，Lei Zhang写道：
>>
>> What's the .gnu_debuglink in libc-2.24.so set to? If it's set to
>> $hash.debug, then you have to pass in the directory that contains
>> $hash.debug and not just /usr/lib/debug/.
>>
>
> $ objdump -s -j .gnu_debuglink /lib/x86_64-linux-gnu/libc-2.24.so
>
> /lib/x86_64-linux-gnu/libc-2.24.so:     file format elf64-x86-64
>
> Contents of section .gnu_debuglink:
>  0000 34353066 36653336 32383738 36356430  450f6e36287865d0
>  0010 39336561 32303962 38356132 32323230  93ea209b85a22220
>  0020 39393235 66662e64 65627567 00000000  9925ff.debug....
>  0030 89e47a7f                             ..z.
>
> Then I try to make path of debug dir longer to reference $hash.debug, the
> result is the same with before.

As a sanity check, you can try running strace -e trace=file dump_syms
..., and see if dump_syms successfully opens $hash.debug.

I checked the filename in the .gnu_debuglink in the `dpkg -L libc6-dbg`. 

The concrete path for it is "/usr/lib/debug/.build-id/79/450f6e36287865d093ea209b85a222209925ff.debug"

So I modify the debug dir again as "/usr/lib/debug/.build-id/79/".

Then it crashed. The dmesg shows:

dmesg: read kernel buffer failed: Operation not permitted

 

This is very interesting now. :)

[Lei Zhang]

On Thu, Aug 3, 2017 at 2:23 PM, 慕冬亮 <mudongl...@gmail.com> wrote:
>
>
> 在 2017年8月3日星期四 UTC-4下午4:44:26，Lei Zhang写道：

> So I modify the debug dir again as "/usr/lib/debug/.build-id/79/".
>
> Then it crashed. The dmesg shows:

Yes, as I mentioned, dump_syms will crash. There's something about the
DWARF format that it doesn't understand.

> dmesg: read kernel buffer failed: Operation not permitted
>
> This is very interesting now. :)

I don't know why you want to look in dmesg when a program crashes, but
the failure you are seeing is unrelated to anything dump_syms did.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=842226#15

[慕冬亮]

在 2017年8月3日星期四 UTC-4下午5:32:29，Lei Zhang写道：

On Thu, Aug 3, 2017 at 2:23 PM, 慕冬亮 <mudongl...@gmail.com> wrote:
>
>
> 在 2017年8月3日星期四 UTC-4下午4:44:26，Lei Zhang写道：
> So I modify the debug dir again as "/usr/lib/debug/.build-id/79/".
>
> Then it crashed. The dmesg shows:

Yes, as I mentioned, dump_syms will crash. There's something about the
DWARF format that it doesn't understand.

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51

#1  0x00007ffff6ba83fa in __GI_abort () at abort.c:89

#2  0x00007ffff6b9fe37 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x48346d "size == 4 || size == 8", 

    file=file@entry=0x483484 "../../breakpad/src/common/dwarf/bytereader.cc", line=line@entry=58, 

    function=function@entry=0x4834e6 "void dwarf2reader::ByteReader::SetAddressSize(uint8)") at assert.c:92

#3  0x00007ffff6b9fee2 in __GI___assert_fail (assertion=0x48346d "size == 4 || size == 8", 

    file=0x483484 "../../breakpad/src/common/dwarf/bytereader.cc", line=58, function=0x4834e6 "void dwarf2reader::ByteReader::SetAddressSize(uint8)")

    at assert.c:101

#4  0x00000000004064ae in (anonymous namespace)::ByteReader::SetAddressSize (this=0x7fffffffd478, size=124 '|')

    at ../../breakpad/src/common/dwarf/bytereader.cc:58

#5  0x000000000040aa2f in (anonymous namespace)::CompilationUnit::ReadHeader (this=0x7fffffffd250)

    at ../../breakpad/src/common/dwarf/dwarf2reader.cc:273

#6  0x000000000040ac70 in (anonymous namespace)::CompilationUnit::Start (this=0x7fffffffd250) at ../../breakpad/src/common/dwarf/dwarf2reader.cc:299

#7  0x0000000000458204 in (anonymous namespace)::LoadDwarf<google_breakpad::ElfClass64> (dwarf_filename=..., elf_header=0x7ffff6779000, 

    big_endian=false, handle_inter_cu_refs=true, module=0x4c2130) at ../../breakpad/src/common/linux/dump_symbols.cc:297

#8  0x0000000000456fff in (anonymous namespace)::LoadSymbols<google_breakpad::ElfClass64> (obj_file=..., big_endian=false, 

    elf_header=0x7ffff6779000, read_gnu_debug_link=false, info=0x7fffffffd9c8, options=..., module=0x4c2130)

    at ../../breakpad/src/common/linux/dump_symbols.cc:685

#9  0x0000000000453235 in (anonymous namespace)::ReadSymbolDataElfClass<google_breakpad::ElfClass64> (elf_header=0x7ffff7e2e000, obj_filename=..., 

    debug_dirs=..., options=..., out_module=0x7fffffffdb20) at ../../breakpad/src/common/linux/dump_symbols.cc:978

#10 0x0000000000452a6a in (anonymous namespace)::ReadSymbolDataInternal (obj_file=0x7ffff7e2e000 "\177ELF\002\001\001\003", obj_filename=..., 

    debug_dirs=..., options=..., module=0x7fffffffdb20) at ../../breakpad/src/common/linux/dump_symbols.cc:1011

#11 0x000000000045340e in (anonymous namespace)::ReadSymbolData (obj_file=..., debug_dirs=..., options=..., module=0x7fffffffdb20)

    at ../../breakpad/src/common/linux/dump_symbols.cc:1080

#12 0x0000000000453330 in (anonymous namespace)::WriteSymbolFile (obj_file=..., debug_dirs=..., options=..., sym_stream=...)

    at ../../breakpad/src/common/linux/dump_symbols.cc:1024

#13 0x0000000000482cb6 in main (argc=3, argv=0x7fffffffdd28) at ../../breakpad/src/tools/linux/dump_syms/dump_syms.cc:106 

Just one assert problem. I don't know if it is known bug now.

> dmesg: read kernel buffer failed: Operation not permitted
>
> This is very interesting now. :)

I don't know why you want to look in dmesg when a program crashes, but
the failure you are seeing is unrelated to anything dump_syms did.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=842226#15

 I once did project related with crash analysis. So I want to check log in dmesg.

Actually, this "Operation not permitted" is caused by missing super privilege.