dump_syms dwarf state machine bug

[Aleksey Khoroshilov]

Hi everyone! We see a lot of dumps where some frames point to a correct function, but show a line number "0". I decided to dig a little to figure out what's happening.

My conclusion: dump_syms incorrectly interprets dwarf state machine in some cases. This lead to many records with line == 0 and minidump_stackwalk outputting 0 as a result.

Why I think dump_syms is wrong? gdb shows perfectly reasonable line numbers for the same binary (it has a slightly different dwarf state machine). I don't think it's a dwarf generator issue, because the opcode stream looks fine and reasonable.

I used dump_syms from Chromium 115.0.5790.56 checkout to generate symbols for the dump_syms binary on Linux x86_64. Here's an example:

FUNC 4b6d0 114 0 google_breakpad::PageAllocator::Alloc(unsigned long)

4b6d0 e 73 155
4b6de 3 74 155
4b6e1 6 74 155
4b6e7 9 0 155   <-- ???
4b6f0 7 77 155

4b6f7 2 77 155
4b6f9 4 77 155
4b6fd 6 77 155
4b703 3 77 155
4b706 2 77 155
4b708 3 78 155

```

As you can see, we have `4b6e7 9 0 155` which points to line 0 in file 155.

gdb on the other hand treats the range [0x4b6de - 0x4b6f0) as a single record without any intermediate 0-line records:

$ gdb dump_syms

(gdb) info line *0x4b6de

Line 74 of "../../third_party/breakpad/breakpad/src/common/memory_allocator.h" starts at address 0x4b6de <google_breakpad::PageAllocator::Alloc(unsigned long)+14> and ends at 0x4b6f0 <google_breakpad::PageAllocator::Alloc(unsigned long)+32>.

(gdb) info line *0x4b6e7  <-- this is a "zero line" address
Line 74 of "../../third_party/breakpad/breakpad/src/common/memory_allocator.h" starts at address 0x4b6de <google_breakpad::PageAllocator::Alloc(unsigned long)+14> and ends at 0x4b6f0 <google_breakpad::PageAllocator::Alloc(unsigned long)+32>.

(gdb) info line *0x4b6f0  <-- next range

Line 77 of "../../third_party/breakpad/breakpad/src/common/memory_allocator.h" starts at address 0x4b6f0 <google_breakpad::PageAllocator::Alloc(unsigned long)+32> and ends at 0x4b708 <google_breakpad::PageAllocator::Alloc(unsigned long)+56>.

It would be very useful to fix dump_syms dwarf parser to not generate these bogus 0-line addresses.

[Marc Gonzalez]

On 04/07/2023 11:28, Aleksey Khoroshilov wrote:

> Hi everyone!
>
> We see a lot of dumps where some frames point to a correct function,
> but show a line number "0". I decided to dig a little to figure out
> what's happening.
>
> My conclusion: dump_syms incorrectly interprets dwarf state machine
> in some cases. This lead to many records with line == 0 and
> minidump_stackwalk outputting 0 as a result.
>
> Why I think dump_syms is wrong? gdb shows perfectly reasonable line
> numbers for the same binary (it has a slightly different dwarf state
> machine). I don't think it's a dwarf generator issue, because the
> opcode stream looks fine and reasonable.
>
> I used dump_syms from Chromium 115.0.5790.56 checkout to generate

> symbols for the dump_syms binary on Linux x86_64. [snip example]

Hello Aleksey,

In a "we have now come full circle" moment, mozilla has rewritten
some google-breakpad tools (which were, long ago, forked from moz).

You may want to give moz dump_syms a try, I've heard "it works better
than google dump_syms in some situations".

https://github.com/mozilla/dump_syms


Tangent

Some may be interested in these 2 (long!) articles:
https://hacks.mozilla.org/2022/06/everything-is-broken-shipping-rust-minidump-at-mozilla/
https://hacks.mozilla.org/2022/06/fuzzing-rust-minidump-for-embarrassment-and-crashes/

Regards