OSX dumps produce single frame stacks on big executable

[Evgeniy Zolenko]

Hi, 

We have a binary with a HUGE .dSYM -- just under 4Gb, release build of binary itself is 50Mb. Takes 1-2 hours just to run dump_syms on it, but eventually symbols are produced (unless machine dies from swap running out of disk space :)). Resulting sym file is 50Mb.

The problem we have is when walking the stack on socorro. It appears that as soon as stack walker finds itself in the main executable, it stops walking the stack further. So we have very short stacks -- just one line deep into interesting code. 

When running minidump_stackwalker built with trunk of chromium/breakpad repository (and same with older breakpad build) I get errors:

2015-12-12 01:24:07: minidump.cc:1415: ERROR: MinidumpThread has a memory region problem, 0x0+0x0, RVA 0x0x0

...

2015-12-12 01:24:14: minidump_processor.cc:237: ERROR: No memory region for /home/socorro/primaryCrashStore/20151212/name/b1/eb/b1eb441a-7d81-4090-9865-11cf42151212.dump:32/37 id 0x1af276

2015-12-12 01:24:14: basic_code_modules.cc:93: INFO: No module at 0x0

2015-12-12 01:24:14: stackwalker_x86.cc:621: ERROR: Can't get caller frame without memory or stack

Dump itself is generated with crashpad built from the code checked out in the middle of september.

All system and other symbols are found.

Apple crash report generated from the same crash has full stack.

Crashes for smaller and saner executables get walked correctly and produce the same number of frames as apple reports.

I haven't tried updating all pieces to latest source yet (dump_syms and crashpad libraries), haven't noticed any relevant commits that might affect it. 

But otherwise where else should I be looking?

Thanks!

[Mark Mentovai]

Evgeniy Zolenko wrote:

We have a binary with a HUGE .dSYM -- just under 4Gb, release build of binary itself is 50Mb. Takes 1-2 hours just to run dump_syms on it, but eventually symbols are produced (unless machine dies from swap running out of disk space :)). Resulting sym file is 50Mb.

The problem we have is when walking the stack on socorro. It appears that as soon as stack walker finds itself in the main executable, it stops walking the stack further. So we have very short stacks -- just one line deep into interesting code. 

When running minidump_stackwalker built with trunk of chromium/breakpad repository (and same with older breakpad build) I get errors:

2015-12-12 01:24:07: minidump.cc:1415: ERROR: MinidumpThread has a memory region problem, 0x0+0x0, RVA 0x0x0

That might be a problem. Does crashpad_handler have anything to say when it’s writing the dump? Watch its stderr.

Do any of the threads in the dump have valid stack regions? You can run minidump_dump and look at each MDRawThread’s stack.* fields.

2015-12-12 01:24:14: minidump_processor.cc:237: ERROR: No memory region for /home/socorro/primaryCrashStore/20151212/name/b1/eb/b1eb441a-7d81-4090-9865-11cf42151212.dump:32/37 id 0x1af276

2015-12-12 01:24:14: basic_code_modules.cc:93: INFO: No module at 0x0

2015-12-12 01:24:14: stackwalker_x86.cc:621: ERROR: Can't get caller frame without memory or stack

Dump itself is generated with crashpad built from the code checked out in the middle of september.

For best results, you should test with the current trunk, although this code should not have changed much since then.

All system and other symbols are found.

Apple crash report generated from the same crash has full stack.

Crashes for smaller and saner executables get walked correctly and produce the same number of frames as apple reports.

I haven't tried updating all pieces to latest source yet (dump_syms and crashpad libraries), haven't noticed any relevant commits that might affect it. 

But otherwise where else should I be looking?

If the problem is indeed a thread without a captured stack in the minidump, we’ll be looking around ProcessReader::CalculateStackRegion().

[Eugene Zolenko]

Ok, I narrowed it down a bit to CFI section. 

All pieces updated to trunk (dump_syms and minidump_stackwalker from breakpad, crashpad_handler and client code from crashpad). All memory region ERRORs are gone now.

If I hide all symbols from stackwalker, I see full stack (but not symbolicated obviously).

If I generate main symbol with no CFI section (dump_syms -c, either giving it dSYM or omitting it completely), I see full stack, but no source locations and C++ function names are mangled.

If I generate main symbol with CFI section (but omitting dSYM), I'm back to one frame. 

All symbols except the one for main executable are hidden during those runs...

I'll try setting up debugger for minidump_stackwalker tomorrow...

[Mark Mentovai]

Eugene Zolenko wrote:

Ok, I narrowed it down a bit to CFI section. 

All pieces updated to trunk (dump_syms and minidump_stackwalker from breakpad, crashpad_handler and client code from crashpad). All memory region ERRORs are gone now.

Great!

Out of the problems you seem to be having, those “memory region problem” ones were the only things that Crashpad could have been causing. These next few things are all Breakpad-related:

If I hide all symbols from stackwalker, I see full stack (but not symbolicated obviously).

If I generate main symbol with no CFI section (dump_syms -c, either giving it dSYM or omitting it completely), I see full stack, but no source locations and C++ function names are mangled.

If I generate main symbol with CFI section (but omitting dSYM), I'm back to one frame. 

You should be giving both your main executable and, via -g, its companion .dSYM to dump_syms for best results. That should give you demangled names with full namespace and class qualification and parameter types. It should also give you source file names and line numbers, and unless you use -c, CFI.

If you’re doing all of this and are still getting single-frame walks, then it does indeed sound like a CFI problem, but it’d be more appropriate to discuss on google-breakpad-dev.

All symbols except the one for main executable are hidden during those runs...

Have you generated symbols (dump_syms) for any of the other modules?

[Eugene Zolenko]

If I hide all symbols from stackwalker, I see full stack (but not symbolicated obviously).

If I generate main symbol with no CFI section (dump_syms -c, either giving it dSYM or omitting it completely), I see full stack, but no source locations and C++ function names are mangled.

If I generate main symbol with CFI section (but omitting dSYM), I'm back to one frame. 

You should be giving both your main executable and, via -g, its companion .dSYM to dump_syms for best results. That should give you demangled names with full namespace and class qualification and parameter types. It should also give you source file names and line numbers, and unless you use -c, CFI.

 

If you’re doing all of this and are still getting single-frame walks, then it does indeed sound like a CFI problem, but it’d be more appropriate to discuss on google-breakpad-dev.

All symbols except the one for main executable are hidden during those runs...

Have you generated symbols (dump_syms) for any of the other modules?

Yep, normally I'm collecting all symbols for all modules and use dSYMs when available, also I have most OSX system symbols. So in most cases I get all symbols resolved on a crash, with line numbers and links to source control for our own binaries. This one binary (which is also the main thing users are running and crashing) has trouble with stack when CFI section is generated (regardless if I use dSYM or not). Other executables are seemingly unaffected.

I'll poke around more and send this to breakpad-dev, thanks.