Linux Stackwalk No Symbols

[Stephen Laird]

When I run the minidump_stackwalk on my minidump and symbols, I get a stacktrace, but it looks incomplete.  I tested breakpad on a dummy program much like the one in the LinuxStarterGuide wiki, and I was getting function names and line numbers of where the crash occurred.  Now that it's integrated with my larger application I'm not getting that information.  Here's what the stacktrace looks like:

>> Operating system: Linux
>>                   0.0.0 Linux 3.2.60 #18 SMP Fri May 15 17:02:36 EDT 2015 x86_64
>> CPU: amd64
>>      family 6 model 62 stepping 4
>>      4 CPUs
>>
>> Crash reason:  SIGSEGV
>> Crash address: 0x0
>> Process uptime: not available
>>
>> Thread 0 (crashed)
>>  0  myapplication + 0x1057a8
>>     rax = 0x0000000000000000   rdx = 0x00007f147ee65e98
>>     rcx = 0x0000000000017680   rbx = 0x0000000000916080
>>     rsi = 0x0000000000000000   rdi = 0x00007f147ee65e40
>>     rbp = 0x00000000005057a0   rsp = 0x00007fff45d3f7b0
>>     r8 = 0x00007f1480281720    r9 = 0x0000000000000000
>>     r10 = 0x0000000000000022   r11 = 0x0000000000000206
>>     r12 = 0x00000000004bf784   r13 = 0x00007fff45d4cb40
>>     r14 = 0x0000000000000000   r15 = 0x0000000000000000
>>     rip = 0x00000000005057a8
>>     Found by: given as instruction pointer in context
>>
>> Loaded modules:
>> 0x00400000 - 0x00708fff  myapplication  ???  (main)  (WARNING: No symbols, myapplication, 000000000000000000000000000000000)
>> 0x7f147e24d000 - 0x7f147e463fff  libz.so.1.2.7  ???
>> 0x7f147e464000 - 0x7f147e857fff  libcrypto.so.1.0.0  ???
>> 0x7f147e85c000 - 0x7f147eaddfff  libm-2.13.so  ???
>> 0x7f147eade000 - 0x7f147ee64fff  libc-2.13.so  ???
>> 0x7f147ee6a000 - 0x7f147f07ffff  libgcc_s.so.1  ???
>> 0x7f147f080000 - 0x7f147f2dffff  libssl.so.1.0.0  ???
>> 0x7f147f2e0000 - 0x7f147f4e3fff  libdl-2.13.so  ???
>> 0x7f147f4e4000 - 0x7f147f6ecfff  libcrypt-2.13.so  ???
>> 0x7f147f71b000 - 0x7f147fa0cfff  libstdc++.so.6.0.17  ???
>> 0x7f147fa22000 - 0x7f147fc4cfff  libsgutils2.so.2.0.0  ???
>> 0x7f147fc4d000 - 0x7f147fe54fff  librt-2.13.so  ???
>> 0x7f147fe55000 - 0x7f148006cfff  libpthread-2.13.so  ???
>> 0x7f1480071000 - 0x7f1480090fff  ld-2.13.so  ???
>> 0x7fff45d8d000 - 0x7fff45d8dfff  linux-gate.so  ???


Why is it saying "WARNING: No symbols, myapplication, 000000000000000000000000000000000"?  I have the symbol files stored in /symbols/myapplication/{buildID}/myapplication.sym, but it looks like it's trying to find /symbols/myapplication/000000000000000000000000000000000/myapplication.sym.  The output of running minidump_stackwalk gives this line at one point as well:

>> 2015-07-14 14:37:01: stackwalker.cc:97: INFO: Couldn't load symbols for: myapplication|000000000000000000000000000000000

Any thoughts?
Thanks in advance!

~Stephen

[Stephen Laird]

After looking into things it does appear that minidump_stackwalk is looking for the build id 000000000000000000000000000000 instead of the correct build id.  What could cause it to look for the wrong build id?

[Ted Mielczarek]

On Tue, Jul 14, 2015, at 04:13 PM, Stephen Laird wrote:

After looking into things it does appear that minidump_stackwalk is looking for the build id 000000000000000000000000000000 instead of the correct build id.  What could cause it to look for the wrong build id?

 

I have seen this a few times where Breakpad fails to write the build id field in the minidump, usually in out of memory type crashes. Breakpad does avoid using the libc allocator when it writes a minidump, but it does use mmap to allocate memory, so if your process is nearly out of virtual memory this can fail. I think it also mmaps entire binary files into memory to find the build id, which is probably not the best design decision, so if your binary file is large that may not be helping. In Firefox we were able to fix some of these issues by reserving a block of memory that we free in Breakpad's pre-write-minidump callback.

 

-Ted

 

[Stephen Laird]

That's interesting, I hadn't considered that as an option.  My binary is somewhat large, but I don't think I should be running out of memory.  Right now I am still in the testing stages, so I am artificially crashing the application by assigning to a NULL pointer.  But in case running out of memory is the problem, where in the minidump is the build id written?  I could look to see if the build id is being written correctly.

~ Stephen

[Ted Mielczarek]

You can use the minidump_dump tool (built right next to minidump_stackwalk) to inspect the minidump structures. It's pretty verbose, but grepping through the output should let you find what you need.

 

If you're testing with a simple NULL deref I wouldn't expect the scenario I described to be an issue, honestly. It's possible there's some other bug at fault.

 

-Ted

--

You received this message because you are subscribed to the Google Groups "google-breakpad-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-breakpad-d...@googlegroups.com.

To post to this group, send email to google-brea...@googlegroups.com.

Visit this group at http://groups.google.com/group/google-breakpad-discuss.

For more options, visit https://groups.google.com/d/optout.

 

[Stephen Laird]

Thanks, that got me what I needed there.  The build id (or debug_identifier as the minidump_dump called it) was written in as 000000000000000000000000000000 instead of the correct name, so the problem appears to be writing the build id to the minidump. 

Yeah I would agree with you that it's probably not your scenario, although adding some pre-allocated memory for bullet-proofing reasons might be good to do anyways.  Could something in the build process of my binary be at fault perhaps?  That was my original hunch, but I'm not sure.

~ Stephen

[Stephen Laird]

I looked into the possibility of the memory being all used up, and I learned that another component of my application as a whole is very greedy for memory, so there is actually a high likelihood that my problem is what you described after all.  How large a block of memory did you need to allocate for the minidump writing?

~Stephen

[Ted Mielczarek]

We ballparked it by looking at the on-disk size of our largest shared library and adding a bit of a fudge factor. We were actually seeing the problem mostly on Windows, so our solution is pretty Windows-specific, but you should be able to do the same thing elsewhere:

https://hg.mozilla.org/mozilla-central/annotate/e7e69cc8c07b/toolkit/crashreporter/nsExceptionHandler.cpp#l374

 

-Ted

[Stephen Laird]

Thanks, I'll be looking into it.  Hate to switch back and forth on this, but I found I was mistaken about the memory usage in my application.  The other memory-greedy component has been modified to be less greedy, and may not be the culprit of my problem after all.  As a safe-guarding feature I may try pre-allocating memory anyways. 

Would having another custom exception handler present be interfering with the writing of the build ID?

~ Stephen

[Ted Mielczarek]

Breakpad has a "filter callback" that you can pass to the ExceptionHandler constructor that will let you do work before writing a minidump, that's what we use:

https://hg.mozilla.org/mozilla-central/annotate/e7e69cc8c07b/toolkit/crashreporter/nsExceptionHandler.cpp#l1235

https://hg.mozilla.org/mozilla-central/annotate/e7e69cc8c07b/toolkit/crashreporter/nsExceptionHandler.cpp#l977

 

-Ted

[Stephen Laird]

Could you point me to the code segment where the build ID is written into the minidump?  I've been digging through the Breakpad source without much success.  I don't think my problem is a lack of memory, but the build ID is definitely not getting written to the minidump correctly (still all 0s) so I want to try to track down my problem.

Thanks,
~ Stephen

[Stephen Laird]

Reviving this just for an update.  I implemented a fix like you suggested, Ted, where I allocate a block of memory that is freed in the filter callback, but I'm still getting a string of 0s for the debug_identifier in the minidump.  Thanks for the tip though!  If you happen to have any other ideas, shoot them my way.  Otherwise I'll go back to banging my head about it on my own. :-)

~ Stephen

[Stephen Laird]

Hey Ted and others,

I've done a lot more digging, and think that I've found the root of my problem.

The application I am integrating Breakpad handling to is running on a switched root filesystem. On startup, the application mounts several directories to "/mnt/rootfs". Then it does a switch root, and the "/mnt/rootfs" becomes "/". The problem is that Breakpad goes off and looks for my binaries in places like "/mnt/ramdisk/binary" instead of "/binary"

Is there a way to notify Breakpad about this change so that it looks in the correct path?  Right now the easiest fix I can think of is to look at the path name and return a pointer that ignores the extra "/mnt/ramdisk" part and just returns "/binary" like I want, but that seems rather hackish and I'd love to implement a better fix if one is available.

Thanks!
Stephen