minidump-2-core

[Bruce Dawson]

I'm trying to use minidump-2-core on some of the breakpad crash files from our Linux builds and I'm not having much luck. gdb doesn't seem to like them. The core file loads into gdb and it says a few encouraging things, but mostly discouraging things:

 

No symbol table is loaded.  Use the "file" command.
warning: core file may not match specified executable file.
[New LWP 30951]
[New LWP 30925]
[New LWP 30927]
[New LWP 30928]
[New LWP 30941]
[New LWP 30942]
[New LWP 30943]
[New LWP 30944]
[New LWP 30945]
[New LWP 30946]
[New LWP 30947]
[New LWP 30948]
[New LWP 30949]
[New LWP 30954]
[New LWP 30955]
[New LWP 30956]
[New LWP 30957]
[New LWP 30958]
[New LWP 30959]
[New LWP 30960]
[New LWP 30961]
[New LWP 30962]
[New LWP 30964]
[New LWP 30965]
Core was generated by `/home/magazino/.local/share/Steam/SteamApps/fooandbar/Team Fortress 2/hl2_linux'.
#0  0xf0c51136 in ?? ()

I've written a script that goes through the list of modules and uses the breakpad IDs to retrieve the shared objects into the correct directory, so I know that all of our shared objects are there and correct. However gdb warns that the core file might not match the specified executable file, and shows nothing but garbage for the call stacks:

 

(gdb) bt
#0  0xf7727430 in ?? ()
#1  0xf5f85018 in ?? ()
#2  0x1a4c96f8 in ?? ()
#3  0x00000000 in ?? ()

I could reconstruct the binaries in the same layout as the customer's file system, if that is what is needed, but I thought I'd ask first.

 

[Bruce Dawson]

I tried reconstructing the file structure for all of our code, but that didn't work -- the stacks are still no good. I don't know if loading a core file on a different machine is just impossible, or if there is more that I need to do. I guess it would be possible to create a chroot environment that mimics the layout of the system where the program crashed, but I'd have to retrieve binaries for every .so that was loaded, which is probably not going to happen.

[Bruce Dawson]

I figured out the answer to my own question. The important point is that you have to execute the gdb command "add-symbol-file <sopath> <textsectionaddress>" for each so that you want gdb to load.

 

So, in addition to parsing minidump_stackwalk -m to get breakpad debug identifiers to let me find the shared objects in our symbol store, I also parsed out the load addresses. Then I used readelf -S to get the offset of the .text section, added those, and created a gdbcommands.txt file. After running my script I can load the core file like this:

 

    cgdb --core core -x gdbcommands.txt

 

I can now walk the stack and view local variables. A few of them look suspicious, but there's clearly enough value to make this a worthwhile process.