Minidump-2-core
Joslyn Moreci
For Firefox releases older than Firefox 41, you will also need toinstall the relevant release of Firefox (for example fromhere),and add the directory it is in (e.g., C:Program FilesMozillaFirefox 3.6 Beta 1") to the same dialog in which you set up thesymbol server (in case the binary location in the minidump is not thesame as the one on your machine). Note that you can install therelevant release anywhere. Just make sure to configure the symbolserver to the directory where you installed it. For releases from 41onward, the binaries are available on the symbol server.
The Breakpadsourcecontains a tool calledminidump-2-core,which converts Linux minidumps into core files. If you checkout andbuild Breakpad, the binary will be atsrc/tools/linux/md2core/minidump-2-core. Running the binary with thepath to a Linux minidump will generate a core file on stdout which canthen be loaded in gdb as usual. You will need to manually download thematching Firefox binaries, but then you can use the GDB Pythonscript to download symbols.
The minidump-2-core source does not currently handle processingminidumps from a different CPU architecture than the system it wasbuilt for. If you want to use it on an ARM dump, for example, you mayneed to build the tool for ARM and run it under QEMU.
Ted has a few tools that can be built against an already-built copy ofBreakpad to do more targeted inspection. All of these tools assume youhave checked out their source in a directory next to the breakpadcheckout, and that you have built Breakpad in an objdir namedobj-breakpad at the same level.
dump-lookuptakes a minidump and prints values on the stack that are potentialreturn addresses. This is useful when a stack trace looks truncatedor otherwise wrong. It needs symbol files to produce useful output,so you will generally want to have run stackwalk-http to downloadthem first.
get-minidump-instructionsretrieves and displays the memory range surrounding the faultinginstruction pointer from a minidump. You will almost always want torun it with the --disassemble option, which will make it send thebytes through objdump to display the disassembled instructions.If you also give it a path to symbols (see stackwalk-http above)it can download the matching source files from hg.mozilla.org anddisplay source interleaved with the disassembly.
minidump-modulestakes a minidump and prints the list of modules from the crashedprocess. It will print the full path to each module, whereas theSocorro UI only prints the filename for each module for privacyreasons. It also accepts a -v option to print the debug ID for eachmodule, and a -d option to print relative paths to the symbol filesthat would be used instead of the module filenames.
First, setup the necessary files for the debugger. You need both the originalexecutables and/or libraries whose symbols you wish to calculate, and you willneed their corresponding debug information. Googlers: See Setup files fordebuggerfor how to get these for official images.
Googlers: Alternatively, you can try my generate_gdb_command_file script toautomatically generate a gdb command file for adding all the symbol files. Forthe "--top" command-line option, specify the path to where your image ismounted. Then, pass the path to the file to which you redirectedminidump-2-core's stderr.
Sometimes, the gdb backtrace command (Use gdb to show abacktrace)doesn't show a stack trace any better than that of minidump_stackwalk (Useminidump_stackwalk to show a stacktrace).If you think there's more to it than what those two are showing you, try thismethod to naively dump all the known symbol addresses seen on the stack. You'llsee some false positives, but you may just find the name of a function thatseems like a plausible place to look.
First, start from the above step of using gdb to show abacktrace.We can reuse the gdb command file that was generated for it. Googlers: If youdidn't use my generate_gdb_command_file script, you can manually create the gdbcommand file containing any "add-symbol-file" commands you ran.
Second, determine the base address of the stack. The easiest way is to just lookat the stack address in the minidump. This command will set the $sp_addrvariable to the stack address of the first thread in the minidump:
If you want to be smarter, you can run minidump_stackwalk on the minidump (usingthe appropriate Breakpad symbols) and skip past any stack frames you can assumeare accurate. You would do this by looking for the stack pointer of the lastconsecutive call frame found by "call frame info". For example, for issue the stack pointer value would be 0x7e8b3440 giventhe following top-most (i.e. higher address) frames:
Finally, run the following gdb command to dump 1024 words of the stack (i.e.potential addresses). In the output, for any value that corresponds to a knownsymbol, gdb will annotate it with the symbol's name in angle brackets. Knowingthat, we can simply pipe the output to something (in this case, perl) to pullout any annotation it finds in the stack dump.
I have a small qt project with a segmentation fault that I want to track (this is an example and the segmentation fault is manually added for testing purposes, in the real project I do not know the segfaults left there). I decided to use google crashpad so I can create minidump files and then track the issues on my computer. I have been successful in doing so in windows with the example project. However, I am cannot open the dmp files in Linux to debug them.
I have tried using minidump-2-core as explained in this blog on the section Using minidump-2-core on Linux. However, when I run this command I get the following error: This minidump was not generated by Linux or NaCl. Obviously I am using the same computer with Ubuntu 18.04 for running the code that generates the dmp file and to convert it to a core file.
You'll want to build minidump_stackwalk which is a command line tool from the Breakpad repository. You'll also need to generate .sym files using dump_syms which is another tool from the Breakpad repository.
Once the symbol files have been laid out in the correct folder structure you can invoke minidump_stackwalk which will output a symbolicated stack trace and a list of modules that were loaded at runtime:
I've see google-breakpad which is meant to create a "minidump" when a process crashes. In google-breakpad there are two utilities core2md and minidump-2-core which at first glance should be able to convert a core file to minidump format, and then back to a core file from the minidump with just the stack information. The problem with this is that core2md requires the process information from under /proc/$PID/ and that's not something I have.
Probably your best bet is to integrate google breakpad into your Linux process so that when a crash occurs it is google breakpad recording the crash instead of the OS creating a core file. This will get you the information that you want, and then you can either use minidump_stackwalk to dump call stacks (handy for summarizing lots of crashes) or you can use minidump-2-core to create a core file.
It is unfortunate that Linux does not seem to have an effective way to create minimal core files with just the contents of the threads' stacks. These files (minidumps in the Windows world) are extremely useful -- they have a very high information-to-size ratio which allows easy collection of millions of crash dumps.
So, the way GDB finds the list of shared libraries that have been
loaded is by traversing a linked list maintained by the dynamic
linker. (It's a documented interface, so this isn't as much of a
kludge as it may sound.) However, a minidump doesn't contain data
segment contents, only stack contents, so the actual linked list is
gone. I suppose minidump-2-core could build a fake version of it that
GDB would innocently accept, but minidump-2-core doesn't do that right
now.
Minidumps are small, concise crash dumps, written by the Google Breakpad library. They contain the crash reason, up to 8 KB of stack memory per thread, plus environment (OS, CPU, ...) and process information (loaded modules). The resulting files are much smaller than traditional core dumps and can be collected over networks much more quickly. Symbol resolution happens on the developer side, so there is no need to ship debug symbols together with Impala binaries. The Breakpad library is widely used, e.g. in Chrome, Firefox, Android and developed as an open source project. Minidump files written by Impala have names like 7dd118b5-fd6a-9098-79215292-715fcff9.dmp. A more detailed introduction to Breakpad can be found here.
The Impala repository contains a script called dump_breakpad_symbols.py, which can be used to convert debug symbols from various sources into Breakpad's own format. The script will write a directory of files, which can then be used by Breakpad's minidump_stackwalk tool to resolve the symbols. Breakpad symbol files are identified by hashes, so you can dump symbols for multiple binaries into the same destination folder.
As you can see the script picks up all ELF files it can find, including some unit tests. Dumping symbols from the tests is not necessary and can take several minutes. If you don't want this you can specify all files to extract manually.
If you obtained a minidump file from a host that had been installed using RPM packages, then you will have to retrieve matching versions of the RPM package that is installed on the host and its corresponding debuginfo package. For example, for release versions of Cloudera's distribution of Impala, you can find these files here:
Minidumps can be converted into core files, which can be inspected with gdb. To do so, call the minidump-2-core tool. Then you should be able to open the core as usual. (TODO: It doesn't seem to work like this).
I've had great success with the Linux build, however, on Windows, I've gotten as far as getting a crash dump, extracting symbols from the .exe using dump_syms utility, and using minidump_stackwalk to check out the crash and line number.
e59dfda104