confused about how to use breakpad on Linux

[Yubao Liu]

Hi,

I'm not sure whether it's OK that minidump_stackwalk doesn't show
function names and line numbers.

$ cat test.cpp
#include <iostream>
#include "client/linux/handler/exception_handler.h"

static bool dumpCallback(const char* dump_path,
const char* minidump_id,
void* context,
bool succeeded)
{
std::cerr << "Minidump file created: " << dump_path << "/" <<
minidump_id << ".dmp\n";
return succeeded;
}

void hello()
{
int a = 2/0;
}

int main(int argc, char* argv[])
{
google_breakpad::ExceptionHandler eh(".", NULL, dumpCallback,
NULL, true);
hello();
return 0;
}

$ ( cd google-breakpad; ./configure; make; )
$ g++ -g -I google-breakpad/src/ -o test test.cpp google-breakpad/src/
client/linux/libbreakpad_client.a
test.cpp: In function ‘void hello()’:
test.cpp:16: warning: division by zero
$ ./test
Minidump file created: ./7c2b2bb7-7b15-53ae-7cacc238-759a349d.dmp
Floating point exception
$ google-breakpad/src/processor/minidump_stackwalk ./
7c2b2bb7-7b15-53ae-7cacc238-759a349d.dmp
2010-11-08 13:39:37: minidump_processor.cc:264: INFO: Processing
minidump in file ./7c2b2bb7-7b15-53ae-7cacc238-759a349d.dmp
2010-11-08 13:39:37: minidump.cc:3500: INFO: Minidump opened
minidump ./7c2b2bb7-7b15-53ae-7cacc238-759a349d.dmp
2010-11-08 13:39:37: minidump.cc:3545: INFO: Minidump not byte-
swapping minidump
2010-11-08 13:39:37: minidump.cc:3902: INFO: GetStream: type
1197932545 not present
2010-11-08 13:39:37: minidump.cc:3902: INFO: GetStream: type
1197932546 not present
2010-11-08 13:39:37: minidump.cc:1945: INFO: MinidumpModule could not
determine version for /home/liuyb/home/jff/bitsway/test
2010-11-08 13:39:37: minidump.cc:1945: INFO: MinidumpModule could not
determine version for /lib/libc-2.11.2.so
2010-11-08 13:39:37: minidump.cc:1945: INFO: MinidumpModule could not
determine version for /lib/libgcc_s.so.1
2010-11-08 13:39:37: minidump.cc:1945: INFO: MinidumpModule could not
determine version for /lib/libm-2.11.2.so
2010-11-08 13:39:37: minidump.cc:1945: INFO: MinidumpModule could not
determine version for /usr/lib/libstdc++.so.6.0.13
2010-11-08 13:39:37: minidump.cc:1945: INFO: MinidumpModule could not
determine version for /lib/ld-2.11.2.so
2010-11-08 13:39:37: minidump.cc:1945: INFO: MinidumpModule could not
determine version for linux-gate.so
2010-11-08 13:39:37: minidump_processor.cc:116: INFO: Minidump ./
7c2b2bb7-7b15-53ae-7cacc238-759a349d.dmp has CPU info, OS info, no
Breakpad info, exception, module list, thread list, no dump thread,
and requesting thread
2010-11-08 13:39:37: minidump_processor.cc:150: INFO: Looking at
thread ./7c2b2bb7-7b15-53ae-7cacc238-759a349d.dmp:0/1 id 0x5885
2010-11-08 13:39:37: minidump.cc:307: INFO: MinidumpContext: looks
like AMD64 context
2010-11-08 13:39:37: minidump.cc:307: INFO: MinidumpContext: looks
like AMD64 context
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26376d0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26377b8
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x1b6150824
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0xf27028
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0xf294b8
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0xf29298
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0xf27028
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0xf294b8
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0xf29298
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2637701
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0xf27040
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0xf29410
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0xf29460
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0xf29490
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x400000003
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0xf27028
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26377b8
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x100000000
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0xfeef377475b3c6b6
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26377b0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x11093b29873c6b6
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x5c5b536da3c6b6
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fff00000000
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26377b8
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x1
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26377b0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26377a8
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x1c
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x1
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd263816e
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at 0x0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638175
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638188
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26381a1
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26381ba
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26381cf
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26381df
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26381eb
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd263823b
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd263825c
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd263826e
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638283
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26382a3
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26382ae
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd263879e
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26387b0
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd26387dd
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638cc4
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638cdd
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638d2b
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638d40
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638d57
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638d68
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638d89
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638d9d
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638dae
2010-11-08 13:39:37: basic_code_modules.cc:88: INFO: No module at
0x7fffd2638e37
2010-11-08 13:39:37: minidump_processor.cc:258: INFO: Processed ./
7c2b2bb7-7b15-53ae-7cacc238-759a349d.dmp
2010-11-08 13:39:37: minidump.cc:3472: INFO: Minidump closing minidump
Operating system: Linux
0.0.0 Linux 2.6.32-5-amd64 #1 SMP Wed Oct 20
00:05:22 UTC 2010 x86_64
CPU: amd64
family 6 model 23 stepping 10
2 CPUs

Crash reason: SIGFPE
Crash address: 0x4018df

Thread 0 (crashed)
0 test + 0x18df
rbx = 0xd26376b0 r12 = 0x00401780 r13 = 0xd26377b0 r14 =
0x00000000
r15 = 0x00000000 rip = 0x004018df rsp = 0xd2637600 rbp =
0xd2637600
Found by: given as instruction pointer in context
1 test + 0x19c3
rip = 0x004019c4 rsp = 0xd2637610
Found by: stack scanning
2 test + 0x1863
rip = 0x00401864 rsp = 0xd2637630
Found by: stack scanning
3 test + 0xba2f
rip = 0x0040ba30 rsp = 0xd26376b0
Found by: stack scanning
4 test + 0x177f
rip = 0x00401780 rsp = 0xd26376c0
Found by: stack scanning
5 test + 0x177f
rip = 0x00401780 rsp = 0xd26376d0
Found by: stack scanning
6 libc-2.11.2.so + 0x1ec4c
rip = 0xb6138c4d rsp = 0xd26376e0
Found by: stack scanning
7 test + 0x18e6
rip = 0x004018e7 rsp = 0xd2637700
Found by: stack scanning
8 test + 0x177f
rip = 0x00401780 rsp = 0xd2637718
Found by: stack scanning
9 test + 0xba2f
rip = 0x0040ba30 rsp = 0xd2637760
Found by: stack scanning
10 test + 0x177f
rip = 0x00401780 rsp = 0xd2637788
Found by: stack scanning
11 test + 0x17a8
rip = 0x004017a9 rsp = 0xd26377a0
Found by: stack scanning

Loaded modules:
0x00400000 - 0x0040dfff test ??? (main)
0x7f59b611a000 - 0x7f59b6271fff libc-2.11.2.so ???
0x7f59b647b000 - 0x7f59b6490fff libgcc_s.so.1 ???
0x7f59b6691000 - 0x7f59b6710fff libm-2.11.2.so ???
0x7f59b6913000 - 0x7f59b6a08fff libstdc++.so.6.0.13 ???
0x7f59b6c27000 - 0x7f59b6c44fff ld-2.11.2.so ???
0x7fffd27ff000 - 0x7fffd27fffff linux-gate.so ???

[Ted Mielczarek]

Hi,

You're missing a step here, which is that you need to convert the
debugging symbols from the native format to Breakpad's preferred text
format[1]. If you run `make` in src/tools/linux/dump_syms, you'll find
a dump_syms binary there. Running that on your "test" binary will
produce a symbol file as output. In order for minidump_stackwalk to
use symbol files, they need to be in a specific directory structure
(the directory structure comes from Microsoft's symbol server). You'll
note that the first line of the output starts with MODULE. This line
has all the info needed to create the proper directory structure. It
should look like:
MODULE Linux x86_64 <UUID> test

You'll need to create a directory like:
mkdir -p symbols/test/<UUID>
Then put the output of dump_syms in a file in that directory:
dump_syms test > symbols/test/<UUID>/test.sym

If you'd like, you can use the symbolstore.py script that I wrote for
Firefox builds to make this process simpler:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py

It can run dump_syms for you and place the output in the proper
directory structure.

Now you can run minidump_stackwalk and pass it the directory as an
additional argument:
google-breakpad/src/processor/minidump_stackwalk ./
7c2b2bb7-7b15-53ae-7cacc238-759a349d.dmp ./symbols

You should now have function name and source line information. I
suppose all of this should go on the Breakpad wiki somewhere!

-Ted

1. http://code.google.com/p/google-breakpad/wiki/SymbolFiles

[Yubao Liu]

Hi,

Thank you very much, I got it successfully.

I write a little note about the full steps, could you paste it to
google-breakpad's
wiki page?

http://groups.google.com/group/google-breakpad-discuss/web/google-breakpad-howto.txt

Regards,
Yubao Liu

On Nov 8, 8:39 pm, Ted Mielczarek <t...@mielczarek.org> wrote:
> Hi,
>
> You're missing a step here, which is that you need to convert the
> debugging symbols from the native format to Breakpad's preferred text
> format[1]. If you run `make` in src/tools/linux/dump_syms, you'll find
> a dump_syms binary there. Running that on your "test" binary will
> produce a symbol file as output. In order for minidump_stackwalk to
> use symbol files, they need to be in a specific directory structure
> (the directory structure comes from Microsoft's symbol server). You'll
> note that the first line of the output starts with MODULE. This line
> has all the info needed to create the proper directory structure. It
> should look like:
> MODULE Linux x86_64 <UUID> test
>
> You'll need to create a directory like:
> mkdir -p symbols/test/<UUID>
> Then put the output of dump_syms in a file in that directory:
> dump_syms test > symbols/test/<UUID>/test.sym
>
> If you'd like, you can use the symbolstore.py script that I wrote for

> Firefox builds to make this process simpler:http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/t...

[Ted Mielczarek]

On Mon, Nov 8, 2010 at 9:54 PM, Yubao Liu <yuba...@gmail.com> wrote:
> Hi,
>
> Thank you very much, I got it successfully.
>
> I write a little note about the full steps,  could you paste it to
> google-breakpad's
> wiki page?
>
> http://groups.google.com/group/google-breakpad-discuss/web/google-breakpad-howto.txt

I took your steps and wrote up a more detailed wiki page:
http://code.google.com/p/google-breakpad/wiki/LinuxStarterGuide

Hopefully that will help you and others in the future.

Regards,
-Ted

[Yubao Liu]

Hi,

Thank you very much, that wiki page is very informative, so I deleted
the
little note uploaded some days ago.

Regards,
Yubao Liu

On Nov 9, 9:26 pm, Ted Mielczarek <t...@mielczarek.org> wrote:

> On Mon, Nov 8, 2010 at 9:54 PM, Yubao Liu <yubao....@gmail.com> wrote:
> > Hi,
>
> > Thank you very much, I got it successfully.
>
> > I write a little note about the full steps,  could you paste it to
> > google-breakpad's
> > wiki page?
>

> >http://groups.google.com/group/google-breakpad-discuss/web/google-bre...