Examiner the minidump produced by breakpad

[Graham]

Hi,

I was wondering if someone would be able to help me with a problem we
are having with breakpad.

I've created a simple win32 test program to in order to develop an out
of process crash handler for us within in our application. I've
managed to successfully create a minidump from this faulting program
(in this example, it's an access violation which occurs).

However, when I open the minidump in WinDbg, the call stack doesn't
seem to include the actual faulting function (See the stack trace at
the bottom of this message).

In the client process an exception handler is instantiated in WinMain
using the following code



ExceptionHandler exception_handler( temp_path.get_as_unicode(),
NULL,
&on_mini_dump_created,
NULL,
ExceptionHandler::HANDLER_ALL,
MiniDumpNormal,
unicode_pipe_name.c_str(),
NULL );

on_minidump_create simply return false.



In the server process, the crash generation server is instantiated in
WinMain with using the following code.

CrashGenerationServer server( lpCmdLine,
NULL,
NULL,
NULL,
&on_client_dump_request,
NULL,
on_client_exited,
NULL,
true,
&temp_path );

on_client_dump_request call MessageBox to display the path of the dump
file.


The stack I get when I examine the dump file WinDbg is as follows.



0018f538 758d0bdd ntdll!NtWaitForMultipleObjects+0x15
0018f5d4 74ef1a2c KERNELBASE!WaitForMultipleObjectsEx+0x100
0018f61c 74ef4208 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0018f638 00405ffb kernel32!WaitForMultipleObjects+0x18
0018f658 0040624a breakpad_test_app!
google_breakpad::CrashGenerationClient::SignalCrashEventAndWait(void)
+0x4b [d:\ffa_dev\google-breakpad-read-only\src\client\windows
\crash_generation\crash_generation_client.cc @ 325]
0018f664 00404df3 breakpad_test_app!
google_breakpad::CrashGenerationClient::RequestDump(struct
_EXCEPTION_POINTERS * ex_info = 0x004010a0, struct MDRawAssertionInfo
* assert_info = 0x00000000)+0x5a [d:\ffa_dev\google-breakpad-read-only
\src\client\windows\crash_generation\crash_generation_client.cc @ 289]
0018f704 00405846 breakpad_test_app!
google_breakpad::ExceptionHandler::WriteMinidumpWithException(unsigned
long requesting_thread_id = 0x4010a0, struct _EXCEPTION_POINTERS *
exinfo = 0x00000000, struct MDRawAssertionInfo * assertion =
0x0071a590)+0x53 [d:\ffa_dev\google-breakpad-read-only\src\client
\windows\handler\exception_handler.cc @ 723]
0018f720 74f3003f breakpad_test_app!
google_breakpad::ExceptionHandler::HandleException(struct
_EXCEPTION_POINTERS * exinfo = 0x771c21d7)+0x46 [d:\ffa_dev\google-
breakpad-read-only\src\client\windows\handler\exception_handler.cc @
434]
0018f7a8 771c21d7 kernel32!UnhandledExceptionFilter+0x127
0018f7b0 771c20b4 ntdll!__RtlUserThreadStart+0x62
0018f7c4 771c1f59 ntdll!_EH4_CallFilterFunc+0x12
0018f7ec 77196ab9 ntdll!_except_handler4+0x8e
0018f810 77196a8b ntdll!ExecuteHandler2+0x26
0018f834 77196a2d ntdll!ExecuteHandler+0x24
0018f8c0 77160143 ntdll!RtlDispatchException+0x127
0018f8c0 00401578 ntdll!KiUserExceptionDispatcher+0xf
0018fc60 74dc62fa breakpad_test_app!WndProc(struct HWND__ * hWnd =
0x001b0f0e, unsigned int message = 0x111, unsigned int wParam =
0x8003, long lParam = 0n0)+0xb8 [d:\ffa_dev\breakpad_test
\breakpad_test_app\breakpad_test_app.cpp @ 231]
0018fc8c 74dc6d3a user32!InternalCallWinProc+0x23
0018fd04 74dc77c4 user32!UserCallWinProcCheckWow+0x109
0018fd64 74dc7bca user32!DispatchMessageWorker+0x3bc
0018fd74 004012e7 user32!DispatchMessageA+0xf
0018fef8 0040650f breakpad_test_app!WinMain(struct HINSTANCE__ *
hInstance = 0x00400000, struct HINSTANCE__ * hPrevInstance =
0x00000000, char * lpCmdLine = 0x004d8d02 "--- memory read error at
address 0x004d8d02 ---", int nCmdShow = 0n5)+0x237 [d:\ffa_dev
\breakpad_test\breakpad_test_app\breakpad_test_app.cpp @ 129]
0018ff88 74ef339a breakpad_test_app!__tmainCRTStartup(void)+0x140 [f:
\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c @ 589]
0018ff94 77189ed2 kernel32!BaseThreadInitThunk+0xe
0018ffd4 77189ea5 ntdll!__RtlUserThreadStart+0x70
0018ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

Do I need to do anything else to get a meaningful stack trace?

Thanks

--Graham

[Ted Mielczarek]

On Wed, Jul 20, 2011 at 6:43 AM, Graham <gda...@hotmail.com> wrote:
> Hi,
>
> I was wondering if someone would be able to help me with a problem we
> are having with breakpad.
>
> I've created a simple win32 test program to in order to develop an out
> of process crash handler for us within in our application.  I've
> managed to successfully create a minidump from this faulting program
> (in this example, it's an access violation which occurs).
>
> However, when I open the minidump in WinDbg,  the call stack doesn't
> seem to include the actual faulting function  (See the stack trace at
> the bottom of this message).

Using WinDBG, you'll want to type the ".ecxr" command to load the
exception context before asking for a stack trace. By default, WinDBG
will show you the stack from the context where the dump was written,
which is just inside the exception handler thread, as you can see.
(Each thread in the minidump has an associated context which contains
CPU state. There's also an exception record that contains its own
context, capturing the state of the faulting thread at the time of the
exception.)

-Ted

[Graham]

Hi,

Thank you very much for your help.

Regards

--Graham

[pradyumna]

Hello,
I build my project with libbreakpad_cient.a in debug mode (in RHEL
5.5)...and later followed the all steps to produce the symbol files
and then stripped the debug symbols from end state by using objcopy
command...and I made the project to crash.... when stack walker
analyzed the dump, it didn't gave me any human readable output.. Its
just giving following information..

2011-08-04 03:27:23: minidump.cc:3731: INFO: Minidump closing minidump
Operating system: Linux
0.0.0 Linux 2.6.18-194.el5 #1 SMP Tue Mar 16
21:52:39 EDT 2010 x86_64
CPU: x86
GenuineIntel family 6 model 26 stepping 5
2 CPUs

Crash reason: SIGSEGV
Crash address: 0xf0f0a167

Thread 8 (crashed)
0 genexbd.159 + 0x601f
eip = 0xf099601f esp = 0xf0b4a000 ebp = 0xf0b4d018 ebx =
0xf0a03741
esi = 0x000000af edi = 0xf0df95b0 eax = 0xf0f0a167 ecx =
0x00000000
edx = 0xf0e51110 efl = 0x00010286
Found by: given as instruction pointer in context
1 genexbd.159 + 0x4c64
eip = 0xf0994c65 esp = 0xf0b4d020 ebp = 0xf0b4d048
Found by: previous frame's frame pointer
2 mdi_glbrtl.276 + 0x6434
eip = 0xf0d6f435 esp = 0xf0b4d050 ebp = 0xf0b4d0a8
Found by: previous frame's frame pointer
3 oen_srvcom.268 + 0x35511
eip = 0xf44ab512 esp = 0xf0b4d0b0 ebp = 0xf0b4d278
Found by: previous frame's frame pointer
4 oen_srvcom.268 + 0x1dd96
eip = 0xf4493d97 esp = 0xf0b4d280 ebp = 0xf0b4d318
Found by: previous frame's frame pointer
5 oen_srvcom.268 + 0x1b928
eip = 0xf4491929 esp = 0xf0b4d320 ebp = 0xf0b4d3a8
Found by: previous frame's frame pointer
6 libpthread-2.5.so + 0x5831
eip = 0x009bb832 esp = 0xf0b4d3b0 ebp = 0xf0b4d498
Found by: previous frame's frame pointer
7 libc-2.5.so + 0xd1e0d
eip = 0x00924e0e esp = 0xf0b4d4a0 ebp = 0x00000000
Found by: previous frame's frame pointer

Am I missing any thing here...

Regards
Pradyumna

[Ted Mielczarek]

On Thu, Aug 4, 2011 at 4:29 AM, pradyumna <prady...@gmail.com> wrote:
> Hello,
> I build my project with libbreakpad_cient.a in debug mode (in RHEL
> 5.5)...and later followed the all steps to produce the symbol files
> and then stripped the debug symbols from end state by using objcopy
> command...and I made the project to crash.... when  stack walker
> analyzed the dump, it didn't gave me any human readable output.. Its
> just giving following information..

<...>

> Am I missing any thing here...

Hi Pradyumna,

When you produced the symbol files, you should have generated a
directory tree containing all of the symbol files. When you run
minidump_stackwalk, you should pass that directory as the second
parameter, as described here:
http://code.google.com/p/google-breakpad/wiki/LinuxStarterGuide#Processing_the_minidump_to_produce_a_stack_trace

Does your symbol directory have a directory named "genexbd.159"?

Regards,
-Ted

[pradyumna]

Thank you... Ted...

On Aug 4, 4:21 pm, Ted Mielczarek <t...@mielczarek.org> wrote:

> On Thu, Aug 4, 2011 at 4:29 AM, pradyumna <pradyum...@gmail.com> wrote:
> > Hello,
> > I build my project with libbreakpad_cient.a in debug mode (in RHEL
> > 5.5)...and later followed the all steps to produce the symbol files
> > and then stripped the debug symbols from end state by using objcopy
> > command...and I made the project to crash.... when  stack walker
> > analyzed the dump, it didn't gave me any human readable output.. Its
> > just giving following information..
> <...>
> > Am I missing any thing here...
>
> Hi Pradyumna,
>
> When you produced the symbol files, you should have generated a
> directory tree containing all of the symbol files. When you run
> minidump_stackwalk, you should pass that directory as the second

> parameter, as described here:http://code.google.com/p/google-breakpad/wiki/LinuxStarterGuide#Proce...