Stack MD5 hashes

[JC]

Hi!

We use CrashRpt in our software suite, and I must say a very big
thanks for the super useful tool! We also use the crprober to
automatically process the crash reports and add/append to our issue
tracking system, depending on whether we detect duplicates or not.

I have a question:
We were wondering how the MD5 hash of the crash is calculated? What I
want to get at is, we have observed the following two scenarios:
1. Stack traces from different applications with different stacks
produce the same MD5 sum.
2. Stack traces from the same application with the same stack produce
different MD5 sums.

I think (suspect, rather) that in the both cases, a large part of the
problem is that the stack trace is 'incomplete' because quite a bit of
it is in MS libraries like ntdll.dll and kernel32.dll etc. We have
this cached on the machine where crprober is run, but if you look at
the folder structure, it is not feasible to specify all the folders on
the command line (207 folders at the moment, with one or more
subfolders each).

I was wondering if there is a way to tell crprober about Microsoft
Symbols? Or should it be extended by you guys? I'm guessing you have
already looked into this, and found it infeasible to implement?

I would appreciate some feedback, maybe you have had similar requests/
incidents and have an alternative solution. It is not a huge problem,
AFAIK scenario 1 has only occurred once (maybe twice), scenario 2 is a
bit more prevalent (but I guess this is to be expected).

[zexspectrum]

Hi, the stack trace's MD5 sum is generated by concatenating all meaningful stack frames and calculating MD5 hash of that string.

For example, assume you have this stack:

1. kernel32.dll!0x260234a43

2. user32.dll!0x567345234

3. somemodule.exe!function1()+0x50f [D:\somefile.cpp:500]

4. ntdll.dll!0xff4341235

5. anothermodule.exe!function1()+0x130 [D:\anotherfile.cpp:1] 

crprober.exe will take stack frames ## 3 and 5, because they contain full symbol information, not only the program counter address. So, in theory, it is possible that different stacks produce similar MD5 sums. 

There is a way to tell crprober.exe about Microsoft symbols. You can acomplish this through a symbol server. For example, see this http://msdn.microsoft.com/en-us/windows/hardware/gg462988

[JC]

Thanks for the reply.

> crprober.exe will take stack frames ## 3 and 5, because they contain full
> symbol information, not only the program counter address. So, in theory, it
> is possible that different stacks produce similar MD5 sums.

OK thanks for this information. So I guess the 'solution' would be to
supply as much symbol information as possible.

> There is a way to tell crprober.exe about Microsoft symbols. You can
> acomplish this through a symbol server.

I have read this, and I have tried as follows:

crprober.exe /f C:\CrashReports
\068de2f7-659d-4569-8135-4a402c8dcdcd.zip /o "" /sym "C:
\CrashReports;C:\Path\To\EXE\symbols;SRV*C:\Development
\MS_Symbols*http://msdl.microsoft.com/download/symbols"

but it seems this does not load symbol information for MS symbols.

From crprober:

== Stack trace for thread 0xbd8 ==

Frame
ntdll.dll!0x77ed065a
ntdll.dll!0x77ed54f1
[Frames below may be incorrect and/or missing]
0xb8
0xb8
0x82d498
0x82d590
0x82d508
kernel32.dll!0x77cf57dd
0x82d5b0
0x82d550

but from VS2010:
3032 0 Main Thread Main Thread NtRequestWaitReplyPort Normal

ntdll.dll!NtRequestWaitReplyPort() + 0xa bytes
ntdll.dll!CsrClientCallServer() + 0x61 bytes
kernel32.dll!CsrBasepCreateActCtx() + 0xbd bytes
kernel32.dll!BasepCreateActCtx() - 0x213 bytes
kernel32.dll!CreateActCtxW() + 0x50d bytes
wininet.dll!00000000630338ac()
[Frames below may be incorrect and/or missing, no symbols loaded for
wininet.dll]

So I think I'm doing something wrong. I'll tinker a bit more and let
you know.

[JC]

> So I think I'm doing something wrong. I'll tinker a bit more and let you know.

All my tinkering thus far have been fruitless. Could you maybe point
me in the right direction?

[zexspectrum]

 You can try something of these:

http://www.codeproject.com/Articles/14148/Debugging-Symbols 

http://www.codeproject.com/KB/debug/symbolserver.aspx 

[JC]

May I ask you, if you specify something like this:

bin\crprober.exe /f crash_report.zip /o "" /sym SRV*C:\Development
\MS_Symbols*http://msdl.microsoft.com/download/symbols

does it load the MS symbol information? I have now gone through the
source code, seems like it _should_ be handled by
CMiniDumpReader::Open:148 SymInitializeW, but I cannot get it to load
the MS symbol information, even though everything is available on my
system. Am I misunderstanding something?

[zexspectrum]

I've not ever used the symbol server, so can't advice you. I'll add an issue to the issue tracking, saying something like "Improve the documentation - write about using symbol server with crprober"

[JC]

OK, I have now finally figured out how to use it :)

Instead of specifying simply "SRV*C:\Development\MS_Symbols*http://msdl.microsoft.com/download/symbols", you have to use a 'two-pronged' approach:

First, download the symbols with symchk, like so:

"C:\Program Files\Debugging Tools for Windows (x64)\symchk" /r c:\windows\system32 /s SRV*C:\Development\MS_Symbols\MicrosoftPublicSymbols*http://msdl.microsoft.com/download/symbols

Then, for crprober, specify only the cache path:

crprober.exe /sym C:\Development\MS_Symbols;extra_paths other_options

No need to also include the 'MicrosoftPublicSymbols' part, as SymInitialize searches the folder recursively. The MicrosoftPublicSymbols path was put there by Visual Studio -- so if one includes this, then it is also re-used by VS!