Case sensitivity in finding .sym files

[Bruce Dawson]

I recently hit a case where some of our Windows crashes were not finding symbols. And yet, and when I looked in our breakpad symbol directory I could see the .sym files, clear as day.

 

The problem became more obvious when I manually ran minidump_stackwalk. It was looking for qtgui4.sym and the actual file was named QtGui4.sym.

 

Oops.

 

These crashes are from Windows machines, where case is often preserved but not always, but we do our analysis on Linux machines. A scan of our breakpad symbol directory showed that about 0.5% of our symbol files had inconsistent case.

 

I'm not sure what the best solution is here. I can fix problems as they crop up but that is expensive because I often won't hear about problems. I can modify minidump_stackwalk to be case insensitive (scan the directory), but I prefer minimizing the changes I make to the breakpad code. I can modify minidump_stackwalk to always look for lowercase files and modify our build steps to ensure that we always produce lowercase files.

 

Any thoughts? I'm sure that other people must be hitting these problems.

 

[Ted Mielczarek]

I can't say that I've seen this in our system, but we get such a large
volume of reports it's possible that it's happened and I just haven't
noticed. We do have a cron job that spits out a list[1] of all unique
DLL/debug ID combinations from each day's worth of Windows crash reports
(I use this to feed a script that fetches symbols from Microsoft's
symbol server), so it probably wouldn't be too hard to see if we get
reports that exhibit this problem.

That being said, canonicalizing all filenames to lowercase in dump_syms
/ SimpleSymbolSupplier doesn't sound like a bad idea, despite the
backwards-compatibility problem.

-Ted

1. https://crash-analysis.mozilla.com/crash_analysis/modulelist/

[Bruce Dawson]

I found some customized code in our version of minidump_stackwalk that tries with all lower-case module names if the .sym file isn't found. So apparently we hit and resolved this problem quite a while ago. Something apparently went wrong with our plan to only use lower-case module names in the .sym files, but that's easily fixed by running a script to lower-case all of the .sym files.

 

I'm still not sure where the mixed-case .sym files came from, but I'll monitor and try to figure that out.

[Bruce Dawson]

BTW, what our version of minidump_stackwalk does is try to find the .sym file with the symbol name as given, and if that doesn't work then try again with an all lower-case version. Thus this avoids any backwards-compatibility problems.

 

I also tweaked our symbol publishing scripts to make sure that they consistently use lowercase, and ran another script on the symbol store to lowercase the existing files, so now I know that everything works.

--
You received this message because you are subscribed to the Google Groups "google-breakpad-discuss" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-breakpad-discuss/-/iFbQUA3txIMJ.

To post to this group, send email to google-brea...@googlegroups.com.
To unsubscribe from this group, send email to google-breakpad-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-breakpad-discuss?hl=en.