Crash stacks from breakpad are often much worse with VC2010 builds

[Benjamin Smedberg]

Just a note: the crash stacks being reported for VC2010 often have
missing or incorrect frames, and in general are significantly worse than
the stacks from VC2005. This is because the VC2010 linker is producing
PDB files which contain stack-walking program instructions which are
unknown to the breakpad processor.

See for instance
https://crash-stats.mozilla.com/report/index/492d83a5-dbe8-4c81-b6e7-5c4342120203
where frames 6, 8-9, 12-17, 19, 21-22 are bogus and we stop walking the
stack at 23 well before we should. This seems to be a common theme
across many crash reports.

I'm currently looking into this in the google-breakpad mailing lists,
but the stackwalking programs are not documented by Microsoft and were
originally reverse-engineered, so we may have to spend some time reverse
engineering the new instructions as well.
http://groups.google.com/group/google-breakpad-dev/browse_thread/thread/64fd4503d0752633

In the meantime, I am not certain whether the current experience is bad
enough that we should consider switching back to VC2005 until it is
resolved. I'm happy to hear debate on this issue!

--BDS

[Justin Wood (Callek)]

My theory is, if we don't anticipate getting this stack unwind fixed in
the next 4 weeks max, we should backout this from trunk now and re-land
MSVC2010 for Firefox 14. Preferrably fixed within the next 2 at least.

Being unable to (easily) check crash stats properly for almost 6 weeks
of time in a given train is, imo, a major problem even if thats just
trunk. It does mean that we won't be able to properly see an explosive
trunk crash for many weeks after it hurts.

--
~Justin Wood (Callek)

[Kyle Huey]

On Sat, Feb 4, 2012 at 6:30 PM, Benjamin Smedberg <benj...@smedbergs.us>wrote:

> In the meantime, I am not certain whether the current experience is bad
> enough that we should consider switching back to VC2005 until it is
> resolved. I'm happy to hear debate on this issue!
>

Does this normally affect the first few frames? If Socorro is still
aggregating crashes properly, and we can download the minidumps to see
what's really going on, I think we can live with it for a bit.

I'm a bit biased here, of course ;-)

- Kyle

[Benjamin Smedberg]

This does not appear to affect the first few frames much. This is partly
because most of our append/skiplist frames are system libraries or not
in libxul so that are not as aggressively optimized with MSVC2010.
However, I'm not sure that downloading minidumps is a good medium-term
solution, since it requires special security access and make our
volunteer crash-triage community much less able to help.

--BDS