Shutdown hangs are very common

[Vladan D]

Background: Firefox shutdown hangs are turned into shutdown crashes by a watchdog thread [1] that forces a crash if shutdown hasn't completed within 1 minute. Thanks to the watchdog and the Windows profile unlocker [2], shutdown hangs aren't as frustrating as they used to be. However, shutdown hangs might still be causing data loss and they are indicative of potentially-serious bugs in the code.


According to this graph of Firefox crash rate history, shutdown hangs (crashes) make up about one third of all browser crashes [3]:

https://crash-analysis.mozilla.com/rkaiser/crash-report-tools/longtermgraph/?fxrel

I've been told shutdown hangs often don't get enough attention.
Should fixing shutdown hangs be higher priority?
And if so, should we allow features with shutdown hangs to be released?


Notes:
1. Force Firefox crash if shutdown hangs https://bugzilla.mozilla.org/show_bug.cgi?id=1038342
2. win32 implementation of nsIProfileUnlocker https://bugzilla.mozilla.org/show_bug.cgi?id=286355
3. The graph above shows that the overall crash rate jumped up by roughly a third when the watchdog code shipped in Firefox 36. Hover over the "36" box on the blue line

[Ryan VanderMeulen]

Windows mochitest-bc shutdown hangs have been on of the top oranges in
our automation for months now. See bug 1121145. Would be great if we
could get more eyes on the problem.

-Ryan

[Mike Conley]

>> Should fixing shutdown hangs be higher priority?

Starting and shutting down properly (and quickly), IMO, fall into the
"core quality" / "no compromise" / "we have to get this right" bucket.

So making it higher priority sounds aligned with the Three Pillars.

>> And if so, should we allow features with shutdown hangs to be released?

Probably depends on the feature and how frequent / severe the shutdown
hangs are with it. Not sure we can make a blanket judgement right here
and now about it, unless we want to go hard-line on this.

[Vladan D]

KaiRo pointed out another reason to reduce shutdown hang rates on IRC: it's lousy UX. The crash-reporter dialog pops up a minute after the user closed Firefox

[Kyle Huey]

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

The last five logs in that bug are all hanging in QuotaClient code. I'll
take a look.

- Kyle

[David Rajchenbach-Teller]

On 06/07/15 22:48, Vladan D wrote:
> KaiRo pointed out another reason to reduce shutdown hang rates on IRC: it's lousy UX. The crash-reporter dialog pops up a minute after the user closed Firefox

That's bug 1137941, but how would that reduce shutdown hangs?


--
David Rajchenbach-Teller, PhD
Performance Team, Mozilla

[Vladan D]

On Monday, July 6, 2015 at 5:00:36 PM UTC-4, David Rajchenbach-Teller wrote:
> On 06/07/15 22:48, Vladan D wrote:
> > KaiRo pointed out another reason to reduce shutdown hang rates on IRC: it's lousy UX. The crash-reporter dialog pops up a minute after the user closed Firefox
>
>
> That's bug 1137941, but how would that reduce shutdown hangs?

I think you misunderstood, I wasn't talking about the reducing the watchdog timeout :)
I was saying that shutdown hangs result in bad Crash Reporter UX, so that is one more reason to *fix* shutdown hangs, i.e. reduce the shutdown hang rate by fixing the causes of hangs

[David Major]

> Should fixing shutdown hangs be higher priority?
> And if so, should we allow features with shutdown hangs to be released?

I admit to skimming over shutdown-hang signatures when looking at
topcrash lists. In my experience they're often longstanding issues that
are difficult to diagnose and rarely end in a successful fix. Given
infinite resources, we should absolutely work on these hangs. In
practice though, we always have more bugs than time, so I tend to focus
my efforts on other topcrashes that we have a better chance of fixing.

A possible exception is for new, never-before-seen shutdown hangs. Those
are likely to be more actionable since we often have a clear regression
range. I think it would be fair to consider those a blocker for the
associated feature.

[David Rajchenbach-Teller]

Ah, right.

On 06/07/15 23:05, Vladan D wrote:
> I think you misunderstood, I wasn't talking about the reducing the watchdog timeout :)
> I was saying that shutdown hangs result in bad Crash Reporter UX, so that is one more reason to *fix* shutdown hangs, i.e. reduce the shutdown hang rate by fixing the causes of hangs
>

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Karl Tomlinson]

Vladan D. writes:

> Should fixing shutdown hangs be higher priority?

_exit() after profile-before-change notification would be the
many-holes-with-one-plug bug to prioritize.

https://wiki.mozilla.org/XPCOM_Shutdown
https://bugzilla.mozilla.org/show_bug.cgi?id=662444

[Kyle Huey]

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

I think most of our hangs are before that. The one RyanVM pointed out
certainly is.

- Kyle

[jmat...@mozilla.com]

Bug 1160459 was filed on the QuotaClient problem, it's a parent of the test failure bug. There's some discussion in there about the problem which you were involved in fyi.

[Georg Fritzsche]

Can we fix the UX?
Presumably we will never have zero shutdown hangs and there may be
different/better ways to prompt the user about it.

[David Rajchenbach-Teller]

On 07/07/15 21:30, Georg Fritzsche wrote:
> Can we fix the UX?
> Presumably we will never have zero shutdown hangs and there may be
> different/better ways to prompt the user about it.

We have had a discussion at Whistler about fixing the UX of crashes (bug
1177121, iirc).

To sum up: we can entirely get rid of the user-visible crash reporter
for shutdown hangs/crashes.

Cheers,
David