Crashes happening during tests just became easier to look at

[Florian Quèze]

Until a couple days ago, when a test crashed, in the treeherder Failure Summary, you saw a line like:
 PROCESS-CRASH | application crashed [@ js::ctypes::ConvertToJS] | testing/xpcshell/test/unit/test_crash.js
To see the stack of the crash, you had to open the log viewer, and see something like: https://treeherder.mozilla.org/logviewer?job_id=530780616&repo=try&task=JBda2XLCQH2vsdO60YM_XQ.0&lineNumber=10476

Thanks to a few recent changes, a similar crash now looks like:
 PROCESS-CRASH | 5df45dee-6eae-5377-8cd4-120136139a6c | application crashed [@ js::ctypes::ConvertToJS] | testing/xpcshell/test/unit/test_crash.js
with the minidump id linking to a crash viewer where you can see the stacks: https://fqueze.github.io/aretestsfastyet/crash-viewer.html?url=https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FRYg0tmzHQia_MtrU9Hqyeg%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2F5df45dee-6eae-5377-8cd4-120136139a6c.json

The links to the crash viewer are also available from the log viewer: https://treeherder.mozilla.org/logviewer?job_id=532374171&repo=try&task=RYg0tmzHQia_MtrU9Hqyeg.0&lineNumber=4288

As you might know, for xpcshell tests, we only report failures for tests that failed twice during the same job (a first time when running the tests in parallel, and a second time when re-running sequentially the tests that failed in parallel). This means lots of crashes occur during xpcshell test jobs without ever being visible on Treeherder. If you are interested in seeing these crashes that didn't turn any job orange, you can have a look at them from this dashboard:
https://fqueze.github.io/aretestsfastyet/crashes.html

--

Florian Quèze

[Emilio Cobos Álvarez]

I didn't know I needed this, but it looks really nice, thanks Florian!

I wonder if this also improves the situation of crash stacks on android
(where the crash stack is usually in the logcat output rather than the
log), inspecting those crashes now also a one-click operation? If so
that'd be awesome.

Thanks!

-- Emilio

On 10/29/25 5:53 PM, Florian Quèze wrote:
> Until a couple days ago, when a test crashed, in the treeherder Failure
> Summary, you saw a line like:
>  PROCESS-CRASH | application crashed [@ js::ctypes::ConvertToJS] |
> testing/xpcshell/test/unit/test_crash.js
> To see the stack of the crash, you had to open the log viewer, and see
> something like: https://treeherder.mozilla.org/logviewer?
> job_id=530780616&repo=try&task=JBda2XLCQH2vsdO60YM_XQ.0&lineNumber=10476

> <https://treeherder.mozilla.org/logviewer?
> job_id=530780616&repo=try&task=JBda2XLCQH2vsdO60YM_XQ.0&lineNumber=10476>

>
> Thanks to a few recent changes, a similar crash now looks like:
>  PROCESS-CRASH | 5df45dee-6eae-5377-8cd4-120136139a6c | application
> crashed [@ js::ctypes::ConvertToJS] | testing/xpcshell/test/unit/
> test_crash.js
> with the minidump id linking to a crash viewer where you can see the
> stacks: https://fqueze.github.io/aretestsfastyet/crash-viewer.html?
> url=https%3A%2F%2Ffirefox-ci-

> tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FRYg0tmzHQia_MtrU9Hqyeg%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2F5df45dee-6eae-5377-8cd4-120136139a6c.json <https://fqueze.github.io/aretestsfastyet/crash-viewer.html?url=https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FRYg0tmzHQia_MtrU9Hqyeg%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2F5df45dee-6eae-5377-8cd4-120136139a6c.json>

>
> The links to the crash viewer are also available from the log viewer:
> https://treeherder.mozilla.org/logviewer?
> job_id=532374171&repo=try&task=RYg0tmzHQia_MtrU9Hqyeg.0&lineNumber=4288

> <https://treeherder.mozilla.org/logviewer?
> job_id=532374171&repo=try&task=RYg0tmzHQia_MtrU9Hqyeg.0&lineNumber=4288>

>
> As you might know, for xpcshell tests, we only report failures for tests
> that failed twice during the same job (a first time when running the
> tests in parallel, and a second time when re-running sequentially the
> tests that failed in parallel). This means lots of crashes occur during
> xpcshell test jobs without ever being visible on Treeherder. If you are
> interested in seeing these crashes that didn't turn any job orange, you
> can have a look at them from this dashboard:

> https://fqueze.github.io/aretestsfastyet/crashes.html <https://
> fqueze.github.io/aretestsfastyet/crashes.html>
>
> --
> Florian Quèze
>
> --
> You received this message because you are subscribed to the Google
> Groups "firef...@mozilla.org" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to firefox-dev...@mozilla.org <mailto:firefox-
> dev+uns...@mozilla.org>.
> To view this discussion visit https://groups.google.com/a/mozilla.org/d/
> msgid/firefox-dev/
> CAFu36SprZV1BLT3FPxzMPHBZCUeirYwuQzZBX7JHw%2B0W1K7axA%40mail.gmail.com
> <https://groups.google.com/a/mozilla.org/d/msgid/firefox-dev/
> CAFu36SprZV1BLT3FPxzMPHBZCUeirYwuQzZBX7JHw%2B0W1K7axA%40mail.gmail.com?
> utm_medium=email&utm_source=footer>.

[Florian Quèze]

On Wed, Oct 29, 2025 at 9:05 PM Emilio Cobos Álvarez <emi...@mozilla.com> wrote:

 

I wonder if this also improves the situation of crash stacks on android
(where the crash stack is usually in the logcat output rather than the
log)

It works by uploading the structured output of minidump-stackwalk as an artifact. Before my changes, we dismissed that JSON file after logging most of its content as plain text in the log. My crash viewer is just a way to show nicely the content of these JSON files that are now uploaded. So if you didn't have any relevant data in the job's log, it's unlikely that my changes improved the situation.

--

Florian Quèze