Shorter xpcshell test timeouts

[Florian Quèze]

tl;dr: xpcshell tests now timeout by default in 30s instead of 5 minutes and you might need to add requesttimeoutfactor annotations to new long running tests.

Even though most xpcshell tests run pretty quickly (the median run time is 1.1s on Linux opt), the default timeout in the xpcshell test harness was 5 minutes. This meant that a test timing out could delay the entire test job significantly (up to 10 minutes of the worker remaining idle, as we run failing tests a second time).

I recently changed the default timeout value to 30s, which is still plenty for most tests. For tests that consistently need longer, the requesttimeoutfactor annotation can be added in the test manifest to multiply that default timeout for a specific test. 

You can find documentation about the timeout factor annotations at:

https://firefox-source-docs.mozilla.org/testing/xpcshell/index.html#:~:text=requesttimeoutfactor

To minimize the number of tests that needed annotations, I looked at the runtime differences between platforms, and added per-platform timeout factors in the taskcluster test job definitions: https://searchfox.org/firefox-main/rev/48f8a74041be2973d7df8ded3ad5e64c2c00e8dd/taskcluster/kinds/test/xpcshell.yml#123-137

The run time information used to decide which tests typically take longer was taken from the marker durations in resource usage profiles of test jobs (on treeherder when a test job is selected, press 'g' to open the resource use profile). I have a work-in-progress dashboard that presents this aggregated data at https://fqueze.github.io/aretestsfastyet/xpcshell-timings.html While this is still prototype-quality, feel free to have a look if you would like to see how your tests perform.

If you are curious, the work on timeouts happened in https://bugzilla.mozilla.org/show_bug.cgi?id=1995058, and as a bonus, now that we have all long running tests annotated, the test harness can use these annotations to start slow tests early so we don't wait for them at the end of the parallel phase of test jobs (https://bugzilla.mozilla.org/show_bug.cgi?id=1995627).

--

Florian Quèze