Newbie: built Firefox, tests fail. What now?

[Maurice Asimov]

Hey,

I checked out the source from mozilla-central, built with debug symbols, tried to run some tests (make check and make xpcshell-tests). There were errors -- while the build is all green on TBPL (for my OS, which happens to be Lion).

What now? And is it possible to get a compact summary of tests that failed and a possibility to rerun the failed tests? A ** make: error 2 ** is kind of cryptic, and scrolling through endless lines of console output to find the tests that failed seems .. well, seems like I'm doing something wrong.

Could I get a push in the right direction?

[Gregory Szorc]

I'd recommend using mach to run tests since the user experience is
superior to make (e.g. you just type paths to tests instead of setting
environment variables).

$ ./mach mochitest-browser browser/base/content/test

# Wait for some failures.

$ ./mach mochitest-browser --rerun-failures


While mach doesn't currently have the same option for xpcshell tests, it
does write out <objdir>/.mozbuild/xpcshell.xunit.xml which contains a
machine readable version of the test execution summary, including failed
tests and their output. Bug 729098 tracks hooking that up intelligently.
That should be a relatively easy change to implement.

[Maurice Asimov]

I didn't know about mach, thanks, this will help a lot.

But the question is still open:

If the tests are all green on tbpl, while I get test failures for my local build (same rev) -- what should I do? Obviously something's amiss, can I help in any way?

[Gregory Szorc]

If you can make your test pass locally while still having it green on
TBPL, you should try submitting the patch and see what happens. It's
quite possible you found a legitimate bug or incompatibility with a
platform we don't fully support!

> tests, it does write out <objdir>/.mozbuild/xpcshell.__xunit.xml