mochitest logging changes

Showing 1-4 of 4 messages
mochitest logging changes Nathan Froyd 2/27/14 7:00 AM
Hi,

Mochitest (M1, M2, M3, M4, M5, and M-oth on TBPL) logging has seen several changes over the past couple of weeks.  The intent of these changes is to reduce the runtime of the tests, particularly for tests which produce many messages, and reduce the log file sizes.  The current status on inbound, post-bug 977198 landing, is:

- Tests with no failures (all TEST-PASS or TEST-KNOWN-FAIL) produce only TEST-START and TEST-END messages.  This is similar to what xpcshell has been doing for some time now.

- If there is a TEST-FAIL or a TEST-UNEXPECTED-PASS, then the failure itself is logged, along with some number of messages (currently capped at 100) since the start of the test or the last failure.  The intent here was to provide most of the benefits of suppressing messages while still showing some record of what the test was doing prior to the failure.

- If you run a single test locally, then all the messages produced by the test will be shown.

(Please note that these rules do not apply to browser-chrome and metro tests, due to the different logging framework they use.  Hooray for consistency.)

If you require more context for your TEST-FAILs, then you may call SimpleTest.requestCompleteLog() in your test and all messages after the call will be displayed.  Please treat this as a debugging aid for Try or for intermittent oranges; it shouldn't be necessary to have SimpleTest.requestCompleteLog() calls lying around in our test suite.

For those interested in the nuts and bolts of the changes, you can refer to https://bugzilla.mozilla.org/show_bug.cgi?id=937181 and its dependent/blocking bugs.

Thanks,
-Nathan
Re: mochitest logging changes Boris Zbarsky 2/27/14 8:30 AM
On 2/27/14 10:00 AM, Nathan Froyd wrote:
> - If there is a TEST-FAIL or a TEST-UNEXPECTED-PASS, then the failure itself is logged, along with some number of messages (currently capped at 100) since the start of the test or the last failure.

Does that include test timeouts?

Do I read this right that this will show the first 100 messages from the
test, not the last 100 messages before the failure?  Or am I
misunderstanding?

-Boris
Re: mochitest logging changes Benjamin Smedberg 2/27/14 1:00 PM
On 2/27/2014 10:00 AM, Nathan Froyd wrote:
> - If there is a TEST-FAIL or a TEST-UNEXPECTED-PASS, then the failure itself is logged, along with some number of messages (currently capped at 100) since the start of the test or the last failure.  The intent here was to provide most of the benefits of suppressing messages while still showing some record of what the test was doing prior to the failure.
Is there a reason why it's not every message since the start of the
current test or the 100 most recent, whichever is greater? I imagine
that seeing the entire output of the single test that failed might be
useful in many circumstances.

>
> - If you run a single test locally, then all the messages produced by the test will be shown.
Is there a mach flag which can force full logging of an entire run
whether it's a single test or an entire directory/manifest?

--BDS

Re: mochitest logging changes Nathan Froyd 2/27/14 2:04 PM
----- Original Message -----
> On 2/27/2014 10:00 AM, Nathan Froyd wrote:
> > - If there is a TEST-FAIL or a TEST-UNEXPECTED-PASS, then the failure
> > itself is logged, along with some number of messages (currently capped at
> > 100) since the start of the test or the last failure.  The intent here was
> > to provide most of the benefits of suppressing messages while still
> > showing some record of what the test was doing prior to the failure.
> Is there a reason why it's not every message since the start of the
> current test or the 100 most recent, whichever is greater? I imagine
> that seeing the entire output of the single test that failed might be
> useful in many circumstances.

Sorry, my explanation was bogus.  The intent and implementation is that you see the last 100 messages produced by the test.  Of course it is possible that you would see fewer if your test hasn't produced that many messages yet.

> > - If you run a single test locally, then all the messages produced by the
> > test will be shown.
> Is there a mach flag which can force full logging of an entire run
> whether it's a single test or an entire directory/manifest?

No.  Bikeshedding on the proper option spelling (I believe Honza has weighed in that --verbose would be sufficient) welcome in bug 969437.

-Nathan