Standardized assertion methods

151 views
Skip to first unread message

Mike de Boer

unread,
Jun 2, 2014, 6:37:13 AM6/2/14
to Mozilla dev-platform mailing list mailing list
Dear unit test writers,

As a happy few of you might already know, we introduced a standalone, versatile class of assertion methods with Assert.jsm[1], which implements the CommonJS Unit Testing specification version 1.1[2].

These methods were already available to you in the global `Assert` namespace in XPCShell-test, Mochitest-browser and Mochitest-chrome tests.

Since last Friday[3], each assertion method in Assert.jsm is available in the global scope of a unit test as well.
Now we can say that the ‘old’ XPCShell-test assertion methods are deprecated in favour of the Assert.jsm ones.

Here’s a short table of what this means in practice:

* do_check_eq(a, b) —> equal(a, b)
* do_check_neq(a, b) —> notEqual(a, b)
* do_check_true(expr) —> ok(expr)
* do_check_false(expr) —> ok(!expr)
* do_check_null(expr) —> equal(expr, null)
* do_check_matches(a, b) —> deepEqual(a, b) (undocumented XPCShell-test feature)

Please note that we did NOT replace all occurrences with the Assert.jsm equivalents, we merely marked them as ‘deprecated’ in the docs[4].

We’re planning to do the same for Mochitest-browser tests in bug 1018226.

Now, dear writer and reviewer of unit tests, I’d like to ask if you could take note of this change and apply it to new tests that’ll land in the tree.

Finally, I’d like to thank Gregory Szorc, Ted Mielczarek, Blair McBride and many others who helped getting this change worked out, integrated and so on.

But we’re not done yet! There are more things we can do to make reading, writing and tracing tests easier and more fun!


Have fun,

Mike.


[1] https://developer.mozilla.org/en/docs/Mozilla/JavaScript_code_modules/Assert.jsm
[2] http://wiki.commonjs.org/wiki/Unit_Testing/1.1
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1014482
[4] https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests
[5] https://bugzilla.mozilla.org/show_bug.cgi?id=1018226

Till Schneidereit

unread,
Jun 2, 2014, 6:56:22 AM6/2/14
to Mike de Boer, Mozilla dev-platform mailing list mailing list, JS Internals list
Should we adopt these for SpiderMonkey's test suites, too? Porting tests
between suites isn't something that's done frequently, but there are people
writing tests for more than one suite, and being able to use the same
assertion methods everywhere would be helpful for that.
> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

Neil

unread,
Jun 2, 2014, 7:41:59 AM6/2/14
to
Mike de Boer wrote:

> * do_check_eq(a, b) �> equal(a, b)
>
There's also strictEqual(a, b) for those like me who were wondering.

--
Warning: May contain traces of nuts.

Mike de Boer

unread,
Jun 2, 2014, 7:49:02 AM6/2/14
to Till Schneidereit, Mozilla dev-platform mailing list mailing list, JS Internals list
Yes, that’d be very nice to have! In a perfect world, all the test suites/ runners we have end up using the same assertion methods. This would indeed greatly improve the portability of individual tests.

Thanks for suggesting this, Till!

Ms2ger

unread,
Jun 2, 2014, 8:28:34 AM6/2/14
to
On 06/02/2014 01:49 PM, Mike de Boer wrote:
> Yes, that’d be very nice to have! In a perfect world, all the test
> suites/ runners we have end up using the same assertion methods. This
> would indeed greatly improve the portability of individual tests.

As I said before (but was ignored), the more sensible approach to
achieve that goal would be to adopt the testharness.js methods, which
are also more consistently named, have better behaviour for the
shorter-name methods, and are documented more clearly.

HTH
Ms2ger

Paolo Amadini

unread,
Jun 2, 2014, 11:39:42 AM6/2/14
to Mike de Boer, Mozilla dev-platform mailing list mailing list
On 6/2/2014 11:37 AM, Mike de Boer wrote:
> Since last Friday[3], each assertion method in Assert.jsm is available in the global scope of a unit test as well.
> Now we can say that the �old� XPCShell-test assertion methods are deprecated in favour of the Assert.jsm ones.

I think it's a very good thing that finally someone is doing the work
of moving various different test suites towards unified assertion
methods (regardless of the actual choice of which methods they are)!

> Here�s a short table of what this means in practice:
>
> * do_check_eq(a, b) �> equal(a, b)
> * do_check_neq(a, b) �> notEqual(a, b)
> * do_check_true(expr) �> ok(expr)
> * do_check_false(expr) �> ok(!expr)
> * do_check_null(expr) �> equal(expr, null)
> * do_check_matches(a, b) �> deepEqual(a, b) (undocumented XPCShell-test feature)

Have you considered requiring test cases to use the the "Assert."
namespace explicitly? I would find that style more readable, and also
assertions easier to find when scanning the code. And they're still
shorter than current assertion functions in xpcshell tests.

Especially the less-known "Assert.throws(<function>, <exception>)" seems
to read better than just "throws(<function>, <exception>)".

For example compare this snippet from the referenced bug...

+ equal(aPacket.type, "paused");
+ let env = aPacket.frame.environment;
+ equal(env.type, "function");
+ equal(env.function.name, "banana3");
+ let parent = env.parent;
+ equal(parent.type, "block");
+ ok("banana3" in parent.bindings.variables);
+ parent = parent.parent;
+ equal(parent.type, "function");
+ equal(parent.function.name, "banana2");
+ parent = parent.parent;
+ equal(parent.type, "block");
+ ok("banana2" in parent.bindings.variables);
+ parent = parent.parent;
+ equal(parent.type, "block");
+ ok("banana2" in parent.bindings.variables);
+ parent = parent.parent;
+ equal(parent.type, "function");
+ equal(parent.function.name, "banana");

...with this modified one:

+ Assert.equal(aPacket.type, "paused");
+ let env = aPacket.frame.environment;
+ Assert.equal(env.type, "function");
+ Assert.equal(env.function.name, "banana3");
+ let parent = env.parent;
+ Assert.equal(parent.type, "block");
+ Assert.ok("banana3" in parent.bindings.variables);
+ parent = parent.parent;
+ Assert.equal(parent.type, "function");
+ Assert.equal(parent.function.name, "banana2");
+ parent = parent.parent;
+ Assert.equal(parent.type, "block");
+ Assert.ok("banana2" in parent.bindings.variables);
+ parent = parent.parent;
+ Assert.equal(parent.type, "block");
+ Assert.ok("banana2" in parent.bindings.variables);
+ parent = parent.parent;
+ Assert.equal(parent.type, "function");
+ Assert.equal(parent.function.name, "banana");

Much easier to visually separate what the test is doing from the
intermixed calls that check the intermediate results.

Cheers,
Paolo

Gijs Kruitbosch

unread,
Jun 2, 2014, 11:51:31 AM6/2/14
to Paolo Amadini, Mike de Boer
Warning: pet peeve coming up.

I agree we should separate these things more clearly. I don't think
making people type the same 7 characters repeatedly is a good way to do
that.

I think we should be more liberal with using blank lines instead. It's
too often I see 30-line blocks of code with not a single line of
whitespace separating logical blocks, leaving the (vegetarian or
beef-greedy) reader the tedious tasks of separating out the meatballs
from the metaphorical spaghetti code.

Concretely, IMO in the code you cite there should be a blank line before
each of the 'parent' reassignments.

~ Gijs
> ....with this modified one:

Ehsan Akhgari

unread,
Jun 2, 2014, 11:59:45 AM6/2/14
to Mike de Boer, Mozilla dev-platform mailing list mailing list
mochitest-browser and mochitest-chrome have traditionally used the
SimpleTest APIs. I'm not aware of the benefits that Assert.jsm
provides, but do we really want to diverge these APIs with
mochitest-plain? I'm _pretty_ sure that the answer is no for
mochitest-chrome at least. Can you please help me understand why we're
making this change to mochitest-chrome at least?

Thanks!
Ehsan

On 2014-06-02, 6:37 AM, Mike de Boer wrote:
> Dear unit test writers,
>
> As a happy few of you might already know, we introduced a standalone, versatile class of assertion methods with Assert.jsm[1], which implements the CommonJS Unit Testing specification version 1.1[2].
>
> These methods were already available to you in the global `Assert` namespace in XPCShell-test, Mochitest-browser and Mochitest-chrome tests.
>
> Since last Friday[3], each assertion method in Assert.jsm is available in the global scope of a unit test as well.
> Now we can say that the �old� XPCShell-test assertion methods are deprecated in favour of the Assert.jsm ones.
>
> Here�s a short table of what this means in practice:
>
> * do_check_eq(a, b) �> equal(a, b)
> * do_check_neq(a, b) �> notEqual(a, b)
> * do_check_true(expr) �> ok(expr)
> * do_check_false(expr) �> ok(!expr)
> * do_check_null(expr) �> equal(expr, null)
> * do_check_matches(a, b) �> deepEqual(a, b) (undocumented XPCShell-test feature)
>
> Please note that we did NOT replace all occurrences with the Assert.jsm equivalents, we merely marked them as �deprecated� in the docs[4].
>
> We�re planning to do the same for Mochitest-browser tests in bug 1018226.
>
> Now, dear writer and reviewer of unit tests, I�d like to ask if you could take note of this change and apply it to new tests that�ll land in the tree.
>
> Finally, I�d like to thank Gregory Szorc, Ted Mielczarek, Blair McBride and many others who helped getting this change worked out, integrated and so on.
>
> But we�re not done yet! There are more things we can do to make reading, writing and tracing tests easier and more fun!

Mike de Boer

unread,
Jun 2, 2014, 12:02:19 PM6/2/14
to Paolo Amadini, Mozilla dev-platform mailing list mailing list
On 02 Jun 2014, at 17:39, Paolo Amadini <paolo....@amadzone.org> wrote:
>
> Have you considered requiring test cases to use the the "Assert."
> namespace explicitly? I would find that style more readable, and also
> assertions easier to find when scanning the code. And they're still
> shorter than current assertion functions in xpcshell tests.
>

I agree with you on that point, it makes the assertions easier to notice, so that’s why we’ve kept the `Assert.` namespace alive. In fact, I ‘forgot’ to remove the namespaced versions from https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests!

It’s up to a matter of taste, I guess… Both are possible at the moment.

Paolo Amadini

unread,
Jun 2, 2014, 12:25:03 PM6/2/14
to Gijs Kruitbosch, Mike de Boer
On 6/2/2014 4:51 PM, Gijs Kruitbosch wrote:
> Concretely, IMO in the code you cite there should be a blank line before
> each of the 'parent' reassignments.

I definitely agree, and I would also use the "Assert." prefix to make
the separation between action and check clearer (while if I understand
correctly you'd prefer the shorter option).

Cheers,
Paolo

Paolo Amadini

unread,
Jun 2, 2014, 12:27:37 PM6/2/14
to Ehsan Akhgari, Mike de Boer
On 6/2/2014 4:59 PM, Ehsan Akhgari wrote:
> I'm _pretty_ sure that the answer is no for
> mochitest-chrome at least.

Are we running these tests out-of-tree in other environments?

Paolo

Ehsan Akhgari

unread,
Jun 2, 2014, 2:08:11 PM6/2/14
to Paolo Amadini, dev-pl...@lists.mozilla.org, Mike de Boer
On Mon, Jun 2, 2014 at 12:27 PM, Paolo Amadini <paolo....@amadzone.org>
wrote:
Do you mean by just opening the page in the browser? I don't think that's
a common use case. But why do you ask?

--
Ehsan
<http://ehsanakhgari.org/>

Paolo Amadini

unread,
Jun 2, 2014, 3:34:59 PM6/2/14
to Ehsan Akhgari, Mike de Boer
On 6/2/2014 7:08 PM, Ehsan Akhgari wrote:
>> On 6/2/2014 4:59 PM, Ehsan Akhgari wrote:
>>> I'm _pretty_ sure that the answer is no for
>>> mochitest-chrome at least.
>>
>> Are we running these tests out-of-tree in other environments?
>
> Do you mean by just opening the page in the browser? I don't think that's
> a common use case. But why do you ask?

It seems to me that if we don't have external compatibility needs, we
might as well move mochitests to use a set of assertion methods that is
the same as xpcshell and maybe other test suites.

Having common assertions makes it trivial to reuse snippets, move test
cases between environments when needed, or even share portions of tests
using test-only modules.

Cheers,
Paolo

Boris Zbarsky

unread,
Jun 2, 2014, 4:14:53 PM6/2/14
to
On 6/2/14, 3:34 PM, Paolo Amadini wrote:
> It seems to me that if we don't have external compatibility needs, we
> might as well move mochitests to use a set of assertion methods that is
> the same as xpcshell and maybe other test suites.

Yes, but imho we should be moving xpcshell in the direction of the ok/is
bits that mochitests use instead of moving mochitests in the direction
of longer function names.

-Boris

Ehsan Akhgari

unread,
Jun 2, 2014, 4:47:37 PM6/2/14
to Boris Zbarsky, dev-pl...@lists.mozilla.org
Yes, this was precisely my point.

--
Ehsan
<http://ehsanakhgari.org/>

Gavin Sharp

unread,
Jun 2, 2014, 5:33:37 PM6/2/14
to Ehsan Akhgari, Boris Zbarsky, dev-platform
Do either of you have reasoning for that other than "it looks better
to me"? I personally think consistency trumps any personal preferences
based on length/concision, as long as what we end up with isn't
unreasonably long/verbose.

Gavin

On Mon, Jun 2, 2014 at 4:47 PM, Ehsan Akhgari <ehsan....@gmail.com> wrote:
> On Mon, Jun 2, 2014 at 4:14 PM, Boris Zbarsky <bzba...@mit.edu> wrote:
>
> Yes, this was precisely my point.
>
> --
> Ehsan
> <http://ehsanakhgari.org/>

Ehsan Akhgari

unread,
Jun 2, 2014, 6:42:34 PM6/2/14
to Gavin Sharp, Boris Zbarsky, dev-platform
On 2014-06-02, 5:33 PM, Gavin Sharp wrote:
> Do either of you have reasoning for that other than "it looks better
> to me"? I personally think consistency trumps any personal preferences
> based on length/concision, as long as what we end up with isn't
> unreasonably long/verbose.

I have two reasons:

1. I believe verbosity for test assertion functions is a bad thing.
is/ok are easier to type/read than the Assert.jsm types. I assume that
people familiar with CommonJS (which Assert.jsm seems to be based on)
would perhaps be trained to ignore the verbosity of those functions, but
people who write mochitest-chrome tests very frequently would probably
not fall into that category.

2. I also value consistency more than my personal preferences, and based
on that, using the existing APIs in some tests and the new APIs in other
tests (even if we agreed that #1 above doesn't matter) is strictly worse
than the status quo.

Also, I'm not sure where the original discussion happened, and what the
reasons are, but if the goal is to be more consistent, it seems like we
need to move xpcshell tests to use the SimpleTest APIs since that seems
to be the only test framework that we have which currently doesn't use
it (ignoring testharness.js of course, and reftests don't really have an
assertion API). I don't think adding a third (fourth considering th.js)
and trying to transition mochitest-browser/chrome to it gradually gets
us close to that goal.

Cheers,
Ehsan

Chris Peterson

unread,
Jun 2, 2014, 7:24:54 PM6/2/14
to
On 6/2/14, 3:42 PM, Ehsan Akhgari wrote:
> 2. I also value consistency more than my personal preferences, and based
> on that, using the existing APIs in some tests and the new APIs in other
> tests (even if we agreed that #1 above doesn't matter) is strictly worse
> than the status quo.

btw, in the mozilla.dev.tech.javascript-engine.internals fork of this
thread, bz and David Bruant pointed out that W3C's testharness and
TC39's test262 each use yet another set of assertion function names. Any
tests we import from those test suites will need glue code to integrate
with our test harness(es).


chris


Martin Thomson

unread,
Jun 2, 2014, 7:26:53 PM6/2/14
to Chris Peterson, dev-pl...@lists.mozilla.org
On 2014-06-02, at 16:24, Chris Peterson <cpet...@mozilla.com> wrote:

> btw, in the mozilla.dev.tech.javascript-engine.internals fork of this thread, bz and David Bruant pointed out that W3C's testharness and TC39's test262 each use yet another set of assertion function names. Any tests we import from those test suites will need glue code to integrate with our test harness(es).

Sounds like an area ripe for standardisation…I mean bike-shedding.

Boris Zbarsky

unread,
Jun 2, 2014, 9:35:41 PM6/2/14
to
On 6/2/14, 5:33 PM, Gavin Sharp wrote:
> Do either of you have reasoning for that other than "it looks better
> to me"?

My personal experience is that when I try to write xpcshell tests the
amount of time it takes to type the test function names is very
noticeable and actively interrupts my thinking about what I actually
want to test... I find it much simpler to write mochitests than
xpcshell tests for this reason.

I'm quite willing to believe this is not the case for everyone else, of
course.

> I personally think consistency trumps any personal preferences
> based on length/concision

Of course given the existence of testharness we're not going to get
consistency in mochitest anyway, even with this change.

> as long as what we end up with isn't unreasonably long/verbose.

I think what xpcshell has now and what testharness says and what's being
proposed (with the "Assert." prefix) are unreasonably long/verbose.

-Boris

Boris Zbarsky

unread,
Jun 2, 2014, 9:44:21 PM6/2/14
to
On 6/2/14, 7:26 PM, Martin Thomson wrote:
> Sounds like an area ripe for standardisation

Note that standardizing several test suites on the same API might not
work all that well when they have different goals and operating
parameters, because what can you end up with is an API that doesn't
quite fit any of them very well...

-Boris

Boris Zbarsky

unread,
Jun 2, 2014, 9:50:39 PM6/2/14
to
On 6/2/14, 9:44 PM, Boris Zbarsky wrote:
> On 6/2/14, 7:26 PM, Martin Thomson wrote:
>> Sounds like an area ripe for standardisation
>
> Note that standardizing several test suites on the same API might not
> work all that well when they have different goals and operating
> parameters

Specifically, testharness and xpcshell are the ones I'm thinking about
here. Mochitest is somewhere in between. The proposed API would work
fine for mochitest and xpcshell, but not for testharness.

-Boris

Dao

unread,
Jun 3, 2014, 2:29:42 AM6/3/14
to
On 03.06.2014 00:42, Ehsan Akhgari wrote:
> Also, I'm not sure where the original discussion happened,

Ditto. For a change that affects pretty much all mozilla-central
hackers, I would have expected a public proposal and a feedback round on
this list.

(My initial reaction was similar to Ehsan's and Boris', i.e. why not
start with the SimpleTest API, maybe make adjustments here and there
where strictly needed, and then have xpcshell adopt that?)

James Graham

unread,
Jun 3, 2014, 4:52:45 AM6/3/14
to dev-pl...@lists.mozilla.org
In fact, for testharness.js tests (and the W3C web-platform-tests in
general) the plan is to have a dedicated test harness (bug 945222). This
is already up and running on tbpl on Cedar and will be turned on for
mozilla-central as soon as the intermittents are under control (Linux is
looking good, Windows has some issues with WebVTT tests, OSX shows a
little more variability).

As a result, in the near future we won't need glue code between
testharness.js tests and other kinds of tests.

FWIW I think the main problem with the CommonJS assertions is their
their semantics. For example:

* Test assertions shouldn't silently type-cast, but ok, equal and
notEqual all do that. Their brevity compared to strictEqual and
notStrictEqual means that they are likely to be much more widely used.

* notStrictEqual and notDeepEqual are terrible names that actively
mislead about what the functions do.

* deepEqual has, as far as I can tell, underspecified semantics. I can't
tell if it is supposed to recurse into nested objects and, if so,
whether there is supposed to be be any loop prevention (the spec talks
about "equivalent values" for every key, without defining what this means).

* throws doesn't seem to provide any way to test that the type or
properties of the thrown object.

I know we have test harnesses using assertion functions that already
have some of these problems, but I don't see the benefit of adding a new
set of methods with exactly the same issues.

Mike de Boer

unread,
Jun 3, 2014, 6:22:17 AM6/3/14
to James Graham, Mozilla dev-platform mailing list mailing list
1. The `Assert.*` namespace is optional and may be omitted. This module is also present in the addon-sdk and used _with_ that namespace, usually with a lowercase `assert.*`. Please pick whatever suits your fancy.

2. testharness.js, Mochitest, XPCShell’s head.js and other suite-runners that we use in-tree are needlessly monolithic. They mix defining assertion methods, test setup, test definition, test teardown in one silo and dogmatically impose a test definition style. Their lack of modularity costs us flexibility in adopting and/ or promoting TDD development. They are, however, great for writing regression tests and that’s what we use them for.

3. Mochitest is a framework we adopted years ago (I don’t know how many, though) and the number of upstream changes/ improvements have been limited to none. In fact, I don’t see Mochitest used by any JS community in the wild anymore and it’s not like I haven’t tried looking. The saddening result, however, is that our status quo is a testing framework that we are reluctant to update or change. It is written in a different coding style than Mozilla’s, which makes it alien enough to present a bar for change.
Consider the boilerplate needed for writing chrome Mochitests: each .xul test file I open redefines an `ok()` method that is a shorthand to `window.opener.wrappedJSObject.SimpleTest.ok` and use _only_ that assertion method to test our code. We need to improve there, I think, and I’m not sure if Mochitest is the way to go here.
We can bikeshed even more about the ‘Simple’ in ‘SimpleTest’, but I’m keeping my cool ;-)

4. None of the test-suites promote modularity and needlessly dictate a reporting style. What I mean by this is that there’s no way to hook different reporting styles in a test runner to promote TDD, for example. What does automation use to detect test failures? TAP[1] is something efficient and optimised for machine-reading, but we parse output lines in a format that is far from an industry standard. We humans delve through a whole bunch of scroll back to find the test case/ assertion we’re interested in. We even rely on our tooling to repeat all the failing tests at the end of a run.

5. Assertion semantics are indeed poorly specified, across the board. Our switch from `do_check_matches()` to `deepEqual()` even revealed a buggy implementation there, which we didn’t know about. Apart from that, it was largely undocumented, not fully covered by unit tests except for the pathological cases. I’m actually a bit scared of what I’ll find in Mochitest[3]
Type coercion is something specifiable, but I’m not sure whether that is something `ok`, `equal` and family should implement guards for. If there’s a wish/ call for more specific assertion methods like `is_truthy`, `is_falsy` and variants for all possible coercion traps, I think there’s room in Assert.jsm to add those.
We are in the sad status quo that all assertion methods in all test suites are underspecified to some degree. The fact that these methods are an integral part of each suite makes it harder to change that. Assert.jsm breaks away from that approach to make these improvements possible to a wider audience. If we agree that more spec’ing is needed, we might as well fork the spec[2] to MDN and collectively work it out.

6. Having various definitions of assertion methods spread out across different suites makes maintaining them a bit harder. I’m specifically thinking about `deepEqual()` here; I was thinking about adding more robust support for Iterables that can be found in ‘new' JS primitives like Set and Map. Wouldn’t it be nice if we could add support for recursively iterating over their values and asserting scalars for equality?

7. Names of assertion methods are an excellent reason for bikeshedding. The main reason for the amount of time it took for the spec[2] to be formalised was exactly this, IIRC. Never mind that, like I said before: I’m fine with forking the spec and adding aliases for each assertion method if need be. I mostly care about the fact that we can implement them in one place.

With a lengthy post like this, it often happens that one steps on ones toe. I apologise for that in advance.

Mike.

[1] http://en.wikipedia.org/wiki/Test_Anything_Protocol
[2] http://wiki.commonjs.org/wiki/Unit_Testing/1.1
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1018226


On 03 Jun 2014, at 10:52, James Graham <ja...@hoppipolla.co.uk> wrote:

> On 03/06/14 00:24, Chris Peterson wrote:
> In fact, for testharness.js tests (and the W3C web-platform-tests in
> general) the plan is to have a dedicated test harness (bug 945222). This
> is already up and running on tbpl on Cedar and will be turned on for
> mozilla-central as soon as the intermittents are under control (Linux is
> looking good, Windows has some issues with WebVTT tests, OSX shows a
> little more variability).
>
> As a result, in the near future we won't need glue code between
> testharness.js tests and other kinds of tests.
>
> FWIW I think the main problem with the CommonJS assertions is their
> their semantics. For example:
>
> * Test assertions shouldn't silently type-cast, but ok, equal and
> notEqual all do that. Their brevity compared to strictEqual and
> notStrictEqual means that they are likely to be much more widely used.
>
> * notStrictEqual and notDeepEqual are terrible names that actively
> mislead about what the functions do.
>
> * deepEqual has, as far as I can tell, underspecified semantics. I can't
> tell if it is supposed to recurse into nested objects and, if so,
> whether there is supposed to be be any loop prevention (the spec talks
> about "equivalent values" for every key, without defining what this means).
>
> * throws doesn't seem to provide any way to test that the type or
> properties of the thrown object.
>
> I know we have test harnesses using assertion functions that already
> have some of these problems, but I don't see the benefit of adding a new
> set of methods with exactly the same issues.

James Graham

unread,
Jun 3, 2014, 6:57:37 AM6/3/14
to dev-pl...@lists.mozilla.org
I'm not sure I grasp your overall point, but I have a few comments.

On 03/06/14 11:22, Mike de Boer wrote:
> 1. The `Assert.*` namespace is optional and may be omitted. This
> module is also present in the addon-sdk and used _with_ that
> namespace, usually with a lowercase `assert.*`. Please pick whatever
> suits your fancy.

FWIW I consider that like other code tests should be optimised for
maintainability. Taking the position "pick whatever" doesn't help with this.

> 2. testharness.js, Mochitest, XPCShell�s head.js and other
> suite-runners that we use in-tree are needlessly monolithic.

It's not needless if you want to run in non-Mozilla environments where
jaascript modules don't yet exist. For platform-level tests, being able
to cross check our implementation against other implementations has
considerable value.

> They mix
> defining assertion methods, test setup, test definition, test
> teardown in one silo and dogmatically impose a test definition style.
> Their lack of modularity costs us flexibility in adopting and/ or
> promoting TDD development. They are, however, great for writing
> regression tests and that�s what we use them for.

I'm not sure I understand this criticism. testharness.js and Mochitest,
at least, are not really intended for writing unit tests, but for
writing functional tests. I have had no problem using testharness.js in
a setup where a comprehensive upfront testsuite was written in parallel
with the code, which seems to be pretty close to a TDD ethos. I don't
think that technical problems are preventing us adopting this
development methodology.

(Maybe for testing frontend code one can use mochitest for unit tests. I
don't know how that works).

> 4. None of the test-suites promote modularity and needlessly dictate
> a reporting style. What I mean by this is that there�s no way to hook
> different reporting styles in a test runner to promote TDD, for
> example. What does automation use to detect test failures? TAP[1] is
> something efficient and optimised for machine-reading, but we parse
> output lines in a format that is far from an industry standard. We
> humans delve through a whole bunch of scroll back to find the test
> case/ assertion we�re interested in. We even rely on our tooling to
> repeat all the failing tests at the end of a run.

Yes, the way we deal with test output has historically been suboptimal.
We are in the process of fixing that as we speak. We have developed a
json-based protocol [1] for test output. This is already being used for
web-platform-tests, and for FirefoxOS certsuite. There is current work
in progress for converting mochitest and marionette tests to this
format. Other suites will follow.

As you say, once we are using structured logging rather than trying to
parse human-readable logging, it should be possible to do a lot more
interesting things with the logging results. The structured logging
package already comes with formatters for a small number of formats
including, for example, something XUnit XML compatible. There are also
lots of ideas for how to improve the user interface to test results in
automation. These will come after the launch of treeherder.

> 5. Assertion semantics are indeed poorly specified, across the board.
> Our switch from `do_check_matches()` to `deepEqual()` even revealed a
> buggy implementation there, which we didn�t know about. Apart from
> that, it was largely undocumented, not fully covered by unit tests
> except for the pathological cases. I�m actually a bit scared of what
> I�ll find in Mochitest[3] Type coercion is something specifiable, but
> I�m not sure whether that is something `ok`, `equal` and family
> should implement guards for. If there�s a wish/ call for more
> specific assertion methods like `is_truthy`, `is_falsy` and variants
> for all possible coercion traps, I think there�s room in Assert.jsm
> to add those. We are in the sad status quo that all assertion methods
> in all test suites are underspecified to some degree. The fact that
> these methods are an integral part of each suite makes it harder to
> change that. Assert.jsm breaks away from that approach to make these
> improvements possible to a wider audience. If we agree that more
> spec�ing is needed, we might as well fork the spec[2] to MDN and
> collectively work it out.

Changing the semantics of things that people are already using seems
like a uphill battle. I think if you wanted to introduce a common set of
assertion names across Mozilla harnesses, starting from CommonJS rather
than starting from a discussion of actual requirements was the wrong
approach.

> 7. Names of assertion methods are an excellent reason for
> bikeshedding. The main reason for the amount of time it took for the
> spec[2] to be formalised was exactly this, IIRC. Never mind that,
> like I said before: I�m fine with forking the spec and adding aliases
> for each assertion method if need be. I mostly care about the fact
> that we can implement them in one place.

At least for testharness.js that would directly conflict with the hard
requirement for cross-browser compatibility. The specific choice of
methods here would also conflict with other lower level goals like
consistency of style and ease of writing good tests.

[1] http://mozbase.readthedocs.org/en/latest/mozlog_structured.html

Mike de Boer

unread,
Jun 3, 2014, 7:27:12 AM6/3/14
to James Graham, Mozilla dev-platform mailing list mailing list
Please see my comments inline.

On 03 Jun 2014, at 12:57, James Graham <ja...@hoppipolla.co.uk> wrote:

> I'm not sure I grasp your overall point, but I have a few comments.
>
> On 03/06/14 11:22, Mike de Boer wrote:
>> 1. The `Assert.*` namespace is optional and may be omitted. This
>> module is also present in the addon-sdk and used _with_ that
>> namespace, usually with a lowercase `assert.*`. Please pick whatever
>> suits your fancy.
>
> FWIW I consider that like other code tests should be optimised for
> maintainability. Taking the position "pick whatever" doesn't help with this.
>
>> 2. testharness.js, Mochitest, XPCShell’s head.js and other
>> suite-runners that we use in-tree are needlessly monolithic.
>
> It's not needless if you want to run in non-Mozilla environments where
> jaascript modules don't yet exist. For platform-level tests, being able
> to cross check our implementation against other implementations has
> considerable value.
>
>> They mix
>> defining assertion methods, test setup, test definition, test
>> teardown in one silo and dogmatically impose a test definition style.
>> Their lack of modularity costs us flexibility in adopting and/ or
>> promoting TDD development. They are, however, great for writing
>> regression tests and that’s what we use them for.
>
> I'm not sure I understand this criticism. testharness.js and Mochitest,
> at least, are not really intended for writing unit tests, but for
> writing functional tests. I have had no problem using testharness.js in
> a setup where a comprehensive upfront testsuite was written in parallel
> with the code, which seems to be pretty close to a TDD ethos. I don't
> think that technical problems are preventing us adopting this
> development methodology.
>
> (Maybe for testing frontend code one can use mochitest for unit tests. I
> don't know how that works).
>
>> 4. None of the test-suites promote modularity and needlessly dictate
>> a reporting style. What I mean by this is that there’s no way to hook
>> different reporting styles in a test runner to promote TDD, for
>> example. What does automation use to detect test failures? TAP[1] is
>> something efficient and optimised for machine-reading, but we parse
>> output lines in a format that is far from an industry standard. We
>> humans delve through a whole bunch of scroll back to find the test
>> case/ assertion we’re interested in. We even rely on our tooling to
>> repeat all the failing tests at the end of a run.
>
> Yes, the way we deal with test output has historically been suboptimal.
> We are in the process of fixing that as we speak. We have developed a
> json-based protocol [1] for test output. This is already being used for
> web-platform-tests, and for FirefoxOS certsuite. There is current work
> in progress for converting mochitest and marionette tests to this
> format. Other suites will follow.

Do you have bug numbers where I can follow that work in progress? It sounds awesome!

>
> As you say, once we are using structured logging rather than trying to
> parse human-readable logging, it should be possible to do a lot more
> interesting things with the logging results. The structured logging
> package already comes with formatters for a small number of formats
> including, for example, something XUnit XML compatible. There are also
> lots of ideas for how to improve the user interface to test results in
> automation. These will come after the launch of treeherder.
>
>> 5. Assertion semantics are indeed poorly specified, across the board.
>> Our switch from `do_check_matches()` to `deepEqual()` even revealed a
>> buggy implementation there, which we didn’t know about. Apart from
>> that, it was largely undocumented, not fully covered by unit tests
>> except for the pathological cases. I’m actually a bit scared of what
>> I’ll find in Mochitest[3] Type coercion is something specifiable, but
>> I’m not sure whether that is something `ok`, `equal` and family
>> should implement guards for. If there’s a wish/ call for more
>> specific assertion methods like `is_truthy`, `is_falsy` and variants
>> for all possible coercion traps, I think there’s room in Assert.jsm
>> to add those. We are in the sad status quo that all assertion methods
>> in all test suites are underspecified to some degree. The fact that
>> these methods are an integral part of each suite makes it harder to
>> change that. Assert.jsm breaks away from that approach to make these
>> improvements possible to a wider audience. If we agree that more
>> spec’ing is needed, we might as well fork the spec[2] to MDN and
>> collectively work it out.
>
> Changing the semantics of things that people are already using seems
> like a uphill battle. I think if you wanted to introduce a common set of
> assertion names across Mozilla harnesses, starting from CommonJS rather
> than starting from a discussion of actual requirements was the wrong
> approach.

That’s not what we’re doing here! Changing semantics is a non-goal, merging assertion methods into one re-usable module is.

Taking the CommonJS spec as an umbrella for these simple assertion methods is a causality and I think it helps provide a common, immediate understanding for new contributors who’d like to write test for the code they contribute.

Sure, the semantics of `do_check_matches()` changed. But that method was only used in two locations, its use not promoted in any wiki page and its implementation lossy.

>
>> 7. Names of assertion methods are an excellent reason for
>> bikeshedding. The main reason for the amount of time it took for the
>> spec[2] to be formalised was exactly this, IIRC. Never mind that,
>> like I said before: I’m fine with forking the spec and adding aliases
>> for each assertion method if need be. I mostly care about the fact
>> that we can implement them in one place.
>
> At least for testharness.js that would directly conflict with the hard
> requirement for cross-browser compatibility. The specific choice of
> methods here would also conflict with other lower level goals like
> consistency of style and ease of writing good tests.
>
> [1] http://mozbase.readthedocs.org/en/latest/mozlog_structured.html

Gijs Kruitbosch

unread,
Jun 3, 2014, 7:49:26 AM6/3/14
to Mike de Boer, James Graham
On 03/06/2014 12:27, Mike de Boer wrote:
> <snip>
>>> 5. Assertion semantics are indeed poorly specified, across the board.
>>> Our switch from `do_check_matches()` to `deepEqual()` even revealed a
>>> buggy implementation there, which we didn�t know about. Apart from
>>> that, it was largely undocumented, not fully covered by unit tests
>>> except for the pathological cases. I�m actually a bit scared of what
>>> I�ll find in Mochitest[3] Type coercion is something specifiable, but
>>> I�m not sure whether that is something `ok`, `equal` and family
>>> should implement guards for. If there�s a wish/ call for more
>>> specific assertion methods like `is_truthy`, `is_falsy` and variants
>>> for all possible coercion traps, I think there�s room in Assert.jsm
>>> to add those. We are in the sad status quo that all assertion methods
>>> in all test suites are underspecified to some degree. The fact that
>>> these methods are an integral part of each suite makes it harder to
>>> change that. Assert.jsm breaks away from that approach to make these
>>> improvements possible to a wider audience. If we agree that more
>>> spec�ing is needed, we might as well fork the spec[2] to MDN and
>>> collectively work it out.
>>
>> Changing the semantics of things that people are already using seems
>> like a uphill battle. I think if you wanted to introduce a common set of
>> assertion names across Mozilla harnesses, starting from CommonJS rather
>> than starting from a discussion of actual requirements was the wrong
>> approach.
>
> That�s not what we�re doing here! Changing semantics is a non-goal, merging assertion methods into one re-usable module is.
>
> Taking the CommonJS spec as an umbrella for these simple assertion methods is a causality

What are you trying to say here? I don't understand. I believe
"causality" is not the word you want here.

~ Gijs

James Graham

unread,
Jun 3, 2014, 7:52:22 AM6/3/14
to dev-pl...@lists.mozilla.org
On 03/06/14 12:27, Mike de Boer wrote:

>>> 4. None of the test-suites promote modularity and needlessly dictate
>>> a reporting style. What I mean by this is that there�s no way to hook
>>> different reporting styles in a test runner to promote TDD, for
>>> example. What does automation use to detect test failures? TAP[1] is
>>> something efficient and optimised for machine-reading, but we parse
>>> output lines in a format that is far from an industry standard. We
>>> humans delve through a whole bunch of scroll back to find the test
>>> case/ assertion we�re interested in. We even rely on our tooling to
>>> repeat all the failing tests at the end of a run.
>>
>> Yes, the way we deal with test output has historically been suboptimal.
>> We are in the process of fixing that as we speak. We have developed a
>> json-based protocol [1] for test output. This is already being used for
>> web-platform-tests, and for FirefoxOS certsuite. There is current work
>> in progress for converting mochitest and marionette tests to this
>> format. Other suites will follow.
>
> Do you have bug numbers where I can follow that work in progress? It sounds awesome!

Yes, sorry I should have included some.

The metabug for structured logging is bug 916295
The Mochitest work is happening in bug 886570 and Marionette in bug 956739.

Treeherder is using pivotal tracker rather than bugzilla, so I don't
have a bug number for that one, but the "story" for basic integration is
[1].

>> Changing the semantics of things that people are already using seems
>> like a uphill battle. I think if you wanted to introduce a common set of
>> assertion names across Mozilla harnesses, starting from CommonJS rather
>> than starting from a discussion of actual requirements was the wrong
>> approach.
>
> That�s not what we�re doing here! Changing semantics is a non-goal, merging assertion methods into one re-usable module is.
>
> Taking the CommonJS spec as an umbrella for these simple assertion methods is a causality and I think it helps provide a common, immediate understanding for new contributors who�d like to write test for the code they contribute.
>
> Sure, the semantics of `do_check_matches()` changed. But that method was only used in two locations, its use not promoted in any wiki page and its implementation lossy.

I was under the impression that you were proposing landing CommonJS
method support and then forking the commonjs spec to improve the
semantics. I may have misunderstood.

I think I would need some evidence to back up the hypothesis that new
contributers are unusually likely to know CommonJS compared to the N
other test frameworks that exist. For example most automation
contributers have a better working knowledge of Python than Javascript
and would be more comfortable with something that looks like Python's
unittest module. I imagine for front end developers there would be
different biases. For platform developers the story is likely to be
different again.

In any case, learning new assertion names isn't something that strikes
me as being a particularly high barrier to entry. With testharness.js
the only complaint I recall about the names is that they favour
explicitness over brevity. Confusion for people moving from other js
test frameworks has more often come from the differences in high-level
philosophy.

[1] https://www.pivotaltracker.com/s/projects/749519/stories/70575430

Mike de Boer

unread,
Jun 3, 2014, 8:39:54 AM6/3/14
to Gijs Kruitbosch, James Graham, Mozilla dev-platform mailing list mailing list
Indeed I meant to say the following:

Taking the CommonJS spec as an umbrella for these simple assertion methods is more of a nice side-effect than it was the primary objective we started off with.
I think it helps provide a common, immediate understanding for new contributors who’d like to write test for the code they contribute, as the add-on SDK and the NodeJS community already use it exclusively.

On 03 Jun 2014, at 13:49, Gijs Kruitbosch <gijskru...@gmail.com> wrote:

> On 03/06/2014 12:27, Mike de Boer wrote:
>> <snip>
>>>> 5. Assertion semantics are indeed poorly specified, across the board.
>>>> Our switch from `do_check_matches()` to `deepEqual()` even revealed a
>>>> buggy implementation there, which we didn’t know about. Apart from
>>>> that, it was largely undocumented, not fully covered by unit tests
>>>> except for the pathological cases. I’m actually a bit scared of what
>>>> I’ll find in Mochitest[3] Type coercion is something specifiable, but
>>>> I’m not sure whether that is something `ok`, `equal` and family
>>>> should implement guards for. If there’s a wish/ call for more
>>>> specific assertion methods like `is_truthy`, `is_falsy` and variants
>>>> for all possible coercion traps, I think there’s room in Assert.jsm
>>>> to add those. We are in the sad status quo that all assertion methods
>>>> in all test suites are underspecified to some degree. The fact that
>>>> these methods are an integral part of each suite makes it harder to
>>>> change that. Assert.jsm breaks away from that approach to make these
>>>> improvements possible to a wider audience. If we agree that more
>>>> spec’ing is needed, we might as well fork the spec[2] to MDN and
>>>> collectively work it out.
>>>
>>> Changing the semantics of things that people are already using seems
>>> like a uphill battle. I think if you wanted to introduce a common set of
>>> assertion names across Mozilla harnesses, starting from CommonJS rather
>>> than starting from a discussion of actual requirements was the wrong
>>> approach.
>>
>> That’s not what we’re doing here! Changing semantics is a non-goal, merging assertion methods into one re-usable module is.

Boris Zbarsky

unread,
Jun 3, 2014, 8:50:57 AM6/3/14
to
On 6/3/14, 6:22 AM, Mike de Boer wrote:
> Their lack of modularity costs us flexibility in adopting and/ or promoting TDD development.

Mike, I'm very curious about this part. Do you have a link offhand to a
more detailed explanation of the issues here?

Note that none of us think Mochitest is perfect by any means. And I
think we all agree (or at least I agree!) that both the behavior an the
names of the old xpcshell test functions is terrible.

I do think we should be very intentional about adopting something new,
both in terms of semantics (mochitest is() using == is a mistake we
should not duplicate in the short-name comparison function in the new
setup) and in terms of naming. That doesn't mean we shouldn't adopt it,
but we should aim to fix the known problems when we do so, because it's
unlikely that we'll get a second chance to change around the names to
address them.

-Boris

Boris Zbarsky

unread,
Jun 3, 2014, 8:54:32 AM6/3/14
to
On 6/3/14, 8:39 AM, Mike de Boer wrote:
> I think it helps provide a common, immediate understanding for new contributors who�d like to write test for the code they contribute, as the add-on SDK and the NodeJS community already use it exclusively.

I think there's a bit of functional area bias here, fwiw. I've seen
very few (if any) Gecko contributors coming from either of those
communities, for example.

-Boris

Boris Zbarsky

unread,
Jun 3, 2014, 9:07:12 AM6/3/14
to
On 6/3/14, 8:50 AM, Boris Zbarsky wrote:
> I do think we should be very intentional about adopting something new,
> both in terms of semantics (mochitest is() using == is a mistake we
> should not duplicate in the short-name comparison function in the new
> setup) y

One other note.

The checkin so far preserved the assertion semantics in xpcshell,
afaict: failure throws and terminates the test.

I assume that the mochitest version will use a different reporter that
doesn't throw-and-terminate, to preserve the current semantics of
mochitest assertions. (If this assumption is incorrect, we need to have
a separate discussion about that.) If so, we'll have the same methods
but different semantics in the different harnesses. Not sure how much
of a problem that is in practice... not least because I don't think
people actually write xpcshell tests all that much.

-Boris

Mike de Boer

unread,
Jun 3, 2014, 9:16:54 AM6/3/14
to Boris Zbarsky, Mozilla dev-platform mailing list mailing list
On 03 Jun 2014, at 14:54, Boris Zbarsky <bzba...@mit.edu> wrote:

> On 6/3/14, 8:39 AM, Mike de Boer wrote:
>> I think it helps provide a common, immediate understanding for new contributors who’d like to write test for the code they contribute, as the add-on SDK and the NodeJS community already use it exclusively.
>
> I think there's a bit of functional area bias here, fwiw. I've seen very few (if any) Gecko contributors coming from either of those communities, for example.

Yes, I’m biased there I think. I hope that will change in the (near) future. Right now, the main area for NodeJS enthusiasts at Mozilla to go to is Services.

<following is slightly off-topic>

I want Firefox frontend development to be more mainstream, or at least be able to move faster and experiment more with JS development trends. (Note that Assert.jsm is not about a trend, it’s quite past that and several years old already).

Indeed, I’m used to the NodeJS/ Mocha flow of writing tests as fast, or even faster, as writing the implementation of a feature. I could group tests, isolate one, hook in a debugger at any point and more. This is something I miss while working on Fx and to be honest: I still crave for more than a year later.

Writing wrappers in python around things to improve the current situation like a band-aid isn’t the way I’m used to fix things; I like to take the bull by the horns[1]

I’d like to ask _why_ structured logging needs to be bolted on top of what we have currently? Is it more work to fix the real problem? Are we less comfortable doing things in JS?

We all want the same thing, I’m pretty sure of that. I’m just asking difficult questions to understand the decision of the direction we’re taking.

Mike.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=867742


>
> -Boris

Mike de Boer

unread,
Jun 3, 2014, 9:21:17 AM6/3/14
to Boris Zbarsky, Mozilla dev-platform mailing list mailing list
On 03 Jun 2014, at 15:07, Boris Zbarsky <bzba...@mit.edu> wrote:

> On 6/3/14, 8:50 AM, Boris Zbarsky wrote:
>> I do think we should be very intentional about adopting something new,
>> both in terms of semantics (mochitest is() using == is a mistake we
>> should not duplicate in the short-name comparison function in the new
>> setup) y
>
> One other note.
>
> The checkin so far preserved the assertion semantics in xpcshell, afaict: failure throws and terminates the test.

Yes, deliberately so.

>
> I assume that the mochitest version will use a different reporter that doesn't throw-and-terminate, to preserve the current semantics of mochitest assertions. (If this assumption is incorrect, we need to have a separate discussion about that.) If so, we'll have the same methods but different semantics in the different harnesses. Not sure how much of a problem that is in practice... not least because I don't think people actually write xpcshell tests all that much.

Ideally, run-until-failure would be a configurable option per test suite. For example, in automation we’d continue on failure and locally we’d stop on failure. Additionally, that’d be configurable per test and per run.

Anyhow, Assert.jsm supports pluggable reporters, specifically meant to support any style. Everything will stay the way it currently is.

>
> -Boris

Mike de Boer

unread,
Jun 3, 2014, 9:29:03 AM6/3/14
to Boris Zbarsky, Mozilla dev-platform mailing list mailing list
On 03 Jun 2014, at 14:50, Boris Zbarsky <bzba...@mit.edu> wrote:

> On 6/3/14, 6:22 AM, Mike de Boer wrote:
>> Their lack of modularity costs us flexibility in adopting and/ or promoting TDD development.
>
> Mike, I'm very curious about this part. Do you have a link offhand to a more detailed explanation of the issues here?

Nope, you got me there - I generalised too easily. This statement is based on personal experience, not science.

>
> Note that none of us think Mochitest is perfect by any means. And I think we all agree (or at least I agree!) that both the behavior an the names of the old xpcshell test functions is terrible.
>
> I do think we should be very intentional about adopting something new, both in terms of semantics (mochitest is() using == is a mistake we should not duplicate in the short-name comparison function in the new setup) and in terms of naming. That doesn't mean we shouldn't adopt it, but we should aim to fix the known problems when we do so, because it's unlikely that we'll get a second chance to change around the names to address them.

I’d say that now that we’ve spliced it all out to a separate module, we’re more free to change things and make amendments.

I don’t think it’s too late to change anything. I agree that `is()` paired with `==` was ill-advised, so now we have `equal()` and `strictEqual()` as well.
To preserve backward compat, all the old function names of XPCShell assertions do still work. I was planning to do the same with Mochi.

Mike.

>
> -Boris

Gijs Kruitbosch

unread,
Jun 3, 2014, 9:39:31 AM6/3/14
to Mike de Boer
On 03/06/2014 14:16, Mike de Boer wrote:
> Indeed, I�m used to the NodeJS/ Mocha flow of writing tests as fast, or even faster, as writing the implementation of a feature. I could group tests, isolate one, hook in a debugger at any point and more. This is something I miss while working on Fx and to be honest: I still crave for more than a year later.


So I'm not used to the "NodeJS/Mocha" 'flow' of writing tests. Can you
explain what the big difference there is? I find it hard to believe the
names of the assertion functions are the one big thing making tests take
much longer to write...

Node is inherently single-threaded (unless you bolt on extra stuff),
which makes the runtime environment a lot more controlled than say, a
web browser. IME, the main issue in writing correct browser mochitest
tests (which is what I write 95% of the time) is focus, timeouts, and
event sequences (ensuring you wait for X before doing Y), OS differences
(and sometimes, try/staging vs. my local machine differences) and making
it easy and obvious (how) to trigger UI actions in a test, and the ease
of debugging (which has gotten better now that we have
--break-on-failure and --jsdebugger).

That all seems to have preciously little to do with assertions, log
output, or TDD per se. In other words, if we want to make the experience
of writing tests easier, I don't think a unified assertion framework is
where we should be focusing our effort.

Can you expand on why you think that that is such an important component
here, and how it compares with mocha/node?

~ Gijs

Mike de Boer

unread,
Jun 3, 2014, 9:45:18 AM6/3/14
to Gijs Kruitbosch, Mozilla dev-platform mailing list mailing list
I understand all that and I *think* you missed the header mentioning I was going off-topic…

Mike.

On 03 Jun 2014, at 15:39, Gijs Kruitbosch <gijskru...@gmail.com> wrote:

> On 03/06/2014 14:16, Mike de Boer wrote:
>> Indeed, I’m used to the NodeJS/ Mocha flow of writing tests as fast, or even faster, as writing the implementation of a feature. I could group tests, isolate one, hook in a debugger at any point and more. This is something I miss while working on Fx and to be honest: I still crave for more than a year later.

Gijs Kruitbosch

unread,
Jun 3, 2014, 9:48:16 AM6/3/14
to Mike de Boer
I don't think this is off-topic. I'm essentially asking: why did you
focus on this, and why in this way? More broadly, I'm asking what it is
you're missing from node/mocha.

If you think that needs its own topic, feel free to fork the summary.

In any case, discussions about the ease of use of our test framework(s)
seem eminently on-topic for m.d.platform.

~ Gijs

On 03/06/2014 14:45, Mike de Boer wrote:
> I understand all that and I *think* you missed the header mentioning I was going off-topic�
>
> Mike.
>
> On 03 Jun 2014, at 15:39, Gijs Kruitbosch <gijskru...@gmail.com> wrote:
>
>> On 03/06/2014 14:16, Mike de Boer wrote:
>>> Indeed, I�m used to the NodeJS/ Mocha flow of writing tests as fast, or even faster, as writing the implementation of a feature. I could group tests, isolate one, hook in a debugger at any point and more. This is something I miss while working on Fx and to be honest: I still crave for more than a year later.

James Graham

unread,
Jun 3, 2014, 10:07:43 AM6/3/14
to dev-pl...@lists.mozilla.org
On 03/06/14 14:16, Mike de Boer wrote:

> Writing wrappers in python around things to improve the current
> situation like a band-aid isn�t the way I�m used to fix things; I
> like to take the bull by the horns[1]
>
> I�d like to ask _why_ structured logging needs to be bolted on top of
> what we have currently? Is it more work to fix the real problem? Are
> we less comfortable doing things in JS?

I'm not sure what "wrappers in python" you have in mind, but I think
that there are a couple of important points to address here.

The first is a general point about testing at Mozilla. There is a lot
more going on than just testing of js code. We have tests written in,
and for, C++, Javascript, Python, HTML, CSS and probably a bunch more
things that I have forgotten. In terms of requirements it's pretty
different from what's needed to test a small project entirely
implemented in a single language.

As it happens a lot of the orchestration of testing is implemented in
Python. I doubt js was even a viable choice at the time this
infrastructure was originally written, and now we have a set of mature
libraries for dealing with lots of the incidental complexity of testing
Mozilla products, like controlling system processes, and setting up B2G
devices. This code is largely decoupled from the tests themselves and
hopefully isn't something that most developers should need to care about.

But the reason that we don't just throw it all away and start again is
because doing so would be a huge cost for extremely uncertain benefit.
That doesn't mean that we can't work to improve things where they are
bad; I'm sure we all have areas that we think are ripe for change.

In terms of structured logging in particular; I don't know why you think
it's "bolted on", or why it isn't fixing the real problem. To be honest
I don't know what you think "the real problem" is. Structured logging is
basically just an API for communicating the results of tests to anyone
that cares to be notified of them, be it tbpl, mach, treeherder, an app
running in your browser that collects test results, or whatever. The
fact that it currently has a Python implementation just reflects the
fact that a lot of our test infrastructure is Python, and doesn't mean
that there's anything Python-specific about it. One could certainly
implement StructuredLog.jsm that would be entirely interoperable with
the Python code. Indeed, if you are writing a test harness that works
entirely in javascript, I would strongly encourage you to do just that.
The work on converting Mochitest to the new format might well have some
code that you could reuse.

If there are particular requirements that you think structured logging
doesn't meet this is probably a good time to discuss them, since we
haven't deployed it to too many places. Perhaps that discussion would be
better off-list since it might not be of interest to everyone.

Mike de Boer

unread,
Jun 3, 2014, 10:35:25 AM6/3/14
to Mozilla dev-platform mailing list mailing list, Gijs Kruitbosch
I started to summarise the things I’d like to see in a JS unit test runner here[1]:

* mini-core.
* Async support as a base. We’ve added `add_task()` methods where possible, but we haven’t made it a core feature of the legacy suites in use today. Generators yielding Promises are now possible, but I think we can do better. This leads to the following point...
* It’d be nice if a suite would be extensible is such a way that it’d be possible to plug-in static analysis tooling to catch race conditions, run linters, common coding pattern validations, common coding error traps, etc.
All this should be possible to add separately as modules, so we don’t end up with 3000+ LOC frameworks that’ll become harder and harder to maintain over time.
* Pluggable reporters to spew out different type of logging formats. TAP, spec, XUnit, to (log)file, etc, all configurable.
* Pluggable code coverage tooling
* Allow to single out specific tests to aid in local debugging
* Allow to disable specific tests without the need to comment them out, so they will be registered as ‘skipped’ (depending on the reporter)
* Support pluggable JS debugger support. Perhaps by running the DevTools remote in a separate XUL window?
* Pluggable environments; the runner should be able to adapt to any JS environment, be it XPCShell, Mochi/ Fx browser, b2g, etc.
* Should be independent of the assertion style used. Assert.jsm could be bundled by default.
* It should be covered 100% by tests itself, Inception-style.
* It should provide hooks to setup a test suite, an individual test, to tear down an individual test and a test suite.
* It‘d be nice to support multiple styles of writing tests: BDD, TDD and QUnit.

I expect this’ll raise eyebrows for some and tears of joy for others… please all respond!

Mike.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=867742

On 03 Jun 2014, at 15:48, Gijs Kruitbosch <gijskru...@gmail.com> wrote:

> I don't think this is off-topic. I'm essentially asking: why did you focus on this, and why in this way? More broadly, I'm asking what it is you're missing from node/mocha.
>
> If you think that needs its own topic, feel free to fork the summary.
>
> In any case, discussions about the ease of use of our test framework(s) seem eminently on-topic for m.d.platform.
>
> ~ Gijs
>
> On 03/06/2014 14:45, Mike de Boer wrote:
>> I understand all that and I *think* you missed the header mentioning I was going off-topic…
>>
>> Mike.
>>
>> On 03 Jun 2014, at 15:39, Gijs Kruitbosch <gijskru...@gmail.com> wrote:
>>
>>> On 03/06/2014 14:16, Mike de Boer wrote:
>>>> Indeed, I’m used to the NodeJS/ Mocha flow of writing tests as fast, or even faster, as writing the implementation of a feature. I could group tests, isolate one, hook in a debugger at any point and more. This is something I miss while working on Fx and to be honest: I still crave for more than a year later.

Mike de Boer

unread,
Jun 3, 2014, 10:38:46 AM6/3/14
to James Graham, Mozilla dev-platform mailing list mailing list
James, thanks so much for the additional background information about testing at Mozilla.

I’m currently following the bugs you mentioned earlier and am looking forward to their results!

Mike.

On 03 Jun 2014, at 16:07, James Graham <ja...@hoppipolla.co.uk> wrote:

> On 03/06/14 14:16, Mike de Boer wrote:
>
>> Writing wrappers in python around things to improve the current
>> situation like a band-aid isn’t the way I’m used to fix things; I
>> like to take the bull by the horns[1]
>>
>> I’d like to ask _why_ structured logging needs to be bolted on top of

Gijs Kruitbosch

unread,
Jun 3, 2014, 11:11:21 AM6/3/14
to Mike de Boer
On 03/06/2014 15:35, Mike de Boer wrote:
> I started to summarise the things I�d like to see in a JS unit test runner here[1]:
>
> * mini-core.
> * Async support as a base. We�ve added `add_task()` methods where possible, but we haven�t made it a core feature of the legacy suites in use today. Generators yielding Promises are now possible, but I think we can do better. This leads to the following point...
What's your concrete proposal for "we can do better"? (see also the
following point, but your point above is sufficiently vague that I don't
know if it's the same as the following or not) How is it being generally
available not a "core feature" ?
> * It�d be nice if a suite would be extensible is such a way that it�d be possible to plug-in static analysis tooling to catch race conditions, run linters, common coding pattern validations, common coding error traps, etc.
This is attempting to put the cart before the horse. We don't have
linters or pattern validators in use today. When someone steps up to
work on them, of course we'd all like them to be modular. But the 3
(debatably...) distinct examples of "pluggable things" you gave (race
detectors, linters, pattern validators) each require different things
from the test runner. It'll basically be the job of whoever wants to
integrate such systems to also write the integration points - which
makes sense, because pre-emptively adding the integration points is
likely to result in a mismatch with the eventual requirements of the
tool (e.g. do you do runtime or static race detection, do you do regex-
or parse-tree-based linting, and so on).

IOW, I don't see what such extensions would require of the test runners
that they don't do today, and/or how you propose the test runner could
do better in "supporting" as-yet-undetermined extensions. We'd do better
to start at writing the static/runtime analysis stuff, and then
integrating it, than we would lamenting about the lack of "support" in
the test framework.
> All this should be possible to add separately as modules, so we don�t end up with 3000+ LOC frameworks that�ll become harder and harder to maintain over time.
> * Pluggable reporters to spew out different type of logging formats. TAP, spec, XUnit, to (log)file, etc, all configurable.
AIUI jgraham is working on this.
> * Pluggable code coverage tooling
This is basically the same as your point above about static analysis -
cart before the horse (although I hear releng are working on code coverage!)
> * Allow to single out specific tests to aid in local debugging
You can do this already.
> * Allow to disable specific tests without the need to comment them out, so they will be registered as �skipped� (depending on the reporter)
And this.
> * Support pluggable JS debugger support. Perhaps by running the DevTools remote in a separate XUL window?
And this (although not for the SDK tests because they're dumb, and not
for xpcshell - the latter would require significant devtools work, AIUI,
less so than test runner work, because the debugger protocol has
concepts of things like windows and tabs, which doesn't really happen in
xpcshell-land; in any case the integration wouldn't really be
"pluggably" the same, even if we stuck it all behind the same
--jsdebugger flag, because mochitest-* get the integration for free from
the running browser).

Note that historically, many of our xpcshell tests actually tested
native code (XPCOM implementations), so the native debugger integration
(which /is/ present) was much more useful. I'm not sure that still holds.
> * Pluggable environments; the runner should be able to adapt to any JS environment, be it XPCShell, Mochi/ Fx browser, b2g, etc.
This doesn't really make sense. The environments are very different
(you've forgotten android, which is java-based, and AFAIK we sadly have
no build system tests (which would be python-based)). Whether or not you
have a chrome or content DOM to work with makes a big difference for the
type of test you write; you can't realistically run 99% of
mochitest-plain/chrome/browser/a11y as xpcshell tests, and there's
basically no reason to want to do the reverse (which goes for crashtests
and reftests as well).
> * Should be independent of the assertion style used. Assert.jsm could be bundled by default.
> * It should be covered 100% by tests itself, Inception-style.
AFAIK SimpleTest, SpecialPowers, xpcshell and other bits of the
mochitest framework all have tests. If the tests are incomplete, I'm
sure nobody would object to you writing more, but it seems hard to
believe this interfering with your ability to quickly write new tests.
> * It should provide hooks to setup a test suite, an individual test, to tear down an individual test and a test suite.
I'm not sure what the benefit of the "test suite" concept is for our
purposes, besides what directory filtering and --start-at and --end-at
already give us. It mostly just sounds like busy-work in terms of the
manifest administration of the tests into test suites.
> * It�d be nice to support multiple styles of writing tests: BDD, TDD and QUnit.
Our current test suites are set up based on the runtime requirements of
the test (ie needs a DOM, needs chrome privileges, needs both, ...).
What do you think we need to do to "support" different styles of tests?
Rearchitecturing all of our test suites isn't a realistic option. Adding
more options to the list purely because some people prefer

when(fooManager.handlingBar).thenExpect(fooManager.bar()).toReturn(5);

over

is(fooManager.bar(), 5, "bar should return 5");

doesn't seem a productive use of our time.


As a small counter-proposal, I'd think we'd get a better cost/effort
ratio for e.g. making the add-on SDK tests work like mochitests, because
the mochitest test runner is much more full-featured than cfx, the
requirements of the tests are very similar, and "debugging"/running the
SDK tests is just painful right now (mostly because the runner is terrible).

More generally, I'm afraid that, as jgraham already pointed out in
another thread, you're underestimating the variety of tests we have, as
well as the different requirements for each of them, and the
complications that each of the test runners have to deal with. Sure, it
would be nice, from a certain perspective, to have the One True
Framework for running tests, but given the different requirements of the
different environments we support (b2g, android, native, js, dom, html,
css, chrome/content, outside-process (marionette)) I don't think it's an
achievable goal.

~ Gijs

Joshua Cranmer 🐧

unread,
Jun 3, 2014, 11:17:38 AM6/3/14
to
On 6/3/2014 8:39 AM, Gijs Kruitbosch wrote:
> On 03/06/2014 14:16, Mike de Boer wrote:
>> Indeed, I’m used to the NodeJS/ Mocha flow of writing tests as fast,
>> or even faster, as writing the implementation of a feature. I could
>> group tests, isolate one, hook in a debugger at any point and more.
>> This is something I miss while working on Fx and to be honest: I
>> still crave for more than a year later.
>
>
> So I'm not used to the "NodeJS/Mocha" 'flow' of writing tests. Can you
> explain what the big difference there is? I find it hard to believe
> the names of the assertion functions are the one big thing making
> tests take much longer to write...

I'm used to xpcshell tests more than mochitests, and the biggest
difference by far between xpcshell and mocha that I'm aware of is that
mocha counts tests at finer granularity: xpcshell tests work on a
file-by-file basis, whereas mocha tests work at the level of individual
test('Name', function() {}) calls. With the right framework support,
this makes it much easier to debug and diagnose single failures when
you're testing a parser function, since you can enable and test only a
single instance instead of setting a breakpoint and continue'ing twelve
times to get to the one you want [fwiw, I have 822 tests in just 6 files
for one of my suites, although most of those are defined in giant array
comparisons].

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

Paolo Amadini

unread,
Jun 2, 2014, 11:39:42 AM6/2/14
to Mike de Boer, Mozilla dev-platform mailing list mailing list
On 6/2/2014 11:37 AM, Mike de Boer wrote:
> Since last Friday[3], each assertion method in Assert.jsm is available in the global scope of a unit test as well.
> Now we can say that the �old� XPCShell-test assertion methods are deprecated in favour of the Assert.jsm ones.

I think it's a very good thing that finally someone is doing the work
of moving various different test suites towards unified assertion
methods (regardless of the actual choice of which methods they are)!

> Here�s a short table of what this means in practice:
>
> * do_check_eq(a, b) �> equal(a, b)
> * do_check_neq(a, b) �> notEqual(a, b)
> * do_check_true(expr) �> ok(expr)
> * do_check_false(expr) �> ok(!expr)
> * do_check_null(expr) �> equal(expr, null)
> * do_check_matches(a, b) �> deepEqual(a, b) (undocumented XPCShell-test feature)

Have you considered requiring test cases to use the the "Assert."
namespace explicitly? I would find that style more readable, and also
assertions easier to find when scanning the code. And they're still
shorter than current assertion functions in xpcshell tests.

Especially the less-known "Assert.throws(<function>, <exception>)" seems
to read better than just "throws(<function>, <exception>)".

For example compare this snippet from the referenced bug...

+ equal(aPacket.type, "paused");
+ let env = aPacket.frame.environment;
+ equal(env.type, "function");
+ equal(env.function.name, "banana3");
+ let parent = env.parent;
+ equal(parent.type, "block");
+ ok("banana3" in parent.bindings.variables);
+ parent = parent.parent;
+ equal(parent.type, "function");
+ equal(parent.function.name, "banana2");
+ parent = parent.parent;
+ equal(parent.type, "block");
+ ok("banana2" in parent.bindings.variables);
+ parent = parent.parent;
+ equal(parent.type, "block");
+ ok("banana2" in parent.bindings.variables);
+ parent = parent.parent;
+ equal(parent.type, "function");
+ equal(parent.function.name, "banana");

...with this modified one:

+ Assert.equal(aPacket.type, "paused");
+ let env = aPacket.frame.environment;
+ Assert.equal(env.type, "function");
+ Assert.equal(env.function.name, "banana3");
+ let parent = env.parent;
+ Assert.equal(parent.type, "block");
+ Assert.ok("banana3" in parent.bindings.variables);
+ parent = parent.parent;
+ Assert.equal(parent.type, "function");
+ Assert.equal(parent.function.name, "banana2");
+ parent = parent.parent;
+ Assert.equal(parent.type, "block");
+ Assert.ok("banana2" in parent.bindings.variables);
+ parent = parent.parent;
+ Assert.equal(parent.type, "block");
+ Assert.ok("banana2" in parent.bindings.variables);
+ parent = parent.parent;
+ Assert.equal(parent.type, "function");
+ Assert.equal(parent.function.name, "banana");

Much easier to visually separate what the test is doing from the
intermixed calls that check the intermediate results.

Cheers,
Paolo

L. David Baron

unread,
Jun 3, 2014, 11:19:33 AM6/3/14
to dev-pl...@lists.mozilla.org, Mike de Boer
On Tuesday 2014-06-03 15:21 +0200, Mike de Boer wrote:
> On 03 Jun 2014, at 15:07, Boris Zbarsky <bzba...@mit.edu> wrote:
> > I assume that the mochitest version will use a different reporter that doesn't throw-and-terminate, to preserve the current semantics of mochitest assertions. (If this assumption is incorrect, we need to have a separate discussion about that.) If so, we'll have the same methods but different semantics in the different harnesses. Not sure how much of a problem that is in practice... not least because I don't think people actually write xpcshell tests all that much.
>
> Ideally, run-until-failure would be a configurable option per test suite. For example, in automation we’d continue on failure and locally we’d stop on failure. Additionally, that’d be configurable per test and per run.

Locally I generally want to see all the failures, since seeing the
complete set of failures is often a much better hint as to the cause
of the failures than just seeing the first one.

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla https://www.mozilla.org/ 𝄂
Before I built a wall I'd ask to know
What I was walling in or walling out,
And to whom I was like to give offense.
- Robert Frost, Mending Wall (1914)
signature.asc

Ehsan Akhgari

unread,
Jun 3, 2014, 11:39:16 AM6/3/14
to Boris Zbarsky, dev-pl...@lists.mozilla.org
On 2014-06-02, 9:35 PM, Boris Zbarsky wrote:
> On 6/2/14, 5:33 PM, Gavin Sharp wrote:
>> Do either of you have reasoning for that other than "it looks better
>> to me"?
>
> My personal experience is that when I try to write xpcshell tests the
> amount of time it takes to type the test function names is very
> noticeable and actively interrupts my thinking about what I actually
> want to test... I find it much simpler to write mochitests than
> xpcshell tests for this reason.
>
> I'm quite willing to believe this is not the case for everyone else, of
> course.
>
>> I personally think consistency trumps any personal preferences
>> based on length/concision
>
> Of course given the existence of testharness we're not going to get
> consistency in mochitest anyway, even with this change.
>
>> as long as what we end up with isn't unreasonably long/verbose.
>
> I think what xpcshell has now and what testharness says and what's being
> proposed (with the "Assert." prefix) are unreasonably long/verbose.

So, what is the decision about what was mentioned in the original post
about reviewers requiring using Assert.jsm in mochitest-chrome/browser?
I mostly care about the former, and I don't think anything in the
subthread that got started with my reply addresses that point at all.

Cheers,
Ehsan

Gregory Szorc

unread,
Jun 3, 2014, 12:08:05 PM6/3/14
to Mike de Boer, Boris Zbarsky, Mozilla dev-platform mailing list mailing list
On 6/3/14, 6:16 AM, Mike de Boer wrote:
> Writing wrappers in python around things to improve the current situation like a band-aid isn�t the way I�m used to fix things; I like to take the bull by the horns[1]
>
> I�d like to ask _why_ structured logging needs to be bolted on top of what we have currently? Is it more work to fix the real problem? Are we less comfortable doing things in JS?

"Structured logging" is a fancy way of saying "machine readable output."
The goal of structured logging is to facilitate machine readable output
from our automation. The goal of machine readable output is to enable
better result viewing, reports, dashboards, etc.

Currently, we have a single stream of output that is read by both humans
and machines. By catering to both consumers, we've produced output that
is inefficient for both. As a human, you hate having to sift through
dozens of lines of context to arrive at the meat. As a computer (or a
human programming a computer), you hate having to parse logs/strings
because it is slow, cumbersome, and fragile. Please read [1] for more
context.

The goal being worked towards is to have two output streams: 1 optimized
for humans and 1 optimized for machines. Since most of our test
harnesses have Python underneath, it makes sense to send machine
readable output from JavaScript/Gecko to Python and have Python "fork"
the stream into human and machine optimized versions (as opposed to
sending 2 streams to Python and maintaining dual output streams in each
test flavor, which is more effort to maintain).

Doing things in Python also has the advantage that the debugging,
testing, and standard library environment is more mature than JavaScript
(though the scales are shifting towards JS). Things like process
management are so much easier in Python. We could write more things in
JavaScript, sure. But we'd be creating more work for ourselves. At the
end of the day, I argue our effort is better spent providing products
like Firefox and Firefox OS, not advance the state of JavaScript
automation. Would advancing JavaScript automation be a noble goal, sure.
But I think the path of least resistance wins out here. And right now
that path means Python over JavaScript (at least for existing automation).

[1]
http://gregoryszorc.com/blog/2012/12/06/thoughts-on-logging---part-1---structured-logging/

Mike de Boer

unread,
Jun 3, 2014, 12:13:21 PM6/3/14
to Ehsan Akhgari, Boris Zbarsky, Mozilla dev-platform mailing list mailing list
I was mainly pointing reviewers of (new) XPCShell tests towards Assert.jsm methods *without* prefix. I’ll remove them from the MDN docs as well. The namespace will remain optional. (Some tests already use it, which is perfectly fine).

As for Mochitest-chrome and/ or browser: I’d like to defer that discussion and decision-making to bug 1018226 for those interested.

Mike.

Ehsan Akhgari

unread,
Jun 3, 2014, 12:19:43 PM6/3/14
to Mike de Boer, Boris Zbarsky, Mozilla dev-platform mailing list mailing list
On 2014-06-03, 12:13 PM, Mike de Boer wrote:
> On 03 Jun 2014, at 17:39, Ehsan Akhgari <ehsan....@gmail.com> wrote:
>
>> On 2014-06-02, 9:35 PM, Boris Zbarsky wrote:
>>> On 6/2/14, 5:33 PM, Gavin Sharp wrote:
>>>> Do either of you have reasoning for that other than "it looks better
>>>> to me"?
>>>
>>> My personal experience is that when I try to write xpcshell tests the
>>> amount of time it takes to type the test function names is very
>>> noticeable and actively interrupts my thinking about what I actually
>>> want to test... I find it much simpler to write mochitests than
>>> xpcshell tests for this reason.
>>>
>>> I'm quite willing to believe this is not the case for everyone else, of
>>> course.
>>>
>>>> I personally think consistency trumps any personal preferences
>>>> based on length/concision
>>>
>>> Of course given the existence of testharness we're not going to get
>>> consistency in mochitest anyway, even with this change.
>>>
>>>> as long as what we end up with isn't unreasonably long/verbose.
>>>
>>> I think what xpcshell has now and what testharness says and what's being
>>> proposed (with the "Assert." prefix) are unreasonably long/verbose.
>>
>> So, what is the decision about what was mentioned in the original post about reviewers requiring using Assert.jsm in mochitest-chrome/browser? I mostly care about the former, and I don't think anything in the subthread that got started with my reply addresses that point at all.
>
> I was mainly pointing reviewers of (new) XPCShell tests towards Assert.jsm methods *without* prefix. I�ll remove them from the MDN docs as well. The namespace will remain optional. (Some tests already use it, which is perfectly fine).

Fair enough. Thanks for updating the wiki!

> As for Mochitest-chrome and/ or browser: I�d like to defer that discussion and decision-making to bug 1018226 for those interested.

OK, I added my r- there, for whatever that would be worth!

Cheers,
Ehsan

Boris Zbarsky

unread,
Jun 3, 2014, 1:05:28 PM6/3/14
to
On 6/3/14, 9:29 AM, Mike de Boer wrote:
> Nope, you got me there - I generalised too easily. This statement is based on personal experience, not science.

I'm not looking for science, necessarily. I'm looking for an
understanding of the problems we're trying to solve.

My basic issue is that from my perspective what I'm seeing is a request
that everyone change their workflow because the new setup is better
without a clear description of the problems that the new setup is trying
to solve. I'm sure there are problems that need solving here; I'm just
trying to get a handle on what they are.

> To preserve backward compat, all the old function names of XPCShell assertions do still work. I was planning to do the same with Mochi.

Sure. The big question is what we do moving forward for new tests.

The way I see it, a test harness has the user-facing API (what test
writers call) and it has a backend that tracks/reports the failures.
Improvements to the backend are generally a no-brainer in my book.
Changes to the user-facing API are something we should be more careful
with...

-Boris

Boris Zbarsky

unread,
Jun 3, 2014, 1:14:28 PM6/3/14
to
On 6/3/14, 11:17 AM, Joshua Cranmer 🐧 wrote:
> I'm used to xpcshell tests more than mochitests, and the biggest
> difference by far between xpcshell and mocha that I'm aware of is that
> mocha counts tests at finer granularity: xpcshell tests work on a
> file-by-file basis, whereas mocha tests work at the level of individual
> test('Name', function() {}) calls. With the right framework support,
> this makes it much easier to debug and diagnose single failures

Indeed. I will assert that the current (now old?) xpcshell harness is
pretty terrible in terms of usability. It was one of the first
harnesses we added and we've learned a lot since then. Basically, my
take on the xpcshell tests is that pretty much any change would be an
improvement. Our other harnesses are not in that situation, though.

-Boris

Boris Zbarsky

unread,
Jun 3, 2014, 1:14:45 PM6/3/14
to
On 6/3/14, 11:19 AM, L. David Baron wrote:
> Locally I generally want to see all the failures, since seeing the
> complete set of failures is often a much better hint as to the cause
> of the failures than just seeing the first one.

Yes, exactly.

-Boris

Gavin Sharp

unread,
Jun 3, 2014, 1:49:24 PM6/3/14
to Boris Zbarsky, dev-platform
On Mon, Jun 2, 2014 at 9:35 PM, Boris Zbarsky <bzba...@mit.edu> wrote:
> I think what xpcshell has now and what testharness says and what's being
> proposed (with the "Assert." prefix) are unreasonably long/verbose.

I suspected this is where we'd end up :) "Reasonability" is just as
subjective as aesthetics.

I really have a hard time accepting at face value the argument
"Assert.notEqual (or other shorter variants) is unreasonably long to
type/paste repeatedly". Hacking on Gecko you have to frequently type
much longer things :) I can certainly buy "it's longer than what I'm
used to", and even "incremental effort is required" - just not
"incremental effort is required and that effort is non-negligible
given other factors" :)

Gavin

Ehsan Akhgari

unread,
Jun 3, 2014, 1:56:29 PM6/3/14
to Gavin Sharp, Boris Zbarsky, dev-platform
Sure. But the point is, what does the proposed change buy us to make
this "subjective" burden be worth it? That is what's not clear in this
thread so far. At the lack of a good reason to change things,
maintaining the status quo should be the default. ;)


Cheers,
Ehsan

Gavin Sharp

unread,
Jun 3, 2014, 2:17:47 PM6/3/14
to Ehsan Akhgari, Boris Zbarsky, dev-platform
I won't argue that a great case has been made :) But I see inherent value
in consistency (both in the implementations and in the user-exposed API)
for assertions across our in-tree test suites (or at least, across
mochitest-based harnesses and xpcshell). Do you disagree?

Gavin


On Tue, Jun 3, 2014 at 1:56 PM, Ehsan Akhgari <ehsan....@gmail.com>
wrote:

Ehsan Akhgari

unread,
Jun 3, 2014, 2:28:32 PM6/3/14
to Gavin Sharp, Boris Zbarsky, dev-platform
On 2014-06-03, 2:17 PM, Gavin Sharp wrote:
> I won't argue that a great case has been made :) But I see inherent
> value in consistency (both in the implementations and in the
> user-exposed API) for assertions across our in-tree test suites (or at
> least, across mochitest-based harnesses and xpcshell). Do you disagree?

Not at all. I thin