QUnit and CommonJS
John Resig
Not sure if people here noticed but as of just this past week QUnit is now capable of running in NarwhalJS (existing as a package/module). I just saw some of the discussion regarding the standardization of method names and it's a bit problematic for us.
Right now QUnit provides 3 methods:
- ok (equivalent to the proposed 'assert')
- equals (equivalent to the proposed 'same')
- same (sort of equivalent to the proposed 'equal' - only it's completely recursive, going much deeper) For example:
same({foo: [ 0, 1, 2 ]}, {foo: [ 0, 1, 2 ]}, "This will pass.");
I have no problem switching 'ok' to 'assert' - and providing the negative versions of these methods - but it's going to be very hard for us to support the conventions proposed by the spec, especially since our same and equals are the complete opposite of the spec's same and equal.
Additionally, the order of the arguments for equals/same are backwards from the proposed same/equal (we want the actual first and the expected second).
Finally, I find it quite odd that an AssertionError would be thrown on a failing test - this would make it impossible to get past the first failing test in a suite (unless I'm mis-understanding the proposed spec).
I realize that there's voting currently going on for the method names, so I'm sorry to find out about this so late in the game, but I consider this to be a huge problem.
--John
Ash Berlin
On 6 Oct 2009, at 21:32, John Resig wrote:
> Hey Everyone -
>
> Not sure if people here noticed but as of just this past week QUnit
> is now capable of running in NarwhalJS (existing as a package/
> module). I just saw some of the discussion regarding the
> standardization of method names and it's a bit problematic for us.
>
> Right now QUnit provides 3 methods:
> - ok (equivalent to the proposed 'assert')
> - equals (equivalent to the proposed 'same')
> - same (sort of equivalent to the proposed 'equal' - only it's
> completely recursive, going much deeper) For example:
> same({foo: [ 0, 1, 2 ]}, {foo: [ 0, 1, 2 ]}, "This will pass.");
>
> I have no problem switching 'ok' to 'assert' - and providing the
> negative versions of these methods - but it's going to be very hard
> for us to support the conventions proposed by the spec, especially
> since our same and equals are the complete opposite of the spec's
> same and equal.
Okay that would be confusing, but part of the spec is that the unit
test stuff is just a minimal set to use in the interop tests
basically. The reason for the AssertiontError is to be able to use
multiple suites of assert functions co-operatively together (see later
in this mail).
>
> Additionally, the order of the arguments for equals/same are
> backwards from the proposed same/equal (we want the actual first and
> the expected second).
So this is the order i'm used to from perl: is ($got, $expected,
$test_name); so i personally dont mind this. Infact i'd probably
favour this.
>
> Finally, I find it quite odd that an AssertionError would be thrown
> on a failing test - this would make it impossible to get past the
> first failing test in a suite (unless I'm mis-understanding the
> proposed spec).
So throwing an error is focused around the method of multiple small
test functions. Take this example from a regression in flusspferd
(not, names and arg order probably won't match the proposed spec. Or
even exist in any spec):
__JS__
require.paths.unshift('test/js/lib/modules-test');
let test = require('test'),
asserts = test.asserts;
exports.test_RequireId = function () {
// Not sure what require.id should be. it ceratinly shouldn't be
'system'
asserts.same(require.id, "test/js/modules.js");
}
exports.test_InstanceOf = function () {
var a1 = require('a1');
var a2 = require('a2');
asserts.instanceOf(a1.array(), Array, "Array in a1 is instanceof
main Array");
asserts.instanceOf(a2.array(), Array, "Array in a2 is instanceof
main Array");
asserts.instanceOf([1,2,3], Array, "sanity check");
}
var global = this;
exports.test_varPolution = function() {
require('a2');
asserts.same("a2" in global, false, "no a2 variable");
}
if (require.main == module)
test.runner(exports);
__END__
so each of those asserts.* functions will throw an error on failure.
But that would only stop the individual function, not the entire test
run. The test runner is supposed to catch these, handle the failure
then move onto the next function. From your confusion I think this
might need to be made more obvious as to what the behaviour is.
(Note: that the above asserts use the form (got, expected) that was my
natural inclination for param order. I for some reason thought we
agreed to have it the other way around. But of course, because of the
AssertionError I can write my own test library and never use the
'offical' ones in any of my code)
Kris Kowal
> Not sure if people here noticed but as of just this past week QUnit is now
> capable of running in NarwhalJS (existing as a package/module). I just saw
> some of the discussion regarding the standardization of method names and
> it's a bit problematic for us.
I think we'll have to add some sort of wrapper to use it as a test
runner for CommonJS module tests.
> Right now QUnit provides 3 methods:
> - ok (equivalent to the proposed 'assert')
> - equals (equivalent to the proposed 'same')
> - same (sort of equivalent to the proposed 'equal' - only it's completely
> recursive, going much deeper) For example:
> same({foo: [ 0, 1, 2 ]}, {foo: [ 0, 1, 2 ]}, "This will pass.");
5.4 states that "equal" equivalence is determined by all values being
equivalent, so it's recursive.
> I have no problem switching 'ok' to 'assert' - and providing the negative
> versions of these methods - but it's going to be very hard for us to support
> the conventions proposed by the spec, especially since our same and equals
> are the complete opposite of the spec's same and equal.
I don't mind going from assert.assert to assert.ok. It wasn't in the
list of nominations, so a recount might be in order.
We could avoid the (equals, same) confusion by backpedalling to (eq,
ne, is, isnt) where there are clear analogs.
> Additionally, the order of the arguments for equals/same are backwards from
> the proposed same/equal (we want the actual first and the expected second).
> Finally, I find it quite odd that an AssertionError would be thrown on a
> failing test - this would make it impossible to get past the first failing
> test in a suite (unless I'm mis-understanding the proposed spec).
Right, as Ash points out, the tests are meant to be small, usually one
assert per test. This is to enable errors to be isolated to a test,
instead of trashing subsequent tests. My original designs for the
unit testing system also were more of a logging system with long lines
of assertions, but this is probably more resilient, albeit more
verbose. It's a trade-off that almost all the prior art was in
agreement on.
> I realize that there's voting currently going on for the method names, so
> I'm sorry to find out about this so late in the game, but I consider this to
> be a huge problem.
I do not mind stopping to bring you on board.
Kris Kowal
Dean Landolt
[snip]
so each of those asserts.* functions will throw an error on failure.
But that would only stop the individual function, not the entire test
run. The test runner is supposed to catch these, handle the failure
then move onto the next function. From your confusion I think this
might need to be made more obvious as to what the behaviour is.
The behavior is implicit in the specification for "run" (points 10-12), but it could be made more explicit. But I really like the AssertionError idea, and not just for the reasons Ash specifies. In some cases a fail-early model will prevent having to wade through too many tests failures, and it allows you to group your tests (and subtests) logically to choose the level of granularity you prefer.
Though, I can't help but be bothered a bit by the (admittedly conventional) exports.testSomeTest convention as specified. As has been pointed out in a lot of the JSGI threads, JS has sane namespaces, and namespaces are good. Why a string prefix and not a exports.test object?
(Sorry if this feedback is a so late to the party. I've kept it to myself because I'd rather have any spec than no spec.)
(Note: that the above asserts use the form (got, expected) that was my
natural inclination for param order. I for some reason thought we
agreed to have it the other way around. But of course, because of the
AssertionError I can write my own test library and never use the
'offical' ones in any of my code)
Except for modules where you intend dependency-free interoperability amongst implementations, right? :)
John Resig
> Not sure if people here noticed but as of just this past week QUnitOkay that would be confusing, but part of the spec is that the unit
> is now capable of running in NarwhalJS (existing as a package/
> module). I just saw some of the discussion regarding the
> standardization of method names and it's a bit problematic for us.
>
> Right now QUnit provides 3 methods:
> - ok (equivalent to the proposed 'assert')
> - equals (equivalent to the proposed 'same')
> - same (sort of equivalent to the proposed 'equal' - only it's
> completely recursive, going much deeper) For example:
> same({foo: [ 0, 1, 2 ]}, {foo: [ 0, 1, 2 ]}, "This will pass.");
>
> I have no problem switching 'ok' to 'assert' - and providing the
> negative versions of these methods - but it's going to be very hard
> for us to support the conventions proposed by the spec, especially
> since our same and equals are the complete opposite of the spec's
> same and equal.
test stuff is just a minimal set to use in the interop tests
basically.
At least as it stands I find QUnit's terminology to be more intuitive. 'equals' equates to ===, 'same' is more wishy-washy (which makes sense, since it's doing a deep comparison). I'd be happy with another alternative to 'same' (deepEqual?) but as it stands this is going to cause problems for us.
Yeah, I'm not a huge fan of this. In QUnit we listen for all assertions and display the results cumulatively (so for an individual test block you could have 2 failing, 3 passing, 5 total tests - a quick demo can be found here: http://docs.jquery.com/QUnit). The important point being that every time you run the test suite you're getting an identical number of total tests out as a result (even if some assertions are failing). The only time in which the total number will differ is if there's a catastrophic failure (a real, unintended, exception is thrown).
This brings up three other concerns:
Forcing a module to export test_FOO methods is really counter-intuitive. Not only are you exposing additional exports but all the method names are forced into cryptic camelCase style (or similar). A better alternative would be to to have a single method that you can call that has a function which contains all the assertions. For example, in QUnit it's:
test("This is the nice, descriptive, name for the tests.", function(){
assert.... etc.
});
Additionally, within these test blocks there is no way to require that a specific number of assertions be run (if there's a logic error in your code and you accidentally skip an assertion you would never catch it with the current set up). In QUnit we have two ways to specify this:
test("foo", 5, function(){
// there will be 5 assertions in here, if not, a failure will be logged
});
test("foo", function(){
expect(5);
// ditto
});
IMO, I'd be fine with just the optional argument one.
Finally, there is no way to handle asynchronous test blocks (it assumes that everything will finish synchronously). Someone can correct me if I'm wrong but I think it's assumed that asynchronous code can exist in CommonJS land (especially if it's running in a browser).
We have two ways of handing this in QUnit:
testAsync("foo", function(){
setTimeout(function(){
assert(true, "I run late!");
start(); // re-start the tests running again
}, 13);
});
test("foo", function(){
stop();
setTimeout(function(){
assert(true, "I run late!");
start(); // re-start the tests running again
}, 13);
});
Both are identical, I'm flexible on the specifics.
Now, obviously, this is all expanding the scope of the current proposal quite dramatically - I may be ok punting on it in favor of getting a simpler API out. That being said, I think it would be good to push a solid (and easy to use!) API out in the first go around.
--John
Dean Landolt
5.4 states that "equal" equivalence is determined by all values being
> Right now QUnit provides 3 methods:
> - ok (equivalent to the proposed 'assert')
> - equals (equivalent to the proposed 'same')
> - same (sort of equivalent to the proposed 'equal' - only it's completely
> recursive, going much deeper) For example:
> same({foo: [ 0, 1, 2 ]}, {foo: [ 0, 1, 2 ]}, "This will pass.");
equivalent, so it's recursive.
I don't mind going from assert.assert to assert.ok. It wasn't in the
> I have no problem switching 'ok' to 'assert' - and providing the negative
> versions of these methods - but it's going to be very hard for us to support
> the conventions proposed by the spec, especially since our same and equals
> are the complete opposite of the spec's same and equal.
list of nominations, so a recount might be in order.
We could avoid the (equals, same) confusion by backpedalling to (eq,
ne, is, isnt) where there are clear analogs.
Considering QUnit will probably see a fair bit of use, it would be tragic to have to juggle two competing, almost opposite meanings for the same functions. I hereby surrender my +1s to the longer names in favor of the shorter ones (even though they violate the literate naming convention).
John Resig
On Tue, Oct 6, 2009 at 1:32 PM, John Resig <jer...@gmail.com> wrote:I think we'll have to add some sort of wrapper to use it as a test
> Not sure if people here noticed but as of just this past week QUnit is now
> capable of running in NarwhalJS (existing as a package/module). I just saw
> some of the discussion regarding the standardization of method names and
> it's a bit problematic for us.
runner for CommonJS module tests.
We can certainly detect to see if we're running in a CommonJS environment and expose a separate API (we already detect if we are and use the exports mechanism) but it would be best to have a non-conflicting API from the start (even better would be having an identical API).
> Right now QUnit provides 3 methods:5.4 states that "equal" equivalence is determined by all values being
> - ok (equivalent to the proposed 'assert')
> - equals (equivalent to the proposed 'same')
> - same (sort of equivalent to the proposed 'equal' - only it's completely
> recursive, going much deeper) For example:
> same({foo: [ 0, 1, 2 ]}, {foo: [ 0, 1, 2 ]}, "This will pass.");
equivalent, so it's recursive.
Ah, ok, I stand corrected. Good!
I don't mind going from assert.assert to assert.ok. It wasn't in the
list of nominations, so a recount might be in order.
I'd be fine leaving it as just assert - I know that it's a name that most people are familiar with (ok was kind of a weird choice on our end, I wouldn't have a problem moving from ok to assert).
We could avoid the (equals, same) confusion by backpedalling to (eq,
ne, is, isnt) where there are clear analogs.
I'm mixed on these names but pleased if we can use it to reach a solid consensus. I assume eq/ne would be === and is/isnt would be the deep comparison?
Right, as Ash points out, the tests are meant to be small, usually one
assert per test. This is to enable errors to be isolated to a test,
instead of trashing subsequent tests. My original designs for the
unit testing system also were more of a logging system with long lines
of assertions, but this is probably more resilient, albeit more
verbose. It's a trade-off that almost all the prior art was in
agreement on.
Hmm, ok. I guess I can understand the logic behind the decision, although I'm not a huge fan. I like to have an identical number of tests being run every time it helps to spot egregious errors. Additionally it's not always the case that the first assertion failure is the most obvious case - being able to see where everything is going wrong can help to lead to a fuller understanding of the problem.
> I realize that there's voting currently going on for the method names, soI do not mind stopping to bring you on board.
> I'm sorry to find out about this so late in the game, but I consider this to
> be a huge problem.
Thank you for being so magnanimous, I really appreciate it. I feel bad that I'm so late to the game here but I absolutely want to help!
--John
Ash Berlin
On 6 Oct 2009, at 22:18, John Resig wrote:
>
> I don't mind going from assert.assert to assert.ok. It wasn't in the
> list of nominations, so a recount might be in order.
>
> I'd be fine leaving it as just assert - I know that it's a name that
> most people are familiar with (ok was kind of a weird choice on our
> end, I wouldn't have a problem moving from ok to assert).
ok() is what perl uses (it also has an explicit pass(msg); fail(msg);
but you hardly ever see those).
>
> We could avoid the (equals, same) confusion by backpedalling to (eq,
> ne, is, isnt) where there are clear analogs.
>
> I'm mixed on these names but pleased if we can use it to reach a
> solid consensus. I assume eq/ne would be === and is/isnt would be
> the deep comparison?
Thats not what it looks like form the wiki: "Deep equivalence
assertion ... assert.eq"
>
> Right, as Ash points out, the tests are meant to be small, usually one
> assert per test. This is to enable errors to be isolated to a test,
> instead of trashing subsequent tests. My original designs for the
> unit testing system also were more of a logging system with long lines
> of assertions, but this is probably more resilient, albeit more
> verbose. It's a trade-off that almost all the prior art was in
> agreement on.
>
> Hmm, ok. I guess I can understand the logic behind the decision,
> although I'm not a huge fan. I like to have an identical number of
> tests being run every time it helps to spot egregious errors.
> Additionally it's not always the case that the first assertion
> failure is the most obvious case - being able to see where
> everything is going wrong can help to lead to a fuller understanding
> of the problem.
I used to do this in perl (which I code as my day job.) I've stopped
after I read this: http://www.shadowcat.co.uk/blog/matt-s-trout/a-cunning-no_plan/
Dean Landolt
>I used to do this in perl (which I code as my day job.) I've stopped
> Hmm, ok. I guess I can understand the logic behind the decision,
> although I'm not a huge fan. I like to have an identical number of
> tests being run every time it helps to spot egregious errors.
> Additionally it's not always the case that the first assertion
> failure is the most obvious case - being able to see where
> everything is going wrong can help to lead to a fuller understanding
> of the problem.
after I read this: http://www.shadowcat.co.uk/blog/matt-s-trout/a-cunning-no_plan/
If we're standardizing on the behavior of the test runner, something like the done_testing; statement mentioned at the bottom of that article come in handy and serve as a useful compromise to counting tests.
Kris Kowal
> Forcing a module to export test_FOO methods is really counter-intuitive. Not
> only are you exposing additional exports but all the method names are forced
> into cryptic camelCase style (or similar).
In the Narwhal codebase, there are a bunch of tests that look like:
exports['test Long Description'] = function () {};
> A better alternative would be to
> to have a single method that you can call that has a function which contains
> all the assertions. For example, in QUnit it's:
>
> test("This is the nice, descriptive, name for the tests.", function(){
> assert.... etc.
> });
>
> Additionally, within these test blocks there is no way to require that a
> specific number of assertions be run (if there's a logic error in your code
> and you accidentally skip an assertion you would never catch it with the
> current set up). In QUnit we have two ways to specify this:
>
> test("foo", 5, function(){
> // there will be 5 assertions in here, if not, a failure will be logged
> });
I'm a fan of this style, but one nice thing about the exports setup is
that the test runner doesn't care whether it's actually receiving a
module's exports. You can organize tests in an object tree by
whatever means you chose. In Narwhal we often create aggregate tests
with the notation:
exports.testSubmodule = require("./submodule");
It's pretty hard to associate a test with a module object with a
function call in the context of CommonJS. We could set up loaders to
optionally curry the id of a calling module on decorated exports, but
this scared a lot of people off when I last brought it up.
Alternately, we could attach a member function to the module object so
it could use "this" to discover the calling module id.
module.test = require("runner").test;
// module.id is the calling module
module.test(desc, n, block);
But, that'd be confusing boilerplate.
> Finally, there is no way to handle asynchronous test blocks (it assumes that
> everything will finish synchronously). Someone can correct me if I'm wrong
> but I think it's assumed that asynchronous code can exist in CommonJS land
> (especially if it's running in a browser).
>
> We have two ways of handing this in QUnit:
>
> testAsync("foo", function(){
> setTimeout(function(){
> assert(true, "I run late!");
> start(); // re-start the tests running again
> }, 13);
> });
> test("foo", function(){
> stop();
> setTimeout(function(){
> assert(true, "I run late!");
> start(); // re-start the tests running again
> }, 13);
> });
We would be very likely, I think, to use promises for this. We should
punt asynchroneity to version 2. It would probably take twice as long
to get consensus with async speced.
Really, what we're trying to do here is to establish conventions for
unit tests to accompany CommonJS specifications to verify compliance.
We don't need much to do this initially, and we're not looking to
constrain extensions to the system. This is more about how tests
should be written than how testers should be written.
Kris Kowal
John Resig
> I don't mind going from assert.assert to assert.ok. It wasn't in theok() is what perl uses (it also has an explicit pass(msg); fail(msg);
> list of nominations, so a recount might be in order.
>
> I'd be fine leaving it as just assert - I know that it's a name that
> most people are familiar with (ok was kind of a weird choice on our
> end, I wouldn't have a problem moving from ok to assert).
but you hardly ever see those).
In another life I use to be a Perl guy - that's probably where I picked that convention up.
> We could avoid the (equals, same) confusion by backpedalling to (eq,Thats not what it looks like form the wiki: "Deep equivalence
> ne, is, isnt) where there are clear analogs.
>
> I'm mixed on these names but pleased if we can use it to reach a
> solid consensus. I assume eq/ne would be === and is/isnt would be
> the deep comparison?
assertion ... assert.eq"
Ehhhh :-/ I just feel like if we have a method name equal/equals/eq it should be immediately equivalent to doing A === B (equals method results in equality test).
I used to do this in perl (which I code as my day job.) I've stopped
> Right, as Ash points out, the tests are meant to be small, usually one
> assert per test. This is to enable errors to be isolated to a test,
> instead of trashing subsequent tests. My original designs for the
> unit testing system also were more of a logging system with long lines
> of assertions, but this is probably more resilient, albeit more
> verbose. It's a trade-off that almost all the prior art was in
> agreement on.
>
> Hmm, ok. I guess I can understand the logic behind the decision,
> although I'm not a huge fan. I like to have an identical number of
> tests being run every time it helps to spot egregious errors.
> Additionally it's not always the case that the first assertion
> failure is the most obvious case - being able to see where
> everything is going wrong can help to lead to a fuller understanding
> of the problem.
after I read this: http://www.shadowcat.co.uk/blog/matt-s-trout/a-cunning-no_plan/
This doesn't seem to be compatible with what I'm proposing. In this case they're using plan testing across an entire file's worth of testing (which is impractical and can cause strange results, as he noted). We use it against individual test() groupings in the jQuery test suite and it's helped us immensely (we now have over 2200 tests with many developers contributing - it helps to keep our sanity, for sure).
But, again, expected test count and async tests may be bike-shedding at this point. As long as the final proposal is able to implicitly make sure that these cases can exist (even if they're layered on by the test runner) then I'll be content.
--John
Kris Kowal
> In another life I use to be a Perl guy - that's probably where I picked that
> convention up.
I generally don't mention my Perl days in polite company.
>> Thats not what it looks like form the wiki: "Deep equivalence
>> assertion ... assert.eq"
>
> Ehhhh :-/ I just feel like if we have a method name equal/equals/eq it
> should be immediately equivalent to doing A === B (equals method results in
> equality test).
My reasoning:
* in Python, "is" is used for object identity, essentially ===,
__eq__ is used for user-defined deep-equivalence.
* .equals() in Java is used for deep-equivalence, and == for shallow,
referential identity.
* Perl and shells establish "eq", "ne", "gt", "lt", "ge", "le", but I
don't think have anything relevant to say about distinguishing shallow
and deep equivalence.
* JavaScript isn't clean; == and === are both shades of gray between
shallow and deep equivalence, so it doesn't have anything relevant to
say about distinguishing shallow and deep.
I get a strong "polymorphic, deeply-equivalent" vibe from "equal" or
its ilk, and a strong "is exactly the same object" vibe from "is".
Kris Kowal
Ash Berlin
Thats not what it looks like form the wiki: "Deep equivalence
> We could avoid the (equals, same) confusion by backpedalling to (eq,
> ne, is, isnt) where there are clear analogs.
>
> I'm mixed on these names but pleased if we can use it to reach a
> solid consensus. I assume eq/ne would be === and is/isnt would be
> the deep comparison?
assertion ... assert.eq"
Ehhhh :-/ I just feel like if we have a method name equal/equals/eq it should be immediately equivalent to doing A === B (equals method results in equality test).
I used to do this in perl (which I code as my day job.) I've stopped
after I read this: http://www.shadowcat.co.uk/blog/matt-s-trout/a-cunning-no_plan/
This doesn't seem to be compatible with what I'm proposing. In this case they're using plan testing across an entire file's worth of testing (which is impractical and can cause strange results, as he noted). We use it against individual test() groupings in the jQuery test suite and it's helped us immensely (we now have over 2200 tests with many developers contributing - it helps to keep our sanity, for sure).
"Well, yes, it would catch gimme_a_list() returning nothing. But if we actually care about the number of elements returned, WHY THE HELL AREN'T WE TESTING FOR WHAT?! I mean, if it matters then you should be writing:
my @list = gimme_a_list();cmp_ok(scalar(@list), '==', 3, 'gimme_a_list() returned three entries');foreach my $member (@list) { ...
But, again, expected test count and async tests may be bike-shedding at this point. As long as the final proposal is able to implicitly make sure that these cases can exist (even if they're layered on by the test runner) then I'll be content.
--John
Nathan Stott
Forcing a module to export test_FOO methods is really counter-intuitive. Not only are you exposing additional exports but all the method names are forced into cryptic camelCase style (or similar). A better alternative would be to to have a single method that you can call that has a function which contains all the assertions. For example, in QUnit it's:
test("This is the nice, descriptive, name for the tests.", function(){
assert.... etc.
});
exports["test some stuff"] = function() { ... }
Hannes Wallnoefer
>
> At least as it stands I find QUnit's terminology to be more intuitive.
> 'equals' equates to ===, 'same' is more wishy-washy (which makes sense,
> since it's doing a deep comparison). I'd be happy with another alternative
> to 'same' (deepEqual?) but as it stands this is going to cause problems for
> us.
If you think of === as 'equals' that explains your position. It's hard
to argue about meaning and habits, however the EcmaScript spec and the
Mozilla JS reference both call == "equals operator" and === "strict
equals operator". (Would strictEquals be an option?)
Another point: "equal" is generally used with plural form ("two things
are equal", "equal colors", "all men are created equal"), while "same"
makes mostly sense with singular form ("the same person", "the same
color"). To me this means that "equals" conveys more of a "two things
that are aequivalent" meaning, while "same" leans more towards "one
and the same thing". (Note that I'm not a native speaker, so I may be
getting this wrong.)
>
> test("This is the nice, descriptive, name for the tests.", function(){
> assert.... etc.
> });
One very nice thing about the exports convention is that you don't
need any additional functionality to define tests beyond what the
module mechanism naturally provides, and that it is very easy to
compose test suites from multiple modules and group test output by
module, e.g.:
http://helma.pastebin.com/f427bb2cd
Hannes
Tom Robinson
On Oct 6, 2009, at 2:37 PM, Kris Kowal wrote:
> Really, what we're trying to do here is to establish conventions for
> unit tests to accompany CommonJS specifications to verify compliance.
> We don't need much to do this initially, and we're not looking to
> constrain extensions to the system. This is more about how tests
> should be written than how testers should be written.
Right, IMO the only reason we're even talking about standardizing a
test framework is so we can write interoperable tests for the CommonJS
APIs. Normally it would be too high level for us to be concerned with.
I fully expect there to be other competing test frameworks like QUnit,
and I have no problem with people using them for their own CommonJS
projects. If our test APIs are only ever used for testing the CommonJS
APIs that's fine with me.
You can think of it like RubySpec's MSpec.
That said it would be good if we could at least agree on naming of the
assert functions to minimize confusion.
-tom
Dean Landolt
Amen. But let's not forget that we're also specifying test runner conventions -- it'd be really great to agree on those too.
Dean Landolt
Earlier today I proposed a namespace object such as exports.test (or exports.tests, or something to that affect). It'd be great if you could do an exports.test["some stuff"] instead -- and from a very liberal reading of the spec perhaps you may already (are strings on exports that start with "test" are objects are tests sniffed out recursively?). I think this would be a very elegant pattern and alleviates the camelCaseTestName for those that aren't into that kinda thing.
Charles Jolley
Just wanted to throw in my 2¢ here: The SproutCore JavaScript
framework (which I represent - http://www.sproutcore.com) is planning
to convert the entire platform over to use modules fairly soon. We
have over 5,500 unit tests written against the QUnit API that we will
convert also.
It appears that the only real two issues that keep Qunit and the
current CommonJS proposal apart are method names and argument order.
Since there are unit tests for many different projects out there
written against Qunit, I think adopting the Qunit version of method
names and argument orders would make it easier for existing JS libs to
make their existing code CommonJS compatible. That seems like a win
for the project.
It would means mostly a one-time find/replace job for the folks on
this list who have already written to against the current API while it
would benefit (hopefully) many more developers in the future who won't
have to rewrite their code.
-Charles
Nathan Stott
Unless I'm missing something, it doesn't really matter if QUnit and the CommonJS unit testing API are different.
Dean Landolt
We have over 5,500 unit tests written against the QUnit API that we will
convert also.
5,500? Holy s*** I'm lazy.
It appears that the only real two issues that keep Qunit and the
current CommonJS proposal apart are method names and argument order.
Since there are unit tests for many different projects out there
written against Qunit, I think adopting the Qunit version of method
names and argument orders would make it easier for existing JS libs to
make their existing code CommonJS compatible. That seems like a win
for the project.
It would means mostly a one-time find/replace job for the folks on
this list who have already written to against the current API while it
would benefit (hopefully) many more developers in the future who won't
have to rewrite their code.
Given how young the spec is I don't think there's much of a concern about having to rewrite tests (after all, there's no conformant implementation yet anyway). There have been some persuasive comments about semantics (what do equal/same mean to you?). Just to throw my paint color in, I think identity when I hear same, and equivalence when I hear equal, but to me this is neither here nor there -- I'll just be ecstatic to see a simple unit testing spec ratified, I don't care so much what it looks like.
Christoph Dorn
> I'll just be
> ecstatic to see a simple unit testing spec ratified, I don't care so
> much what it looks like.
+1
Christoph
Kevin Dangoor
Just wanted to throw in my 2¢ here: The SproutCore JavaScript
framework (which I represent - http://www.sproutcore.com) is planning
to convert the entire platform over to use modules fairly soon. We
have over 5,500 unit tests written against the QUnit API that we will
convert also.
This is a very cool development. (Side note: we are reconstructing Bespin around CommonJS modules and SproutCore, and Charles has been very receptive and helpful. I personally can't wait to see the interesting intersections we get when we start having SC bits running on both the client and server!)
It appears that the only real two issues that keep Qunit and the
current CommonJS proposal apart are method names and argument order.
Actually, there are 4 issues:
1. method names
2. argument order
3. test discovery (exports.test* vs. test(...))
4. fail fast or not (throw AssertionError or just log the failure?)
5. use of global names (not mentioned yet, I don't think)
Before I toss out some opinions on these things, there is the general question of "does it matter?"
Arguably, the standard that is being set for CommonJS is "just" for the API. However, I daresay that having a "good enough" API in CommonJS will likely lead to people using it for testing their application code. Python's unittest module is not the best choice for testing Python code, but I'm sure it's still the most common.
So, I think it is worth considering the implications for application developers, not just API implementers.
Personally, I agree with others on this thread about the definition of "equals" and "same" -- I see "equals" as equivalent value and "same" as being the actual same object. I think a lot of us are likely influenced by xUnit. That said, that bit is a total bikeshed and it would probably take me writing just a couple of tests before I get used to the other style.
Given that QUnit is used by a lot of code and is familiar to a fair number of JS programmers, I wouldn't even personally put up a fight about the names and argument order.
Test discovery:
This also strikes me as a bikeshed color thing. One could still build test suites by having the main test module require other modules that call test(). Is there something I'm missing?
Fail fast:
For almost a decade, I've been writing tests using frameworks that fail a test using an exception (thus failing at the first failed assertion). Earlier this year, my colleague Joe Walker wrote a small testing framework for Bespin that works more like QUnit. I thought it would be too noisy... but it's not. So, again, I'm +0 here, as long as there is a mechanism available for custom check functions to log a failure (which I assume would just be a matter of calling ok(false, "fancy check failed!"))
Finally, I would be willing to bet that most QUnit tests are written with the various functions available as globals... eg,
test("a basic test example", function() {
ok( true, "this test is fine" );
var value = "hello";
equals( "hello", value, "We expect value to be hello" );
});
Which is taken from the QUnit page.
I'm personally a really big fan of automated tests. So much so, that I'm going to toss something out that I hope doesn't derail this conversation entirely...
Wouldn't it be nice if test runners could actually provide the basic test functions as free variables to test modules?
Like everyone else here, I'd like to see the test API discussion laid to rest. But I do think this API will have an impact beyond just CommonJS compliance. Also, I'm with Kris on punting putting asynchronous testing in the first version of this spec, because that will almost certainly prolong things. I'm also willing to punt on test functions appearing as free variables in test modules, but I wanted to see what people thought.
Kevin
--
Kevin Dangoor
work: http://labs.mozilla.com/
email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com
Kris Kowal
> Wouldn't it be nice if test runners could actually provide the basic test
> functions as free variables to test modules?
It is within the realm of possibility. We're exploring something like
this with Jake in Narwhal for Jakefiles. However, it only works if we
create custom module loaders for these types of files, which is to say
that they are not CommonJS modules at all. Having frameworks in place
for creating module loaders in pure-javascript helps (for things like
identifier resolution) but I think that we should punt this to a
future version as well. We have a generic module system; I think we
ought to use it.
Kris Kowal
Dion Almaer
Kevin Dangoor
It's definitely fine by me if all of the CommonJS compliance tests are written as
var assert = require("assert");
...
assert.equals(...)
...
And, truth be told, I've experimented in Python with providing free variables in certain contexts and have each time decided that having "normal modules" was better. Test modules are one case where it may be okay, because they aren't likely to be useful in non-test contexts...
Kevin Dangoor
In the Java world, TestNG brought the ability to fail fast or not. There are different use cases where you want one versus another, and I really hope that we can choose. There are also really cool features like remembering the last run, and the test that failed, and starting there (useful when you are iterating quickly).
Starting with the last failed test on a re-run is a great feature.
John Resig
> At least as it stands I find QUnit's terminology to be more intuitive.If you think of === as 'equals' that explains your position. It's hard
> 'equals' equates to ===, 'same' is more wishy-washy (which makes sense,
> since it's doing a deep comparison). I'd be happy with another alternative
> to 'same' (deepEqual?) but as it stands this is going to cause problems for
> us.
to argue about meaning and habits, however the EcmaScript spec and the
Mozilla JS reference both call == "equals operator" and === "strict
equals operator". (Would strictEquals be an option?)
Another point: "equal" is generally used with plural form ("two things
are equal", "equal colors", "all men are created equal"), while "same"
makes mostly sense with singular form ("the same person", "the same
color"). To me this means that "equals" conveys more of a "two things
that are equivalent" meaning, while "same" leans more towards "one
and the same thing". (Note that I'm not a native speaker, so I may be
getting this wrong.)
So I slept on it overnight and realized that I proposed a ton of things here yesterday. I'm going to reduce my proposal - Charles is right - the two main concerns are the equal/same method names and the argument order.
I'd like to make a proposal for the method names:
equal/notEqual (the same as A === B)
equiv/notEquiv (deep, recursive, comparison)
Hannes brought up a good point above - he feels as if 'equal' conveys "two things that are *equivalent*" - which isn't correct. 'equiv' conveys equivalence - and, frankly, it's a much better name than both 'equal' or 'same'. Once you have a method named 'equiv' it becomes much more obvious what the purpose of equal/notEqual is.
This would be an easier API change for QUnit. We would go from equals to equal and same to to equiv - which is pretty manageable and doesn't cause any cognitive overload.
Secondly, it seems as if there's no major concerns with changing the actual/expected argument order. In fact, Ash was already writing tests that assumed it was in the actual/expected order! If we could get this changed in the spec doc that'd be great.
--John
Tom Robinson
not in our current proposal. If it's in "fail fast mode" then the
asserts must throw an AssertionError. If not then they should report
the assertion result through other means.
The point of defining the AssertionError is to make it easy to
differentiate between an assertion and hard errors in the fail fast
case.
-tom
Kris Kowal
> I'd like to make a proposal for the method names:
>
> equal/notEqual (the same as A === B)
> equiv/notEquiv (deep, recursive, comparison)
I think this is equitable. Let's be clear together that it would be
good to eventually support polymorphic override of "equiv" and that
would be the logical method name for that behavior. The name for
equivalence comparison should smell good both in this API, and on
methods.
> Secondly, it seems as if there's no major concerns with changing the
> actual/expected argument order. In fact, Ash was already writing tests that
> assumed it was in the actual/expected order! If we could get this changed in
> the spec doc that'd be great.
I don't mind.
For the failing fast generalism, we can export a flag to opt into
"log" mode instead of "fail" mode, or vise versa.
Kris Kowal
Kevin Dangoor
Hannes Wallnoefer
> I'm cool with all of these as well, but there is still the question of
> calling test() rather than exporting test*. That is another significant
> difference between QUnit and the current proposal. Are people okay with that
> change?
features. It's great that it runs without DOM now, but what does it
have that makes it uniquely valuable for testing CommonJS code that we
don't already have? Unless there is something, I don't see any reason
to bend over backwards to make QUnit fit in.
Hannes
Kevin Dangoor
2009/10/7 Kevin Dangoor <dan...@gmail.com>:
>I don't get it. Looking at the QUnit code, I see mostly DOM-related
> I'm cool with all of these as well, but there is still the question of
> calling test() rather than exporting test*. That is another significant
> difference between QUnit and the current proposal. Are people okay with that
> change?
features. It's great that it runs without DOM now, but what does it
have that makes it uniquely valuable for testing CommonJS code that we
don't already have? Unless there is something, I don't see any reason
to bend over backwards to make QUnit fit in.
Compared to the other APIs that have been discussed and are being discussed, the unit testing API is very simple and has few real world ramifications.
In fact, the fail fast vs. log only thing is the only thing about this that is not bikeshedding. If we choose to paint the unit test bikeshed QUnit color rather than #432e21, that may be a marketing win because of familiarity.
Unless there is something actually wrong with the QUnit API that will cause problems, I suggest that we should put it to a vote rather than spending a lot more time on a fairly simple bit of API. Perhaps sometime tomorrow after people have a little more time to voice any functional objections, I (or anyone for that matter) should kick off a thread to see which way people are leaning.
Dean Landolt
2009/10/7 Kevin Dangoor <dan...@gmail.com>:
>I don't get it. Looking at the QUnit code, I see mostly DOM-related
> I'm cool with all of these as well, but there is still the question of
> calling test() rather than exporting test*. That is another significant
> difference between QUnit and the current proposal. Are people okay with that
> change?
features. It's great that it runs without DOM now, but what does it
have that makes it uniquely valuable for testing CommonJS code that we
don't already have? Unless there is something, I don't see any reason
to bend over backwards to make QUnit fit in.
I don't think it's a matter of making QUnit fit in as much as aligning CommonJS's API with a very popular js unit testing framework. If, for instance, the two have contradicting definitions of key methods we're just confusing the js testing landscape, nascent as it may be.
But still, it may be bending over backward to switch to calling test() over exports.test* + run(). Is there any way to unify these notions?
Kevin Dangoor
But still, it may be bending over backward to switch to calling test() over exports.test* + run(). Is there any way to unify these notions?
Dean Landolt
I think it's all in the test runner. A test runner could be made to work either way or both ways simultaneously. I really don't see a significant difference...On Wed, Oct 7, 2009 at 9:53 PM, Dean Landolt <de...@deanlandolt.com> wrote:But still, it may be bending over backward to switch to calling test() over exports.test* + run(). Is there any way to unify these notions?
Part of what Unit Testing/A specifies is a test runner and a particular style for setting up tests. I don't necessarily buy that a unit testing spec needs to be compatible in style with QUnit (or any other test framework for that matter) but I think it'd be a worthy goal to shoot for. It sounds like folks have some ideas about having the best of both worlds with logging vs. fail fast -- I'm wondering if people have any similar ideas about test structuring.
Kris Kowal
> On Wed, Oct 7, 2009 at 9:53 PM, Dean Landolt <de...@deanlandolt.com> wrote:
>> But still, it may be bending over backward to switch to calling test()
>> over exports.test* + run(). Is there any way to unify these notions?
> I think it's all in the test runner. A test runner could be made to work
> either way or both ways simultaneously. I really don't see a significant
> difference...
The difference really is whether test modules are expressions or
programs. If we go with "test", then requiring a test module makes
that module a test program that runs as it is required, rather than a
test expression that can be run or manipulated out of band.
My pre-commonjs Chiron tests were programs, where there wasn't even a
closure-per-test; a module was the granular unit of testing (organized
with the module name space) and assertion logs were the fine unit.
This was about as simple as a framework can get.
http://code.google.com/p/chironjs/source/browse/trunk/src/test/base/dict.js
With the present proposal, objects are the granular unit, test
functions are the fine unit, assertions provide means to distinguish
failures from errors, and tests are *not* run-as-declared. The QUnit
proposal appears to have scripts as the granular unit, test functions
as the fine unit, with assertion logs and log counts providing failure
details, and they are run as they are declared.
Training the "test" function to construct a test expression that can
be run later would be complicated.
Kris Kowal
Kevin Dangoor
With the present proposal, objects are the granular unit, test
functions are the fine unit, assertions provide means to distinguish
failures from errors, and tests are *not* run-as-declared. The QUnit
proposal appears to have scripts as the granular unit, test functions
as the fine unit, with assertion logs and log counts providing failure
details, and they are run as they are declared.
Is that necessarily so? Perhaps they are collected as they are declared?
Training the "test" function to construct a test expression that can
be run later would be complicated.
var test = function(name, func) {
exports["test" + name] = func;
}
So, the QUnit implementation runs as they are declared, but that need not be a requirement of other test runners which can collect as declared...
Kris Kowal
On Oct 7, 2009, at 7:50 PM, Kevin Dangoor <dan...@gmail.com> wrote:
>
> var test = function(name, func) {
> exports["test" + name] = func;
> }
>
This presumes that an imported API (or injected, but that's not
relevant to a usual module) closes on the exports of each importing
module (which is not possible with the current module system). It is
however possible if we ammend the module system to have a
require.curryId decorator.
I don't really think these are practical.
Kris Kowal
Nathan Stott
Kris Kowal
> As a side note, I really wish it were possible to provide functions, like
> the 'test' in this case, to modules. It would open up so many possibilities
> for coding expressiveness.
We really could not do that with securable modules, but perhaps when
we have a specification for module loaders, we could work injection
into that layer. The loader.load(id) in our Ihab's and my January
proposal returned a module factory function. That function, in that
draft, accepted the values of "require" and "exports" to inject into
the module evaluation. I'm starting to think that it would be more
generally useful if it accepted an object that owned attributes that
should be injected into the module's scope. So, it would look like:
var factory = require.loader.load(id);
factory({require, exports, module, …});
So, with this convention it would be possible to use the module system
for DSL's like this without involving the module memo or affecting the
calling convention for general purpose modules. That's a topic for
another day though.
Kris Kowal
Hannes Wallnoefer
> the 'test' in this case, to modules. It would open up so many possibilities
> for coding expressiveness.
by module x into the local module scope. It can do this because it
works with pythonic aka top-level module scopes[1], which is a fully
compatible extension to the securable module spec. I still think this
wins head and shoulders over pure securable modules, and Helma NG code
is so much cleaner for it.
For example, compare the following two - and that's just the single
property case that require handles relatively well:
var Foo = require("foo").Foo;
include("foo");
[1] http://wiki.commonjs.org/wiki/Modules/PythonicModules (a module
proposal I wrote back then)
Hannes
Kevin Dangoor
Yes, that's certainly true. But, this one still makes my point:
var collectedTests = {};
exports.test = function(name, func) {
collectedTests[name] = func;
}
A test runner does not need to run the tests as it finds them...
Kevin Dangoor
2009/10/8 Nathan Stott <nrs...@gmail.com>:
> As a side note, I really wish it were possible to provide functions, likeHelma NG provides an include(x) that defines all properties exported
> the 'test' in this case, to modules. It would open up so many possibilities
> for coding expressiveness.
by module x into the local module scope. It can do this because it
works with pythonic aka top-level module scopes[1], which is a fully
compatible extension to the securable module spec. I still think this
wins head and shoulders over pure securable modules, and Helma NG code
is so much cleaner for it.
I do agree that this is a nice extension. As a frequent user of Python's "from foo import bar, baz, etc" syntax, it's nice that apps can do something similar.
Kevin
Wes Garland
I do agree that this is a nice extension. As a frequent user of Python's "from foo import bar, baz, etc" syntax, it's nice that apps can do something similar.
You know, if we amended the spec to say that properties of exports must be enumerable, users could do this:
function import(module, scope)
{
var exports = require(module).exports;
for (export in exports)
scope[export] = exports[export];
}
import("module", this);
Simply importing all module symbols into the program's global namespace is a mistake, if you ask me. GPSEE has a legacy include similar to Helma's, but we add a scope option, and by default tie it to the calling scope. This isolates the module from poluting the program's global scope, but still means that we can't have completely hidden symbols in the module. I prefer the stricter namespace of the require()/exports system for these reasons.
Wes
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102