http://wiki.commonjs.org/wiki/Unit_Testing/A
As a reminder, I incorporated our consensus on the argument order and
default messages. I've also opted to simplify the API, removing some
functions that seem convenient but are actually superfluous. The
idioms for isTrue, isFalse, eqTrue, isNaN, isntNan, &c, all boil down
to:
assert.is(true, actual, message);
assert.is(false, …);
assert.is(false, …);
assert.eq(true, …); // truthy
assert.is(NaN, …);
assert.ni(NaN, …);
So, the functions I've opted to include were "eq", "ne", "is", "ni"
(which is new, but also funny, which I think makes it easier to
remember, and analogous to "ne" in at least a couple ways). I also
left a note that, yes, eventually it would be really cool to be able
to send test results to an object implementing a standard logging
interface. However, to get this out the door, I think we can safely
defer that to a future discussion and a version 2 spec.
Is this something we can ratify and start using? Once this is real,
we can start standardizing unit tests in our specification proposals.
Please show your hand on the Wiki.
http://wiki.commonjs.org/wiki/Unit_Testing/A
Kris Kowal
* throws() might be more self-explanatory than error().
>
> IMHO the direction with all these methods adds a pile of unnecessary
> cruft, and misses out on stuff that's actually useful.
>
> When doing tests assert() and assert.not() has been enough to handle
> all
> those cases listed there.
> I don't see the point in obfuscating a few === or =='s (reading a
> method
> isn't quite clear on that) which could be written out on their own
> easily enough.
>
> While writing tests for Wrench.js the big trouble that isn't even
> covered here was arrays, which resulted in me creating:
> assert.match(arr, ["foo", "bar", "baz"]);
>
> So:
> -1 To .eq, .ni, abbreviations, and all these superfluous methods
> +1 To thought on methods like assert.match to deal with things you
> need
> to test that aren't easy to do with reliable builtin methods (like
> comparing arrays based on if their contents match).
Thats my votes too.
As always when testing comes up, I will pimp the function I found on
the internet to do just that:
Ah, yeah, here are Narwhal's:
http://github.com/tlrobinson/narwhal/tree/master/tests/
Kris Kowal
Agreed on superfluous .isEqual, .isTrue, &c. They're the least worthy
kind of cruft. These are equivalent:
assert.is(true, /^.$/.test('a'), 'regexp dot matches char');
assert.isTrue(/^.$/.test('a'), 'regexp dot matches char');
That's almost no savings in readability or expressiveness, or even
brevity. It just burdens the API with an open-ended list of functions
named after constants that all ultimately forward to assert.is and
assert.eq.
As for assert.match, the specification for "assert.eq" does the job,
and can be backed by Ash's equiv.
assert.eq(["foo", "bar", "baz"], ["foo", "bar", "baz"]); // equal but
assert.ni(["foo", "bar", "baz"], ["foo", "bar", "baz"]); // not identical
Kris Kowal
assertTrue() is used 51 times in our tests, so there must be some use
for it for some people. (Among these are 25 uses in Narwhal's
bytearray/bytestring tests, btw.)
>>
>> While writing tests for Wrench.js the big trouble that isn't even
>> covered here was arrays, which resulted in me creating:
>> assert.match(arr, ["foo", "bar", "baz"]);
The equal assertion should do a deep comparison of arrays and return
true for arrays with the same contents.
Hannes
Unfortunately, it would also defeat the aim of making it easily
extensible, apart from monkey patching. I'm still a proponent of
simplicity. Most tests can be broken down to assert.is(expected,
actual, message) or assert.throws(block, [Error]), and anything else
is really convenience that can be added through higher level API's.
Custom assertions need only be functions that throw
AssertionError(expected, actual, message), which seems like the
logical boundary for extension.
Kris Kowal
On Tue, 15 Sep 2009 18:15:35 -0700, Kris Kowal <cowber...@gmail.com>
wrote:
And i think this is the pertient point. We're never going to go get
everything we want, so the Unit test spec, which should be used in the
interop tests. should be simple:
assert.is(expected, actual);
assert.is(expected, actual, message);
assert.throws(block);
assert.throws(block, errorCtor);
assert.throws(block, message);
assert.throws(block, errorCtor, message);
I'm not sure we even need to say how we add other asserts to the core, if
we just mandate that they |throw new require('test').AssertionError( {} ) |
on error. I suggest we use a single object as a dictionary, rather than
un-named params so we can add/extend params in the future.
That way, you can do (assuming you write my-assert.js right:
var check = require('my-assert').assert;
export.testFoo = function() {
check(1).is(2, "1 is 2"); // Will throw new
require('test').AssertionError({got: 1, wanted: 2, message: "1 is 2"});
}
I think this approach gives us the best of both worlds:
1) A simple spec that we can use in the interop tests
2) Allow people to write their own test frameworks
3) But have them work together co-operatively:
var assert = require('test').assert;
var check = require('my-assert').check;
export.testFoo = function() {
assert(2,2, "2 is 2");
check(1).is(2, "1 is 2"); // throw new
require('test').AssertionError({got: 1, wanted: 2, message: "1 is 2"});
}
Please say you are all happy with this, and i will wiki it into a proposal.
+1
But I would add that you should probably also standardize on the base object keys AssertionError can take. And just to bikeshed a little on the names, I really don't like "got" and "wanted". I would think "wanted" should be "expected" and got should be something like "actual" or "relieved" -- a little longer but more clear.
+1
I do think that equivalence and non-equivalence are still basic enough
that *everyone* will need them in their tests. If you mean that "is"
means "equivalence", I'd hope that it be distinguished. I think of
"is" as "identity" (as it means in Python) and "eq" as "equivalence"
(as it means in Perl, shell, and others), and has many complements
like "ne", "lt", "gt", "le", and "ge". I'm not adamantly against
verbose equivalents, but I think that these are clear and widespread.
I am ready to compromise at this point all the way down to this exact
proposal, or even different names for these extensions, but please at
least have a deep equivalence and non-equivalence assertions in the
standard.
Kris Kowal
If the implementation of assert.is() is done with a simple === check,
it won't work for NaN. Because NaN !== NaN.
The assert.is() implementation would then have to have special handling for NaN.
I therefore raise my hand for an assert.isNaN(actual, message).
Ates
Well, after finishing reading through the entire thread and seeing the
general direction in keeping the gamut small, I guess one can always
use:
assert.is(true, isNaN(actual), msg);
Ates
I've cut "ni" and "ne". That leaves "is" and "eq". I've also renamed
"error" to "throws"; I think that's fine.
I presume "is" is not contentious.
I see that both Ash and Hannes have pending intentions, but let's hear
the nominations for alternate names and features we can't live
without. When replying on this thread, please augment the list of
recommendations for each name so we accumulate a list.
A.) no "eq" function.
B.) "eq"
C.) "equals"
D.) "equiv"
E.) "equivalent"
Z.) no "ne" function
Y.) "ne", Kris Kowal
X.) "notEquals"
W.) "notEquiv"
V.) "notEquivalent"
1.) no "isnt" function
2.) "isnt"
3.) "ni" not identical
4.) "notIdentical"
Kris Kowal (B, Z or Y, 1 2 or 3)
> * As long as the equality helpers themselves are spec'd and everyone agrees
> on those too
I think we should respect NaN != NaN in deep comparision. [NaN] !=
[NaN] for the same good reasons NaN != NaN or 1/0 != 2/0. It's a
harbinger of math that has lost it's meaning and is no longer useful
for such things as equivalence comparison.
Java objects throw a monkey wrench, or sabot if you're into monkey
wrenches as I know some of you are, in the machinery. I think we can
safely leave these objects outside the realm of specification.
You're right that #4 is very first draft, though. For example new
Date(0) != new Date(0), but we should treat them as equivalent.
Anybody who's willing to rewrite this section from the perspective a
vetted equivalence comparison library should.
Then there's the issue of polymorphism. Eventually I'd like to
augment this specification to defer to leftHandSide[eqMethodName] if
it exists in the prototype chain but not owned by a literal Object.
But, in the interest of getting this ratified quickly, I think we can
leave that for version 2.
Kris Kowal
http://philrathe.com/articles/equiv handles comparing NaN to NaN and
Date(0) to Date(), both of which return true.
>
> Then there's the issue of polymorphism. Eventually I'd like to
> augment this specification to defer to leftHandSide[eqMethodName] if
> it exists in the prototype chain but not owned by a literal Object.
> But, in the interest of getting this ratified quickly, I think we can
> leave that for version 2.
>
> Kris Kowal
I've just updated the wiki page (
http://wiki.commonjs.org/wiki/Unit_Testing/A ) to change the parameters of
the AssertionError constructor from 3 arguments to a single dict/object
with named keys, and added a section on how to write custom assert
(modules).
-ash
On Fri, 18 Sep 2009 10:52:39 +0100, Ash Berlin
<ash_flu...@firemirror.com> wrote:
>
> http://philrathe.com/articles/equiv handles comparing NaN to NaN and
> Date(0) to Date(), both of which return true.
>
Typo - that was meant to be Date(0) to Date(0)
On Thu, 17 Sep 2009 14:39:52 -0700, Kris Kowal <cowber...@gmail.com>
wrote:
>
> http://wiki.commonjs.org/wiki/Unit_Testing/A
>
> A.) no "eq" function.
> B.) "eq"
> C.) "equals"
> D.) "equiv"
> E.) "equivalent"
>
> Z.) no "ne" function
> Y.) "ne", Kris Kowal
> X.) "notEquals"
> W.) "notEquiv"
> V.) "notEquivalent"
>
> 1.) no "isnt" function
> 2.) "isnt"
> 3.) "ni" not identical
> 4.) "notIdentical"
Kris Kowal (B, Z or Y, 1 2 or 3)
Ash Berlin (B, Y, 2)
But I worry that the spec |eq| doesn't cover enough cases. I would be
happier if it was either far simpler (just compares arguments using |x ==
y| no deepness etc.), or (my prefered behaviour) covers all the cases in
the equiv[0] module. (Yes I know I keep carping on about that module, but
I've been using it for a while and found it to be very good.)
[0] We're all lazy, so here's that link again:
http://philrathe.com/articles/equiv#traps Some of these shouldn't affect us
as the are IE bugs, but the list is still good.
Ates Goral (B, Y or neq, 2)
>
> I agree we should have a basic "isTrue" type function, both
> "assert.equal(foo < bar, true)" and "if (foo >= bar) assert.fail()"
> are blehhh.
And both of these give rubbish diagnostics. The aim with this thread
is to agree on a *minimal* set of asserts that we use in the
interoperability test suite, and have a defined way for other people
to add
>
> Can we just have the top level "assert" object be a function that
> checks it's first argument is true (or truthy?)
This could work.
>
> Of course then we'd have to do var assert = require("test/
> assert").assert; but that wouldn't be a dealbreaker for me, at least.
>
> Have we decided whether equal/notEqual should be strict or not (===
> and !== vs. == and !=), or have both somehow?
Judging be the presence of same/notSame, I took equals to be strict
(===) but this is one problem with the name 'equals' - its not clear
if its === or == based.
>
> Also, just throwing this out there, not sure if I like it myself, but:
>
> assert.equal
> assert.not.equal
> assert.same
> assert.not.same
>
> i.e. have the inverses under the "not" namespace. (doesn't work very
> well for assert.not.throws, etc though)
I prefer the assert.notX form.
assert.isTrue(value, [message])
assert.isFalse(value, [message])
assert.isEqual(value1, value2, [message])
assert.notEqual(value1, value2, [message])
assert.isIdentical(value1, value2, [message])
assert.notIdentical(value1, value2, [message])
assert.throws(func, error, [message])
assert.fail([message])
Rational: More verbose is better. This way we say exactly what we
mean, without introducing any possible ambiguities and potential
confusion.
I agree we should have a basic "isTrue" type function, both
"assert.equal(foo < bar, true)" and "if (foo >= bar) assert.fail()"
are blehhh.
Can we just have the top level "assert" object be a function that
checks it's first argument is true (or truthy?)
Of course then we'd have to do var assert = require("test/
assert").assert; but that wouldn't be a dealbreaker for me, at least.
>
> But why? Looking at SecurableModules it doesn't specify this as
> illegal:
>
> var assert = exports = function(condition, message){
> if(!condition)
> throw new Error("assert failed" + (message ? ": " + message :
> ""));
> };
>
> assert.equal = function ...
> assert.same = function ...
> assert.AssertionError ...
>
> SecurableModules requires "exports" to be an object -- I'd take that
> to mean it could also be a function. This would render the var
> assert = ("test/assert").assert is unnecessary:
My understanding was however that you can't replace the exports
object. The return of require is the original state of the exports
object when the module 'started'. But i can't find the wording that
made me thing about that. Point 1.3 on Modules/SecurableModules kind
of implies this is the case. I know that Flusspferd and Narwhall
behave this way (exports is the initial variable.) I'm fairly certain
that GPSEE does as well.
>
> var assert = require("test/assert");
> assert(truthyTest, "some message");
> // or
> assert.equal(actual, expected, "some message");
>
> I'm already using this trick in persevere and it works nicely. I
> haven't checked in narwhal or any other SecurableModules
> implementations but I can't see why it wouldn't.
>
> And honestly, if we can agree on a spec for AssertionError's
> behavior, I don't see why we need to get all fancy and agree on
> isTrue, notSame, etc. -- the above simple assert function is Good
> Enough for standard lib unit tests (with the addition of a vetted
> equality test lib as noted by Kris Kowal).
Yes! This is the point I've been trying to make. It might be worth not
specing *any* asserts, just the Assertion errors, and the interop
tests define a minimal set that they use.
>
> Can someone be more specific about exactly what we gain from a
> diagnostics standpoint by specing every possible comparison under
> the sun? I'm sure there are some benefits to be had, but can we get
> some specifics?
Its pretty much a case of this
asset.lt(4, 3);
// Expected: < 3
// Actual: 4
except with a nicer message or something. But to restate my point that
Dean also made. Shall we just agree on the format ofthe AssertionError
exception, and the basic format of test files (exports.testX,
test.runner(exports), etc.), and then just write some asserts for the
interop tests as neede?
From the spec:
2) In a module, there is a free variable called "exports", that is an
object that the module may add its API to as it executes.
3) modules must use the "exports" object as the only means of exporting.
The module must use the exports object that the implementation
provided to the module scope. That way the implementation can keep a
reference to the exports object before the module has finished
executing, mainly in order to handle potential cross dependencies
between modules. To be even clearer, maybe 3) should say: modules must
use *that* "exports" object as the only means of exporting.
> Can someone be more specific about exactly what we gain from a diagnostics
> standpoint by specing every possible comparison under the sun? I'm sure
> there are some benefits to be had, but can we get some specifics?
If the goal is solely to agree on a unit testing module for testing
the standard library itself then a minimalist approach is fine. If the
goal is to provide a unit testing module as part of the standard
library api then I think more verbose is better.
Chris
My understanding was however that you can't replace the exports
On 20 Sep 2009, at 19:10, Dean Landolt wrote:
>
> But why? Looking at SecurableModules it doesn't specify this as
> illegal:
>
> var assert = exports = function(condition, message){
> if(!condition)
> throw new Error("assert failed" + (message ? ": " + message :
> ""));
> };
>
> assert.equal = function ...
> assert.same = function ...
> assert.AssertionError ...
>
> SecurableModules requires "exports" to be an object -- I'd take that
> to mean it could also be a function. This would render the var
> assert = ("test/assert").assert is unnecessary:
object. The return of require is the original state of the exports
object when the module 'started'. But i can't find the wording that
made me thing about that. Point 1.3 on Modules/SecurableModules kind
of implies this is the case. I know that Flusspferd and Narwhall
behave this way (exports is the initial variable.) I'm fairly certain
that GPSEE does as well.
Yes! This is the point I've been trying to make.
>
> var assert = require("test/assert");
> assert(truthyTest, "some message");
> // or
> assert.equal(actual, expected, "some message");
>
> I'm already using this trick in persevere and it works nicely. I
> haven't checked in narwhal or any other SecurableModules
> implementations but I can't see why it wouldn't.
>
> And honestly, if we can agree on a spec for AssertionError's
> behavior, I don't see why we need to get all fancy and agree on
> isTrue, notSame, etc. -- the above simple assert function is Good
> Enough for standard lib unit tests (with the addition of a vetted
> equality test lib as noted by Kris Kowal).
From the spec:
On Sun, Sep 20, 2009 at 20:10, Dean Landolt <de...@deanlandolt.com> wrote:
> On Sat, Sep 19, 2009 at 5:56 AM, Tom Robinson <tlrob...@gmail.com> wrote:
>>
>> I agree we should have a basic "isTrue" type function, both
>> "assert.equal(foo < bar, true)" and "if (foo >= bar) assert.fail()"
>> are blehhh.
>>
>> Can we just have the top level "assert" object be a function that
>> checks it's first argument is true (or truthy?)
>>
>> Of course then we'd have to do var assert = require("test/
>> assert").assert; but that wouldn't be a dealbreaker for me, at least.
>
> But why? Looking at SecurableModules it doesn't specify this as illegal:
>
> var assert = exports = function(condition, message){
> if(!condition)
> throw new Error("assert failed" + (message ? ": " + message : ""));
> };
2) In a module, there is a free variable called "exports", that is an
object that the module may add its API to as it executes.
3) modules must use the "exports" object as the only means of exporting.
The module must use the exports object that the implementation
provided to the module scope. That way the implementation can keep a
reference to the exports object before the module has finished
executing, mainly in order to handle potential cross dependencies
between modules. To be even clearer, maybe 3) should say: modules must
use *that* "exports" object as the only means of exporting.
If the goal is solely to agree on a unit testing module for testing
> Can someone be more specific about exactly what we gain from a diagnostics
> standpoint by specing every possible comparison under the sun? I'm sure
> there are some benefits to be had, but can we get some specifics?
the standard library itself then a minimalist approach is fine. If the
goal is to provide a unit testing module as part of the standard
library api then I think more verbose is better.
On Sun, Sep 20, 2009 at 2:57 PM, Chris Zumbrunn <chris.z...@gmail.com> wrote:From the spec:
On Sun, Sep 20, 2009 at 20:10, Dean Landolt <de...@deanlandolt.com> wrote:
> But why? Looking at SecurableModules it doesn't specify this as illegal:
>
> var assert = exports = function(condition, message){
> if(!condition)
> throw new Error("assert failed" + (message ? ": " + message : ""));
> };
2) In a module, there is a free variable called "exports", that is an
object that the module may add its API to as it executes.
3) modules must use the "exports" object as the only means of exporting.
The module must use the exports object that the implementation
provided to the module scope. That way the implementation can keep a
reference to the exports object before the module has finished
executing, mainly in order to handle potential cross dependencies
between modules. To be even clearer, maybe 3) should say: modules must
use *that* "exports" object as the only means of exporting.
Damn. I understand now, but still -- it's such an awesome feature, and I can't think of any other way to implement it without resorting to some kind of wunderbar magic...
Ideas anyone (without changing anything about SecurableModules)? I admit this feature gains us nothing but a little clarity, but it seems so tantalizingly close it should be possible.
On 20 Sep 2009, at 20:15, Dean Landolt wrote:On Sun, Sep 20, 2009 at 2:57 PM, Chris Zumbrunn <chris.z...@gmail.com> wrote:
> But why? Looking at SecurableModules it doesn't specify this as illegal:
>
> var assert = exports = function(condition, message){
> if(!condition)
> throw new Error("assert failed" + (message ? ": " + message : ""));
> };
From the spec:
2) In a module, there is a free variable called "exports", that is an
object that the module may add its API to as it executes.
3) modules must use the "exports" object as the only means of exporting.
The module must use the exports object that the implementation
provided to the module scope. That way the implementation can keep a
reference to the exports object before the module has finished
executing, mainly in order to handle potential cross dependencies
between modules. To be even clearer, maybe 3) should say: modules must
use *that* "exports" object as the only means of exporting.
Damn. I understand now, but still -- it's such an awesome feature, and I can't think of any other way to implement it without resorting to some kind of wunderbar magic...
I'm not even aware of how to do this *with* wunderbars. I know spidermonkey (and rhino) has the __noSuchMethod__ one, but that doesn't work on the object itself. I'm not even sure if its possible to turn a normal Object into something callable via an C APIs.
Ideas anyone (without changing anything about SecurableModules)? I admit this feature gains us nothing but a little clarity, but it seems so tantalizingly close it should be possible.Yeah, I've often wanted this sort of feature. I can think of a few tricks to let you change the value of exports and still have the 'part completed circular deps' dependancy be satisified.
Ideas anyone (without changing anything about SecurableModules)? I admit this feature gains us nothing but a little clarity, but it seems so tantalizingly close it should be possible.Yeah, I've often wanted this sort of feature. I can think of a few tricks to let you change the value of exports and still have the 'part completed circular deps' dependancy be satisified.
Do tell -- I think there's real clarity to be gained from modules being callable.
> This is what I was suggesting as the wunderbar solution, but as far
> as I can tell, in order for it to work across implementations the
> __callable__ (or whatever else) key would have to be spec'd in
> SecurableModules -- and changing SecurableModules, the only ratified
> spec, is probably out of the question (not to mention the fact that
> introducing this kind of impure magic would probably upset some
> folks).
Yes it would need to be added to the spec. There's no other way right
now.
> That leaves me to wonder, is there another way? Does replacing
> "exports" definitely violate the spec?
It doesn't *violate* the spec, but it's not defined, so
implementations are *not* required to support it, thus you cannot rely
on it in CommonJS compatible modules.
>
> As I've inadvertently hijacked this thread (uh, it was about unit
> testing, right?), so this is the last I'll say on the subject -- it
> looks like Ash started another thread so I'll take up this
> conversation there.
>
> So back on topic, what is the absolutely BARE minimum a ServerJS-
> conformant "assert" implementation should provide? Ash says sane
> equivalence testing and a standardized AssertionError. What else is
> really needed?
In my opinion:
truthy ("assert", "isTrue")
identity (===, "is", "same")
equivalence ("eq", "equiv")
throws exception ("throws")
and the inverses of these four.
Kris Kowal
+1
Unfortunately that kind of require is fundamental to SecurableModules.
Because if you write a File and Path class in two separate modules and
they each use a method that uses the other class (such as
complementary toFile and toPath conversions) the only way to get those
classes is to require them.
~Daniel Friesen (DanTMan, Nadir Seen Fire) [http://daniel.friesen.name]
On 20-Sep-09, at 11:25 AM, Ash Berlin <ash_flu...@firemirror.com>
wrote:
>
>
> On 20 Sep 2009, at 19:10, Dean Landolt wrote:
>
>>
>> But why? Looking at SecurableModules it doesn't specify this as
>> illegal:
>>
>> var assert = exports = function(condition, message){
>> if(!condition)
>> throw new Error("assert failed" + (message ? ": " + message :
>> ""));
>> };
>>
For those interested, we've had this discussion before:
http://groups.google.com/group/commonjs/browse_thread/thread/6a6a4fe4ce4c0200/0673da702ddfc50d?#0673da702ddfc50d
Yes, the exports object can't be replaced. I ran into the unfortunate
reason why while implementing require in MonkeyScript Lite.
If you require something and that module retuires or even leads to a
chain that ends up requiring the module you are already in the middle
of requiring the script won't have finished and you need to either
return a partial (Impossible if you allow it to be replaced) or throw
a failure error and tell the programmer not to loop like that.
Unfortunately that kind of require is fundamental to SecurableModules.
Because if you write a File and Path class in two separate modules and
they each use a method that uses the other class (such as
complementary toFile and toPath conversions) the only way to get those
classes is to require them.
In pure-js module loaders, this would involve injecting an expression
that returns the new exports value and reassigning it to the module
instance memo.
// "text" is the content of the module file.
// "id" is the top-level identifier of the module
evaluate = function (text, id) {
return new Function(
"require",
"exports",
"module",
text + "\n//*/\nreturn exports" // hack
);
};
modules[id] = exports;
var factory = loader.load(id);
modules[id] = factory(Require(id), exports, Module(id, path));
That is to say that this feature would be a hack, none worse than any
we've committed in the past.
However, my intention is to support the idea of standardizing a secure
module loader for ES-Harmony in terms of a "hermetic evaluator", an
"eval" that the security folks can trust to construct modules from
suspicious code. The hermetic evaluator would take a program and a
list of names to inject in its lexical scope, and return a function
that executes the program with particular respective values in a
deeply frozen global scope. That, and variations of that idea, would
be an elegant design with many uses. Twisting the hermetic evaluator
so that it returns the last known values of the injected
variables…would be hard to fathom and I don't think it would fly.
Kris Kowal
If you implement CommonJS, you can support that feature. Just don't
expect other implementations to support it.