renaming test-is?

[Stuart Halloway]

Rich mentioned some discomfort with the name "test-is" for a testing
framework to be added to clojure core. I want it in core, and don't
care much what it is called. :-)

Stuart (et al), are you open to a rename? Some random ideas:

Sierra
CLJUnit
XUnit
Cluje
Frak
Testify
Truthiness
Clue
Crank
Claim
Verily

[Stuart Sierra]

I don't care about the name. I'd be honored by the promotion. :) It
can be called clojure.test, cljtest, whatever. I guess I'd like to
keep the word "test" in the name somewhere, so it's obvious what it
does.

I would like to be able to continue working on it, probably by adding
more assertions and reporting methods.

-Stuart Sierra


On Jun 19, 11:11 am, Stuart Halloway <stuart.hallo...@gmail.com>
wrote:

[J. McConnell]

On Fri, Jun 19, 2009 at 1:11 PM, Stuart Halloway
<stuart....@gmail.com> wrote:
>
> Rich mentioned some discomfort with the name "test-is" for a testing
> framework to be added to clojure core. I want it in core, and don't
> care much what it is called. :-)
>
> Stuart (et al), are you open to a rename? Some random ideas:
>

> Testify

I like testify ... keeps "test" in the title. Plus, there is a notion
of false testimony, so it doesn't feel as much like lying as verily
would.

- J.

[Rich Hickey]

On Fri, Jun 19, 2009 at 1:18 PM, Stuart
Sierra<the.stua...@gmail.com> wrote:
>
> I don't care about the name.  I'd be honored by the promotion.  :)  It
> can be called clojure.test, cljtest, whatever.  I guess I'd like to
> keep the word "test" in the name somewhere, so it's obvious what it
> does.
>
> I would like to be able to continue working on it, probably by adding
> more assertions and reporting methods.
>
> -Stuart Sierra

clojure.test works for me.

Yes, please, one of the ideas behind moving to git was that authors of
contrib libraries that get promoted can easily continue working on
them. I envision we will use git pulls for these long-running
modification situations rather than patches.

Rich

[Stuart Halloway]

I have created forks to do all this work:

http://github.com/stuarthalloway/clojure
http://github.com/stuarthalloway/clojure-contrib

Plus a ticket:

http://www.assembla.com/spaces/clojure/tickets/131-Move-Clojure-tests-from-contrib-into-Clojure

And a wiki page:

http://www.assembla.com/wiki/show/clojure/Getting_Tests_Into_Clojure

[Stuart Halloway]

The forks now have working, tests-passing versions of Clojure and
contrib. Decisions made along the way are documented in the wiki page:

> http://www.assembla.com/wiki/show/clojure/Getting_Tests_Into_Clojure

and in the commit history of the forks:

> http://github.com/stuarthalloway/clojure
> http://github.com/stuarthalloway/clojure-contrib

The only "sore thumb" left is clojure.contrib.template. As things
stand, it is needed by clojure and contrib, but doesn't (IMO) pass the
generality test for inclusion in Clojure.

I don't care exactly how tests get into Clojure, but I am passionate
to see it happen somehow. If these forks are acceptable, great! If
not, I hope they will spur the conversation to quickly settle on
something else.

I am sure there are bits of cruft left over from the conversion. For
example, anything that has a manifest of packages (such as Craig
Andera's gen-html-docs) will need to update to match the new package
structure. I am happy to help clean this up, but the clock is ticking.
If we decide quickly, the forks will be easy to merge, otherwise
perhaps not.

Feedback welcomed!

Stu

P.S. It is worth mentioning that these forks were much easier to do
because of the (minimal) tests already in place, and would have been
even easier with a more complete test suite.

[Stuart Sierra]

On Jun 19, 2:48 pm, Stuart Halloway <stuart.hallo...@gmail.com> wrote:
> The only "sore thumb" left is clojure.contrib.template. As things  
> stand, it is needed by clojure and contrib, but doesn't (IMO) pass the  
> generality test for inclusion in Clojure.

FYI, clojure.contrib.template and clojure.contrib.walk are only
required for the "are" macro.

I don't know how many of the core language tests use "are". I rarely
use it, but some people really like it and have asked me not to remove
it.

templates were a wacky idea that I hoped might become generally
useful, but so far I don't think they've been used anywhere except in
test-is.

-Stuart Sierra

[Rich Hickey]

Could 'are' be broken out into, e.g. test-are, and left in contrib? Do
any core clojure tests use 'are'?

[Stuart Halloway]

FWIW, I find walk to be very much in the spirit of Clojure, and
deserving of core regardless of the needs of the test suite. I demo
walk almost every time I show Clojure.

On Fri, Jun 19, 2009 at 7:03 PM, Stuart
Sierra<the.stua...@gmail.com> wrote:
>

[Stuart Halloway]

There are several dozen tests that use "are". It wouldn't be hard to
change them, though. Options are:

(1) rewrite tests to use "is" (less than an hour of work), and put
test-are in contrib. This would let us leave template and walk in
contrib.

(2) leave tests as-is, bringing template and walk into core in some
fashion.

Rich, sounds like you would lean toward (1). Let me know and I will
get this done ASAP so you can merge the forks back in.

Stu

[Rich Hickey]

I briefly looked through the 'ares'. Many are of the form (are (= _1
_2) ...) which could be replaced with a dedicated are= macro.

However, it seems something like 'are' is desirable in order to avoid
a lot of duplication in the tests. I just don't like template.

Could are be defined in terms of anonymous fns? Instead of:

(are (instance? _2 _1) ...)

something like:

(are #(instance? %2 %1) ...)

There seems to be lot of overlap with #().

There's no hiding template once people are writing _1 _2 everywhere in tests.

Restructuring dozens of tests, making them more verbose in the
process, doesn't seem like a good idea.

Rich

On Sat, Jun 20, 2009 at 7:17 AM, Stuart
Halloway<stuart....@gmail.com> wrote:
>
> There are several dozen tests that use "are". It wouldn't be hard to
> change them, though. Options are:
>
> (1) rewrite tests to use "is" (less than an hour of work), and put
> test-are in contrib. This would let us leave template and walk in
> contrib.
>
> (2) leave tests as-is, bringing template and walk into core in some
> fashion.
>
> Rich, sounds like you would lean toward (1). Let me know and I will
> get this done ASAP so you can merge the forks back in.
>


>>

[Stuart Sierra]

On Jun 20, 7:08 am, Rich Hickey <richhic...@gmail.com> wrote:
> Could are be defined in terms of anonymous fns? Instead of:
>
> (are (instance? _2 _1) ...)
>
> something like:
>
> (are #(instance? %2 %1)  ...)

Not easily. The purpose of template is to do anonymous functions at
the syntactic level. You need the literal test expression inside the
"is" macro to get the right error reports. If it's an anonymous
function, the "is" macro doesn't know what kind of test you're doing.
Not a problem if all the tests pass, but harder to debug when they
fail.

The closest approximation with anonymous functions would be:

(map #(is (instance? %2 %1))
(partition 2 list-of-arguments))

-Stuart Sierra

[Stuart Halloway]

I would like to move "are" out of clojure.test, back into a
clojure.contrib.test-extensions or some such Two reasons:

(1) the implementation issues being discussed here

(2) "are" encourages the wrong kind of testing. Tests should test one
thing [1].

I wouldn't go so far as to say that a multi-assertion test is *always*
a bad idea, but making "are" convenient encourages the wrong kind of
tests. Also, I suspect that many of the "are" tests could be written
as a single "is" comparing two sequences.

Stu

[1] http://blog.astrumfutura.com/archives/388-Unit-Testing-One-Test,-One-Assertion-Why-It-Works.html

[Stuart Sierra]

On Jun 20, 10:33 am, Stuart Halloway <stuart.hallo...@gmail.com>
wrote:

> I would like to move "are" out of clojure.test, back into a
> clojure.contrib.test-extensions or some such

Ok by me.

>...

> (2) "are" encourages the wrong kind of testing. Tests should test one
> thing [1].

I agree, though the original reason for "are" was to run the same test
with several different arguments.

-Stuart Sierra

[Rich Hickey]

Are those separable, e.g. the reusable test from the multiple assertion?

Rich

[Stuart Sierra]

On Jun 20, 11:09 am, Rich Hickey <richhic...@gmail.com> wrote:
> Are those separable, e.g. the reusable test from the multiple assertion?

Yes. Not quite as pretty. Just define a macro:

(defmacro check-my-function [x y z]
`(is (= ~x (my-function ~y ~z)))

(deftest test-my-function
(check-my-function 4 5 6)
(check-my-function 5 6 7)
(check-my-function 6 7 8))

You could do this with functions too, but you lose some of the
reporting features. Specifically, you would know that the test as a
whole failed, but not which arguments caused the failure.

As another alternative, the argument to "are" could be a macro-like
function:

(are (fn [x y z] `(= ~x (my-function ~y ~z))
...arguments...)

Less elegant, potentially confusing, but achieves the same goal as
templates.

-Stuart Sierra

[Stuart Halloway]

What about:

(defmacro are [is-expr & arg-lists]
`(do
~@(map (fn [arg-list]
`(is (~is-expr ~@arg-list)))
arg-lists)))

This ties into whatever error messaging "is" does/will do in the
future, if you are willing to tolerate the calling convention:

(are =
[2 (+ 1 1)]
[4 (* 2 2)])
=> true

(are instance?
[String "foo"]
[String 10])
FAIL in ...
expected: (instance? String 10)
actual: java.lang.Integer
=> false

Stu

[Phil Hagelberg]

Stuart Halloway <stuart....@gmail.com> writes:

> I have created forks to do all this work:
>
> http://github.com/stuarthalloway/clojure
> http://github.com/stuarthalloway/clojure-contrib

Anything you could use help with, or is it not a very parallelizable task?

-Phil

[Stuart Halloway]

Hi Phil,

There isn't much implementation work here, but opinions on the choices
being made would be most welcome. Here's where things stand:

Pretty much agreed on (but feel free to question)

* move clojure.contrib.test-is to clojure.test
* move clojure.contrib.stacktrace to clojure.stacktrace (dependency of
* move clojure tests into clojure
* do *not* move clojure.contrib.template into core
* re-implement the tiny number of test that drag in other contrib
dependencies (java-util, combinatorics). I did individual commits for
these so that it will be easy to find them. Once we have a commit that
represents our go-forward strategy, knocking these off is totally
parallelizable.

Still to be decided:

* create a new version of 'are' that does not depend on
clojure.contrib.template, or change the clojure tests to use 'is'
instead of 'are'? Stuart Sierra and I have posted a few different
possible 'are's on this thread if you look back.

I am doing family stuff during the day today, but I should be able to
finish this tonight, once we decide what "this" is.

Stu

[Rich Hickey]

Another possibility is to pass the arity and drop the brackets:

(are [= 2]
2 (+ 1 1)
...)

(the vector is something 'are' could check for to ensure it is the new
syntax)

Personally I'm on the fence about this, some of the 'are' calls are
hard to visually parse, OTOH, putting brackets around single args is
silly. This would leave most of the tests pretty much intact.

This would still require some of the more elaborate tests to be made
into macros for reuse, not necessarily bad. 'is' already does
macroexpansion?

Rich

On Jun 20, 10:42 pm, Stuart Halloway <stuart.hallo...@gmail.com>
wrote:

[Stuart Halloway]

I am on the fence between these options. Stuart? Users of "are"?

Sent from my iPhone

[Mark Engelberg]

On Sun, Jun 21, 2009 at 2:38 PM, Stuart
Halloway<stuart....@gmail.com> wrote:
>
> I am on the fence between these options. Stuart? Users of "are"?
>

I like the way "are" saves typing. I don't like the way the _1 _2
syntax differs from the %1 %2 syntax of anonymous functions.

[Stuart Sierra]

On Jun 21, 12:02 pm, Rich Hickey <richhic...@gmail.com> wrote:
> This would still require some of the more elaborate tests to be made
> into macros for reuse, not necessarily bad. 'is' already does
> macroexpansion?

No, "is" does not do macroexpansion. "is" dispatches on the first
(unevaluated) symbol in the test expression. In effect, it's a mini-
language -- that's how it can do things like (is (thrown? ...)).

-Stuart Sierra

[Stuart Sierra]

On Jun 21, 3:38 pm, Stuart Halloway <stuart.hallo...@gmail.com> wrote:
> I am on the fence between these options. Stuart? Users of "are"?

Frankly, I don't like any of the alternatives to the original "are".
I think you have to accept "are" as it is, with templates, or leave it
out.

Some of the templates used in the language tests cannot be expressed
in the proposed alternatives, such as:
(are (< (- _1 _2) DELTA) ...)
(are (= (distinct [_ _]) [_]) ...)

For perspective, the language tests in Stuart Halloway's branch
contain:
278 uses of "is", and
165 uses of "are", of which
87 use "(are (= _1 _2))", and the rest use
64 distinct other templates.

So I will support one of 3 options:
1. Leave "are" as it is. Move template into Clojure along with test-
is.
2. Rewrite the language tests without "are". Leave "are" as an
extension in contrib.
3. Do not move test-is or language tests into Clojure at this time.

-SS

[Rich Hickey]

But it could macroexpand the test expr prior to dispatch, which would
be a no-op for things like thrown?, right?

Rich

[Stuart Sierra]

On Jun 21, 6:13 pm, Rich Hickey <richhic...@gmail.com> wrote:
> > No, "is" does not do macroexpansion.  "is" dispatches on the first
> > (unevaluated) symbol in the test expression.  In effect, it's a mini-
> > language -- that's how it can do things like (is (thrown? ...)).
>
> But it could macroexpand the test expr prior to dispatch, which would
> be a no-op for things like thrown?, right?

Yes, that should work.
-SS

[Konrad Hinsen]

On 20.06.2009, at 02:42, Stuart Halloway wrote:

> FWIW, I find walk to be very much in the spirit of Clojure, and
> deserving of core regardless of the needs of the test suite. I demo
> walk almost every time I show Clojure.

I agree. However, I'd take out the macroexpand-all function. While it
is a nice demonstration of how to use walk, it is not a faithful
implementation of how macro expansion works, because it doesn't treat
special forms specially.

Konrad.

[Stuart Halloway]

I am fine with option 1 or option 2 below, and happy to take the fork
to either of those destinations. Option 3 I don't care for much. :-)

Stu

[Stuart Sierra]

By the way, we might want to fork a contrib release to match what's in
"Programming Clojure" before we start moving things around.d
-SS

[Frantisek Sodomka]

Getting Clojure core tests from clojure-contrib to clojure itself is a
good idea and I definitely support it.

Some of my ideas:

1) 'are'
'are' is very useful and I like it very much. It simplifies lots of
writing and brings clarity, in my opinion.

For example in data_structures.clj, test-hash-set:

; equal and unique
(are (and (= (hash-set _) #{_})
(= (hash-set _ _) #{_}))
nil
false true
0 42
0.0 3.14
2/3
0M 1M
\c
"" "abc"
'sym
:kw
() '(1 2)
[] [1 2]
{} {:a 1 :b 2}
#{} #{1 2} )

Rewriting this just using 'is' would be really tedious and hard to
read. Ideal case is to have a function or template and just "throw
data at it". Adding more data adds more tests. Function/template
really has to be a macro in order to get the most information (0/1 and
0 is the same at runtime).

I vote to keep 'are'. Template could be included inside the new
clojure.test - it doesn't have to be a library by itself.

2) Removed tests from java-utils and combinatorics
- please do not delete them, just comment them out with the
appropriate message - if deleted, it will be hard to remember to get
them back in
- dependency on combinatorics is at this moment only for all-are - we
could include 'combinations' in test-utils for this support or include
all of clojure.contrib.combinatorics into the core

I am busy these days, but I will chime-in later.

Frantisek


On Jun 19, 10:48 pm, Stuart Halloway <stuart.hallo...@gmail.com>
wrote:

[Rich Hickey]

I definitely don't want template as-is. It does too much, has
confusing overlap with #(), and doesn't work similarly:

;I would expect this to consume 2 args:
(macroexpand '(do-template (check _2) :a :b :c :d))
-> (do (check :a) (check :b) (check :c) (check :d))

(macroexpand '(do-template (check _2 _2) :a :b :c :d))
-> (do (check :a :a) (check :b :b) (check :c :c) (check :d :d))

(macroexpand '(do-template (check _2 _2 _3) :a :b :c :d :e :f))
-> (do (check :a :a :b) (check :c :c :d))

The eval-only-once aspect is also perhaps too tricky. How often is
that leveraged by existing client code?

(macroexpand-1 '(template (let [x (think-hard)] (println "done") (+ _
x))))
-> (clojure.core/let [HOLE_749 (println "done")]
(clojure.core/fn [_1] (let [x (think-hard)] HOLE_749 (+ _1 x))))

Hrm.

However, the replacement facilities of template, and duplication
reduction of are, are useful and perhaps could be packaged
differently.

If templates took an arglist we could get out of _1, _2 magic naming
land:

(are [x] (and (= (hash-set x) #{x})
(= (hash-set x x) #{x}))
nil
false true
0 42 ...)

(are [x y] (< (- x y) DELTA) ...)
(are [x] (= (distinct [x x]) [x]) ...)

Existing tests could be converted by just inserting an arglist taking
_1 etc:

(are [_1 _2] (= _1 _2) ...)

If the eval-once capabilities are important, they could be explicit:

;proposed - macros can use metadata too!
(macroexpand-1 '(clojure.contrib.template/template [_]
(let [x #^lift (think-hard)]
(println "done")
(+ _ x))))

(clojure.core/let [LIFT_760 (think-hard)]
(clojure.core/fn [_]
(let [x LIFT_760]
(println "done")
(+ _ x))))

Thoughts?

Rich

[Stuart Sierra]

On Jun 23, 8:48 am, Rich Hickey <richhic...@gmail.com> wrote:
> I definitely don't want template as-is. It does too much, has
> confusing overlap with #(), and doesn't work similarly:
>
> ;I would expect this to consume 2 args:
> (macroexpand '(do-template (check _2) :a :b :c :d))
> -> (do (check :a) (check :b) (check :c) (check :d))

Good point, I hadn't thought of that.

> The eval-only-once aspect is also perhaps too tricky. How often is
> that leveraged by existing client code?

Rarely, if ever. It was a side-track thought on the way to templates.

> If templates took an arglist we could get out of _1, _2 magic naming
> land:
>
> (are [x] (and (= (hash-set x)   #{x})
>               (= (hash-set x x) #{x}))
>      nil
>      false true
>      0 42 ...)

I like that idea, I'll work on it.

-Stuart

[Stuart Sierra]

On Jun 23, 8:48 am, Rich Hickey <richhic...@gmail.com> wrote:
> If templates took an arglist we could get out of _1, _2 magic naming
> land:

Started working on this by rewriting c.c.template:
http://github.com/stuartsierra/clojure-contrib/tree/new-templates

Example:
(macroexpand '(do-template [x y] (+ y x) 2 4 3 5))
;;=> (do (+ 4 2) (+ 5 3))

Next, need to rewrite "are" to use that syntax.

-Stuart Sierra

[Stuart Sierra]

And here it is:
http://github.com/stuartsierra/clojure-contrib/tree/new-templates

test-is/are updated with new template syntax:

user> (are [x y] (= x (+ y y))
2 1
4 2
5 3)

FAIL in clojure.lang.PersistentList$EmptyList@1 (NO_SOURCE_FILE:1)
expected: (= 5 (+ 3 3))
actual: (not (= 5 6))

-Stuart Sierra

[Stuart Halloway]

Very cool. Should I convert the tests to use this, or are there issues
still undecided?

On Jun 23, 2009, at 3:53 PM, Stuart Sierra

[Stuart Sierra]

On Jun 23, 8:48 am, Rich Hickey <richhic...@gmail.com> wrote:
> The eval-only-once aspect is also perhaps too tricky. How often is
> that leveraged by existing client code?

It's used in clojure.contrib.condt, which was my template-based
version of what later became clojure.core/condp.

I removed condt from the new-templates branch on my github acct.
http://github.com/stuartsierra/clojure-contrib/tree/new-templates

I left in c.c.walk/macroexpand-all, only because swank-clojure depends
on it. The version in c.c.macro-utils is more correct.

-Stuart Sierra

[Rich Hickey]

I'm good with the new templates.

Rich

[Stuart Halloway]

I am porting all the tests to use the new template syntax on the
"gtic" branch of my clojure-contrib fork:

http://github.com/stuarthalloway/clojure-contrib/tree/gtic

My goal is to make a series of well-organized commits so that others
can easily review.

Stu

[Stuart Halloway]

I think I am done. I have updated the Wiki page in Assembla [1] to
document the things moving from contrib into Clojure, and to point to
commits that have everything moved and all tests passing.

Please have a look and let me know if I missed anything. It would be
nice to merge this soon so there is not extra work keeping it in sync.

Stu

[1] http://www.assembla.com/wiki/show/b4-TTcvBSr3RAZeJe5aVNr/Getting_Tests_Into_Clojure

[Rich Hickey]

Ok, this is merged into master on the Clojure side. We can do any
further finessing there. Someone should do contrib.

Thanks to everyone, especially both Stuarts!

Rich

On Jun 24, 1:18 am, Stuart Halloway <stuart.hallo...@gmail.com> wrote:
> I think I am done. I have updated the Wiki page in Assembla [1] to
> document the things moving from contrib into Clojure, and to point to
> commits that have everything moved and all tests passing.
>
> Please have a look and let me know if I missed anything. It would be
> nice to merge this soon so there is not extra work keeping it in sync.
>
> Stu
>

> [1]http://www.assembla.com/wiki/show/b4-TTcvBSr3RAZeJe5aVNr/Getting_Test...

[Stuart Halloway]

I have pushed the changes to contrib. Contrib tests all pass on my
box, but there is lots in contrib that is untested, so take a look at
your contribs. I am happy to help out if you find something weird.

Stu

[Konrad Hinsen]

On 26.06.2009, at 04:26, Stuart Halloway wrote:

> I have pushed the changes to contrib. Contrib tests all pass on my
> box, but there is lots in contrib that is untested, so take a look at
> your contribs. I am happy to help out if you find something weird.

With that change, contrib becomes incompatible with Clojure 1.0,
right? Then it's time to create the "compatible-1.0" branch.

Konrad.

[Tom Faulhaber]

On Thu, Jun 25, 2009 at 9:44 PM, Konrad Hinsen <konrad...@laposte.net> wrote:

With that change, contrib becomes incompatible with Clojure 1.0,
right? Then it's time to create the "compatible-1.0" branch.

I was thinking the exact same thing.