Dynamic test creation?

[AndyK]

I am setting up tests with clojure.test that are driven by a CSV where
each line represents one test case. Right now, there is a single
deftest function that runs all the assertions. That's ok but creates
reporting like 1 test was run with 1000s of assertions. Clojure being
so dynamic, is it possible to create tests on-the-fly and run them
where each dynamic test represents each row so that the reporting says
X tests (where X == number of CSV rows).

I'm fairly new to clojure and quite unfamiliar with the ins-and-outs
of clojure.test.
Any pointers here would be appreciated.

Thank you

[Nate Young]

It absolutely would be possible, and furthermore this is an area where
macros really shine.

I would choose macros because from what you describe, it sounds like
you'd like to write a program that generates a bunch of deftest forms,
and then runs those tests. But you need language facilities like reading
from a csv file in order to do so. Clojure (and indeed all lisps) give
you this ability.

You could write a macro that reads in the CSV file and for each line,
generates a deftest form. Below is a quick sketch of what it might look
like if your CSV file consisted of two columns of values that were
supposed to be equal to each other.

(use 'clojure.test)
(require '[clojure.java.io :as io]
'[clojure.data.csv :as csv])

(defn testdef-form [n [expected actual]]
`(deftest ~(str "testfromline" n)
(is (= ~expected ~actual))))

(defmacro defcsvtests [filename]
(with-open [in-file (io/reader "in-file.csv")]
(let [testdefs (csv/read-csv in-file)]
`(do ~@(map testdef-form (iterate inc 1) testdefs)))))

(defcsvtests "test-cases.csv")

[AndyK]

How would (run-tests 'my-namespace) know to run all those dynamic
tests? I thought that it parsed the namespace for 'my-namespace when
you call it. Or is it that the call to defcsvtests sets off a chain of
macro resolutions before run-tests can even do its thing (so that it
appears to run-tests like a file full of deftests)?

[Nate Young]

On 11/01/2011 03:05 PM, AndyK wrote:
> How would (run-tests 'my-namespace) know to run all those dynamic
> tests? I thought that it parsed the namespace for 'my-namespace when
> you call it. Or is it that the call to defcsvtests sets off a chain of
> macro resolutions before run-tests can even do its thing (so that it

Right. Its that the macro-expansion phase actually reads from the csv
file in order to create a number of deftest forms, and each one then
gets evaluated, so after you've evaluated (defcsvtests), then
(run-tests) will be able to find a number of tests to run.

[AndyK]

I finally had a chance to try this out and it fails with

error: java.lang.ClassCastException: java.lang.String cannot be cast
to clojure.lang.IObj
Compilation failed.

When I substituted in something like this...

(defn prn-form [n scenario]
`(prn ~(str "foo" n) (prn ~(str n " :: " scenario))))

the file did compile.

Is the fact that deftest is also a macro going to cause a problem with
the original idea?

[Alex Baranosky]

Sounds lile you could use Midje's tabular tests.  Or if you want write a acro to generate a tabular fact.  Tje tabular fact will give you good reporting.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[AndyK]

Questions about tabular tests...
* can they take a lazy-seq as input?
* will the tests be parallelized? (i have anywhere from 10k-20k tests
to run)

On Nov 8, 4:16 pm, Alex Baranosky <alexander.barano...@gmail.com>
wrote:

> Sounds lile you could use Midje's tabular tests.  Or if you want write a
> acro to generate a tabular fact.  Tje tabular fact will give you good
> reporting.

[Brian Marick]

On Nov 9, 2011, at 8:37 AM, AndyK wrote:

> Questions about tabular tests...
> * can they take a lazy-seq as input?

Do you mean something like having a computation that generates a stream of test inputs and feeds them into a test evaluator? If so, no: all `tabular's` work happens at compile/macroexpansion time.

However, the underlying test-execution-and-reporting function is available and takes test descriptions as maps, so you could do something like

(map midje.unprocessed/expect* (source-of-test-maps))

> * will the tests be parallelized? (i have anywhere from 10k-20k tests
> to run)

The code doesn't make any effort to parallelize tests. Tests are normally run as they're encountered at load time. (Unlike clojure.test, the `fact` macro doesn't stash a test-function away for later execution.)

A test runner like the above should be parallelizable, though I didn't make any special effort to ensure that. The default reporter just prints results. You'd probably want one that stashes them away in an atom as they come in. There's already a way to swap in different reporters, so that shouldn't be too hard.

More discussion should probably happen at http://groups.google.com/group/midje

-----
Brian Marick, Artisanal Labrador
Now working at http://path11.com
Contract programming in Ruby and Clojure
Occasional consulting on Agile

[Nate Young]

On 11/08/2011 12:44 PM, AndyK wrote:
> I finally had a chance to try this out and it fails with
>
> error: java.lang.ClassCastException: java.lang.String cannot be cast
> to clojure.lang.IObj
> Compilation failed.

I think I fat-fingered my deftest-form definition:

I originally wrote:
(defn testdef-form [n [expected actual]]
`(deftest ~(str "testfromline" n)
(is (= ~expected ~actual))))

But it should probably be:

(defn testdef-form [n [expected actual]]

`(deftest ~(symbol (str "testfromline" n))
(is (= ~expected ~actual))))

The original, wrong, function would have produced something like this:
(deftest "testfromline27"
(is (= 5 2)))

Which is probably what's giving you the error message you're seeing.
That string should be a symbol:
(deftest testfromline27
(is (= 5 2)))

> When I substituted in something like this...
>
> (defn prn-form [n scenario]
> `(prn ~(str "foo" n) (prn ~(str n " :: " scenario))))
>
> the file did compile.

I'd have to see the rest of your code to comment on what's going on here.

> Is the fact that deftest is also a macro going to cause a problem with
> the original idea?

Macros certainly have composability issues, but its certainly safe to
write a macro that produces some other macro form. In fact, lots of
clojure.core macros use this to great effect (affect?), see ->, cond,
or, and .. to name a few.