[ANN} TestGarden

[Joel VanderWerf]

http://github.com/vjoel/test-garden

TestGarden is a test framework based on Wrong. It's not mature enough to
declare version 0.0.0 or to announce on ruby-talk. I'd relish feedback
from Wrong people.

Synopsis:

require 'test-garden'

test Thing do
thing = Thing.new; teardown {thing.cleanup()}
assert {thing.ok?}

test "assign foo" do
thing.foo = "baz" # does not affect foo in subsequent tests
assert {thing.foo == "baz"}
end

test "compare foo in two instances" do
thing2 = Thing.new; teardown {thing2.cleanup()}
assert {thing.foo == thing2.foo}
end
end

Run the test like so:

ruby test_thing.rb [-v | --verbose] [topic topic ...]

If no topics are given, the verbose output is:

T: Thing
T: Thing: assign foo
T: Thing
T: Thing: compare foo in two instances
4 passed, 0 failed, 0 skipped, 0 errors in Thing

See the README and examples for more details.

[Phlip]

> TestGarden is a test framework based on Wrong. It's not mature enough to
> declare version 0.0.0 or to announce on ruby-talk. I'd relish feedback from
> Wrong people.

>    test Thing do

>      thing = Thing.new; teardown {thing.cleanup()}
>      assert {thing.ok?}

Okay, now take off the assert{}. All lines must return true, else
Wrong swoops down and examines the failing line, in context.

This includes the Assemble lines and Activate lines too. Why would you
assemble a None just to test it? And the activate line can say None ==
activate(), to correct its polarization.

--
  Phlip
  http://c2.com/cgi/wiki?ZeekLand

[Phlip]

Another question: assert{} can do this, right?

>>      assert(' thing not okay '){ thing.ok?}

meaning it can pump out the given string just above the fault diagnostic, right?

[Joel VanderWerf]

On 10/08/2010 08:57 PM, Phlip wrote:
>> TestGarden is a test framework based on Wrong. It's not mature enough to
>> declare version 0.0.0 or to announce on ruby-talk. I'd relish feedback from
>> Wrong people.
>
>> test Thing do
>> thing = Thing.new; teardown {thing.cleanup()}
>> assert {thing.ok?}
>
> Okay, now take off the assert{}. All lines must return true, else
> Wrong swoops down and examines the failing line, in context.

You mean just the following on the line?

think.ok?

But then Wrong won't swoop at all.

With the assert{...}, if #ok? returns false, you get an informative message:

F: Thing: failed assertion, at example/test_thing.rb:11
Expected thing.ok?, but #<Thing:0x7f5d91d8e190> is not ok nil
thing is #<Thing:0x7f5d91d8e190>
0 passed, 1 failed, 0 skipped, 0 errors in Thing

I must have missed your point...

> This includes the Assemble lines and Activate lines too. Why would you
> assemble a None just to test it? And the activate line can say None ==
> activate(), to correct its polarization.

WAit,whuT?

[Steve Conover]

You mean I get to write "test"'s? As in, I have things in my code
called "test"'s?

Sold.

[Joel VanderWerf]

On 10/08/2010 10:38 PM, Steve Conover wrote:
> You mean I get to write "test"'s? As in, I have things in my code
> called "test"'s?
>
> Sold.

it "should not be like this"

:)

[Alex Chaffee]

Sent from my iPhone

On Oct 9, 2010, at 1:38 AM, Steve Conover <scon...@gmail.com> wrote:

> You mean I get to write "test"'s? As in, I have things in my code
> called "test"'s?
>
> Sold.

LOL -- my first thought was "Steve will loooove this API."

There's a lot to like in that synopsis. I like the implicit setup and
inline teardown. Can I keep nesting test blocks (turning them into
more setups)?

The word "teardown" makes less sense without "setup". Maybe rename it
to something like "cleanup" or just "clean" or "undo"?

I'll look at the code when I'm really online (I'm in iPhone purgatory
right now).

This is cool stuff. It's the first test framework that's not just a
skin on top of xUnit -- it's actually doing stuff with procs that you
can't do with methods and modules.

It does feel a bit hardcore -- like "for experts only" -- but maybe
that feeling will pass.

[Steve Conover]

Well what you lose not being on top of test/unit is compatibility with
all the toolsets that integrate with test/unit. For what I do it's
not a loss.

[Alex Chaffee]

Sent from my iPhone

On Oct 9, 2010, at 12:55 AM, Joel VanderWerf
<joelvan...@gmail.com> wrote:

> On 10/08/2010 08:57 PM, Phlip wrote:
>>> TestGarden is a test framework based on Wrong. It's not mature
>>> enough to
>>> declare version 0.0.0 or to announce on ruby-talk. I'd relish
>>> feedback from
>>> Wrong people.
>>
>>> test Thing do
>>> thing = Thing.new; teardown {thing.cleanup()}
>>> assert {thing.ok?}
>>
>> Okay, now take off the assert{}. All lines must return true, else
>> Wrong swoops down and examines the failing line, in context.
>
> You mean just the following on the line?
>
> think.ok?
>
> But then Wrong won't swoop at all.

One night last week I had a similar thought. "why do we need the
braces? Can't I just swoop in and execute whatever's after the 'assert'?

Well, yeah, I could... But in the cold light of day i realized then
I'd lose the binding. You only get one of those from a block. Sorry.

[Phlip]

>>>    test Thing do
>>>      thing = Thing.new; teardown {thing.cleanup()}
>>>      assert {thing.ok?}
>>
>> Okay, now take off the assert{}. All lines must return true, else
>> Wrong swoops down and examines the failing line, in context.
>
> You mean just the following on the line?
>
>        think.ok?
>
> But then Wrong won't swoop at all.

Yes. Instead of sending assert{} to Wrong, send test do...end to Wrong.

And configure Wrong to identify individual lines

To put it another way, every line should be an assertion, so assert{}
becomes redundant, so take it out.

[Joel VanderWerf]

On 10/09/2010 08:59 AM, Phlip wrote:
> To put it another way, every line should be an assertion, so assert{}
> becomes redundant, so take it out.

Oh, I see. But don't you sometimes need lines whose value is false, like:

fish = x.fish(Time.today)
fowl = x.fowl(Time.today)
assert {fish || fowl}

Asserting on every line would be a barrier to breaking assertions up
into assignments and more concise assertions.

[Joel VanderWerf]

On 10/09/2010 02:44 AM, Alex Chaffee wrote:
> There's a lot to like in that synopsis. I like the implicit setup and
> inline teardown. Can I keep nesting test blocks (turning them into more
> setups)?

Yes, the multiple nesting is really what's new IMO. For example, when
you are testing comm protocols, there are usually several stages...
create socket, connect, login, send something, receive something, send
something else... Each stage can be a separate level of nesting, so that
you can reuse the setup code up to each stage for multiple tests that go
in different directions from that stage. See the MockSocket test in the
examples.

If the nesting starts getting deep, you can factor into methods.

> The word "teardown" makes less sense without "setup". Maybe rename it to
> something like "cleanup" or just "clean" or "undo"?

You might be right. I prefer "cleanup", but I was thinking teardown
would be more obvious to test-unit users.

> I'll look at the code when I'm really online (I'm in iPhone purgatory
> right now).
>
> This is cool stuff. It's the first test framework that's not just a skin
> on top of xUnit -- it's actually doing stuff with procs that you can't
> do with methods and modules.

Thanks!

> It does feel a bit hardcore -- like "for experts only" -- but maybe that
> feeling will pass.

Yeah, I haven't used it enough to know if I like it yet. I'm planning on
testing an interval arithmetic library with it today.

[Joel VanderWerf]

On 10/09/2010 08:25 AM, Steve Conover wrote:
> Well what you lose not being on top of test/unit is compatibility with
> all the toolsets that integrate with test/unit. For what I do it's
> not a loss.

Tools? We don't need any steenking tools ;)

You're right. It's less object-oriented than frameworks with test
_instances_, which a tool could interact with. It's not really possible
to, for example, inspect the tests or even to count them without
actually running them.

[Phlip]

> Oh, I see. But don't you sometimes need lines whose value is false, like:
>
>  fish = x.fish(Time.today)
>  fowl = x.fowl(Time.today)
>  assert {fish || fowl}
>
> Asserting on every line would be a barrier to breaking assertions up into
> assignments and more concise assertions.

My original vision:

assert fish || fowl

At fail time, Ruby still _knows_ everything it must reflect, but we
just can't navigate all the way to all of it!

--
  Phlip
  http://c2.com/cgi/wiki?ZeekLand

[Phlip]

>> Well what you lose not being on top of test/unit is compatibility with
>> all the toolsets that integrate with test/unit.  For what I do it's
>> not a loss.
>
> Tools? We don't need any steenking tools ;)

U remind me of my Python Morelia Viridis:

http://c2.com/cgi/wiki?MoreliaViridis

Other BDD frameworks force you to write test cases in their language.
Morelia simply drops into any TestCase class you have and re-uses its
... tools.

--
  Phlip
  http://c2.com/cgi/wiki?ZeekLand

[Joel VanderWerf]

On 10/09/2010 10:20 AM, Phlip wrote:
>> Oh, I see. But don't you sometimes need lines whose value is false, like:
>>
>> fish = x.fish(Time.today)
>> fowl = x.fowl(Time.today)
>> assert {fish || fowl}
>>
>> Asserting on every line would be a barrier to breaking assertions up into
>> assignments and more concise assertions.
>
> My original vision:
>
> assert fish || fowl

LOL. I usually type it this way first and then have to go back and add
the { }.

> At fail time, Ruby still _knows_ everything it must reflect, but we
> just can't navigate all the way to all of it!

Maybe ruby 2.0...

[Joel VanderWerf]

On 10/09/2010 10:23 AM, Phlip wrote:
>>> Well what you lose not being on top of test/unit is compatibility with
>>> all the toolsets that integrate with test/unit. For what I do it's
>>> not a loss.
>>
>> Tools? We don't need any steenking tools ;)
>
> U remind me of my Python Morelia Viridis:
>
> http://c2.com/cgi/wiki?MoreliaViridis

Oh, right. Multiple "when" clauses are like nesting blocks:

Scenario: Split When Blocks, and again
Given some setup
And some condition

When a first trigger occurs
Then something good happens

When another trigger occurs
Then something else happens

corresponds to:

test "Split When Blocks, and again" do
some setup
some condition

test "first trigger" do
a first trigger occurs
assert {something good happens}
end

test "another trigger" do
another trigger occurs
assert {something else happens}
end
end

[Brad]

>> At fail time, Ruby still _knows_ everything it must reflect, but we
>> just can't navigate all the way to all of it!

> Maybe ruby 2.0...

I tried asking Matz last Thursday at the Fukuoka event if he was going
to (in Ruby 2.0--the topic of his talk) add more support for things like
what Phlip was asking for last week (to have access to the call tree,
save stack snapshots, etc.)... I probably didn't do a very good job of
representing Phlip's interests, but nonetheless Matz replied that no, he
wasn't (he said he didn't have the time or resources to add things to
support better tools).

However, he did mention that the NetBeans guys have done a lot in this
area and that we should look into what they've done (and perhaps
open-sourced?)...

He's speaking again on Monday at the Hacker Dojo, if anyone's going and
wants to try asking him more eloquently than I (
http://www.investfk.com/ )! (Be warned: the first 30-60 min is a bunch
of Ruby guys from Japan describing their projects, Matz is the headliner
that only started talking the 2nd hour, at Twilio in SF last Thursday.)

~ Brad

[Phlip]

> I tried asking Matz last Thursday at the Fukuoka event if he was going
> to (in Ruby 2.0--the topic of his talk) add more support for things like
> what Phlip was asking for last week (to have access to the call tree,
> save stack snapshots, etc.)...  I probably didn't do a very good job of
> representing Phlip's interests, but nonetheless Matz replied that no, he
> wasn't (he said he didn't have the time or resources to add things to
> support better tools).

I think all I need is a tiny tweak on the existing journaling, so I
should just try it myself. C-: