<ANN> TeSLa, a Domain Specific Language for Unit Testing

[javie...@gmail.com]

Hi
I just posted version 0.1.0 of TeSLa, a Domain Specific Language (DSL)
for Unit Testing.
You can download TeSLa along with a small example script from
http://theniceweb.com/projects/tesla/tesla.zip (zip)
or here
http://theniceweb.com/projects/tesla/tesla.tar.gz (tar.gz)
I also posted a small article/tutorial explaining the rationale and use
of TeSLa here
http://www.theniceweb.com/JaviersBlog/2005/10/tesla-test-specific-language-for-ruby.html

[James Britt]

This looks quite interesting. Two questions:

1. How does this compare to behavior-driven development?

http://daveastels.com/index.php?p=5

2. The TeSLa docs give this example, comparing Test::Unit and TesLa:

class TestCatalog < Test::Unit::TestCase
def test_add_item
catalog = Catalog.new()
catalog.items = [:item1, :item2, :item3]
catalog.add_item :item4
assert_equal(catalog.size, 4, "length should be 4")
end
end

would look like this in TeSLa

test_method :add_item => [:item4] do
requires {@items = [:item1, :item2, :item3]}
assert {size == 4}
end

How does the TeSLa version know what :item4 is for, or where it goes, or
what 'size' refers to? The example has no reference to Catalog; how
does the code know to test that class?

Thanks,

James
--

http://www.ruby-doc.org - The Ruby Documentation Site
http://www.rubyxml.com - News, Articles, and Listings for Ruby & XML
http://www.rubystuff.com - The Ruby Store for Ruby Stuff
http://www.jamesbritt.com - Playing with Better Toys

[Daniel Sheppard]

>
> How does the TeSLa version know what :item4 is for, or where
> it goes, or what 'size' refers to? The example has no
> reference to Catalog; how does the code know to test that class?

I know nothing about TeSLA, but I can make a guess.

> test_method :add_item => [:item4] do
> requires {@items = [:item1, :item2, :item3]}
> assert {size == 4}
> end

This creates a testcase that will call the "add_item" method on an
instance of the class (I'm guessing that you define test_method within
the context of a class). Before calling the method, it will set the
instance variable @items to the value [:item1, :item2, :item3]. It will
then pass that arguments [:item4] to the method, and afterwards call the
size method and ensure that it equals 4. It does the equivalent of:

c = Catalogue.new
c.instance_eval { @items = [:item1, :item2, :item3] }
c.instance_eval { send(:add_item, :item4) }
raise unless c.instance_eval { size == 4 }

Or, that's my impression....

#####################################################################################
This email has been scanned by MailMarshal, an email content filter.
#####################################################################################

[obscured by code]

Well, it doesn't quite works that way (the implementation, that is... I
tried to avoid using 'eval' as long as I could) but yes, all references
to properties or methods are to an instance of the class where the
tests are specified. If you take a look at the example.rb script
included in the package, you'll notice

test_method :add_item => [:item4] do
requires {@items = [:item1, :item2, :item3]}
assert {size == 4}
end

is indeed declared just bellow the 'add_item' method in class Catalog
and that 'items' is a property of that class. 'test_method' will create
a clean instance of class Catalog and execute the 'add_item' method
with :item4 as a parameter and 'assert' will check that 'size' (of the
Catalog instance) is indeed 4 as long of the 'precondition' stated by
requires was met.
As for behavior-driven development, I never knew it existed, thanks a
lot for the link, I'm going to have a look at it right away.
By the way, I never intended for TeSLa to be a "pure" unit testing
framework, maybe even include concepts from Eiffel's Design By Contract
(so far that's just an idea I'm toying with) so let me know of any
interesting concept you might have about testing in general and I'll
take a look at it.
Again, let me know of ways you think the syntax might be more
readable/intuitive (even though I know that's a subject where people
will never totally agree)
Thanks a lot for the feedback

[James Britt]

obscured by code wrote:
> Well, it doesn't quite works that way (the implementation, that is... I
> tried to avoid using 'eval' as long as I could) but yes, all references
> to properties or methods are to an instance of the class where the
> tests are specified. If you take a look at the example.rb script
> included in the package, you'll notice
>
> test_method :add_item => [:item4] do
> requires {@items = [:item1, :item2, :item3]}
> assert {size == 4}
> end
>
> is indeed declared just bellow the 'add_item' method in class Catalog
> and that 'items' is a property of that class. 'test_method' will create
> a clean instance of class Catalog and execute the 'add_item' method
> with :item4 as a parameter and 'assert' will check that 'size' (of the
> Catalog instance) is indeed 4 as long of the 'precondition' stated by
> requires was met.

Ah. Thank you for the clarification. I often put my unit tests in the
same source file as the class code, and being able to have tests right
next to methods is quite slick.

> As for behavior-driven development, I never knew it existed, thanks a
> lot for the link, I'm going to have a look at it right away.
> By the way, I never intended for TeSLa to be a "pure" unit testing
> framework, maybe even include concepts from Eiffel's Design By Contract
> (so far that's just an idea I'm toying with) so let me know of any
> interesting concept you might have about testing in general and I'll
> take a look at it.

I'm interested in the idea of having executable comments, so that the
rdoc for a method both renders as examples for the reader, and can be
run as unit tests. I have no idea how your work might fit into that,
though. One out-of-the-blue idea: an rdoc directive call that grabs the
TeSLa code and translates it into an example using the normal calling
syntax.

So,

test_method :add_item => [:item4] do
requires {@items = [:item1, :item2, :item3]}
assert {size == 4}
end

would become, in the rendered method docs,

c = Catalog.new
c.items = [:item1, :item2, :item3]
c.add_item( :item4 )
Expected results: c.size == 4

Or something.

Thanks,

James Britt

[itsm...@hotmail.com]

I like this direction.

obscured by code wrote:
> test_method :add_item => [:item4] do
> requires {@items = [:item1, :item2, :item3]}
> assert {size == 4}
> end
>
> is indeed declared just bellow the 'add_item' method in class Catalog
> and that 'items' is a property of that class. 'test_method' will create
> a clean instance of class Catalog and execute the 'add_item' method
> with :item4 as a parameter and 'assert' will check that 'size' (of the
> Catalog instance) is indeed 4 as long of the 'precondition' stated by
> requires was met.

So what part of the code makes {@items = [:...]} true?

You might also find it useful to look at ruby-contract:
http://ruby-contract.rubyforge.org/wiki/wiki.pl?HomePage

[obscured by code]

No part of the code makes that line true, that line is a precondition
to the success of the assert once the method is run. That is,

requires {@items = [:item1, :item2, :item3]}

is executed *before*
add_item => [:item4]
Look at it as initialization code if you like. In other words, it is
*required* for the 'items' property to hold items 1 to 3 so the
addition of a fourth one would make the
assert {size == 4}
line true.
Below is the entire Catalog class from example.rb. Perhaps that would
help make things clearer

class ShoppingCart

def add_item item
@items << item
end

def size
@items.length
end

attr_accessor :items

test_method :add_item => [:item5] do

# requires makes sure certain conditions are met before the
# assert is run

requires {@items = [:item1, :item2, :item3]}
assert {size == 4}
end

end

[itsme213]

"obscured by code" <javie...@gmail.com> wrote in message

> No part of the code makes that line true, that line is a precondition
> to the success of the assert once the method is run.

A precondition is a check with no side effects.

> requires {@items = [:item1, :item2, :item3]}
> is executed *before*
> add_item => [:item4]
> Look at it as initialization code if you like.

Initialization code will set the @items variable, right?

So your "requires ..." is an executed statement that has the effect of
making the precondition true, not just a boolean test, correct?

If so, it makes sense. If not, I am still a bit lost.

[obscured by code]

Well, I think your confusion stems from my use of the word
"precondition". Thinking back, I realize it might have been better if
I'd have used a different wording.
In Eiffel a precondition is indeed (and I quote from Meyer's OOSC) "The
boolean expression which defines the domain", but in this thread I've
been using the word in the sense of:
"A condition that must exist or be established before something can
occur or be considered; a prerequisite."
The word 'requires' in the code is therefore not used in the sense of
the Eiffel keyword but rather to imply that what follows it is a
*requirement* for 'assert' to succeed. In plain English, the following
segment of code

test_method :add_item => [:item4] do

requires {@items = [:item1, :item2, :item3]}

assert {size == 4}
end

should be read as, "it is required for the 'items' property of any
Catalog instance to hold three items if the condition of 'size' being 4
is to be met after the method 'add_item' is called with an 'item' as
parameter".
All this leads me to think maybe a better syntax is, well... required
(no pun intended). Any suggestions regarding this will be very much
welcomed.
As an aside note, I've been reading Dave Astels' proposal of a Behavior
Driven approach (thanks James) and tend to agree with much of what he
proposes. Perhaps I could use some ideas (and syntax) from BDD to make
TeSLa more readable/intuitive. I'd still use Test::Unit underneath,
though, as maintaining tool integration is still very important to me.
Again, I'm very open to suggestions.

[Ryan Leavengood]

On 11/1/05, obscured by code <javie...@gmail.com> wrote:
>
> test_method :add_item => [:item4] do
> requires {@items = [:item1, :item2, :item3]}
> assert {size == 4}
> end

..

> Again, I'm very open to suggestions.

What about:

test_method :add_item => [:item4] do

before {@items = [:item1, :item2, :item3]}
after {size == 4}
end

Or variations thereof.

Ryan

[James Britt]

Ryan Leavengood wrote:

> What about:
>
> test_method :add_item => [:item4] do
> before {@items = [:item1, :item2, :item3]}
> after {size == 4}
> end
>
> Or variations thereof.

Such as

test_method :add_item => [:item4] do

given {@items = [:item1, :item2, :item3]}
expected {size == 4}
end

[Douglas Livingstone]

2005/11/2, James Britt <jam...@neurogami.com>:

>
> Such as
>
> test_method :add_item => [:item4] do
> given {@items = [:item1, :item2, :item3]}
> expected {size == 4}
> end
>

You could probably get off with the "assert" used already, ala:

test_method :add_item => [:item4] do
given {@items = [:item1, :item2, :item3]}

assert {size == 4}
end

Or:

test :adding_an_item do

given { @items = [:item1, :item2, :item3] }

and_on { add_item :item4 }

assert { size == 4 }
end

Where and_on is much the same as given.

Douglas

[Christian Neukirchen]

James Britt <jam...@neurogami.com> writes:

> Ryan Leavengood wrote:
>
>> What about:
>> test_method :add_item => [:item4] do
>> before {@items = [:item1, :item2, :item3]}
>> after {size == 4}
>> end
>> Or variations thereof.
>
> Such as
>
> test_method :add_item => [:item4] do
> given {@items = [:item1, :item2, :item3]}
> expected {size == 4}
> end

Continuing the brainstorm (I used "assuming" before "given", but that
reads better):

given { @items = [:item1, :item2, :item3] }.expect {
add_item_should_increase_size {
given {
@previous_size = size
}
doing { add_item :item4 }
results_in { size > @previous_size }
}

add_item_should_increase_size_by_one {
doing { add_item :item4 }
results_in { size == 4 }
}
}

I'm still not sure whether instance_eval'ing tests is a good idea.

> James
--
Christian Neukirchen <chneuk...@gmail.com> http://chneukirchen.org

[Eric Mahurin]

On 11/3/05, Christian Neukirchen <chneuk...@gmail.com> wrote:

> I'm still not sure whether instance_eval'ing tests is a good idea.

That's my concern too. I'm not sure I trust the validity of of the tests
since they are applied in a way that they are not used. This type of testing
also ties the tests to a particular implementation. If you change the
implementation (not the functionality), you'll likely have to change the
testing since you are directly setting and accessing instance variables.

On the other hand, with some types of objects, you may not want to expose
very much internal state at the API and this type of testing may make it
much easier. I can draw an analogy from this to the hardware world - DFT
(design for test) and ATPG (automatic test pattern generation) - where it is
common practice to add control and observe points for testing (most state
points at least). That is testing for defects, but even for functional tests
it is common to at least have observability at some internal points in a
chip. BTW, this is where I draw many of my software testing ideas from. I
think the software world can learn some things from the hardware world in
terms of testing.

[Christian Neukirchen]

Eric Mahurin <eric.m...@gmail.com> writes:

> On 11/3/05, Christian Neukirchen <chneuk...@gmail.com> wrote:
>
>> I'm still not sure whether instance_eval'ing tests is a good idea.
>
> That's my concern too. I'm not sure I trust the validity of of the tests
> since they are applied in a way that they are not used. This type of testing
> also ties the tests to a particular implementation. If you change the
> implementation (not the functionality), you'll likely have to change the
> testing since you are directly setting and accessing instance variables.

Also, how do you develop test-first with non-existing instance variables?

At least, one should make two groups of test, one that test
"blackbox", and one that test the actual implementation. I'd write
the blackbox test first, and the whitebox tests in parallel to the
implementation. That way, you can oversee the programming and still
test for implementation details. The latter tests will need to be
thrown away on a reimplementation, of course.

> On the other hand, with some types of objects, you may not want to expose
> very much internal state at the API and this type of testing may make it
> much easier. I can draw an analogy from this to the hardware world - DFT
> (design for test) and ATPG (automatic test pattern generation) - where it is
> common practice to add control and observe points for testing (most state
> points at least). That is testing for defects, but even for functional tests
> it is common to at least have observability at some internal points in a
> chip. BTW, this is where I draw many of my software testing ideas from. I
> think the software world can learn some things from the hardware world in
> terms of testing.

I recently searched a bit about "behavior driven design" and found a
paper on "spec driven design"---which turned out to be about hardware.

[Jim Weirich]

On Thursday 03 November 2005 10:41 am, Christian Neukirchen wrote:
> I recently searched a bit about "behavior driven design" and found a
> paper on "spec driven design"---which turned out to be about hardware.

Here are a couple:
http://blog.daveastels.com/files/sdbp2005/BDD%20Intro%20slides.pdf
http://blog.daveastels.com/files/sdbp2005/BDD%20Intro.pdf

--
-- Jim Weirich j...@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

[obscured by code]

Thank you all for your responses, I though on posting here some of my
opinions but it turned out to be too big so I decided I'd better blog
about it. You can find the article here
http://www.theniceweb.com/JaviersBlog/2005/11/ruby-20-and-tesla-upcoming-syntax_04.html
Oh, would you guys please elaborate on "I'm not sure I trust the
validity of the tests since they are applied in a way that they are not
used." I don't really understand what you mean by that.

[Christophe Grandsire]

Selon obscured by code <javie...@gmail.com>:

> Thank you all for your responses, I though on posting here some of my
> opinions but it turned out to be too big so I decided I'd better blog
> about it. You can find the article here
>
http://www.theniceweb.com/JaviersBlog/2005/11/ruby-20-and-tesla-upcoming-syntax_04.html

I know it's not exactly what you asked for when you've written this post, but
your comments on your blog about possible mandatory parentheses in Ruby2
concern me. Can people read Javier's blog and comment on that part? (if
possible by saying that it ain't gonna happen so!) Thanks in advance.
--
Christophe Grandsire.

http://rainbow.conlang.free.fr

It takes a straight mind to create a twisted conlang.

[Eric Mahurin]

Sometimes, you can get different results by calling the method without a
receiver (what you do in this testing infrastructure) vs. with a receiver
(what you do when you use this class). I think the main problem revolves
around the use of #method_missing. If you declare a private method #foo and
also #method_missing, calling #foo without a receiver will give you the
private method #foo and calling #foo with a receiver will put you in
#method_missing. In general you are not testing with the protections that
you have from outside the class. There may be other suttle issues to be
concerned about. I don't think this invalidates your approach, but it should
be a concern.

[Christian Neukirchen]

Jim Weirich <j...@weirichhouse.org> writes:

> On Thursday 03 November 2005 10:41 am, Christian Neukirchen wrote:
>> I recently searched a bit about "behavior driven design" and found a
>> paper on "spec driven design"---which turned out to be about hardware.
>
> Here are a couple:
> http://blog.daveastels.com/files/sdbp2005/BDD%20Intro%20slides.pdf
> http://blog.daveastels.com/files/sdbp2005/BDD%20Intro.pdf

Yeah, I've seen those already. It's all rehashing the same content.
;) I'd like to see code...

> -- Jim Weirich j...@weirichhouse.org http://onestepback.org

[Christian Neukirchen]

Christophe Grandsire <christophe...@free.fr> writes:

> Selon obscured by code <javie...@gmail.com>:
>
>> Thank you all for your responses, I though on posting here some of my
>> opinions but it turned out to be too big so I decided I'd better blog
>> about it. You can find the article here
>>
> http://www.theniceweb.com/JaviersBlog/2005/11/ruby-20-and-tesla-upcoming-syntax_04.html
>
> I know it's not exactly what you asked for when you've written this post, but
> your comments on your blog about possible mandatory parentheses in Ruby2
> concern me. Can people read Javier's blog and comment on that part? (if
> possible by saying that it ain't gonna happen so!) Thanks in advance.

I don't think the behavior will be changed compared to 1.8.2:
assert raises(ZeroDivisionError)
and
attr_accessor :items
will stay valid. (IANM, but I'd be very sad if parens are enforced.)

> Christophe Grandsire.

[Jim Weirich]

On Friday 04 November 2005 02:49 pm, Christian Neukirchen wrote:
> > Here are a couple:
> > http://blog.daveastels.com/files/sdbp2005/BDD%20Intro%20slides.pdf
> > http://blog.daveastels.com/files/sdbp2005/BDD%20Intro.pdf
>
> Yeah, I've seen those already. It's all rehashing the same content.
> ;) I'd like to see code...

gem install rspec

--

-- Jim Weirich j...@weirichhouse.org http://onestepback.org

[obscured by code]

>"Sometimes, you can get different results by calling the method without a
>receiver (what you do in this testing infrastructure)"

Well, it might be a matter of wording, maybe I'm not quite getting what
you say (English is not, after all, my mother tongue), but I totally
fail to see why you think methods get called without a receiver inside
TeSLa. As far as my understanding of what a receiver is, they don't. In
line 31 of tesla.rb you can see the following
@instance = self.new
self being the class where the tests are being defined. Later on,
inside method #assert (line 105 to be precise) you find the following
result = @instance.send(@method_name, *@params)
As you can see, there's no eval_ing of the method being tested, the
method is being sent to a previously created instance of the class in
which both the test and the method being tested are defined. It's true
*right now* you can directly test private methods, but that is
intentional, as the inability to test private functionality is a common
complain from people using other testing frameworks, and as such,
something I wanted TeSLa to offer. Also, testing a private method will
always end up in the private method being called, regardless of
#method_missing being defined in the class or not. I stressed *right
now* because in Ruby 1.9 you can't use Object#send with private
methods, so as it's currently implemented TeSLa won't be able to do
that. By the time Ruby 1.9 is out (in a stable form, that is) I'll be
forced to either create an explicit syntax for doing tests on private
methods of trying to work some inner magic with the newly introduced
Object#funcall.
By the way, the only thing that gets eval_ed in TeSLa are the
Test::Unit test methods, but that's only because I want to rely on
Test::Unit reporting and tool integration. If TeSLa were to do its own
test report (which by the way it does, only it's commented out) there
would be no evals in the code.