Proposal for haXe's unit testing facilities

[Nicolas Juneau]

Hello fellow haXers,

I have been working for a while on a modification of haXe's unit testing
facilities (which can be found there :
https://github.com/njuneau/Unit2). Maybe some of the ideas could be
integrated in future versions of haXe, should you find the modifications
worthy of integration.

First of all, the TestRunner has been modified to look for method
metadata instead of method names in order to find tests. So, instead of
having to write test methods with the "test" prefix, the "@Test"
metadata tag must be added to it. This lets the user choose any name
for his/her test methods and making it clearer in the code what is being
marked as a unit test.

The same mechanic has been applied for the setup and tearDown methods -
they have been stripped from the TestCase class. Instead, the "@Before"
and "@After" tags may be used to mark a method that must be executed
before and after a unit test. For those who are familiar with JUnit, it
should make the transition easier.

The TestRunner is now responsible for creating the TestCase instances.
Adding tests to the TestRunner no longer requires the user to have a
constructor in his/her TestCase implementations - they are added in the
TestRunner using classes instead of instances ( testRunner.add(TestCase)
instead of testRunner.add(new TestCase()) ). The TestRunner will create
empty TestCase instances without calling constructors. The TestCase can
still have initialization mechanics by having a method annotated with
the "@BeforeClass" metadata tag. Added bonus (again inspired by JUnit),
"@AfterClass" can be used to simulate destruction.

Test statuses are no longer contained in TestCase - they have been moved
in TestRunner. The TestRunner is now entirely responsible for handling
the test results. Therefore, the TestCase class is now more lightweight
and error control is, I believe, easier. In theory, any error that
occurs in a unit test can be caught by the TestRunner for processing
using a simple try/catch.

Last but not least, test results are no longer printed to the console by
default. TestRunner and TestCase no longer have any printing/formatting
mechanics. It has been taken out in "output writers". After executing
the TestRunner, the user may send the instance to an OutputWriter
implementation (right now there is only TextOutputWriter), which takes
care of scanning the TestRunner for the results and formatting them for
output. The final result is a string that can be used by the user in any
way he wants - printing to the console, writing to a file... New
OutputWriter implementations could be developed to have, for example,
HTML or XML output.

That's it, in a nutshell. What do you think?

--
Nicolas Juneau

[Dom De Lorenzo]

Hi Nicolas,

Have you looked into other unit testing libraries on haxelib?

munit (https://github.com/massiveinteractive/MassiveUnit) takes an identical approach with metadata driven test classes and adapters for different report formats.

- supports synchronous and asynchronous tests
- js/as3/as2/neko, cpp coming soon
- rich HTML report client
- junit style test reports that are saved out to the file system for reporting and CI
- Integrated code coverage reports (via integration with MCover)

The integrated tooling (haxelib run munit) provides a streamlined TTD workflow :
- Compile and run multiple targets from an hxml build file
- Launch and run test applications in the browser or command line (neko)
- Auto generate test suites based on test classes in a src directory
- Auto generate stub test classes (and/or target classes)

Don't let the 0.9.2.x version number put you off - this is a stable production library with comprehensive documentation and examples on github (https://github.com/massiveinteractive/MassiveUnit)

We've been holding off the 1.0 version until the following (in development) features are released:

- full cpp/nme support
- overhauled server module to decouple runners from report clients thus enabling results integration with IDEs (moving from nekoserver to nodejs and websockets for improved stability and consistant cross platform behavior)
- Ability to compile and execute a single test or test class rather than the entire suite)

Cheers,
Dom

> --
> To post to this group haxe...@googlegroups.com
> http://groups.google.com/group/haxelang?hl=en

[Stephane Le Dorze]

+backed by business..

[Nicolas Juneau]

No, I wasn't aware of the existence of that project - although what I suggest
is not an entirely new unit testing implementation and is not intended to be
an independant library. I stayed in the scope of what is provided by the
original haXe API in order to limit transition effects, should this code be, in
an alternate reality or not, integrated in haXe.

I haven't had time yet to explore all of what MassiveUnit has to offer (which
is indeed, massive :P ), but from what I see, it could very well implement the
conceptual changes I propose.

I have not added any new feature - the functionnality is essentially the same.
It's only the way the functionnalities are achieved that changes. The goal was
to keep the original API's functionnality, but improve on it by using new
features introduced by the haXe laguage, such as metadata. This way, if the
changes are some day introduced as part of the language, the transition effort
is minimalised.

In short, what I initially asked feedback for was not necessarly "Is Unit2 a
good testing framework in itself", but more "Could haXe benefit from metadata
support in its unit testing framework, Unit2 being an example of how it can be
done". Or, from MassiveUnit's point of view, "Could haXe benefit from metadata
support in its unit testing framework, MassiveUnit being an example of how it
can be done".

If we look at what we have so far, in general, taking into consideration
MassiveUnit:

* Metadata support
* Versatile test reports
* Synchronous / asynchronous tests
* Code coverage reports
* Code / stub generation
* Fine-grained control over test compilation and execution

Are these features that could be part of the haXe API in the future? Would the
community benefit or suffer from such additions (transition issues, workflow
changes)?

[Mike Stead]

Hi Nicolas

I'm personally of the opinion that no framework should be bundled with the core sdk. Unit testing is important but the sdk's sole concern should be to provide a clean, consistent api on which to build upon.

Now that it is part of the sdk it's hard to remove / move to haxelib. Maybe Haxe 3.0 will offer this chance. Or maybe others disagree.

There's a little bit of a discussion on this here http://code.google.com/p/haxe/wiki/ApiChanges

[Nicolas Juneau]

I find haXe's unit tests API to be quite useful - it's very simple to pick up, integrated and works on all platforms. It doesn't cover more advanced functionality that you'd find in a full-fledged testing framework, but it does, in my opinion, cover the basics very well. Looking at haXe's metadata features, I thought it made sense to modify the original functionality to work with it - it seems very well suited for that kind of work. As for the different outputting methods, it was more to accommodate the user in situations where the output was not necessarily standalone in a terminal.

Without implying that it's not suited for more advanced uses, the integrated unit testing facilities must be nice for any newcomer to haXe since said newcomer wouldn't even need to know how haxelib works or search for libraries to test his/her code. Just write your unit tests and run them!

On the other hand, having to maintain a bigger unit testing framework as part of haXe API may not be strategic - as you mentioned, having a clean and consistent API is very important, and diverging too much from the initial design may cause headaches to users. Nevertheless, maybe it could benefit from a little update to take better advantage of haXe's features.

Maybe a good question would be : is there a sufficient interest in haXe's unit testing facilities in order for the packages to be modernized?

Mike Stead a écrit :

--