ANN: RunTests.jl a test running framework

[Andrew Burrows]

Recently I've been working on a new Testing framework RunTests.jl . It is a test runner inspired by python's pytest that aims to improve testing without bloating your code with loads of boilerplate. In its simplest form RunTests.jl can save you from writing run_tests.jl scripts that look like this:

my_tests = ["test/sometests.jl",
            "test/somemoretests.jl",
            "test/evenmoretests.jl"]

println("Running tests:")
for my_test in my_tests
  include(my_test)
end

and allows you to write them, more simply, like this:

using RunTests
exit(run_tests())

But it has more to offer than that! RunTests.jl builds on top of Julia's Base.Test library to make it easy to add structure to your tests. Structuring your tests with RunTests.jl gives the following advantages:

• Using modules and functions to structure test files gives you a natural isolation between tests.
  
• All the tests are run - the tests script doesn't bomb out after the first failure so you can see all your test results at once.
• A summary of how many tests passed/failed is produced so you can judge at a glance how the test run went.
• The STDOUT and STDERR output from each test is captured and reported along with the details of the test failure.
• You get a progress bar showing how far through the tests you are, it is green while all the tests are passing and goes red if any fail
• You can selectively skip tests with @skipif and mark failing tests with @xfail.
• Using @parameterize you can run the same test again again with different parameters and see which pass and which fail.

For more information and examples of how to use it a look at the github repo:

https://github.com/burrowsa/RunTests.jl

My next aim for the project is to better integrate my Fixtures.jl package with it too

All feedback, ideas, issues, pull requests etc. are very welcome.

Cheers

Andy