Lua Unit Testing

[Stephen E. Baker]

Hello,

A discussion started in https://github.com/CorsixTH/CorsixTH/pull/211 about unit testing CorsixTH.  I think it's clear that this is something we want going forward, but the discussion of how to get there is ongoing.  I'm copying this list because I think it's important for people that might not be watching this pull request to be involved in the discussion.

The current state of the discussion is deciding what tools would be best for the job:

JASheppard:

Having looked at the about pages for Busted and Luaunit, I personally think Busted would be a better unit testing framework to use because:
1. Busted is provided by an open source and alive Github project where as Luaunit was made by one programmer and its development ceased years ago.
2. Unlike Luaunit, Busted has a good website and good documentation.
3. Busted supports more unit testing functionality with support for Mocks, Stubs and Spies. Luaunit's about page doesn't mention support for these things.

alanwoolley:
As for the C++ side, Google Test is considered to be one of the best and lightweight. Again, never actually used it personally so I can't comment too much. I'll defer to someone more qualified on that.

[Edvin Linge]

From a first glance one downside of Busted is that it is not so light
weight and comes with its own syntax.
Lunit(x) is just three Lua files that could even be included in our
repository for ease of use, and you still continue to write "normal"
Lua code, just with a "test" prefix on test functions. Luaunit too has
"normal" syntax.

/Edvin

On den 5 mars 2014 18:21:10, Stephen E. Baker wrote:
> Hello,
>
> A discussion started in https://github.com/CorsixTH/CorsixTH/pull/211
> about unit testing CorsixTH. I think it's clear that this is
> something we want going forward, but the discussion of how to get
> there is ongoing. I'm copying this list because I think it's
> important for people that might not be watching this pull request to
> be involved in the discussion.
>
> The current state of the discussion is deciding what tools would be
> best for the job:
>
> JASheppard:
>
> Having looked at the about pages for Busted

> <http://olivinelabs.com/busted/> and Luaunit
> <http://phil.freehackers.org/programs/luaunit/>, I personally think
> /Busted/ would be a better unit testing framework to use because:
> 1. /Busted/ is provided by an open source and alive Github project
> <https://github.com/Olivine-Labs/busted/issues> where as /Luaunit/ was

> made by one programmer and its development ceased years ago

> <https://github.com/luaforge/luaunit>.
> 2. Unlike /Luaunit/, /Busted/ has a good website and good documentation.

> 3. Busted supports more unit testing functionality with support for

> Mocks, Stubs and Spies. /Luaunit's/ about page
> <http://phil.freehackers.org/programs/luaunit/> doesn't mention

> support for these things.
>
> alanwoolley:
> As for the C++ side, Google Test

> <https://code.google.com/p/googletest/> is considered to be one of the

> best and lightweight. Again, never actually used it personally so I
> can't comment too much. I'll defer to someone more qualified on that.
>

> --
> You received this message because you are subscribed to the Google
> Groups "CorsixTH Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to corsix-th-de...@googlegroups.com
> <mailto:corsix-th-de...@googlegroups.com>.
> For more options, visit https://groups.google.com/groups/opt_out.

[Joe#]

I still think personally that Busted would be a better choice than LuaUnit. It wouldn't be as light and simple to integrate as LuaUnit would be, but all of the extra functionality that Busted has will make it easier for unit tests to be made and automated. From what I've read about them today, I think Spy, Mock and Stub functions are useful unit testing framework features: http://www.elijahmanor.com/unit-test-like-a-secret-agent-with-sinon-js/

I think Busted having its own syntax isn't a problem because its syntax is simple and easy to learn.

Plus because Busted is provided by an open source Github project, members of our project could add functionality we require to Busted by submitting pull requests to its repository and then members of that project could help make the commits in these pull requests acceptable.

> <mailto:corsix-th-dev+unsub...@googlegroups.com>.

[Edvin Linge]

Indeed, it all depends on who uses it. A new contributor that is not much used to unittests will have a lower threshold with LuaUnit or lunit, but at the same time if there are already a lot of tests to base your new ones on it should be fine.

As long as it is fairly easy to setup in both Windows and Linux we're good.

/Edvin

To unsubscribe from this group and stop receiving emails from it, send an email to corsix-th-de...@googlegroups.com.

[Joe#]

Is anyone working on integrating a unit testing framework with CorsixTH?

I want to make a unit test for a new commit I'm going to add to my hell deaths pull request, so I will start working on integrating Busted with CorsixTH if no one is already integrating a unit testing framework with CorsixTH.

> <mailto:corsix-th-dev+unsubscri...@googlegroups.com>.

> For more options, visit https://groups.google.com/groups/opt_out.

[Edvin Linge]

I have not started that yet, no, so go ahead. :-)

For more options, visit https://groups.google.com/d/optout.

[Joe#]

I will start working on this today.

[Joe#]

I'm sorry I took ownership of this implementation task nearly a month ago and haven't finished it, I've not even started trying to integrate Busted with CorsixTH because I've not even been able to make Busted run on my Windows 8 OS yet. I guess Lego and the other project administrators must be pretty frustrated with me as a contributor who takes ownership of implementation tasks and then takes forever to get them finished.

I have less free time every day now and I now have a friend who has lots of time to spend playing online games with me because they are having a summer university semester break. Plus when I do spend time programming things, I'm dividing it up my between CorsixTH and a game I've been making. And because I've been enjoying working on my game more than the tasks I've been working on for CorsixTH,  I've been spending most of my time working on my game instead of CorsixTH. I do want to open source my game soon.

So I now want to focus on finishing my hell deaths pull request before I spend any time working on anything else for CorsixTH. And I will spend at least an hour every day doing CorsixTH programming because I want to be a better, more dedicated contributor to this project.

I don't mind if someone else takes over the task of integrating a Lua unit testing framework with CorsixTH.

[Stephen E. Baker]

Thanks for the status update Joe.

Don't beat yourself up over it, CorsixTH is just a hobby for all of us, and life happens.  I don't think anyone else would have been working on the unit testing anyway - things are not very active right now.  I'm looking forward to merging hell deaths :)

On 2014-07-03 12:53 PM, Joe# wrote:

I'm sorry I took ownership of this implementation task nearly a month ago and haven't finished it, I've not even started trying to integrate Busted with CorsixTH because I've not even been able to make Busted run on my Windows 8 OS yet. I guess Lego and the other project administrators must be pretty frustrated with me as a contributor who takes ownership of implementation tasks and then takes forever to get them finished.

I have less free time every day now and I now have a friend who has lots of time to spend playing online games with me because they are having a summer university semester break. Plus when I do spend time programming I'm dividing it up my between CorsixTH and a game I've been making. And because I've been enjoying working on my game more than the tasks I've been working on for CorsixTH,  I've spending most of time working on it instead of CorsixTH. I do want to open source my game soon though.

So I now want to focus on finishing my hell deaths pull request before I spend any time working on anything else for CorsixTH. And I will spend at least an hour every day doing CorsixTH programming because I want to be a more dedicated contributor to this project.

I don't mind if someone else takes over the task of integrating a Lua unit testing framework with CorsixTH.

On Monday, 9 June 2014 15:45:48 UTC+1, Joe# wrote:

[Joe#]

Thanks Stephen.

"And I will spend at least an hour every day doing CorsixTH programming tasks because I want to be a better, more dedicated contributor to this project".

I haven't spent any hours working on CorsixTH tasks since I said this, but today I will start spending at least an hour every day work on CorsixTH programming tasks.

[Lego3]

I have not spent (too much) time trying to figure out how to get busted doing what I believe we need.

The good part:
On Windows it takes some fiddling to get installed and ready to use, whereas on Linux (as usual) it is a matter of two commands and you're done.

The thing I don't understand:
I then tried to use the provided example and just add something from our code base to test. My problem is that I haven't been able to get busted to show anything more than how many tests succeeded and how many failed. Judging from the documentation it looks like you should also get output which tests failed and why, but I don't. It must be some small thing that I just miss out though.

The downside regarding documentation is that in the end everything seems to just point to that one official documentation, or be a small rewrite of it. I haven't really been able to find other good example code anywhere.

Could someone else try it out to see what obvious thing I've missed?

You can basically do something like this on Linux:
1. Go to your checked out CorsixTH directory.
2. sudo apt-get install luarocks
3. luarocks install busted
4. mkdir spec
5. touch spec/test_spec.lua
6. Put the example from http://olivinelabs.com/busted/ in that file
7. busted

/Edvin

[Lego3]

Note that the below list was taken from the top of my head, so there might be some small tweaks that need to be done. :-P

[Alan Woolley]

I've just given it a try on a Ubuntu installation using your instructions, and I have no problems seeing the cause of the failed tests:

./spec/test_spec.lua:28: Expected to be falsy, but value was:
(boolean) true

That's after adding a simple test for:

assert.falsy(true)

which will obviously fail.

-- Alan

[Edvin Linge]

Right, now I think I know why it doesn't work at least... The next step is to understand how to fix it, or how to work with busted together with our filebase.

I don't know that much of how Lua as a language works. Pete created the class system in the beginnning. As soon as I start using that package of files (strict.lua, class.lua etc) busted stops working. strict.lua restricts the usage of globals, which I suspect is the cause.

--

[Stephen E. Baker]

You should be able to stub the functions in strict.lua with no runtime effect, maybe as not_strict.lua.  Include it instead when we want to allow globals i.e. for unit testing.