Prototype of a new shell unit-test library

[Julio Merino]

Hello everyone,

A couple of weeks ago, the thought of implementing a more modern
JUnit-like testing library for sh using the shtk foundations
(http://github.com/jmmv/shtk/) crossed my mind. And I have not been
able to resist prototyping it, particularly because I have been
isolated in planes for a long part of this period thus having a lot of
free time (yay long flights!)

So after this week and a half of intense coding, I have been able to
surpass atf-sh's functionality (_including_ atf-check's!).

The new shtk-based library is:

* fully written in sh (no C++ toolchain),
* has a much cleaner codebase than atf-sh,
* supports assert vs. expect checks (fatal vs. non-fatal, that is),
* is infinitely better tested than atf-sh,
* supports fixtures (aka groups of tests that share a common
setup/teardown process),
* supports unit-testing of sh code (e.g. atf-check's counterpart can
be used to verify sh functions!),
* and implements a simplified process to register test cases.

However, it lacks some minor features provided by atf-sh (like other
check functions) and, more importantly, there is a glaring lack of
documentation (other than the docstrings in the code and the code
samples in them) -- but hey, this is only a very preliminary
prototype.

I don't know yet where this is going. I'm currently planning on trying
to convert shtk's own tests to this library to see how things could
look like, and then we'll see how the results stand against atf-sh.

The point of this email is to let you all know that this prototype
exists and is an invitation for you to take a look. Especially, you
can take a look at unittest.subr's comments at the very top for a
snippet of sample code, and also glance over unittest_inttest.sh for
the integration tests of the library itself.

You can take a look at this new code on the "unittest" branch (which
I'll be *rebasing* frequently while working on it, so don't expect
"git pull" to work on already-fetched branches):

https://github.com/jmmv/shtk/tree/unittest

Thoughts? In particular, what do you think of the API? Also, is a
better testing library for shell something you are interested in?

Thanks!

[Craig Rodrigues]

On Fri, Oct 24, 2014 at 4:14 PM, Julio Merino <jm...@meroh.net> wrote:

Hello everyone,

A couple of weeks ago, the thought of implementing a more modern
JUnit-like testing library for sh using the shtk foundations
(http://github.com/jmmv/shtk/) crossed my mind.

I don't know where you want to go with this, but you
might want to take a look at: https://code.google.com/p/shell2junit/

It is not a comprehensive framework, but it does provide a way
to generate JUnit output from shell scripts.  For some people I've
talked to, they found this to be an easier way to integrate test scripts with Jenkins,

rather than using ATF, and kyua report-junit. 

Also, is a better testing library for shell something you are interested in?

For me, not really, but if you come up with something good and it works, I'll use it.

The two main things, I would like from from kyua/ATF right now are:

-  fix for https://github.com/jmmv/kyua/issues/123 .  The errors reported by kyua when someone botches writing an ATF testcase are confusing to newcomers.

- better integration with languages which are not C, C++, or POSIX shell.  For example,
  Python integration would be useful to me, like what Alan prototyped: https://lists.freebsd.org/pipermail/freebsd-testing/2013-November/000104.html

Outside of codebases like FreeBSD/NetBSD, most people who work in devops or systest groups in companies these days are normally
using other scripting languages besides POSIX shell.

--

Craig