[dev] Suckless unit testing in C?

[Sam Dodrill]

Hey all,

I was wondering what you all use for a suckless style unit testing
framework in C.

Thanks,

--

Sam Dodrill
shado...@gmail.com / xe...@yolo-swag.com

[FRIGN]

On Tue, 24 Feb 2015 13:21:08 -0800
Sam Dodrill <shado...@gmail.com> wrote:

Hey Sam,

> I was wondering what you all use for a suckless style unit testing
> framework in C.

I use {} for all my unit testing needs.

Cheers

FRIGN

--
FRIGN <d...@frign.de>

[Louis Santillan]

I'm a fan of TAP frameworks [0]. Very minimal, easy to roll you own,
consumable by many Unix tools, human & machine readable. clibs links
to a bunch of pre-written libs as well [1]. kazuho had written this
35 line example of a lib [2] that exports all you need to basic TAP,
which is like 75% of all TAP. See an example usage here [3].

[0] http://en.wikipedia.org/wiki/Test_Anything_Protocol
[1] https://github.com/clibs/clib/wiki/Packages#testingquality-assurance
[2] https://github.com/kazuho/picogc/blob/master/t/test.h
[3] https://github.com/lpsantil/picotap/blob/master/t/test.c

[FRIGN]

On Tue, 24 Feb 2015 13:58:16 -0800
Louis Santillan <lpsa...@gmail.com> wrote:

Hey Louis,

> I'm a fan of TAP frameworks [0]. Very minimal, easy to roll you own,
> consumable by many Unix tools, human & machine readable. clibs links
> to a bunch of pre-written libs as well [1]. kazuho had written this
> 35 line example of a lib [2] that exports all you need to basic TAP,
> which is like 75% of all TAP. See an example usage here [3].

you do realize this is written in C++? Come to think of it, what's the
purpose of that? You could use asserts then anyway. And asserts are bad,
same applies to any unit testing in smaller projects.

If you projects grow too large, there might be some reason to have tests,
but as already discussed earlier, if you are careful, things won't out-
grow on a well-designed interface and if you set up tests, they often
end up covering one are but not the other.

Cheers

FRIGN

> [0] http://en.wikipedia.org/wiki/Test_Anything_Protocol
> [1] https://github.com/clibs/clib/wiki/Packages#testingquality-assurance
> [2] https://github.com/kazuho/picogc/blob/master/t/test.h
> [3] https://github.com/lpsantil/picotap/blob/master/t/test.c

--
FRIGN <d...@frign.de>

[Louis Santillan]

Of course I understand there's some C++ stuff in there. Throw it out.
Not necessary. Two methods is all you need, his plan() & ok().
Testing, IMO, is always good. You speak about well-designed
interfaces. How do you know if some implementation detail change
breaks your usage? And lets say this implementation is still
compliant to the interface. A test could help you with that. A
well-designed interface is still not a perfect interface, nor is it
the same thing as perfect/correct usage of such an interface.
However, those things are easy to test. As for test coverage, a
well-designed interface deserves a well-designed test case.

[Eduardo A. Bustamante López]

I've played a bit with caddeus (https://github.com/alvarezp/caddeus), the thing
I like about it is that it fits in a single GNU make makefile.

[Anselm R Garbe]

On 24 February 2015 at 22:21, Sam Dodrill <shado...@gmail.com> wrote:
> I was wondering what you all use for a suckless style unit testing
> framework in C.

If you apply the Unix principle correctly, main() is your unit test.

All you need is creating test input, save the test output and whenever
you call main() with the same test input, you just check for
regressions in the test output compared to the expected test output.

If you need unit tests to test portions of your code, you've probably
not followed the Unix principle well.

Requiring unit tests as in Java, C++, etc. is typically a very good
indicator that your overall software architecture sucks and that it's
incompliant with the Unix principle.

-Anselm

[Eduardo A. Bustamante López]

On Wed, Feb 25, 2015 at 07:45:58AM +0100, Anselm R Garbe wrote:
[...]

> If you apply the Unix principle correctly, main() is your unit test.
>
> All you need is creating test input, save the test output and whenever
> you call main() with the same test input, you just check for
> regressions in the test output compared to the expected test output.

What if you're trying to test a program that requires network connectivity?
It'd be good to be able to mock the networking stuff.

[Anselm R Garbe]

On 25 February 2015 at 08:04, Eduardo A. Bustamante López

If you develop a client against some mock server, you should check if
it wouldn't be a better idea to use the real server or to implement a
real server instead of a mock server to begin with ;) You not only
save a lot of wasted time developing mock stuff, you also benefit from
implementing an interface on both sides, which is always beneficial.

-Anselm

[k0...@shike2.com]

>
> What if you're trying to test a program that requires network connectivity?
> It'd be good to be able to mock the networking stuff.

Write your program to deal with sdtdin and stdout, and comunicate
to the network using nc or inetd.

Regards,

[Louis Santillan]

On Tue, Feb 24, 2015 at 10:45 PM, Anselm R Garbe <gar...@gmail.com> wrote:
> On 24 February 2015 at 22:21, Sam Dodrill <shado...@gmail.com> wrote:
>> I was wondering what you all use for a suckless style unit testing
>> framework in C.
>
> If you apply the Unix principle correctly, main() is your unit test.
>

Which Unix Principle [0][1][2] are you referring to?

[0] http://en.wikipedia.org/wiki/Unix_philosophy#Doug_McIlroy_on_Unix_programming
[1] http://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond.E2.80.99s_17_Unix_Rules
[2] http://en.wikipedia.org/wiki/Unix_philosophy#Mike_Gancarz:_The_UNIX_Philosophy

[hiro]

exactly 0 1 2, stdin stdout, stderr.

[Louis Santillan]

I'm asking an honest question here because there is a lot of wisdom on
mailing list and I very well lack that perspective. How is main() the
unit test? I read that almost as "I don't always test, but when I do,
I do it production."

[Markus Teich]

Heyho Louis,

in this case I can also recommend Chapter 6 of „The Practice of Programming“ by
Brian W. Kernighan and Rob Pike.

--Markus

[Eduardo A. Bustamante López]

On Wed, Feb 25, 2015 at 08:16:52AM +0100, k0...@shike2.com wrote:
> Write your program to deal with sdtdin and stdout, and comunicate
> to the network using nc or inetd.

This is very simplistic.

I'll give you some examples:

- ii: I don't see it using netcat.
- quark: Doesn't use inetd
- surf: depends on webkit for most stuff
- dwm: X11 API

Why aren't these communicating with stdin or stdout?

The reality is that sometimes you have to interface with complex APIs, and a
simple way of testing your own code is to abstract or mock the aspects of these
APIs that your program relies on. Or do you have better suggestions to test
your code, other than "don't write bugs" or "for every change, test all the
possible features that could've been affected by that change"?

[Rian Hunter]

On Wed, Feb 25, 2015 at 07:45:58AM +0100, Anselm R Garbe wrote:

> All you need is creating test input, save the test output and whenever
> you call main() with the same test input, you just check for
> regressions in the test output compared to the expected test output.

Dunno, this seems awfully religious. Not that there is anything wrong with that, I worship a lot of programming gods myself.

Logically though, this advice breaks down for interactive programs with modes of operation that are only accessible after a long sequence of specific user input with output that isn't easily machine-parseable. Unit tests exist for this exact purpose, to give you a way to quickly and easily test these modes. It's much simpler and more robust to call a function and check a linked-list or integer result than to simulate user input and parse the output of ncurses or a framebuffer.

If the subtext of the advice here is to do away with interactive programs, except for the shell, then this philosophy makes sense. Unix started on a teletype device without advanced visual ttys. This advice definitely makes things much more scriptable and that is good. I like git, mpc, xdotool, xsetroot, notify-send, sed, awk, sendmail, etc.

Then again, I like my interactive modal programs too. I like emacs, mutt, irssi, elinks, tmux, st, dwm. These tools are necessary for my daily work. How did the early Unixers get along without these tools? ed, mail, write, I'd guess. Should we go back to that? Personally, I'd rather not.

I'd say that while the Unix philosophy is a very good way to build composable tools for Unix (I would hate if git were an interactive program) but not every user-oriented tool fits exactly fits into that paradigm. When you want an automated way of testing, unit tests can make sense in those cases.

An alternative would be to structure your interactive program as a thin shell whose sole job is to spawn smaller helper commands for functionality and determining what to display. That could work, I guess, but doing that just to follow the Unix philosophy seems a little overly religious. In many cases it would be simpler to have a single program.

[Greg Reagle]

On 02/25/2015 02:35 PM, Eduardo A. Bustamante López wrote:
> This is very simplistic.
>
> I'll give you some examples:
>
> - ii: I don't see it using netcat.
> - quark: Doesn't use inetd
> - surf: depends on webkit for most stuff
> - dwm: X11 API
>
> Why aren't these communicating with stdin or stdout?

I think that is a great question. I would find it very instructive to see that question answered. I am a big fan of the suckless attitude, but I am not an expert suckless programmer like some of the people on this list. If you are inclined, please educate me. I am an eager student. Would it be possible or desirable to make any of the suckless tools/programs to suck even less by writing them to use netcat or inetd and removing all network-specific code? Do one thing and do it well? Is Plan 9 the logical extreme of this practice?

[Markus Teich]

Rian Hunter wrote:
> …

Heyho Rian,

Please wrap your lines to a majority approved sane length. Thanks.

--Markus

[FRIGN]

On Wed, 25 Feb 2015 14:57:15 -0500
Greg Reagle <greg....@umbc.edu> wrote:

> > - ii: I don't see it using netcat.
> > - quark: Doesn't use inetd
> > - surf: depends on webkit for most stuff
> > - dwm: X11 API
> >
> > Why aren't these communicating with stdin or stdout?
>
> I think that is a great question. I would find it very instructive to see
> that question answered. I am a big fan of the suckless attitude, but I am
> not an expert suckless programmer like some of the people on this list. If
> you are inclined, please educate me. I am an eager student. Would it be
> possible or desirable to make any of the suckless tools/programs to suck
> even less by writing them to use netcat or inetd and removing all
> network-specific code? Do one thing and do it well? Is Plan 9 the logical
> extreme of this practice?

Hey Greg,

let's first look at the examples given:

- ii
He can't be serious here. ii is like 500 LOC and doesn't need special unit tests,
also thanks to the simple IRC protocol.

- quark
Wait for the changes coming up. I am the maintainer and Hiltjo is working on some
really cool stuff I'm hoping to merge soon.

-surf
Yeah so what? How are we supposed to debug webkit in quark?

-dwm
Yeah, the Xlib sucks. Next question.

If you have to design a bigger program and want to make it more flexible, having
the program operate on stdin and stdout and using netcat is ingenious, but can
also be an issue.
There are others here who are more knowledgable on networking, but in theory this
should work.

But please guys, don't flood this ml with these strange discussions and write some
code instead. If you want to nag and want to masturbate to something, go to reddit
or the Arch Forums.

[Martti Kühne]

1. You have a part you have to test, write a test program covering the
*important* functionality. If your module contains a few objects with
interfaces, write one test creating, modifying and destroying these
objects as you see fit.
Writing an interface is work, but it is also a set of strings that let
you forget the puppet and yes, you will spend some time debugging
errors *inside* the interfaced objects.

2. If your program does X and uses the above-mentioned interfaces, the
mere fact that your program actually *does* X will be proof enough
that - wait, no, sorry openssl, it was not.
For debugging, you can write state and intermediate values to standard
file descriptors or valgrind/ltrace up to some point, and your
compiler may provide you with features that let you wrap function
calls so that you can profile the program in other ways in which you
see fit, which then still might not give you any more power as the
mentioned tools.

3. If you insist in using a test framework, here, have [0]. The
concept is a single line of preprocessing code you can extend to
whatever you want to do with it. However let me explain for a minute
why I think that mutest might not be such a good idea. Instead you
should read the specification for the programming language [1] and any
further tools and libraries you are using, because whatever interfaces
from third-parties you're going to use are what will be making your
life hell because you will at some point do something with these tools
they might not yet have covered.

cheers!
mar77i

[0] http://www.llucax.com.ar/proj/mutest/
[1] http://iso-9899.info/n1256.html

[Eric Pruitt]

On Wed, Feb 25, 2015 at 09:09:44PM +0100, Markus Teich wrote:
> Please wrap your lines to a majority approved sane length. Thanks.

I see from your headers that you use mutt. It's a lot easier to write a
mail filter (man muttrc && /display_filter) than to change everyone's
behaviours or, in this case, their MUA's behaviours. I've got an awk
script filter that does various things from wrapping lines to localizing
date headers.

Eric

[FRIGN]

On Wed, 25 Feb 2015 14:17:30 -0800
Eric Pruitt <eric....@gmail.com> wrote:

> I see from your headers that you use mutt. It's a lot easier to write a
> mail filter (man muttrc && /display_filter) than to change everyone's
> behaviours or, in this case, their MUA's behaviours. I've got an awk
> script filter that does various things from wrapping lines to localizing
> date headers.

Just wrap your lines, it's basic E-Mail-etiquette.
And it's not just some convenience for Mutt-users. Any mail-client benefits
from properly-wrapped lines and there's no reason to tell every user to
use a script to handle it.
This just pushes the problem onward to people who actually are not responsible
for it.

[Dmitrij D. Czarkoff]

Rian Hunter said:
> In many cases it would be simpler to have a single program.

Looks like you are in a wrong crowd.

--
Dmitrij D. Czarkoff

[Eric Pruitt]

On Thu, Feb 26, 2015 at 12:33:07AM +0100, FRIGN wrote:
> Just wrap your lines, it's basic E-Mail-etiquette.
> And it's not just some convenience for Mutt-users. Any mail-client benefits
> from properly-wrapped lines and there's no reason to tell every user to
> use a script to handle it.
> This just pushes the problem onward to people who actually are not responsible
> for it.

Do you really think people that are using mail clients that don't
automatically wrap text are going to take to manually wrap them? I very
much doubt it, and trying to convert all of them to different mail
clients or take the time to manually wrap lines is a battle that was
lost the day it was started. If anything, you could argue that unwrapped
lines are better since it lets the end-user choose what width they want
their lines wrapped at.

Eric

[M Farkas-Dyck]

On 25/02/2015, Eric Pruitt <eric....@gmail.com> wrote:
> unwrapped
> lines are better since it lets the end-user choose what width they want
> their lines wrapped at.

This.

On 25/02/2015, Markus Teich <markus...@stusta.mhn.de> wrote:
> Please wrap your lines to a majority approved sane length. Thanks.

Sanity is not a democracy.

[Anselm R Garbe]

On 25 February 2015 at 20:52, Rian Hunter <rian+suc...@thelig.ht> wrote:
> On Wed, Feb 25, 2015 at 07:45:58AM +0100, Anselm R Garbe wrote:
>> All you need is creating test input, save the test output and whenever
>> you call main() with the same test input, you just check for
>> regressions in the test output compared to the expected test output.

[..]

> If the subtext of the advice here is to do away with interactive programs, except for the shell, then this philosophy makes sense. Unix started on a teletype device without advanced visual ttys. This advice definitely makes things much more scriptable and that is good. I like git, mpc, xdotool, xsetroot, notify-send, sed, awk, sendmail, etc.
>
> Then again, I like my interactive modal programs too. I like emacs, mutt, irssi, elinks, tmux, st, dwm. These tools are necessary for my daily work. How did the early Unixers get along without these tools? ed, mail, write, I'd guess. Should we go back to that? Personally, I'd rather not.

To me this is rather a question of the software architecture.
Interactive programs should consist and/or incorporate lot's of small
noninteractive programs (one for a particular job) that can be
combined interactively like in a similar shell usecase. Each of such a
small noninteractive program can have a particular "test program" or
more complex test suits can be developed for them. Those
noninteractive programs or parts define an interface (perhaps not a C
API interface, but a shell-like interface), but my emphasis is on
'interface'.

I do see sense in developing (unit) tests for interfaces like
libraries (with native lang bindings) or such noninteractive command
line programs. A (unit) test makes sense if the underlying program or
library function is called from various places/contexts with different
inputs and use by more than some internal code line. (Unit) tests
should test interfaces, not concrete implementations. They should
define what the concrete outcome is of a particular interface.

Demanding to write unit tests for internal code lines or hidden
implementations is rather pointless to me. Honestly how on earth would
a unit test suit for dwm's inner workings look like? Checking if the
linked structs attach/detach work correctly? Rather not. That is not
designed as API interface for external use.

Instead I can imagine bad behaving X clients as "unit" tests for dwm
-- as they would challenge the task of dwm -- _window management_.


Speaking out of experience, I would also pretty much object to do TDD
(test driven development) in a sense that you write test cases first
and the real code afterwards. That barely tends to a robust product,
as the test cases will become quite arbitrary.

My recommendation - and the only one - when it comes to unit tests is
this: whenever you or someone else finds a bug that points out a
non-conforming aspect of an implementation of such an interface,
record a unit test for this, in order to avoid that this specific bug
will happen again.

A good example for this is how'd you develop a compiler for some
language: whenever an input program results in a compiler error/bug
that is non-conforming to the language standard, fix the bug in the
compiler implementation and record the specific input program as unit
test for future uses. During time your unit tests will grow an test
real world problems and not artificial programs that the developer
came up with while developing and that haven been rendered invalid
meanwhile.

-Anselm

[Roberto E. Vargas Caballero]

>
> - ii: I don't see it using netcat

You can use netcat in the program or in the test. This is an example
where you can do the test of ii using simple shell scripts with
netcat. You don't need unit test or mockup.

> - quark: Doesn't use inetd

Because it was not designed by me. I propossed it and there were
people didn't want do it in this way. But again, you can
use netcat to test the program.

> - surf: depends on webkit for most stuff

Yes, and do you can use unit test to test if webkit is correct?
Surf is only a good wrapper about webkit, so if you mockup
webkit you are going to test only the part without problems.
The problems in surf is the interconexion with webkit (and
webkit itself). For example, I compiled surf the other day
in a system without any font (fc-list | wc -l == 0), and surf
generated ramdom cores, in a very nested function of
webkit. How can you test it with unit tests mocking
webkit?

> - dwm: X11 API

You are moving to something different to the original topic,
network testing. When you have graphical programs you need
something different. I worked in a company that used xvkbd
and other program (I don't remember the name now) to read
the content of the window. You can use something like [1]
today.


Regards,

[1] http://www.semicomplete.com/projects/xdotool/xdotool.xhtml

[Roberto E. Vargas Caballero]

>
> A good example for this is how'd you develop a compiler for some
> language: whenever an input program results in a compiler error/bug
> that is non-conforming to the language standard, fix the bug in the
> compiler implementation and record the specific input program as unit
> test for future uses. During time your unit tests will grow an test
> real world problems and not artificial programs that the developer
> came up with while developing and that haven been rendered invalid
> meanwhile.

But in this case you are talking about functional test or test cases,
I feel they are different to unit test, isn't it?

Regards,

[Roberto E. Vargas Caballero]

> student. Would it be possible or desirable to make any of the
> suckless tools/programs to suck even less by writing them to use
> netcat or inetd and removing all network-specific code? Do one thing
> and do it well? Is Plan 9 the logical extreme of this practice?

Depend. This is always the answer.

There are cases where the network code is the central part of the
program, like for example identd, but there are other cases where the
best option is to create a textual protocol, and use nc/inetd, like
the case of ftp/http.

Of course, if you need high performance (that is not the common), you
can not rely in nc/inetd solutions, but you can begin with somethig
simple and create the complex solution only when you need it. This is
also part of the Unix philosophy.


Regards,

[Anselm R Garbe]

On 26 February 2015 at 09:26, Roberto E. Vargas Caballero

Depends. If you imagine the compiler as "unit", then the term "unit"
test would be adequate.

That's why I prefer to just call it test(s), rather. Different people
conglomerate different meanings of such terms...

-Anselm

[FRIGN]

On Thu, 26 Feb 2015 02:28:41 -0500
M Farkas-Dyck <stra...@gmail.com> wrote:

> > lines are better since it lets the end-user choose what width they want
> > their lines wrapped at.
> This.

You don't seem to be very experienced in the area of typesetting. Your eyes
can only read lines with a certain length without issues.
~70 chars per line is not a democratically assumed value, but actually the
result of years of experience and more or less a biological limit.

> Sanity is not a democracy.

Exactly! But having unwrapped lines corresponds to anarchy. And anarchy is
for idiots.

[Eduardo A. Bustamante López]

Hey Roberto, I'm going to stop due to FRIGNs request.

I do find your feedback interesting, and most of your points valid. Thank you
very much.

[sta...@cs.tu-berlin.de]

* FRIGN 2015-02-26 10:43

> M Farkas-Dyck <stra...@gmail.com> wrote:
> > Sanity is not a democracy.
>
> Exactly! But having unwrapped lines corresponds to anarchy. And anarchy is
> for idiots.

A. can diverge badly but also (surprisingly to many) converge to very
much sane results. I'd rather consider the ~70-wrapping as the
latter.

--s (wrapping at whitespace <=72)