memcached Test Suite

[Victor]

Hi,
A few months ago there was a discussion on the mailing list about the
memcached tests and the test suite.
The "official" memcached test suite is written in perl. Some people
are comfortable with perl while others of course aren't. There was a
suggestion about including libmemcached with memcached and writing
tests using that library.
Since most all memcached users know C this should make it easier for
some to understand what the tests are doing and to do the debugging.
It might be easier to expand the test suite with larger tests later on
too...not having to write missing library functions first, just
because that takes some extra time and effort.
The concern with using libmemcached was that it might become difficult
to distinguish between a library bug and a server bug since the tests
aren't minimalist anymore. I guess this is valid if the tests are
supposed to be minimalist but eventually I imagine tests to be there
that will require a library (C or perl or etc..).
Anyway, one answer that came up was to write a very simple light-
weight c library and use that instead of libmemcached. My co-worker
Trond has written a such a library for the purpose of doing testing on
the upcoming binary protocol. I've rewritten many of the perl tests in
C using his library. The tests can switch between using the textual
and the binary protocol. I'm hoping we can include these tests with
memcached when they're ready.
I should also mention that Brad came up with an idea during the
Hackathon - defining a language-independent text format that describes
expected requests and responses from the memcached server. This text
format should be so simple that it would make it easy to add new
tests. The processing of the text could be done with perl while the
command execution could be done with C e.g. Simple tests could be
written in this format. More complex tests too I suppose - but the
text format might not be that simple any longer. I don't have the time
to look more at this now but if somebody has any suggestions it will
be welcome along with all other suggestions on how to improve the test
suite.

-Victor

[Dustin]

On Nov 24, 4:13 am, Victor <victor.kirk...@sun.com> wrote:

> Since most all memcached users know C this should make it easier for
> some to understand what the tests are doing and to do the debugging.

I would never say that using C makes things more understandable than
a more terse language. I'm not a perl fan, though. I'd rather have
comprehensive tests in multiple languages than a One True test suite
sort of thing.

> It might be easier to expand the test suite with larger tests later on
> too...not having to write missing library functions first, just
> because that takes some extra time and effort.

Do note that the memcached test suite I did for bootstrapping the
binary protocol typically took about three lines of code to define any
new command when I was testing:

1) function signature
2) documentation for the function (not hugely necessary in a test
suite, but why not?)
3) the function itself

Some were longer (multiget was the worst at 14 SLOC).

> The concern with using libmemcached was that it might become difficult
> to distinguish between a library bug and a server bug since the tests

This may be true, but I don't think speculation is really helpful.
There will be bugs in the server and bugs in the tests. Right now, we
have test suites outside of the server tree that help us find bugs in
the tests. More buggy tests might be better than fewer tests that
aren't as comprehensive.

> Anyway, one answer that came up was to write a very simple light-
> weight c library and use that instead of libmemcached. My co-worker
> Trond has written a such a library for the purpose of doing testing on
> the upcoming binary protocol. I've rewritten many of the perl tests in
> C using his library. The tests can switch between using the textual
> and the binary protocol. I'm hoping we can include these tests with
> memcached when they're ready.

My entire standalone test suite for binary protocol (including
client) that I used to develop it is 448 lines of rather easy to read
text (plus another 71 for constants). It's a bit out of date (all of
the delete tests currently fail due to packet format changes), but I
think it's a pretty good way to write such tests.

http://github.com/dustin/memcached-test/tree/master/testClient.py

I've also got a bit under 3,000 SLOC of tests using my java client
that goes a lot further (tests every combination of text and binary
over ipv4 and ipv6 with every command I support and hopefully their
edge conditions on inputs and has been useful for finding other things
such as pipelining bugs).

> I should also mention that Brad came up with an idea during the
> Hackathon - defining a language-independent text format that describes
> expected requests and responses from the memcached server. This text
> format should be so simple that it would make it easy to add new
> tests. The processing of the text could be done with perl while the
> command execution could be done with C e.g. Simple tests could be
> written in this format. More complex tests too I suppose - but the
> text format might not be that simple any longer. I don't have the time
> to look more at this now but if somebody has any suggestions it will
> be welcome along with all other suggestions on how to improve the test
> suite.

The idea of a testing DSL sounds good. I imagine it'd look somewhat
similar to the perl tests, but perhaps with less perl. :)