Unit Testing for vim [Was: Vim Lib [Was: List Questions]]
Luc Hermitte
"Ingo Karkat" <sw...@ingo-karkat.de> wrote :
> On 07-Feb-09 12:39, Luc Hermitte wrote:
> > My few requirements, in case it helps:
> > - we should be able to test values with very simple things like:
> > :Assert value < 42
> > - the result should end in the quickfix window (which implies
> > a first pass on functions/files to determine the current line
> > where the Assert is used)
> > - it would be nice to test mappings/commands/abbreviations/...
> > definitions, buffer alterations, etc.
> > - setup/teardown support would be nice also.
>
> Thanks for your list.
I've finally implemented my own unit testing plugin for vim. [1]
This is a first draft, and all comments are welcomed.
The plugin has been strongly inspired by Tom Link's tAssert plugin
(thanks Tom!).
However I've made a different design choice: my plugin acts as a
preprocessor. Thanks to that, I'm able to know the line where an
assertion failure occurred, and I'm also able to handle expressions
with script-local s:variables.
My plugin is made for Unit Testing, and nothing else. We write an
independent UT script, and give it as argument to :UTRun command.
On the other hand, Tom's plugin permits to Design scripts by Contract,
which is also extremely useful. i.e., it makes possible to place
assertions in regular Vim scripts (*-plugins). However by its design
choice, it cannot fills the quickfix window with the list of failed
assertions -- hence my NIH script.
[It could be possible to process regular scripts with my plugin, in
order to do DbC by working on "debug"-enabled plugins, but I'm not sure
it worth it]
BTW, tAssert provides convenience functions that my script don't (yet?).
At first, I wondered if both plugins should be merged. But as the engines
are quite different, I'm not sure it makes any sense.
> What prompted me to start my implementation is that though it's
> trivial to come up with some :Assert commands, these cover the unit
> testing part well (i.e. checking that MyFunc('foo') does return 'bar'),
> but are of little help when testing the effects of custom mappings and
> commands, what most scripts and tips on vim.org are about.
That's true. However, it should be possible to provide convenience
functions that help checking a buffer content.
[1] http://code.google.com/p/lh-vim/wiki/UT
NB: for once, I've written a script that depends on no other
script -- even if it can take advantage of BuildToolsWrapper if it
is detected.
PS: shouldn't the discussion move to vim_use ?
--
Luc Hermitte
http://lh-vim.googlecode.com/
http://hermitte.free.fr/vim/
Tom Link
> preprocessor. Thanks to that, I'm able to know the line where an
> assertion failure occurred
> BTW, tAssert provides convenience functions that my script don't (yet?).
> At first, I wondered if both plugins should be merged.
meant to work as a unit testing framework, I then was more interested
in sprinkling the code with assertions that could easily be turned
off. But with a DbC-related plugin, you'd always want to load it on
startup during (informal) testing. A unit-testing plugin on the other
hand, you'd probably want to load only before running the test, I
suppose. So, a DbC plugin should load fast, a UT-plugin doesn't
necessarily have to. This is also the reason why I'd rather prefer to
strip down my tassert plugin and to leave only the TAssert command and
some utility functions in it.
There is some overlap of course since the utility functions you
mentioned are useful in both contexts. Maybe those functions should go
in a shared autoload file?
One main challenge with a unit testing framework for vim scripts
actually is, as Ingo Karkat pointed out, the utility functions it
provides. IMHO these functions should not only be able to test for
buffer content after a defined series of simulated key presses (via
feedkey()) but also something like window layout etc. It would also be
interesting to combine such a UT plugin with something like Charles
Campbell's PluginKiller[1] to make sure that the results are the same
with different sets of option values. I think CC's plugin is quite
interesting in this respect but I personally don't think it's very
practical to manually simulate all different kinds of user
interactions repeatedly with different settings. AFAIK PluginKiller
doesn't automate (record & replay) this process, does it?
Regards,
Thomas.
[1] http://www.vim.org/scripts/script.php?script_id=1489
Tom Link
> preprocessor. Thanks to that, I'm able to know the line where an
> assertion failure occurred
> BTW, tAssert provides convenience functions that my script don't (yet?).
> At first, I wondered if both plugins should be merged.
Charles Campbell
... q @a, for example),
so I don't have PK automating that process. What'd be great would be to
combine that playback
with the ability to determine if the plugin behaved correctly or not --
but I don't see any way to
generically accomplish that. Certainly the notion of TAssert could be
used to give a partial
feedback of that sort.
Regards,
Chip Campbell
Tony Mechelynck
>>
> I think that one can already do a keystroke recording and playback (qa
> ... q @a, for example),
> so I don't have PK automating that process. What'd be great would be to
> combine that playback
> with the ability to determine if the plugin behaved correctly or not --
> but I don't see any way to
> generically accomplish that. Certainly the notion of TAssert could be
> used to give a partial
> feedback of that sort.
>
> Regards,
> Chip Campbell
Well, at least most of Vim (with the exception, IIUC, of the python
interface) is essentially single-threaded, so unlike developers for some
other applications, Vim developers normally don't get headaches from
race conditions (some of them depending on "how fast you're typing and
how busy is the system") and the sporadic crashes that often go with
them, and even sometimes disappear when you harness the app in a
debugging framework. (Though I have sometimes seen X11 error crashes in
gvim at startup, which spontaneously disappeared when started with the
--sync switch to force synchronous interfacing with the X server.)
Best regards,
Tony.
--
"The Good Ship Enterprise" (to the tune of "The Good Ship Lollipop")
On the good ship Enterprise
Every week there's a new surprise
Where the Romulans lurk
And the Klingons often go berserk.
Yes, the good ship Enterprise
There's excitement anywhere it flies
Where Tribbles play
And Nurse Chapel never gets her way.
See Captain Kirk standing on the bridge,
Mr. Spock is at his side.
The weekly menace, ooh-ooh
It gets fried, scattered far and wide.
It's the good ship Enterprise
Heading out where danger lies
And you live in dread
If you're wearing a shirt that's red.
-- Doris Robin and Karen Trimble of The L.A. Filkharmonics
Tom
> > At first, I wondered if both plugins should be merged.
> strip down my tassert plugin and to leave only the TAssert command and
> some utility functions in it.
removed the UT related stuff from tassert and created a spec plugin
that allows to write script specifications that could look like this:
http://github.com/tomtom/vimtlib/blob/83976d6b1572000b9950241749371206f7e03d59/spec/spec/spec.vim
http://github.com/tomtom/vimtlib/blob/83976d6b1572000b9950241749371206f7e03d59/spec/spec/should.vim
Other than Luc's UT, spec scripts are normal vim scripts (they are not
pre-processed) but they have to be run by the :Spec command. I don't
know yet which approach is better but I plan to integrate the should
functions with his UT since I personally prefer the more descriptive
messages when something goes wrong. There is also a function that runs
a command and then compares the buffer contents with a file (with the
option to ignore changes in whitespace).
Regards,
Thomas.
Ingo Karkat
> I've finally implemented my own unit testing plugin for vim. [1]
> This is a first draft, and all comments are welcomed.
>
> The plugin has been strongly inspired by Tom Link's tAssert plugin
> (thanks Tom!).
>
>
> My plugin is made for Unit Testing, and nothing else. We write an
> independent UT script, and give it as argument to :UTRun command.
>
> On the other hand, Tom's plugin permits to Design scripts by Contract,
> which is also extremely useful. i.e., it makes possible to place
> assertions in regular Vim scripts (*-plugins). However by its design
> choice, it cannot fills the quickfix window with the list of failed
> assertions -- hence my NIH script.
>
I've just published a first version of my testing framework for VIM plugins.
Whereas Tom's plugin is for Design by Contract, and Luc has started developing a
unit testing plugin, my framework allows to write succinct automated regression
test suites. I had been dragged down because of the immense testing effort
involved whenever I made a modification to one of my published plugins. It is
especially tedious to test for different VIM settings and to re-create failure
conditions for custom commands and mappings.
Now I can specify an expected output, e.g. in file 'test001.ok'
expected output
and more
expected output
and/or specify expected messages in file 'test001.msgok'
/\d\+ lines changed/
use an input file 'test001.in'
ExPeCteD OuTpUt
AND MORE
ExPeCteD OuTpUt
write a short test script 'test001.vim'
edit test001.in
normal! gg3guu
write test001.out
quit!
and my framework invokes the test(s) [suites], compares the actual with the
expected output, checks that the messages match, and prints any failures and a
test summary:
2 tests, 2 run: 2 OK, 0 failures, 0 errors.
Unit tests can also be executed; I've chosen the VimTAP plugin because the TAP
output is very easy to parse. However (since the quickfix buffer can be saved to
a file, too), Luc's UT plugin could be used for unit tests just as well.
It's no rocket science, just some (rather ugly) shell script driver, but it has
totally changed the way I maintain and develop my plugins. Here's the link:
http://vim.sourceforge.net/scripts/script.php?script_id=2565
I mention this here in the hope that this may be useful to other script writers,
too, or maybe just to inspire them to do more in the testing area, and
potentially come up with a better alternative. Feedback is welcome.
-- regards, ingo
--
-- Ingo Karkat -- /^-- /^-- /^-- /^-- /^-- /^-- http://ingo-karkat.de/ --
-- http://vim.sourceforge.net/account/profile.php?user_id=9713 --