Looking for test coordinator
Bram Moolenaar
So far we have been testing by hoping for people to install Vim and use
it. This does find problems related to daily use, but I suspect quite a
few things are not yet used and bugs go unnoticed. Then when Vim 7.3 is
released new features get used and bugs are uncovered. Too late!
It would be good if we have someone systematically going through new
features and recent changes, making a list of use cases that need to be
tested. Possibly using the help as a guideline and brainstorming of
situations and border cases that might have problems.
Then we need to have people who take one of the uses cases and run
through them. Verify that it works as expected. Try a few variants
while doing that. This should not take much time.
I have started a shared document for this. it's very rudimentary:
https://docs.google.com/document/edit?id=1CyOvJqKYoTmL4D50w2htYMG7xt5SADcIX6r7zZ978HQ&hl=en&authkey=COfQ_6oD
I need to spend my time on fixing bugs, it would be great if someone can
step forward to coordinate this testing. This includes making the
layout of the document better and keeping an eye on the progress.
I also wonder if a shared document is the best way to do these things.
At least this allows multiple people at the same time making changes.
Most wiki's are not good at that.
Comments?
--
ARTHUR: It is I, Arthur, son of Uther Pendragon, from the castle of Camelot.
King of all Britons, defeator of the Saxons, sovereign of all England!
[Pause]
SOLDIER: Get away!
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD
/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ download, build and distribute -- http://www.A-A-P.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
Marc Weber
> At least this allows multiple people at the same time making changes.
> Most wiki's are not good at that.
If its not versioned I'd prefer git or mercurial ..
> Comments?
Given that there are tons of different compile configuration of Vim and
tons of operating systems Vim runs on .. calling for some human testers
is nice but will not bring any benefits in the future.
What about setting up a test suite (driven by git or mercurial?)?
I could setup a cron-job which pulls latest tests patches once a day and
runs it on Linux.
If we used Ruby test suites it can be run easily on all major OS
systems.
I can't do so today - but within some days.
Comments?
Marc Weber
Bram Moolenaar
Marc Weber wrote:
> > I also wonder if a shared document is the best way to do these things.
> > At least this allows multiple people at the same time making changes.
> > Most wiki's are not good at that.
>
> If it is versioned then why not ? It can be accessed easily by everyone.
> If its not versioned I'd prefer git or mercurial ..
There is a history. Using a versioning system doesn't allow easy
editing, tables, etc.
> > Comments?
> Given that there are tons of different compile configuration of Vim and
> tons of operating systems Vim runs on .. calling for some human testers
> is nice but will not bring any benefits in the future.
>
> What about setting up a test suite (driven by git or mercurial?)?
There already is a "make test". Extending this is also a good idea.
And perhaps moving slower tests to a separate target, so that one can do
a quick check after building and a more comprehensive test.
> I could setup a cron-job which pulls latest tests patches once a day and
> runs it on Linux.
I run the tests quite often, this won't catch much more.
> If we used Ruby test suites it can be run easily on all major OS
> systems.
>
> I can't do so today - but within some days.
>
> Comments?
I prefer a Vim-script solution. One problem with this is that one
cannot check what is actually on the display. E.g., to check for the
conceal bugs that have been fixed recently.
--
Clothes make the man. Naked people have little or no influence on society.
-- Mark Twain (Samuel Clemens) (1835-1910)
Tony Mechelynck
I think the problem is severalfold:
- There are the existing tests, which are run automatically by "make
test". Maybe they could be divided in several sets, or maybe not.
- Some of the new feature haven't yet got automated tests. They shouldbe
identified, and tests written if possible.
- Some features might not be amenable to automated tests. Maybe a
standardized set of "manual tets" could be set up (with unambiguous
instructions, and "do you see this" questions at critical points).
- Then there is always live-testing, just using the new version
day-to-day with no specific guidelines. This is less systematic, and may
miss some bugs, but it may also sometimes find bugs nobody had thought
to test.
All these cases can be further divided depending on whether you use
- the latest stable version (currently 7.2.446 b619655b31db 2010-07-12
23:38 +0200)
- the latest beta milestone (currently IIUC 7.3c f766a1c87f69
2010-07-25 20:53 +0200)
- the latest bleeding-edge development build (currently 7.3c
33148c37f3c9 2010-07-28 19:38 +0200, but I don't know if the next push
will arrive within minutes, hours, or days).
Best regards,
Tony.
--
"I'm not under the alkafluence of inkahol that some thinkle peep I am.
It's just the drunker I sit here the longer I get."
Benjamin R. Haskell
>
> So far we have been testing by hoping for people to install Vim and
> use it. This does find problems related to daily use, but I suspect
> quite a few things are not yet used and bugs go unnoticed. Then when
> Vim 7.3 is released new features get used and bugs are uncovered. Too
> late!
>
> It would be good if we have someone systematically going through new
> features and recent changes, making a list of use cases that need to
> be tested. Possibly using the help as a guideline and brainstorming
> of situations and border cases that might have problems.
>
> Then we need to have people who take one of the uses cases and run
> through them. Verify that it works as expected. Try a few variants
> while doing that. This should not take much time.
>
> I have started a shared document for this. it's very rudimentary:
> https://docs.google.com/document/edit?id=1CyOvJqKYoTmL4D50w2htYMG7xt5SADcIX6r7zZ978HQ&hl=en&authkey=COfQ_6oD
>
> I need to spend my time on fixing bugs, it would be great if someone
> can step forward to coordinate this testing. This includes making the
> layout of the document better and keeping an eye on the progress.
>
> I also wonder if a shared document is the best way to do these things.
> At least this allows multiple people at the same time making changes.
> Most wiki's are not good at that.
>
> Comments?
Is anyone aware of anything like a terminal-emulator version of
Selenium? I've thought for several things like this that that would be
incredibly useful.
Brief summary of Selenium is that it's an automated web application
testing toolkit. It consists of a scripting language with capabilities
like (pseudocode):
open(URL)
type 'bhaskell' in the 'username' field
click the button with name 'login'
and corresponding tests, like:
verify that the text 'logged in' appears in the current page
ensure that there's a 'div' with id attribute 'maincontent'
It has support for a wide range of browsers (ie/moz/chromium/opera).
And it supports creating template profiles (so, starting Firefox with
GreaseMonkey already installed). I've used it at my current job to test
things via a headless X server (Xvfb).
For Vim's purposes, a similar framework with support for a wide range of
terminal emulators (for non-gvim) and the major GUI frameworks (Vim
supports GTK and ...?) would be very handy.
Scripts like:
execute(vim -U NONE -u NONE -N) " start vim
type ':e testfile' " edit a new file
type 'isometext<esc>' " insert 'sometext'
click and drag from column 3-6 " highlight the 'mete' portion
type 'o' " open new line
middle-click on terminal " terminal-paste the 'mete'
Tests like:
ensure that the buffer displays "sometext\nmete"
--
Best,
Ben
Raúl Núñez de Arenas Coronado
Tony Mechelynck <a...@gmail.com> skribis:
> - Some of the new feature haven't yet got automated tests. They
> shouldbe identified, and tests written if possible.
Yes, any automation possible should be done. Given my low "kyu" in Vim,
I don't have the needed expertise to do this unfortunately.
> - Some features might not be amenable to automated tests. Maybe a
> standardized set of "manual tets" could be set up (with unambiguous
> instructions, and "do you see this" questions at critical points).
YES! Darn it, I was going to suggest that minutes before I read your
message. If manual, but systematic, tests are done, anyone can
volunteer. I can do any manual test that is needed, as long as it is
systematic because then I don't need to do additional work, thinking
about possible code paths, etc. I think anyone can afford to spend some
time following some simple instructions to make a manual test.
> - Then there is always live-testing, just using the new version
> day-to-day with no specific guidelines. This is less systematic, and
> may miss some bugs, but it may also sometimes find bugs nobody had
> thought to test.
The keyword here, I'm afraid, is "less systematic". Systematic tests are
needed, even if they need human interaction.
Not that my opinion counts, but IMHO this would help future development
of Vim. Making test systematic is a very good idea, specially for tests
that require human interaction.
The shared document is a very good place to put descriptions of such
tests. In the beginning, they don't need to be exhaustive or even
precise, that's the advantage of collaborative systems, they can be
refined with time.
I volunteer to make as many tests requiring human interaction as needed,
as long as they're specified as a set of fixed steps. This way I can use
small pieces of my spare time to test.
--
Raúl "DervishD" Núñez de Arenas Coronado
Linux Registered User 88736 | http://www.dervishd.net
The revolution will not be televised
Dominique Pellé
> So far we have been testing by hoping for people to install Vim and use
> it. This does find problems related to daily use, but I suspect quite a
> few things are not yet used and bugs go unnoticed. Then when Vim 7.3 is
> released new features get used and bugs are uncovered. Too late!
A couple of points:
1/ Human tests are certainly complementary with automated tests.
Having said that, I'm mostly in favor of adding more automated
tests (make test) or improving coverage of existing ones. I've
never looked at how to write those tests myself, but I can give
it a try.
2/ One thing that surprised me in Vim source code at first
(but I got used to it) is the lack of asserts. Why not put
asserts in Vim code? I think they could catch problems
during development and they can be disabled for official
builds. Personally, I would build my own Vim with asserts
enabled to be able to find and report more bugs. They also
add semantic information when reading the code and can
even help to improve accuracy of Coverity checkers.
3/ Speaking of Coverity, the Coverity scan page shows that there
are 25 uninspected Vim defects (http://scan.coverity.com/rung2.html).
I don't know how up-to-date this is and I don't have access to it,
but would it be possible to look at them before releasing Vim-7.3?
4/ Some bugs can also be found if we test Vim on different OS.
How about keeping track of whether Vim successfully compiles and
passes all tests on:
- Linux (x86, x86_64, ARM, MIPS, ... )
- Windows 95, 98, NT, 2000, XP, Vista, 7 (32 & 64 bits)
- Cygwin-1.5, Cygwin-1.7
- OS-X
- OpenBSD, HP-UX, Solaris, AIX...
- and the more exotic zOS, vms, qnx, AmigaOS, BeOS, RISC-OS (others?)
We can ask users to submit short summaries like this and collect
them somewhere:
Vim version: Vim-7.3c (2427:33148c37f3c9)
OS: Ubuntu-10.04, x86
configure: --with-features=huge --enable-gui=gnome2
compiler: gcc-4.4.3
compilation: OK, no warnings.
make test: ALL DONE
Tested by: Dominique Pelle
File vim/src/Makefile contains this kind of information but
it looks old.
5/ Many bugs have also been found thanks to Valgrind. I use
it often to check Vim but I'm sure that I did not think of every
configurations and odd cases. I wrote a help page in case
others want to also test Vim with Valgrind:
http://dominique.pelle.free.fr/valgrind.txt.html
Latest version of Valgrind in SVN is supposed to run on
OS-X and Linux ARM (I don't have access to those OS).
Maybe some developers have access to other runtime checkers
or static analyzers?
> I have started a shared document for this. it's very rudimentary:
> https://docs.google.com/document/edit?id=1CyOvJqKYoTmL4D50w2htYMG7xt5SADcIX6r7zZ978HQ&hl=en&authkey=COfQ_6oD
What? Not a document that I can edit with Vim? :-)
Regards
-- Dominique
Steve Hall
>
> 1/ Human tests are certainly complementary with automated tests.
> Having said that, I'm mostly in favor of adding more automated
> tests (make test) or improving coverage of existing ones. I've
> never looked at how to write those tests myself, but I can give
> it a try.
Are we discussing strictly build tests or also usage testing? I see a
big difference between the two but the importance of both.
As for the latter, Cream uses an extremely large number of Vim
features just to start and in doing so tests a wide range of code and
conditions. (I think it has uncovered a Vim bug only a couple of times
in the last 10 years. So far, no problems with 7.3c BETA.)
Couldn't a script/test suite be created that methodically tests all
the features, with new tests added for each new feature? I would think
a collection of statements could be added to a log file. Here's one
method of data collection we use:
let s:debug = "\n INITIALIZATION:\n\n"
function! Foo_function(dir)
" collect the passed {dir} variable as test data...
let s:debug = s:debug . "Foo_function(): {dir} == \"" . a:dir .
"\"\n"
[... rest of script ...]
function! Collect_debug_info_local()
return s:debug
endfunction
This could easily be extended to test any Vim feature.
> 4/ Some bugs can also be found if we test Vim on different OS.
> How about keeping track of whether Vim successfully compiles and
> passes all tests on:
>
[...]
> We can ask users to submit short summaries like this and collect
> them somewhere:
>
> Vim version: Vim-7.3c (2427:33148c37f3c9)
> OS: Ubuntu-10.04, x86
> configure: --with-features=huge --enable-gui=gnome2
> compiler: gcc-4.4.3
> compilation: OK, no warnings.
> make test: ALL DONE
> Tested by: Dominique Pelle
The file data collection and formatting should be automated so the
user just has to do something like:
:vimtests
or similar and email the resulting data file or output. I do this with
Cream and it because the user only has to run one command, I can
collect an exhaustive amount of data that would otherwise take hours
to run. Our script labels, formats, and even inserts folds in the log
for easy navigation.
--
Steve Hall [ digitect dancingpaper com ]
Christian Brabandt
That reminds me, that I have once read about a test tool, that the
author of the rss reader newsbeuter has written, to test newsbeuter.
Ah, here is the announcement (sorry, German only):
http://synflood.at/blog/index.php?/archives/758-Automatisiertes-Text-User-Interface-Testing.html
and the github page:
http://github.com/akrennmair/tuitest
regards,
Christian
James Vega
> 4/ Some bugs can also be found if we test Vim on different OS.
> How about keeping track of whether Vim successfully compiles and
> passes all tests on:
>
> - Linux (x86, x86_64, ARM, MIPS, ... )
> - Windows 95, 98, NT, 2000, XP, Vista, 7 (32 & 64 bits)
> - Cygwin-1.5, Cygwin-1.7
> - OS-X
> - OpenBSD, HP-UX, Solaris, AIX...
> - and the more exotic zOS, vms, qnx, AmigaOS, BeOS, RISC-OS (others?)
We sort of get this (at least for Linux) with the Debian packages. As
of this email, Debian's packages get built on 14 different
architectures. As part of the build, I run the test suite, albeit only
with the binary that produces the vim binary package[0]. I could look
into changing that to use a binary with the language interpreters
enabled so those portions of the test suite get run.
I monitor the build logs[1] when I upload a new version of the package
to Debian for any test failures and then triage failures. We have a
porter team for each architecture (in some state of activity) which can
be consulted if there's an architecture-specific problem that's flagged.
[0]: --enable-cscope --with-features=huge --enable-multibyte --without-x
--enable-gui=no, and --enable-gpm on linux kernels.
[1]: https://buildd.debian.org/vim
--
James
GPG Key: 1024D/61326D40 2003-09-02 James Vega <jame...@jamessan.com>
Marc Weber
> Selenium?
cat the-testsuite.bin | vim ...
expect can record sessions. I don't know how easy it is to write test
cases that way. The problem is that you can't look at Vim while its
doing its work.
XTest could be another option - I never used it though.
Marc Weber
Nazri Ramliy
>> Is anyone aware of anything like a terminal-emulator version of
>> Selenium?
What if we had some helper command that dumps how the
current screen look like so that it can be compared with
some "expected_dump.txt"?
Something along the lines of:
FILE *fd = fopen("dump.txt", "w");
int r, c;
for (r = 0; r < screen_Rows; r++)
for (c = 0; c < screen_Columns; c++)
fputc(ScreenLines[r * screen_Columns + c], fd);
fputc('\n', fd);
fclose(fd);
I think this would help in testing stuff that would not be
easy to test currently like 'relativenumber', and
completion-related stuff.
Vim can then be told to dump the screen to a file by setting
up hooks, say, "dump the screen whenever this or that event
happen", or maybe done through the "--remote-send/expr" or
by sending signals.
This implies that the test must be done in a pre-set terminal
size.
The file "expected_dump.txt" would of course be prepared
through the same facility.
I think the BOE (benefit-to-effort, as in signal-to-noise) ratio
would be quite high.
Worth a shot?
Hmm or we can just make use of the hardcopy feature in screen
(the terminal multiplexer)?
nazri.
Bram Moolenaar
Nazri Ramliy wrote:
Using the lowest level information about the screen is a good idea. We
don't test the communication with the terminal then, but that's a
separate issue, and there have been very few problems in that area the
last few years.
We can use the remote facility from one Vim to control another Vim.
Basically:
Start Vim: :!vim --servername VIMTEST -u {test vimrc}
Send command: :call remote_send("VIMTEST", ":edit file\<CR>")
Evaluate effect: :echo remote_expr("VIMTEST", screen_char(0, 0))
The only thing missing is the function to obtain the properties of a
screen character, such as screen_char(). That's easy to implement.
This only works when compiled with the remote server feature, thus it
won't work to test a small version. Not a real problem.
We can't get output at the command line this way. Accessing the message
history should help, but it doesn't store all the output. Perhaps a
testing mode can be enabled to store all output on the command line and
a function to obtain it.
A few more generic things would be needed, such as a check that there
was no error message.
Perhaps someone can give this a start to test the conceal feature? That
would be an excellent example of what we should be able to test this
way.
--
OLD WOMAN: Well, how did you become king, then?
ARTHUR: The Lady of the Lake, her arm clad in the purest shimmering samite,
held Excalibur aloft from the bosom of the water to signify by Divine
Providence ... that I, Arthur, was to carry Excalibur ... That is
why I am your king!
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD
/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
Tony Mechelynck
> From: Dominique_Pell�, Wed, July 28, 2010 5:18 pm
There is already
:runtime bugreport.vim
in the plain-vanilla distribution, which collects a huge lot of
information � in fact, usually too much for my health. ;-)
Oh, and BTW, how come your mailer always breaks threading when replying?
Best regards,
Tony.
--
There once was a fiesty young terrier
Who liked to bite girls on the derriere.
He'd yip and he'd yap,
Then leap up and snap;
And the fairer the derriere the merrier.
Steve Hall
[...]
>
> There is already
>
> :runtime bugreport.vim
>
> in the plain-vanilla distribution, which collects a huge lot of
This is a lot of raw data about variables, mappings, abbreviations,
registers, etc. I suppose we can assume that the setting of these Vim
features means that they worked, but there don't appear to be any
tests that interface with the user such as running system calls,
interpreted languages, dialogs, typed entry, etc.
> Oh, and BTW, how come your mailer always breaks threading when
> replying?
I sometimes have to use a browser-based mailer that is less than
respectable.