test suite for vim

[Martin Guy]

Hi, fellow vi-lovers!
I maintain another vi clone, xvi, and have developed an automated
test suite for POSIX compliance. Unlike the official POSIX test suite,
which only uses : commands, writes the files out and checkc their
contents, xvi's one runs the editor in a virtual terminal and scrapes
the visual output to check screen correctness. Needless to say, vim
passes almost all of the tests, including hanoi and maze, but it does
spot a few bugs, incompatabilities and POSIX uncompliances. Some
"failures" are false, due to the startup screen not being blank.

To try it out, install expect, tclsh and tk and

git clone http://github.com/martinwguy/xvi
cd xvi/test
VI=vim make check

Of course you would be welcome to incorporate the test framework into
vim itself.

Thanks for your work - vim has helped enormously in correcting bugs in xvi.

Blessings

M

[Christian Brabandt]

Can you show sample output of failed tests?

Best,
Christian
--
In der Welt kommt's nicht drauf an, dass man die Menschen kenne,
sondern dass man im Augenblick klüger sei als der vor uns Stehende.
Alle Jahrmärkte und Marktschreier geben Zeugnis.
-- Goethe, Maximen und Reflektionen, Nr. 235

[Bram Moolenaar]

Interesting. You do run the tests with $VIM_POSIX set? That makes Vim
run as Posix-compatible as it can.

Screenshot tests can be a nice generic testing tool, but can also check
for irrelevant changes. Do you apply some kind of mask to ignore parts
that don't matter?

I'm not sure Posix complience is even moderately important. Vim is
considered the reference by now.

--
Your mouse has moved. Windows must be restarted for the change
to take effect. Reboot now?

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Martin Guy]

On 10/03/2020, Bram Moolenaar <Br...@moolenaar.net> wrote:
> Margin Guy wrote:

! :D

> Interesting. You do run the tests with $VIM_POSIX set? That makes Vim
> run as Posix-compatible as it can.

Same result, 13 "failures", so either VIM is already a POSIX hero, or
the differences are so small not to notice. Incidentally, there was
already "POSIXLY_CORRECT". Maybe it should spot that too? Just a
thought.

Actually, vim only reacts to VIM_POSIX to set (it says) different
defaults for "shm" and "cpo", but I see no difference in the setting
of cpo with and without.

> Do you apply some kind of mask to ignore parts
> that don't matter?

Yes, it detects whether you're testing vim in a few places and drops
some false positives, but not everywhere.

> Vim is considered the reference by now.

The most widely diffuse vi, for sure, but I consider
classic/tradditional/heirloom vi as the reference (Bill Joy's
original) for what it "should" do, with those few POSIX modifications
to that, more to make sure that complex scripts like maze and hanoi
work everywhere, not just in the vi that they were written in.

Ti give an idea, here's the list from one test, "Interrupted"

- Control-C's status line should be "Interrupted" instead of a help
message (POSIX llimply says, "the terminal shall be alerted"). If it
matters, maybe it could switch behaviour if *POSIX*

- If you press ^C after one character of a two-character command, vim
does abort the 2-letter command but says nothing. Same thing if
interrupted in insert, replace and command modes, or when running a
recursive "map!" like
:map! x a[Ctrl-V][Enter]xy[Enter]
ax
(PS Congrats on spotting recursive "map"s instead. vi's the only one
that does that!)

Cheers

M