Advice needed: best practices for vim plugin testing

81 views
Skip to first unread message

Felipe Vieira

unread,
Oct 19, 2017, 2:10:06 PM10/19/17
to vim_use
Hi everyone,

I've been trying to develop a plugin and I'm used to writing testing for the
softwares I develop. The problem is that I cannot find a suitable testing
platform for vim plugins. This makes me feel uncomfortable in pushing
improvements made on my own fork of a bigger project (this may adversely impact
hundreds of users, and I think the codebase is complex enough; tests would
force some adherence to what is already coded and improve the plugin itself).

I have tried a couple of other vim plugins for testing with little/no success.

**What currently is the best practice for developing vim plugins?**
(preferably an official vim testing platform for plugins)

It is needless to say how important is to develop tests for software (this is
beyond the point here).

I don't thinks this information is very relevant but:

* I have some experience with vimscript
* I program in python most of the time
* The plugin is: https://github.com/python-mode/python-mode (my own fork:
https://github.com/fmv1992/python-mode)
* I am willing to learn whatever it takes to be able to develop testing for my
plugin (as long as it is something 'official/stable')

Best,

Felipe.

Marcin Szamotulski

unread,
Oct 19, 2017, 5:37:11 PM10/19/17
to vim...@googlegroups.com
Hi Felipe,

I've been missing a testing suite for vim too. One possible solution is
to use the same technique that vim is using itself to test the code.
There are now bunch of assert functions (:h assert_equal(), etc.).

Another thing that would be awesome is to have a PureScript backend
for VimL and develop vim plugins in a strongly typed language. That would
rule out a lot of bugs. There is Python backend for Purescript which
one could use to write plugins that are using vim python interface.

Cheers,
Marcin
signature.asc

Felipe M. Vieira

unread,
Oct 19, 2017, 9:51:09 PM10/19/17
to vim...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Marcin,

thanks for the tips.

I was aware of the 'assert_' family. They are useful if someone is checking
function correctness and the like.

However my concerns are broader as I'm trying to figure out ways of having a
vim project maintained by different people but enforcing that their coding
standards/modifications adhere to a test framework.

As such I was thinking of ways of vim syntax checking (if proposed git/github
modifications were syntactically correct) and their results as well (for
instance having a vim command issued like 'PythonEnfoceCodingStandard' and be
sure that a test buffer changed).

And to be 100% honest I don't know exactly what I want. That's why I do know
that there is a gap/need to contribute to vim in this point but I don't think I
have enough practice with coding to tackle this issue myself.

I'll let everyone know if I stumble upon something useful. Meanwhile lets hope
others contribute to this post.

Best,

- --
Felipe Martins Vieira
I have transitioned to a new pgp key. Check my statement at:
https://drive.google.com/open?id=1Hoaedh6tIss0P7TYHYZFIqyOy8jBbb7FNhYtO2RzCb8
New Key Fingerprint: 4723 61F6 81DE A694 4060 D2BE ABD6 D936 A5B9 9720
Mobile +55 11 971 066 250
-----BEGIN PGP SIGNATURE-----

iHUEARYIAB0WIQTPlFCAHE56WnFmPNa09RkZHowVJwUCWelWHgAKCRC09RkZHowV
JxMBAPwJobcxUzKXs1vUNb+oyXo6QZCN00v/7ROZf+ESPEFeIAEAgU00MKgzqMnJ
n3usjrOqC+0RVcYXS+XRWS/CunsAfgk=
=sEeB
-----END PGP SIGNATURE-----

Marcin Szamotulski

unread,
Oct 20, 2017, 3:09:38 AM10/20/17
to vim...@googlegroups.com
For testing syntax one could use `synstack()` and `synIDaddr()` to test
if the file has the right syntax groups.

Best regards,
Marcin
signature.asc

lith

unread,
Oct 20, 2017, 3:29:00 AM10/20/17
to vim_use
> I have tried a couple of other vim plugins for testing with little/no success.

Which one did you try? Are you looking for something like vader <https://github.com/junegunn/vader.vim>?

Regards

Luc Hermitte

unread,
Oct 20, 2017, 4:10:58 AM10/20/17
to vim use
Hi,

> However my concerns are broader as I'm trying to figure out ways of
> having a
> vim project maintained by different people but enforcing that their
> coding
> standards/modifications adhere to a test framework.

So, you're not looking for unit testing as there exists quite a few projects, but for linters it seems like vint
https://github.com/Kuniwak/vint

I've even seen an external tool for github (1) that automatically tries to execute the linter on code pushed -- alas I can't remember the name, or did it stop supporting vim, I can't tell.

Anyway, I usually just unit-test my plugin with a mixture of my vim-UT (1) and of vimrunner (3) that I've even registered in travis-ci which gives me little nice badges (4).

(1) https://github.com/marketplace
(2) https://github.com/LucHermitte/vim-UT/
(3) https://github.com/LucHermitte/vim-UT/blob/master/doc/rspec-integration.md
(4) https://github.com/LucHermitte/lh-vim-lib#lh-vim-lib-v400---

--
Luc Hermitte

tyru

unread,
Oct 20, 2017, 5:43:47 AM10/20/17
to vim...@googlegroups.com


2017/10/20 16:29 "lith" <mini...@gmail.com>:

> I have tried a couple of other vim plugins for testing with little/no success.

Which one did you try? Are you looking for something like vader <https://github.com/junegunn/vader.vim>?

Or if you want a full stack one, themis.vim is also the thing.


Regards

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_use" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marcin Szamotulski

unread,
Oct 20, 2017, 8:36:40 AM10/20/17
to vim...@googlegroups.com
Nice work, thanks for the links.

Best regards,
Marcin
signature.asc

Charles Campbell

unread,
Oct 20, 2017, 10:05:45 AM10/20/17
to vim...@googlegroups.com
Felipe Vieira wrote:
> Hi everyone,
>
> I've been trying to develop a plugin and I'm used to writing testing for the
> softwares I develop. The problem is that I cannot find a suitable testing
> platform for vim plugins. This makes me feel uncomfortable in pushing
> improvements made on my own fork of a bigger project (this may adversely impact
> hundreds of users, and I think the codebase is complex enough; tests would
> force some adherence to what is already coded and improve the plugin itself).
>
> I have tried a couple of other vim plugins for testing with little/no success.
> <snip>
Have you tried pchk? (http://www.drchip.org/astronaut/vim/index.html#PCHK)

The idea: create a file with a number of commands.

* Run :PChkTry and fix your plugin until you have it working. (your
plugin will be running in a separate vim window)
* Run :PChkMake -- this will, at every PChkSnapshot, take a hash of
the display of the remote vim window's contents and save it (along with
some geometry information, such as qty of windows, size of windows, etc)
* In the future: use :PChk -- that will compare the current plugin
performance with that of its previous performance (ie. if the hash is
the same, pretty much the display has not changed)

An excerpt from its help:

*PChkTry* *PChkMake* *PChk* *PChkStep*
Start a test by editing it, then run one of the following four commands:

:PChkTry : try the test. Will stop at :PChkSnapshot lines
:PChkMake : make an Expected/tst###.out file
:PChk : compare snapshot hashes currently obtained
with previously obtained expected test results
:PChkStep : step through each line of a test, not just at
:PChkSnapshots.


Test File Commands *PChkSnapshot* *PChkFeedkeys* *PChkPause*

:PChkSnapshot : runs a hashing algorithm over the remote display.
Result is saved in/compared with Expected/tst###.out
:PChkFeedkeys ... : send keys to server. ^X transla
*PChkTry* *PChkMake* *PChk* *PChkStep*
Start a test by editing it, then run one of the following four commands:

:PChkTry : try the test. Will stop at :PChkSnapshot lines
:PChkMake : make an Expected/tst###.out file
:PChk : compare snapshot hashes currently obtained
with previously obtained expected test results
:PChkStep : step through each line of a test, not just at
:PChkSnapshots.


Test File Commands *PChkSnapshot* *PChkFeedkeys* *PChkPause*

:PChkSnapshot : runs a hashing algorithm over the remote display.
Result is saved in/compared with Expected/tst###.out
:PChkFeedkeys ... : send keys to server. ^X translates to control
characters (ie. ^M is a ENTER)
:PChkPause : pause the test (uses |input()|). One may
* <cr> =continue
* c =check
* s =step
* t =try
:..cmd : send cmd to server to execute..
^\s*#... : echomsg comment in client vim (will not be sent
to server)
..cmd.. : send cmd to server to execute as a normal command
*..reply.. : netrw issues prompt; reply to it with this string
tes to control
characters (ie. ^M is a ENTER)
:PChkPause : pause the test (uses |input()|). One may
* <cr> =continue
* c =check
* s =step
* t =try
:..cmd : send cmd to server to execute..
^\s*#... : echomsg comment in client vim (will not be sent
to server)
..cmd.. : send cmd to server to execute as a normal command
*..reply.. : netrw issues prompt; reply to it with this string


An example of a test file (for netrw): (in a file called tst003)

# Make a file: % (called TmpFile)
# Delete the file
:let g:netrw_pchk= 1
:cd dir001
:e .
:PChkSnapshot
%
*TmpFile
:call setline(1,"testing")
:PChkSnapshot
:w
:e .
:PChkSnapshot
:call search("TmpFile","wc")
:PChkSnapshot
D
*y
:PChkSnapshot

The :... commands are vim commands. Without a ":" are vim normal-mode
commands/maps.

Hope that helps,
Chip Campbell

Felipe M. Vieira

unread,
Oct 22, 2017, 10:04:07 PM10/22/17
to vim...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hey all,

thanks for all of your replies. I will dutifully check those soon!

With so many options I feel like I was not looking at the right places.

I shall give a feedback on those in a couple of weeks.

Best,

- --
Felipe Martins Vieira
I have transitioned to a new pgp key. Check my statement at:
https://drive.google.com/open?id=1Hoaedh6tIss0P7TYHYZFIqyOy8jBbb7FNhYtO2RzCb8
New Key Fingerprint: 4723 61F6 81DE A694 4060 D2BE ABD6 D936 A5B9 9720
Mobile +55 11 971 066 250
-----BEGIN PGP SIGNATURE-----

iHUEARYIAB0WIQTPlFCAHE56WnFmPNa09RkZHowVJwUCWe1NpQAKCRC09RkZHowV
J850AQCj4MWdkM4HIardApr0W89QpuvbKV4wI6mlL9G9jq6K7wEA7tUVbT+IC6g2
Dkx6NqTRlITbzzoNMfyfjPumbW6mTAo=
=Tpmg
-----END PGP SIGNATURE-----

Lifepillar

unread,
Oct 23, 2017, 11:38:13 AM10/23/17
to vim...@googlegroups.com
On 19/10/2017 23:36, Marcin Szamotulski wrote:
> On 11:10 Thu 19 Oct , Felipe Vieira wrote:
>> Hi everyone,
>>
>> I've been trying to develop a plugin and I'm used to writing testing for the
>> softwares I develop. The problem is that I cannot find a suitable testing
>> platform for vim plugins.
>
> [...]
>
> Hi Felipe,
>
> I've been missing a testing suite for vim too. One possible solution is
> to use the same technique that vim is using itself to test the code.
> There are now bunch of assert functions (:h assert_equal(), etc.).

Here you may find a minimal script based on Vim functions:

https://gist.github.com/lifepillar/b5018945561e024eeb9fc57650fc5d61

Just write the tests at the top of the script using the above-mentioned
assert functions, then source it.

Depending of what you are looking for, that might be all you need. For
something more sophisticated than that, you may take a look at Vim's
own test suite, from which my snippet is derived.

Enjoy,
Life.

Felipe M. Vieira

unread,
Feb 12, 2018, 12:07:37 PM2/12/18
to vim...@googlegroups.com
Hi everyone,

Thanks for all the valuable suggestions.

The method I found out to be the most suitable, dependecy free and sane
(this is highly personal) was to write bash tests which in turn invokes
vim with a given number of flags and which sources a given script which
ends with either "qall!" (exit code 0) or "cquit!" (exit code 1). You
can check an example of such tests in:

1. https://github.com/python-mode/python-mode/blob/develop/tests/test.sh
2. https://github.com/fmv1992/vim_dictionary/blob/dev/tests/test.sh

Example:

Bash invokes:
$ vim -i NONE -u /tmp/vimdict_vimrc -c 'source ./test_procedures_vimscript/lookup1.vim' /tmp/vimdict_disposable.py

File `./test_procedures_vimscript/lookup1.vim` has:
~~~~~~
" [...]
" Put assertions before.
call assert_true(bufexists('vim_dictionary-scratch'))

if len(v:errors) > 0
echom "Exiting lookup1 with errors" . join(v:errors, '|')
cquit!
else
qall!
endif
~~~~~~

It is also worth noting that vim's `verbosefile` option is valuable as
well.

Best wishes for the vim community,
signature.asc
Reply all
Reply to author
Forward
0 new messages