[vim/vim] Add tests for tiny and small builds (#6696)

[K.Takata]

• Add testtiny (and tiny) make target.
• Bring back some old style tests (test20, etc.) which can be executed on tiny builds.
• Delete test1.
  • New style tests don't need it anymore.
  • All old style tests can be run on tiny builds, so there's no need to generate small.vim.
• Refactor testdir/Make_ming.mak.
  • Add report target like other makefiles.
  • Convert fileformat of old style tests in dostmp like testdir/Make_dos.mak.
  • Always use cmd.exe for the shell, because it is difficult to support both cmd.exe and unix shell in one makefile.

---

You can view, comment on, or merge this pull request online at:

  https://github.com/vim/vim/pull/6696

Commit Summary

• Add tests for tiny and small builds
• Refactor testdir/Make_ming.mak
• Add testtiny target
• Fix tests
• Update makefiles
• Adjust some more
• Update doc
• Update makefile

File Changes

• M .travis.yml (6)
• M Filelist (6)
• M Makefile (2)
• M runtime/doc/testing.txt (6)
• M src/Make_mvc.mak (18)
• M src/Makefile (9)
• M src/testdir/Make_all.mak (25)
• M src/testdir/Make_dos.mak (17)
• M src/testdir/Make_ming.mak (95)
• M src/testdir/Makefile (49)
• M src/testdir/runtest.vim (4)
• A src/testdir/test20.in (27)
• A src/testdir/test20.ok (10)
• A src/testdir/test22.in (13)
• A src/testdir/test22.ok (4)
• A src/testdir/test23.in (15)
• A src/testdir/test23.ok (2)
• A src/testdir/test24.in (0)
• A src/testdir/test24.ok (32)
• A src/testdir/test28.in (0)
• A src/testdir/test28.ok (2)
• A src/testdir/test36.in (108)
• A src/testdir/test36.ok (99)
• A src/testdir/test41.in (24)
• A src/testdir/test41.ok (3)
• A src/testdir/test42.in (0)
• A src/testdir/test42.ok (0)
• M src/testdir/test_options.vim (4)

Patch Links:

• https://github.com/vim/vim/pull/6696.patch
• https://github.com/vim/vim/pull/6696.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[codecov[bot]]

Codecov Report

Merging #6696 into master will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@

##           master    #6696      +/-   ##

==========================================

- Coverage   88.25%   88.24%   -0.01%     

==========================================

  Files         147      147              

  Lines      160451   160451              

==========================================

- Hits       141603   141593      -10     

- Misses      18848    18858      +10     

Impacted Files	Coverage Δ
src/netbeans.c	76.07% <0.00%> (-0.23%)	⬇️
src/gui.c	63.30% <0.00%> (-0.15%)	⬇️
src/getchar.c	85.97% <0.00%> (-0.15%)	⬇️
src/search.c	91.33% <0.00%> (-0.11%)	⬇️
src/ex_docmd.c	94.10% <0.00%> (-0.06%)	⬇️
src/message.c	88.47% <0.00%> (-0.05%)	⬇️
src/ex_getln.c	91.33% <0.00%> (-0.05%)	⬇️
src/version.c	92.13% <0.00%> (ø)
src/vim9compile.c	93.03% <0.00%> (ø)
src/gui_gtk_x11.c	58.73% <0.00%> (+0.04%)	⬆️
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d91bcb...014bbd2. Read the comment docs.

[Bram Moolenaar]

"make test" should run all the tests. The "tiny" tests will always work and the new style tests will quickly bail out if the +eval feature is missing.
E.g. the first targets in testdir/Make_dos.mak should probably be:

nongui: nolog tinytests newtests report

tiny: nolog tinytests report

tinytests: $(SCRIPTS_TINY_OUT)

[K.Takata]

@k-takata pushed 1 commit.

• 4d558aa Run tiny tests also on nongui and gui tests, etc.

—
You are receiving this because you are subscribed to this thread.

View it on GitHub or unsubscribe.

[K.Takata]

@k-takata pushed 1 commit.

• 8576422 Update Make_amiga.mak

—
You are receiving this because you are subscribed to this thread.

[K.Takata]

Updated based on your suggestion.
Also updated Make_amiga.mak.
(Not sure how to do with Make_vms.mms.)

[K.Takata]

@k-takata pushed 1 commit.

• 530a06e Update Make_vms.mms

—
You are receiving this because you are subscribed to this thread.

[K.Takata]

It seems that tests on GitHub Actions are sometimes unstable. E.g.

Failures: 
	From test_autocmd.vim:
	Found errors in Test_bufunload_all():
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[454]..function RunTheTest[39]..Test_bufunload_all line 25: Expected True but got 0

Need some wait here?
https://github.com/vim/vim/blob/7ac616cb0a52bc72b449e19cf9db93bee116c15a/src/testdir/test_autocmd.vim#L1452-L1453
I will look into this later.

[Yegappan Lakshmanan]

Hi,

On Wed, Aug 12, 2020 at 4:06 PM K.Takata <vim-dev...@256bit.org> wrote:

It seems that tests on GitHub Actions are sometimes unstable. E.g.

Yes. I have seen this and other failures intermittently. Also, it looks like the GitHub

actions are run even for changes to the private branches also. I think the actions

should be run only for the pull requests.

Regards,

Yegappan

[vim-dev ML]

[K.Takata]

Also, it looks like the GitHub actions are run even for changes to the private branches also.

Do you mean that it runs when you push a commit into your (forked) repository?
If you don't like it, you can turn off GHA from the settings of your repository.

—
You are receiving this because you commented.

[Yegappan Lakshmanan]

Hi,

On Wed, Aug 12, 2020 at 4:25 PM K.Takata <vim-dev...@256bit.org> wrote:

Also, it looks like the GitHub actions are run even for changes to the private branches also.

Do you mean that it runs when you push a commit into your (forked) repository?
If you don't like it, you can turn off GHA from the settings of your repository.

Yes. I have now turned it off.

Thanks,

Yegappan

[vim-dev ML]

[Bram Moolenaar]

Closed #6696 via b96a32e.

—
You are receiving this because you commented.

[Bram Moolenaar]

Thanks for the PR. I renumbered the tests to avoid gaps. Git can't diff binary files, I hope I got them right.

—
You are receiving this because you commented.

[K.Takata]

Thank you for merging.
I have some comments:

• I thought that test1 was not needed anymore. But, VMS has test77a and it sources small.vim. Maybe, test1 is still needed only for VMS. (But test1 can be simplified because tiny.vim, mbyte.vim and mzscheme.vim are not used now.)
• Now the clean target of testdir/Makefile doesn't delete tiny.vim, small.vim, mbyte.vim and mzscheme.vim.
  I'm rethinking whether this was too early. If someone runs the tests in an old version, then updates the source code and runs make clean, then small.vim and other files will be left behind.
  And there's inconsistency. testdir/Make_dos.mak and testdir/Make_ming.mak still clean those files.
• Should testdir/README.txt mention about old style tests?
• Are the tiny tests enough? You said in the other PR:

  Just in case we manage to do something stupid, like a misplaced #ifdef causes them to exit, or .vimrc is not sourced, stuff like that.

—
You are receiving this because you commented.

[Bram Moolenaar]

Ken Takata wrote:

> Thank you for merging.
> I have some comments:
>

> * I thought that test1 was not needed anymore. But, VMS has test77a

> and it sources small.vim. Maybe, test1 is still needed only for VMS.
> (But test1 can be simplified because tiny.vim, mbyte.vim and
> mzscheme.vim are not used now.)

We can do the same as in runtest.vim. I hope someone with access to a
VMS system can verify this works.

> * Now the `clean` target of `testdir/Makefile` doesn't delete

> tiny.vim, small.vim, mbyte.vim and mzscheme.vim.
> I'm rethinking whether this was too early. If someone runs the tests
> in an old version, then updates the source code and runs `make

> clean`, then small.vim and other files will be left behind.

I don't think we need to worry about this. In case it happens and gets
noticed, it's easy enough to delete the files manually. But mostly the
files will not be noticed, and they are tiny, thus it doesn't really
matter. I have quite a few of these stray file laying around. That's
why the "Filelist" file is a bit strict about what to include.
Fortunately, when adding a file git always requires explicitly adding
it, that also avoids mistakes. And I have a script that checks what
Filelist matches with what's in the git repository.

> And there's inconsistency. `testdir/Make_dos.mak` and
> `testdir/Make_ming.mak` still clean those files.

I'll remove those lines.

> * Should `testdir/README.txt` mention about old style tests?

I'll add a few words.

> * Are the tiny tests enough? You said in the other [PR](https://github.com/vim/vim/pull/6683#issuecomment-672205194):

> > Just in case we manage to do something stupid, like a misplaced #ifdef causes them to exit, or .vimrc is not sourced, stuff like that.

It's probably fine as it is now. In case we find a mistake that gets
fixed we can add a test for that.

--
hundred-and-one symptoms of being an internet addict:
199. You read this entire list of symptoms, looking for something
that doesn't describe you.

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

—
You are receiving this because you commented.

[K.Takata]

It's probably fine as it is now. In case we find a mistake that gets fixed we can add a test for that.

I got it.
Also, thank you for fixing my mistake in 8.2.1456.

—
You are receiving this because you commented.