[vim/vim] Failing tests, possibly due to 50 ms sleeps (#6754)
Jakub Kądziołka
We're observing nondeterministic test failures, like these:
Failures:
From test_autocmd.vim:
Found errors in Test_CursorHold_autocmd():
Run 1:
command line..script /tmp/guix-build-vim-full-8.2.1471.drv-0/source/src/testdir/runtest.vim[454]..function RunTheTest[39]..Test_CursorHold_autocmd line 21: Expected ['1', '2', '3'] but got ['1', '3']
Run 2:
command line..script /tmp/guix-build-vim-full-8.2.1471.drv-0/source/src/testdir/runtest.vim[488]..function RunTheTest[39]..Test_CursorHold_autocmd line 21: Expected ['1', '2', '3'] but got ['1', '2']
Run 3:
command line..script /tmp/guix-build-vim-full-8.2.1471.drv-0/source/src/testdir/runtest.vim[488]..function RunTheTest[39]..Test_CursorHold_autocmd line 21: Expected ['1', '2', '3'] but got ['1', '2']
Flaky test failed too often, giving up
From test_terminal3.vim:
Found errors in Test_term_mouse():
Run 1:
Caught exception in Test_term_mouse(): Vim(call):E684: list index out of range: 0 @ command line..script /tmp/guix-build-vim-full-8.2.1471.drv-0/source/src/testdir/runtest.vim[454]..function RunTheTest[39]..Test_term_mouse, line 73
Run 2:
Caught exception in Test_term_mouse(): Vim(let):E684: list index out of range: 0 @ command line..script /tmp/guix-build-vim-full-8.2.1471.drv-0/source/src/testdir/runtest.vim[488]..function RunTheTest[39]..Test_term_mouse, line 41
Run 3:
Caught exception in Test_term_mouse(): Vim(call):E684: list index out of range: 0 @ command line..script /tmp/guix-build-vim-full-8.2.1471.drv-0/source/src/testdir/runtest.vim[488]..function RunTheTest[39]..Test_term_mouse, line 64
Run 4:
Caught exception in Test_term_mouse(): Vim(let):E684: list index out of range: 0 @ command line..script /tmp/guix-build-vim-full-8.2.1471.drv-0/source/src/testdir/runtest.vim[488]..function RunTheTest[39]..Test_term_mouse, line 41
Run 5:
Caught exception in Test_term_mouse(): Vim(call):E684: list index out of range: 0 @ command line..script /tmp/guix-build-vim-full-8.2.1471.drv-0/source/src/testdir/runtest.vim[488]..function RunTheTest[39]..Test_term_mouse, line 64
Flaky test failed too often, giving up
A thing these tests have in common is that 50 ms sleep calls are used for synchronization. It seems that this is too small a number, even when the test is marked as flaky. Ideally, something like "wait until file is written" or "wait until child vim is waiting for keystrokes again" would be used instead of a hardcoded length of time, but I understand this is not always possible.
Perhaps it would be desirable to increase this delay? I understand that this has an effect on test suite speed, but if the tests are always looped due to flakiness, maybe this would help reduce that, and as a result speed the test suite up?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Yegappan Lakshmanan
vim-dev ML
test_autocmd.vim and test_terminal3.vim
and check whether that fixes this problem?
Thanks,
Yegappan
Jakub Kądziołka
I am currently running 10 back-to-back builds to test your suggested change. Though, the above commit only resolves this partially, right?
—
You are receiving this because you commented.
Jakub Kądziołka
It seems that replacing term_wait with TermWait does indeed fix this issue. A quick grep suggests that term_wait is being used in some other tests - maybe it would be worth it to phase it out entirely?
—
You are receiving this because you commented.
Yegappan Lakshmanan
It seems that replacing term_wait with TermWait does indeed fix this issue. A quick grep suggests that term_wait is being used in some other tests - maybe it would be worth it to phase it out entirely?
vim-dev ML
Jakub Kądziołka
Ah, I didn't notice that commit. Thank you!
—
You are receiving this because you commented.