[vim/vim] Screen dumps may omit lines for test files with lines longer than 75 columns (Issue #14245)

Aliaksei Budavei

unread,

Mar 20, 2024, 7:13:29 PMMar 20

to vim/vim, Subscribed

Steps to reproduce

This problem showed up a few times as noted in the following
PRs: #14105, #14120, #14150.

It can be reproduced as follows:

Copy a dots.dots file into runtime/syntax/testdir/input:

1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
9. . . . . . . . . . . . .   winwidth(0): 75 . . . . . . . . . . . . . . . 10
11 . . . . . . . . . . . .  winheight(0): 19 . . . . . . . . . . . . . . . 12
13 . . . . . . . . . . . . . .  ruler  . . . . . . . . . . . . . . . . . . 14
15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
27 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
29 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
31 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
33 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
35 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
37 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
41 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
43 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
45 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
47 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
49 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
51 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
53 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
55 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
57 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
59 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
61 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
63 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
65 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
67 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
69 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
71 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
73 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
75 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
77 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
79 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
81 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
83 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
85 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
87 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
89 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
91 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
93 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
95 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
97 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
99 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
101. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
103. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
105. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
107. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
109. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
111. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
113. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
115. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

Run tests for the syntax files:

cd runtime/syntax/
make clean test

Note that with the following manoeuvre you can negotiate the
absence of ../../src/vim for local experimenting.

cd src/
test -e vim || ln -s $(command -v vim)
cd ../runtime/
export VIMRUNTIME=$(pwd)
cd syntax/
make clean test

# cd src/ && unlink vim
# unset VIMRUNTIME

Examine a series of newly generated screen-dump files,
paying attention to the continuity of line numbers on the
wrapped-over lines:

ls testdir/failed/
../../src/vim --clean testdir/input/dots.dots \
  '+call term_dumpload("testdir/failed/dots_00.dump")' \
  '+call term_dumpload("testdir/failed/dots_01.dump")' \
  '+call term_dumpload("testdir/failed/dots_02.dump")' \
  '+call term_dumpload("testdir/failed/dots_03.dump")' \
  '+call term_dumpload("testdir/failed/dots_99.dump")'

Out of all 58 (116) lines of 76-78 columns each, the runs of
present lines are: 1-18-@@@, 31-48-@@@, 67-84-@@@, 103-116,
99-116. Now, @@@-lines are parts of lines in their own
right, but their content is hidden, so they are as good as
missing. As can be seen, the missing lines are:
@@@-20-30 (12), @@@-50-66 (18), @@@-86-98 (14).

Expected behaviour

No lines should be missing.

Version of Vim

v9.1.0192

Environment

Any.

Logs and stack traces

No response

—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.

Aliaksei Budavei

unread,

Mar 20, 2024, 7:19:13 PMMar 20

to vim/vim, Subscribed

With the following script, current problematic input/ files
can be discovered.

let outfile = '/tmp/longlines.txt'

while 1
    call cursor(1, 1)
    let bufname = bufname('%')

    while 1
        let lnum = search('\%76v.', 'eW')

        if lnum < 1
            break
        endif

        let width = strdisplaywidth(getline(lnum))
        call writefile([printf("%4d: %4d: %s", width, lnum, bufname)], outfile, 'a')
    endwhile

    try
        next
    catch /\<E16[35]:/
        quit
    endtry
endwhile

cd runtime/syntax/testdir/input/
vim --clean -S /tmp/collector.vim *

Only three files out of all listed by the script have some
lines missing:

sh_05.sh (line 357 => no 356 in sh_05_20.dump, an empty
line).
sh_07.sh (line 70 => no 68 and 69 in sh_07_04.dump,
two empty lines).
vim_keymap.vim (lines ?? => no lines from 7 to 14 in
vim_keymap_01.dump) (Almost every line in this file has
lots of trailing space, yet it is unclear whether any
of such space is significant. And it is a good specimen
to test against).

For the remaining files, the stroke of luck is in the fact
that the lines extending over 75 columns appear among either
the first lines or the last lines of a file and so end up in
*00.dump or *99.dump, respectively.

The fourth file, sh_08.sh, should have # as its first line
for sh_08_04.dump; but this line is also in sh_08_99.dump.

—
Reply to this email directly, view it on GitHub.

You are receiving this because you are subscribed to this thread.

Aliaksei Budavei

unread,

Mar 20, 2024, 7:51:08 PMMar 20

to vim/vim, Subscribed

Two related deficiencies can be identified with the current
implementation of testdir/runtest.vim with respect to
wrapped lines.

The cursor advanced with the G command would hop over
wrapped lines or wouldn't move at all in a degenerate case
of a single long line (and so other lines vanish):

https://github.com/vim/vim/blob/978178823b7c62a0249411f3d1f584f8a3144c5d/runtime/syntax/testdir/runtest.vim#L213-L214

To address it, let's consider the gj command for movement:

	let topline += 18
	" With "g^", align the contents of &ruler for the current "gj" and
	" the succeeded "G".
	call term_sendkeys(buf, '18gjztg^')

This would require a cooperative adjustment for linecount to
correctly maintain the progress of a Vim instance driven
with term_sendkeys().

The line count calculation:

https://github.com/vim/vim/blob/978178823b7c62a0249411f3d1f584f8a3144c5d/runtime/syntax/testdir/runtest.vim#L103

should, in addition, allow for as many new "extra" lines as
are claimed by the ends of the wrapped lines. This adjusted
total should be arrived at in a Vim instance that lays out
parts of a test file in its 75x20 window for screen dumping
and where strdisplaywidth() can be taken advantage of; and
since this would be another Vim instance, distinct from the
one that drives the logic of testdir/runtest.vim, it would
be necessary to arrange a once only IPC between them to hand
over the total, with some busy waiting to manage asynchrony.

Knowing that out of the three OS platforms for which jobs
are scheduled in .github/workflows/ci.yml, XTerm is the
terminal of choice for testing on linux and mac; I propose
that we leverage its API (:help terminal-api) for IPC, and
use a plain file as a fallback for any platform (e.g. Linux
consoles at Ctrl-Alt-F1 etc.). In the current prototype,
there is ipc/dispatch.vim, whose sole purpose is to pick
some IPC implementation. It can be seen that a ping message
will be once sent in order to probe whether XTerm API is
supported for this Vim instance, despite to what &term may
advertise or what term_setapi() may have imposed.

This is still work in progress, try:

cp testdir/input/vim_keymap.vim{,.fix}

then, change as follows:

  for fname in glob('input/*.fix', 1, 1)

and run make clean test again. The count for gj needs to
vary, here: [22, 18, 16]. It is trivial to exchange a list
of offsets instead of a line count, but it is still unclear
as to what to look for for offsets.

—
Reply to this email directly, view it on GitHub.

You are receiving this because you are subscribed to this thread.

Christian Brabandt

unread,

Apr 2, 2024, 3:19:45 PMApr 2

to vim/vim, Subscribed

Thanks for working on this. Yes I agree that making this change is beneficial. The xterm api may be nice, but can we keep it simple and just always use the file as IPC? Or is this too slow?

This whole thing is getting a bit complex. We need to make sure we still understand what is going on, how to debug this and how to (re-)create all the screendumps (also this needs to be understandable for new contributors.

—
Reply to this email directly, view it on GitHub.

You are receiving this because you are subscribed to this thread.

Aliaksei Budavei

unread,

Apr 2, 2024, 5:57:18 PMApr 2

to vim/vim, Subscribed

All right, let's use a file only. I'll try to carve out
some time this week to work on this.

—
Reply to this email directly, view it on GitHub.

You are receiving this because you are subscribed to this thread.

Aliaksei Budavei

unread,

May 25, 2024, 4:36:44 PMMay 25

to vim/vim, Subscribed

Another promising approach for not losing lines is currently
in the works.

Instead of trying to pre-calculate vertical line offsets for
long lines and further adjusting these values for zt and
[count]gj, let's execute various moving commands and rely on
line marking and additional movement to position lines at
desired offsets, carefully preserving compatibility for the
&scrolloff and &ruler values inherited from defaults.vim.
Now that there is no ahead-of-time arithmetic for offsets,
there is nothing to relay via IPC before screen dumping, so
a file for exchange of data is no longer necessary; but in
order to keep track of progress through an input/ file,
let's simply read the rightmost end of the ruler line from
the terminal buffer, looking for All or Bot for a cue to
finish intermediate screen dumping.

It turns out that a few more files should be generated, with
one for each of some input/ files, for current line progress
calculations are somewhat inexact and make some bottom lines
only recorded in *_99.dump files, and even when all lines
are short. On the other hand, a few less files with closed-
folds lines should be generated, with current implementation
failing to determine whether a range of lines is collapsed
to a fold. (As folded lines do not wrap their &foldtext
contents, text after column 75 will still be cut as before.)

With these changes applied, we can push the lossless line
length limit from winwidth(1) (75) characters up to
winwidth(1) * winheight(1) (1425) characters.

—
Reply to this email directly, view it on GitHub.

You are receiving this because you are subscribed to this thread.

Aliaksei Budavei

unread,

May 25, 2024, 4:51:43 PMMay 25

to vim/vim, Subscribed

Here is another topic branch with a new set of proposed
changes.

For reviewers, a couple head-scratchers to watch for when
looking at dumps/ files viewed through term_dumpload():

*_01.dump files may have 6 bottom lines from *_00.dump
files repeated owing to &scrolloff, but will have other
number of lines repeated when some of them are wrapped;
note that this variance directly comes from changing
&scrolloff.
vim_keymap_*.dump can be challenging; remember to have
their input/ source confined to the 75 columns width
to align wrapped over lines when comparing file contents.

Otherwise, *_02.dump should start with the last line of
*_01.dump, *_03.dump should start with the last line of
*_02.dump, and so on (if the last line is wrapped, then all
wrapped parts should still be at the start of another file).
A *.dump file named with the greatest number but not 99
must have the last lines of its input/ source.

I imagine, reviewing dumps/ in batches over several days
would be the least demanding and exhausting.

I have run into another asynchronous oddity that manifested
itself in a delayed (outraceable?) :redraw. It seems as if
:redraws executed in the terminal buffer can be postponed;
however, additionally executing :redraw from its parent
process buffer appears to do the job. There is one such
:redraw put into a new IsWinNumOneAtEOF() function right
before the ruler line from the terminal buffer is queried.

—
Reply to this email directly, view it on GitHub.

You are receiving this because you are subscribed to this thread.

Aliaksei Budavei

unread,

Jun 7, 2024, 4:21:10 PM (9 days ago) Jun 7

to vim/vim, Subscribed

@chrisbra, @dkearns, have you had any chance to try out the
latest solution to this problem of missing lines in dumps/
files?

—
Reply to this email directly, view it on GitHub.

You are receiving this because you are subscribed to this thread.

Christian Brabandt

unread,

Jun 9, 2024, 1:05:43 PM (8 days ago) Jun 9

to vim/vim, Subscribed

Thanks. We can certainly try it out, if this is more reliable. What is the idea with the self-test please? Can you document this please?

—
Reply to this email directly, view it on GitHub.

You are receiving this because you are subscribed to this thread.

Aliaksei Budavei

unread,

Jun 9, 2024, 3:35:13 PM (7 days ago) Jun 9

to vim/vim, Subscribed

The files housed in input/selftestdir/ are additional test
files that proved useful for development and refinement of
the proposed algorithm and can be valuable for any future
improvement (or replacement) of its parts. Since the syntax
is made-up, these files should not get in the way of regular
syntax file testing.

I have pushed a input/selftestdir/README.txt file saying
about that much.

Another immediate (?) change to consider is the removal of
all *_99.dump files and the code handling their generation:

https://github.com/vim/vim/blob/aa925eeb97bd294d4a5253a3194949a37cbc8365/runtime/syntax/testdir/runtest.vim#L221-L226

The new algorithm turns a *_99.dump file obsolete because
all lines written in such a file will already be present in
lower-numbered *.dump files. Besides, should the total
*.dump count for any input/ file ever break a hundred
(input/vim_ex_commands.vim is at 66), we would have to face
the E953 error from term_dumpwrite() for an already claimed
*_99.dump name.

—
Reply to this email directly, view it on GitHub.

You are receiving this because you are subscribed to this thread.

Aliaksei Budavei

unread,

Jun 10, 2024, 3:21:10 PM (6 days ago) Jun 10

to vim/vim, Subscribed

There are several ‘orphaned’ dumps/ files generated by some
other means or often ‘exposed’ at a later time (there are no
matching input/ files anymore):

html_{0{0..7},99}.dump (input/html.html before 110dd9),
vim_{00,99}.dump (input/vim.vim before 4e043b),
vim_ex_func_{00,99}.dump (no initial file at 61887b),
vim_syntax_{0{0..8},99}.dump (input/vim_syntax.vim
before e5c9ba),
yaml.yaml_{0{0..8},99}.dump (no initial file at cc7597).

The above Vim files are likely outdated duplicates, but not
vim_ex_func_*, and can be removed, can't they, @dkearns?

Because of past CI failures, the HTML input/ file is masked
for now. Its dumps/ files need to catch up with the changes
made for testdir/runtest.vim at a2adde (set t_ZH= t_ZR=).
Can these settings also make CI happy with HTML?

Looking at the above YAML files, there are many wrapped long
lines in them, so their absent input/ file could be a good
source to try on the new algorithm.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.

dkearns

unread,

Jun 10, 2024, 3:39:33 PM (6 days ago) Jun 10

to vim/vim, Subscribed

The above Vim files are likely outdated duplicates, but not vim_ex_func_*, and can be removed, can't they, @dkearns?

Yes, thanks. There's so many artifacts created with these tests, perhaps we can check for spurious dump files as part of the test run?