[vim/vim] Vim doesn't preserve the order of messages when callbacks are defined for job_start() or term_start() (Issue #16354)

[bfrg]

Steps to reproduce

Run the following script with vim --clean -S test.vim:

vim9script

const script: list<string> =<< trim EOF
    #!/bin/sh
    printf >&2 '1 stderr\n2 stderr\n'
    printf '3 stdout\n'
EOF

const cmd = '/tmp/test.sh'
writefile(script, cmd)
setfperm(cmd, 'rwxr-xr-x')

const ptybuf = term_start(cmd, {
    err_cb: (ch: channel, msg: string) => {
        echowindow 'stderr:' msg
    },
    out_cb: (ch: channel, msg: string) => {
        echowindow 'stdout:' msg
    },
})

Result: The order of the output of test.sh is not preserved when out_cb and err_cb are set. There's also some weird indentation that's not supposed to be there:

screenshot-2024-12-31_150231.png (view on web)

Also notice the output of :messages:

stdout: 3 stdout^M
stderr: 1 stderr^@2 stderr

There's an extra ^M character for some reasons.

The same behavior can be observed when job_start() is used instead of term_start().

I've looked into the source code and it seems like Vim saves the output (stderr and stdout) from a job but it won't save the order they are received in. Then in the main event loop it just checks if there are messages from a job and if so, it will invoke its callbacks. It looks like out_cb is always invoked before err_cb. And because the shell script prints to stderr and stdout almost simultaneously, the stdout part is processed first.

When adding a short sleep command between the two printf command, the output is processed in the right order, but for some reasons there are unexpected indentations:

#!/bin/sh
printf >&2 '1 stderr\n2 stderr\n'
sleep 0.1
printf '3 stdout\n'

screenshot-2024-12-31_152329.png (view on web)

Expected behaviour

I would have expected the following output in the terminal buffer:

1 stderr
2 stderr
3 stdout

Version of Vim

9.1.954

Environment

Linux, any terminal.

—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.Message ID: <vim/vim/issues/16354@github.com>

[Ben Jackson]

stdout and stderr are both synchronised streams at an OS level. This is not solvable and it’s not a vim problem.

This could happen with any program attempting to read stdout and stderr of some subprocess from different streams.

The only solution is to redirect stderr to stdout so they are synchronised at source.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: <vim/vim/issues/16354/2566577602@github.com>

[bfrg]

Then why does it work as expected when both callbacks are removed?

—
Reply to this email directly, view it on GitHub.

You are receiving this because you are subscribed to this thread.Message ID: <vim/vim/issues/16354/2566629041@github.com>

[Christian Brabandt]

Closed #16354 as completed via 9e55474.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: <vim/vim/issue/16354/issue_event/23792657432@github.com>

[bfrg]

bfrg left a comment (vim/vim#16354)

@h-east Thank you for the fix! The indentation issue is definitively fixed.

But for some reasons the order is still not always preserved.

Running

1. vim --clean -S test.vim: order is preserved. Output:

1. 1 stderr
   2 stderr
   3 stdout
   

2. vim --clean -S test.vim test.vim: order is not preserved. Output:

   3 stdout
   1 stderr
   2 stderr
   

3. vim -u NONE -S test.vim test.vim order is preserved. Output:

1. 1 stderr
   2 stderr
   3 stdout
   

I'm not sure why sourcing and editing test.vim is causing an issue in vim --clean.

Tested on Linux.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: <vim/vim/issues/16354/4165612334@github.com>

[h_east]

h-east left a comment (vim/vim#16354)

@bfrg Currently, maintaining the output order of stdout and stderr with callbacks is difficult.
I have added information regarding this to the documentation.
Please check it.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: <vim/vim/issues/16354/4177603126@github.com>

[h_east]

h-east left a comment (vim/vim#16354)

Hi @bfrg,

FYI, patch 9.2.0224 (#19776), which fixed this issue, turned out to cause a regression (#20041) for callers that rely on receiving raw chunks from term_start()'s out_cb/err_cb — vim-fugitive's :Git status output got garbled because embedded newlines were being consumed.

I've opened #20045 to partially revert that patch:

• Kept: the LF→CR+LF conversion for the terminal buffer indentation, and the stderr-before-stdout ordering in channel_parse_messages() (your original issues 1 & 2).
• Reverted: the per-line splitting of the RAW callback data. out_cb/err_cb will again receive the raw chunk as read from the OS, including embedded NL and any trailing CR from the PTY line discipline.

For your original use case, the ^M and ^@ you observed in :messages are rendering artifacts of a raw chunk that contains embedded NL / trailing CR. In RAW mode, handling that is the callback's responsibility. A small helper does it cleanly:

vim9script

def Handle(prefix: string, msg: string)
    for line in split(msg, '\n')
        const clean = substitute(line, '\r$', '', '')
        if clean != ''
            echom prefix .. ': ' .. clean
        endif
    endfor
enddef

const ptybuf = term_start(cmd, {
    err_cb: (ch: channel, msg: string) => Handle('stderr', msg),
    out_cb: (ch: channel, msg: string) => Handle('stdout', msg),
})

This produces the expected:

stderr: 1 stderr
stderr: 2 stderr
stdout: 3 stdout

#20045 also updates :help term_start() to make this contract explicit. Sorry for the partial revert — this seems to be the most compatible way to satisfy both your case and existing callers that parse multi-line chunks.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today!
You are receiving this because you are subscribed to this thread.Message ID: <vim/vim/issues/16354/4304615758@github.com>