[vim/vim] Terminal left in normal mode when execuiting interactive shell command from auto command (#7562)
George Brown
Describe the bug
When running an interactive shell command with ! from an auto command it appears that the terminal is left in normal mode rather than being returned to application mode.
This can be observed as the arrow keys not functioning as expected.
I'm reasonably sure this issue is normal/application or CSI/SS3 issue as the faulty state can be reminded with.
!tput smkx
To Reproduce
The following test.vim file serves as reproducer. I would note this does not seem to be specific to the QuickFixCmdPost event.
autocmd QuickFixCmdPost * !less test.vim
doautocmd QuickFixCmdPost
Run the following.
vim --clean -S test.vim
Then press <Left> to get the error.
E349: No identifier under cursor
Whilst in this faulty state let us enter <C-V><Left> in insert mode to see what Vim is receiving. Rather telling as [D would have produced the previously seen error.
^[[D
By comparison when not on this state we get the following.
^[OD
In both faulty and good states querying set t_kl? shows the same.
t_kl <Left> ^[O*D
As a note prefixing the test with the following makes no difference.
let &t_TI = ""
let &t_TE = ""
Expected behavior
Behaviour of keys when returning from an interactive command should be the same, whether that command was typed by the user in the command-line or run form an auto command.
Environment (please complete the following information):
- Vim version
VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Jun 3 2020 15:02:02)
macOS version
Included patches: 1-891
Compiled by gbr...@lappy.local
Huge version without GUI. Features included (+) or not (-):
+acl -farsi +mouse_sgr +tag_binary
+arabic +file_in_path -mouse_sysmouse -tag_old_static
+autocmd +find_in_path +mouse_urxvt -tag_any_white
+autochdir +float +mouse_xterm -tcl
-autoservername +folding +multi_byte +termguicolors
-balloon_eval -footer +multi_lang +terminal
+balloon_eval_term +fork() -mzscheme +terminfo
-browse +gettext +netbeans_intg +termresponse
++builtin_terms -hangul_input +num64 +textobjects
+byte_offset +iconv +packages +textprop
+channel +insert_expand +path_extra +timers
+cindent +ipv6 -perl +title
-clientserver +job +persistent_undo -toolbar
+clipboard +jumplist +popupwin +user_commands
+cmdline_compl +keymap +postscript +vartabs
+cmdline_hist +lambda +printer +vertsplit
+cmdline_info +langmap +profile +virtualedit
+comments +libcall -python +visual
+conceal +linebreak -python3 +visualextra
+cryptv +lispindent +quickfix +viminfo
-cscope +listcmds +reltime +vreplace
+cursorbind +localmap +rightleft +wildignore
+cursorshape -lua -ruby +wildmenu
+dialog_con +menu +scrollbind +windows
+diff +mksession +signs +writebackup
+digraphs +modify_fname +smartindent -X11
-dnd +mouse -sound -xfontset
-ebcdic -mouseshape +spell -xim
+emacs_tags +mouse_dec +startuptime -xpm
+eval -mouse_gpm +statusline -xsmp
+ex_extra -mouse_jsbterm -sun_workshop -xterm_clipboard
+extra_search +mouse_netterm +syntax -xterm_save
system vimrc file: "$VIM/vimrc"
user vimrc file: "$HOME/.vimrc"
2nd user vimrc file: "~/.vim/vimrc"
user exrc file: "$HOME/.exrc"
defaults file: "$VIMRUNTIME/defaults.vim"
fall-back for $VIM: "/Users/gbrown/local/vim/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DMACOS_X -DMACOS_X_DARWIN -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc -L/usr/local/lib -o vim -lm -lncurses -liconv -lintl -framework AppKit
- OS: macOS 11.1
- Terminal: Apple terminal
Additional context
I've observed the same on my Linux desktop running Ubuntu 20.10 in Gnome Terminal.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Bram Moolenaar
I cannot reproduce this (using an xterm). I see:
:!less test.vim
test.vim: No such file or directory
shell returned 1
Press ENTER or type command to continue
And then things appear to be working normally, after pressing Enter.
George Brown
:!less test.vim
test.vim: No such file or directory
Did you create the example as test.vim? Sorry my steps to reproduce could be more verbatim.
$ cat << EOF > test.vim
heredoc> autocmd QuickFixCmdPost * !less test.vim
heredoc> doautocmd QuickFixCmdPost
heredoc> EOF
$ vim --clean -S test.vim
Should show the file in less, after exiting less a Press ENTER or type command to continue prompt is shown, then press <Left>.
lacygoill
In xterm, the terminal option 't_RV' needs to be cleared to reproduce the issue. Run this shell command
vim -Nu NONE -S <(cat <<'EOF'
set t_RV=
call writefile([], '/tmp/somefile')
autocmd QuickFixCmdPost * !less /tmp/somefile
doautocmd QuickFixCmdPost
EOF
)
Then, press q to close less(1), then press <Left>; E349 is unexpectedly raised:
E349: No identifier under cursor
lacygoill
It seems the issue is caused by a terminal whose $TERM is set with xterm (or some derivative), while it's not fully xterm-compatible. Because I can't reproduce the issue in gnome-terminal with my default settings (I've configured gnome-terminal so that $TERM is gnome-256color). But I can if I temporarily reset $TERM with xterm-256color.
George Brown
Interesting, setting $TERM to gnome-256color without touching t_RV I still hit this issue most of the time.
However I do appear to see an exception to this on macOS with xterm using my now slighly older build of Vim (v8.2.0891). This also appears to extend to the default version of Vim shipped in macOS (v8.2.0850).
| Issue presetns? | Vim version | OS | Terminal | $TERM |
|---|---|---|---|---|
| Yes | v8.2.2243 | Ubuntu 20.10 | Gnome terminal 3.38.0 | xterm |
| Yes | v8.2.2243 | Ubuntu 20.10 | Gnome terminal 3.38.0 | xterm-256color |
| Yes | v8.2.2243 | Ubuntu 20.10 | Gnome terminal 3.38.0 | gnome-256color |
| Yes | v8.2.2243 | Ubuntu 20.10 | xterm (353) | xterm |
| Yes | v8.2.2243 | Ubuntu 20.10 | xterm (353) | xterm-256color |
| Yes | v8.2.2243 | macOS 11.1 | Terminal.app | xterm |
| Yes | v8.2.2243 | macOS 11.1 | Terminal.app | xterm-256color |
| Yes | v8.2.2243 | macOS 11.1 | xterm (326) | xterm |
| Yes | v8.2.2243 | macOS 11.1 | xterm (326) | xterm-256color |
| Yes | v8.2.0891 | Ubuntu 20.10 | Gnome terminal 3.38.0 | xterm |
| Yes | v8.2.0891 | Ubuntu 20.10 | Gnome terminal 3.38.0 | xterm-256color |
| Yes | v8.2.0891 | Ubuntu 20.10 | Gnome terminal 3.38.0 | gnome-256color |
| Yes | v8.2.0891 | Ubuntu 20.10 | xterm (353) | xterm |
| Yes | v8.2.0891 | Ubuntu 20.10 | xterm (353) | xterm-256color |
| Yes | v8.2.0891 | macOS 11.1 | Terminal.app | xterm |
| Yes | v8.2.0891 | macOS 11.1 | Terminal.app | xterm-256color |
| No | v8.2.0891 | macOS 11.1 | xterm (326) | xterm |
| No | v8.2.0891 | macOS 11.1 | xterm (326) | xterm-256color |
| Yes | v8.2.0850 | macOS 11.1 | Terminal.app | xterm |
| Yes | v8.2.0850 | macOS 11.1 | Terminal.app | xterm-256color |
| No | v8.2.0850 | macOS 11.1 | xterm (326) | xterm |
| No | v8.2.0850 | macOS 11.1 | xterm (326) | xterm-256color |
I've tried to bisect this peculiarity on macOS. But when building even v.8.2.0850 (albeit with changes from v8.2.1310 required to compile on my version of macOS) said build ends up producing the issue even under xterm.
Bram Moolenaar
> In xterm, the terminal option `'t_RV'` needs to be cleared to reproduce the issue. Run this shell command
>
> vim -Nu NONE -S <(cat <<'EOF'
> set t_RV=
> call writefile([], '/tmp/somefile')
> autocmd QuickFixCmdPost * !less /tmp/somefile
> doautocmd QuickFixCmdPost
> EOF
> )
>
>
> E349: No identifier under cursor
just typing ":!less /tmp/somefile" does the same.
What happens is that Vim receives ESC [ D for the Left cursor key.
This is not recognized, the Left key is expected to produce ESC O D.
The Esc key is then used to leave the Enter prompt and the remaining [ D
is a command to search for the identifier under the cursor.
At the Enter prompt the terminal hasn't been switched back to termcap
mode yet, because that might switch to the alternate screen and you
can't see the shell command output. Making the Left key produce a
different Escape sequence is a side effect.
This could perhaps be fixed by going to half-termcap mode, only
outputting some of the codes and sending t_ti later. However, the
effect is unpredictable, the termcap codes may contain anything.
Another way would be to also recognize the alternate form of the Left
key. But we would need to do the same for many special keys.
Since it's been like this for ages without complaints, and trying to fix
this will most likely have side effects, I think we should just leave
it.
--
Latest survey shows that 3 out of 4 people make up 75% of the
world's population.
/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
George Brown
Thanks for the info. I can also reproduce it without the autocommand,
just typing ":!less /tmp/somefile" does the same.
That's odd I've only been able to get this with an auto command.
Another way would be to also recognize the alternate form of the Left
key. But we would need to do the same for many special keys.
I'm guessing this meant to sound unappealing? If so I agree. What is being received from the terminal has changed from what Vim was expecting (and indeed previously receiving). I don't think it's Vim's responsibility to handle such a change.
Bram Moolenaar
cursor keys, but they start with Esc [ 1. I'm not sure why the
requirement for that number one comes from, perhaps it works if we leave
it out:
--- ../prev/term.c 2020-12-18 19:49:52.389571687 +0100
+++ term.c 2020-12-29 14:50:48.493170891 +0100
@@ -914,10 +914,10 @@
{K_RIGHT, IF_EB("\033O*C", ESC_STR "O*C")},
{K_LEFT, IF_EB("\033O*D", ESC_STR "O*D")},
// An extra set of cursor keys for vt100 mode
- {K_XUP, IF_EB("\033[1;*A", ESC_STR "[1;*A")},
- {K_XDOWN, IF_EB("\033[1;*B", ESC_STR "[1;*B")},
- {K_XRIGHT, IF_EB("\033[1;*C", ESC_STR "[1;*C")},
- {K_XLEFT, IF_EB("\033[1;*D", ESC_STR "[1;*D")},
+ {K_XUP, IF_EB("\033[;*A", ESC_STR "[;*A")},
+ {K_XDOWN, IF_EB("\033[;*B", ESC_STR "[;*B")},
+ {K_XRIGHT, IF_EB("\033[;*C", ESC_STR "[;*C")},
+ {K_XLEFT, IF_EB("\033[;*D", ESC_STR "[;*D")},
// An extra set of function keys for vt100 mode
{K_XF1, IF_EB("\033O*P", ESC_STR "O*P")},
{K_XF2, IF_EB("\033O*Q", ESC_STR "O*Q")},
--
It was recently discovered that research causes cancer in rats.
George Brown
So I see this has been closed with the approach of dealing with the altered input. Indeed this does leads to a situation where the arrows keys seemingly function as expected.
However it doesn't change the fact that executing an interactive command changes our terminal state, which doesn't seem right. Again I'm only seeing this with an auto command and not when starting from the command-line. Which makes me feel like the execution from an auto command could possibly be "fixed" as execution from the command-line seems to behave without side effects.
Bram Moolenaar
> So I see this has been closed with the approach of dealing with the
> altered input. Indeed this does leads to a situation where the arrows
> keys seemingly function as expected.
>
> However it doesn't change the fact that executing an interactive
> command changes our terminal state, which doesn't seem right. Again
> I'm only seeing this with an auto command and not when starting from
> the command-line. Which makes me feel like the execution from an auto
> command could possibly be "fixed" as execution from the command-line
> seems to behave without side effects.
exactly right. Vim operates in RAW mode, external commands expect
cooked mode. Switching modes is required to make external commands work
properly.
As mentioned, the question is when to switch back to RAW mode. The
external command may have displayed output and the user probably wants
to read that. That's why we have the hit-Enter prompt. If we switch
back to RAW mode after displaying the prompt, it's possible this causes
the displayed text to be cleared, esp. when switching between the
primary and alternate screen. That is a problem, since the prompt
itself is probably also not visible, and the user would have to blindly
type something.
--
hundred-and-one symptoms of being an internet addict:
39. You move into a new house and setup the Wifi router before
unpacking any kitchen stuff.
/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
George Brown
Sorry to keep repeating this but I don't understand why this only occurs with an auto command.
I known in you've previously mentioned that you were able to trigger this from a command-line invocation but I cannot reproduce this on either of my systems.