Adding terminal emulator support in Vim

[Bram Moolenaar]

For a long time I have been wondering whether it would be a good idea to
add a terminal emulator inside Vim. It's a dangerous thing, it can
easily grow out of proportions and be a maintenance nightmare. At the
same time, it can be very, very useful.

The reason I've wanted this is to debug Vim over an ssh connection. I
have a nice setup locally, but when I'm not at home I always suffer from
many limitations, which usually means I postpone the debugging until I'm
home again. There are plugins that offer some of this, but debugging
Vim requires running it in a terminal.

Another reason... wait: TWO reasons to want a terminal emulater are:
1. For debugging.

2. Properly be able to run external commands in the GUI. This has been
in the todo list for a long time. Even ":!less file.txt" doesn't
work properly.

Also,... wait, wait, wait; THREE reaons for a terminal emulator are:

1. For debugging.

2. For running external commands in the GUI..

3. For running tests with a remote controlled Vim. The current test
setup allows for a lot, but there are still tests where the test
script and the test itself interfere. With a terminal window the
test is able to run Vim, send it keystrokes and check for the
expected output. This is only possible with a built-in terminal
emulator.

And it fits in with.... WAIT! Among the reasons to add a terminal
emulator window are such reasons as:

1. For debugging.

2. For running external commands in the GUI.

3. For running tests with a remote controlled Vim.

4. Run a job conveniently on all platforms and interact with it. Allows
for jobs that prompt the user, keeping an eye on progress, etc.,
without the need to start another xterm or console window.

Some may say that this invites users to run a shell in a Vim window,
which is not what an editor should be doing. That's true. But one can
actually already do this with a job that is connected to a buffer. I
haven't heard much complaints about that. We do need to set priorities,
supporting the above mentioned reasons is the most important.

I am currently exploring using libvterm for the implementation. It
looks like this will work. Still a lot of stuff to implement, which
might not work that well, we'll have to see. I hope to send out an
initial patch soon, we can discuss more then.

--
"Oh, no! NOT the Spanish Inquisition!"
"NOBODY expects the Spanish Inquisition!!!"
-- Monty Python sketch --

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Nikolay Aleksandrovich Pavlov]

Debugging and tests can actually be easy to achive if you take
libvterm and Python with pexpect, I did that for powerline. (Though
Neovim with a “remote UI” based on msgpack is far more convenient
setup for most tests, excluding testing specifically terminal UI and
not just a supposed screen state, we have some of them which use
Neovim run inside a :terminal inside a Neovim which exports screen
state via its RPC API.) 4 also, but there is a big difference between
making Python “build time dependency needed for a subset of tests” and
“runtime dependency needed for a set of plugins”.

The interesting thing is that despite Neovim has built-in terminal
emulator which allows creating special terminal buffers (I do hope you
will look at that before implementing your own variant), support has
never gone far enough to use it for `:!`, though after `:!` in Neovim
started to work through pipes and stopped displaying colors or working
with interactive apps correctly this feature started to be rather
desired. Scripting interaction with such buffers is possible both via
“switch to buffer and do :normal” and “use jobsend()” (obviously,
second variant is best practice most of time).

>
> --
> "Oh, no! NOT the Spanish Inquisition!"
> "NOBODY expects the Spanish Inquisition!!!"
> -- Monty Python sketch --
>
> /// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
> /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
> \\\ an exciting new programming language -- http://www.Zimbu.org ///
> \\\ help me help AIDS victims -- http://ICCF-Holland.org ///
>

> --
> --
> You received this message from the "vim_dev" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>
> ---
> You received this message because you are subscribed to the Google Groups "vim_dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[R0b0t1]

On Sun, Jul 2, 2017 at 3:46 PM, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
> For a long time I have been wondering whether it would be a good idea to
> add a terminal emulator inside Vim. It's a dangerous thing, it can
> easily grow out of proportions and be a maintenance nightmare. At the
> same time, it can be very, very useful.
>
> The reason I've wanted this is to debug Vim over an ssh connection. I
> have a nice setup locally, but when I'm not at home I always suffer from
> many limitations, which usually means I postpone the debugging until I'm
> home again. There are plugins that offer some of this, but debugging
> Vim requires running it in a terminal.
>
> Another reason... wait: TWO reasons to want a terminal emulater are:
> 1. For debugging.
>

This doesn't seem like a very good reason. You should be able to debug
a program remotely, but you'll still be running the debugger on the
machine running the debugged program. How are you currently debugging
Vim?

> 2. Properly be able to run external commands in the GUI. This has been
> in the todo list for a long time. Even ":!less file.txt" doesn't
> work properly.
>

This is a fairly good thing to want but I'm not sure adding a terminal
emulator to Vim is the way to accomplish this. It's possible to rather
accurately detect the terminal emulator in use on Linux and then
launch it (if not, it could be a variable). On Windows, it might be a
good idea to call AllocConsole and then write to it.

While potentially more annoying than some alternatives it seems like
it might be okay if they were expecting the new window.

> Also,... wait, wait, wait; THREE reaons for a terminal emulator are:
>
> 1. For debugging.
>
> 2. For running external commands in the GUI..
>
> 3. For running tests with a remote controlled Vim. The current test
> setup allows for a lot, but there are still tests where the test
> script and the test itself interfere. With a terminal window the
> test is able to run Vim, send it keystrokes and check for the
> expected output. This is only possible with a built-in terminal
> emulator.
>

I know quite a bit about UI automation. What is it that the tests need
to do and how do they currently do that? I only ask because I don't
understand how the setup you described is prevented from doing what it
needs to do by the lack of a terminal emulator, although something may
need to be done differently for the GUI version (as it doesn't run in
a terminal emulator).

> And it fits in with.... WAIT! Among the reasons to add a terminal
> emulator window are such reasons as:
>
> 1. For debugging.
>
> 2. For running external commands in the GUI.
>
> 3. For running tests with a remote controlled Vim.
>
> 4. Run a job conveniently on all platforms and interact with it. Allows
> for jobs that prompt the user, keeping an eye on progress, etc.,
> without the need to start another xterm or console window.
>
> Some may say that this invites users to run a shell in a Vim window,
> which is not what an editor should be doing. That's true. But one can
> actually already do this with a job that is connected to a buffer. I
> haven't heard much complaints about that. We do need to set priorities,
> supporting the above mentioned reasons is the most important.
>
> I am currently exploring using libvterm for the implementation. It
> looks like this will work. Still a lot of stuff to implement, which
> might not work that well, we'll have to see. I hope to send out an
> initial patch soon, we can discuss more then.
>

Most of the very bad reasons I could give are mitigated by using
libvterm, so as far as I know there's no grave reason you should
completely avoid adding terminal emulation to Vim. I can, however,
attest that it will be quite a bit of work even with what libvterm
provides you.

If adding a terminal emulator solves all of the reasons given in an
elegant enough fashion then it makes sense to go forward with it
despite the issues I brought up. Unfortunately, things like #2 and #4
when on Windows seem like "pretend to be Cygwin" which sounds like
project creep so massive it is actually impossible to accomplish.
Getting around those two issues seems like it is generally
accomplished by installing Cygwin. I'm not sure how that solution
interacts with gVim, but it sounds like it is avoided - throughout my
use of Vim there have been many suggestions to use it inside of Cygwin
when on Windows.

My conservative response to points #1 and #3 is likely due to my
background with Linux. I see no reason why those things should be
added to Vim merely because they exist elsewhere and adding them would
take effort which could be spent doing something else. On Windows the
situation may be different (I currently know of no way to do those
things without using Cygwin), but on some level I think it needs to be
acknowledged that there will be operating system differences that may
not be reconcilable inside of Vim.

R0b0t1.

P.S. I'm actually rather startled to learn that neovim has a built in
terminal emulator.

[Nikolay Aleksandrovich Pavlov]

There is a very limited terminal emulator in GVim. AFAIR the main
reasoning behind adding terminal emulator into Neovim were exactly GUI
and making things more uniform: the “very limited terminal emulator”
does not even support colors, it is only as good as “create pty and
set $TERM to dumb”.

And, BTW, Neovim is planning on making its terminal emulator work on
Windows, and this has nothing to do with cygwin. Basically “terminal
emulator on Windows” means one of two things or both of them combined:

1. Escape codes interpreter. Easy to achieve, you just work exactly
like on linux, the only problem is the differences between starting
Windows and linux processes, but that is covered by libuv. Despite the
fact that “terminal escape codes ‘language’” is full of legacy,
interpreter for it is smaller then interpreter for VimL.
2. Emulator which is able to somehow receive and interpret [Windows
console API][winconapi]. This is far harder, totally different from
unix escape sequences … and already covered by winpty which is able to
do the translation. I am not sure what you would need as a dependency,
but [PR which adds support for terminal on Windows][neovimpr] uses no
new third-party packages for building and just lays its hands on
[winpty-agent.exe][wpty-agent] (711K in [MSVC 0.4.3
release][winpty-0.4.3] which is the largest among the variants) and
[winpty.dll][winptydll] which is 625K. This is not much by the modern
standards, especially given that you need Qt as second most mature
Neovim UI aside from lua UI used for testing is nvim-qt (first is
built-in terminal UI which is *nix-specific).

And yes, [that PR][neovimpr] means that Windows Neovim is going to
have :terminal support.

[wpty-agent]: https://github.com/neovim/neovim/pull/6383/files#diff-7abbb8315fea3e74ba20ce24ac297519R429
[winptydll]: https://github.com/neovim/neovim/pull/6383/files#diff-7abbb8315fea3e74ba20ce24ac297519R443
[winpty-0.4.3]: https://github.com/rprichard/winpty/releases/tag/0.4.3
[winconapi]: https://msdn.microsoft.com/en-us/library/windows/desktop/ms682073(v=vs.85).aspx
[neovimpr]: https://github.com/neovim/neovim/pull/6383

[R0b0t1]

That is fine, but a terminal emulator is only as useful as the
utilities you can run in it. Unless I'm mistaken gVim isn't shipped
with a bundle of Unix-alike command line utilties, so what are you
going to run in this terminal emulator that nothing on the system by
default can use?

Some things I could see conceivably being run in it might work, but if
that's what you're doing - why not use Cygwin? Using that single
project solves almost every problem that open source projects have
porting themselves to Windows by virtue of providing POSIX compliance.

> And, BTW, Neovim is planning on making its terminal emulator work on
> Windows, and this has nothing to do with cygwin. Basically “terminal
> emulator on Windows” means one of two things or both of them combined:
>
> 1. Escape codes interpreter. Easy to achieve, you just work exactly
> like on linux, the only problem is the differences between starting
> Windows and linux processes, but that is covered by libuv. Despite the
> fact that “terminal escape codes ‘language’” is full of legacy,
> interpreter for it is smaller then interpreter for VimL.
> 2. Emulator which is able to somehow receive and interpret [Windows
> console API][winconapi]. This is far harder, totally different from
> unix escape sequences … and already covered by winpty which is able to
> do the translation. I am not sure what you would need as a dependency,
> but [PR which adds support for terminal on Windows][neovimpr] uses no
> new third-party packages for building and just lays its hands on
> [winpty-agent.exe][wpty-agent] (711K in [MSVC 0.4.3
> release][winpty-0.4.3] which is the largest among the variants) and
> [winpty.dll][winptydll] which is 625K. This is not much by the modern
> standards, especially given that you need Qt as second most mature
> Neovim UI aside from lua UI used for testing is nvim-qt (first is
> built-in terminal UI which is *nix-specific).
>
> And yes, [that PR][neovimpr] means that Windows Neovim is going to
> have :terminal support.
>

The second is a decent solution, I am just surprised someone cared
enough to write it, especially seeing as Cygwin already exists.

As someone who mainly uses Linux and interacts with plenty of people
who use Linux - I honestly can't see many people using that feature.
The people who use the command line daily would do all of those things
using other pre-existing programs. While I don't know every usecase it
seems like a lot of work to support *just* Windows.

That's not to criticize Bram's suggestion just for what it is about.
It just seems like an awful lot of work and code and external
libraries when another perfectly fine solution already exists: Cygwin.

[mattn]

Hi, Bram.

I have implementation (but PoC/WIP) that works on Windows.

https://qiita-image-store.s3.amazonaws.com/0/665/67e1a2e1-c2dc-847e-20b6-5b26a0f62b89.gif

This is in vim/terminal branch in my repository.

https://github.com/mattn/vim/tree/terminal

[Phil Dobbin]

Sounds like a great idea to me. I spend all day in either Vim or
gnome-terminal so it'd sit very well with me.

Cheers Phil.

--
If you are firmly enough rooted in your masculinity,
it shouldn't bother you much to have a woman on top of you.
It's a good deal more relaxing, it's often more satisfying for both parties
and you can see a lot more of what's going on
A Case Of Lone Star - Kinky Friedman

[Bram Moolenaar]

Nikolay Pavlov wrote:

> 2017-07-02 23:46 GMT+03:00 Bram Moolenaar <Br...@moolenaar.net>:
> >
> > For a long time I have been wondering whether it would be a good idea to
> > add a terminal emulator inside Vim. It's a dangerous thing, it can
> > easily grow out of proportions and be a maintenance nightmare. At the
> > same time, it can be very, very useful.
>

> [...]

>
> > I am currently exploring using libvterm for the implementation. It
> > looks like this will work. Still a lot of stuff to implement, which
> > might not work that well, we'll have to see. I hope to send out an
> > initial patch soon, we can discuss more then.
>
> Debugging and tests can actually be easy to achive if you take
> libvterm and Python with pexpect, I did that for powerline. (Though
> Neovim with a “remote UI” based on msgpack is far more convenient
> setup for most tests, excluding testing specifically terminal UI and
> not just a supposed screen state, we have some of them which use
> Neovim run inside a :terminal inside a Neovim which exports screen
> state via its RPC API.) 4 also, but there is a big difference between
> making Python “build time dependency needed for a subset of tests” and
> “runtime dependency needed for a set of plugins”.
>
> The interesting thing is that despite Neovim has built-in terminal
> emulator which allows creating special terminal buffers (I do hope you
> will look at that before implementing your own variant), support has
> never gone far enough to use it for `:!`, though after `:!` in Neovim
> started to work through pipes and stopped displaying colors or working
> with interactive apps correctly this feature started to be rather
> desired. Scripting interaction with such buffers is possible both via
> “switch to buffer and do :normal” and “use jobsend()” (obviously,
> second variant is best practice most of time).

I have looked at terminal.c in neovim, it gives some ideas. However,
both libvterm and this use of it are largely without documentation or
comments. I see quite a few things in the neovim implementation where I
wonder why it's done that way (in other words: looks like a bad choice).
E.g. having two Vim windows on one terminal seems weird. And the round
trip through a buffer to display text is weird, especially since
attributes are obtained in a completely different way. Bound to
introduce bugs with two conversions that seem unnecessary.

Still more exploring to do. I also intend to add more documentation to
libvterm along the way. I hope Paul Evans includes my changes.

--
Common sense is what tells you that the world is flat.

[Bram Moolenaar]

Interesting. What kind of terminal does it pretend to be? With
libvterm it's like an xterm. On MS-Windows commands expect to be
running in a console, that is quite different. So only programs ported
from Unix would run in libvterm.

--
# echo reboot >universe
# chmod +x universe
# ./universe

[mattn]

On Monday, July 3, 2017 at 6:16:51 PM UTC+9, Bram Moolenaar wrote:
> Yasuhiro Matsumoto wrote:
>
> > I have implementation (but PoC/WIP) that works on Windows.
> >
> > https://qiita-image-store.s3.amazonaws.com/0/665/67e1a2e1-c2dc-847e-20b6-5b26a0f62b89.gif
> >
> > This is in vim/terminal branch in my repository.
> >
> > https://github.com/mattn/vim/tree/terminal
>
> Interesting. What kind of terminal does it pretend to be? With
> libvterm it's like an xterm. On MS-Windows commands expect to be
> running in a console, that is quite different. So only programs ported
> from Unix would run in libvterm.

This is not terminal. This is gvim.exe (with guioptions= ). This doesn't any external libraries.

- mattn

[R0b0t1]

Yes, but your terminal implementation must adhere to some standard.
Most Unix derived virtual terminals use inline control characters to
manipulate the state of the terminal. On Windows, a terminal is
created by operating system code as a desktop window and doesn't
implement a control code standard. As it always existed as some form
of abstraction and never copied physical hardware, there is a set of
system calls that interacts with the same code that creates a terminal
as a desktop window.[1]

This is what I was trying to point out when I mentioned Cygwin. The
project may have merit (and I think it does) but it is far less useful
for all Windows users of Vim based on assumptions that are being built
into it.

[1] You can make cmd.exe or powershell.exe interpret most VT control
codes but this doesn't do anything for you if you need to make a
Windows program run in a VT environment.

[Bram Moolenaar]

We will have to let the user decide whether to open a terminal emulator
in a Vim window, in which it's possible to run Vim and perhaps gdb, or a
Windows console, which can do anything. But it won't be in a Vim window
(I believe that is impossible, unless we fake it by positioning the
console on top of Vim). We already have this code for the GUI.

The important part is that the commands and functions we add to Vim for
terminal support work on Unix, Mac and Windows, even when it works a
little bit differently.

--
hundred-and-one symptoms of being an internet addict:
106. When told to "go to your room" you inform your parents that you
can't...because you were kicked out and banned.

[Nikolay Aleksandrovich Pavlov]

If I am not mistaking, winpty is a thing which makes it possible:
quoting winpty README

> The software works by starting the winpty-agent.exe process with a new, hidden console window, which bridges between the console API and terminal input/output escape codes. It polls the hidden console's screen buffer for changes and generates a corresponding stream of output.

>
> The important part is that the commands and functions we add to Vim for
> terminal support work on Unix, Mac and Windows, even when it works a
> little bit differently.
>
> --
> hundred-and-one symptoms of being an internet addict:
> 106. When told to "go to your room" you inform your parents that you
> can't...because you were kicked out and banned.
>
> /// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
> /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
> \\\ an exciting new programming language -- http://www.Zimbu.org ///
> \\\ help me help AIDS victims -- http://ICCF-Holland.org ///
>

[skywind3000]

Another possible choise:
http://www.dotnetsurfers.com/blog/2013/12/15/developer-tools-screencast-7-conemu/

[Dominique Pellé]

Bram Moolenaar wrote:

> Some may say that this invites users to run a shell in a Vim window,
> which is not what an editor should be doing. That's true. But one can
> actually already do this with a job that is connected to a buffer.

I don't think that having a terminal in Vim should be taboo.
Vim users tend to like using the terminal. Combining the
goodness of Vim and the terminal seems very useful.
I'd love to run a terminal in Vim window, and switch
mode so that the buffer behaves either like a terminal or
like a regular buffer, so we can search, copy text (etc) using
vim commands in the terminal window.

Regards
Dominique

[Bram Moolenaar]

That sounds very useful. Yasuhiro's patch seems to do something
similar, but without the escape codes. It's actually closer to libvterm
that way. But winpty might cover some more stuff.

--
hundred-and-one symptoms of being an internet addict:

107. When using your phone you forget that you don't have to use your
keyboard.

[mattn]

Hi, all.

Good news. I'm working implement of :terminal using winpty/libvterm on Windows.

http://i.imgur.com/lrm9hr0.gif

Currently, buggy and flacky, But I'll send pull-request in few days or cupple of weeks.

https://github.com/mattn/vim/tree/windows-terminal

[Bram Moolenaar]

Looks interesting.

Looking at the requirements of winpty, this is really difficult to
build, requires random tools. Probably can't build it with just MSVC.
Also, it appears to run some executable, for which I don't see the
reason and is likely to cause trouble later.

We need to figure out a simple way to do this. It's already quite hard
to build Vim on MS-Windows anyway...
Perhaps it works to include the header file and provide the rest as
a pre-build .dll.

--
hundred-and-one symptoms of being an internet addict:

193. You ask your girlfriend to drive home so you can sit back with
your PDA and download the information to your laptop

[mattn]

On Thursday, July 20, 2017 at 7:48:15 PM UTC+9, Bram Moolenaar wrote:
> Yasuhiro Matsumoto wrote:
>
> > Good news. I'm working implement of :terminal using winpty/libvterm on Windows.
> >
> > http://i.imgur.com/lrm9hr0.gif
> >
> > Currently, buggy and flacky, But I'll send pull-request in few days or cupple of weeks.
> >
> > https://github.com/mattn/vim/tree/windows-terminal
>
> Looks interesting.
>
> Looking at the requirements of winpty, this is really difficult to
> build, requires random tools. Probably can't build it with just MSVC.
> Also, it appears to run some executable, for which I don't see the
> reason and is likely to cause trouble later.
>
> We need to figure out a simple way to do this. It's already quite hard
> to build Vim on MS-Windows anyway...
> Perhaps it works to include the header file and provide the rest as
> a pre-build .dll.

Yes, I'm thinking that Windows support of terminal will be optional. And if user put winpty.dll and winapty-agent.exe in the PATH, :terminal works, but currently work in progress.