[PATCH] Non-blocking job control for vimscript

[Thiago Padilha]

This patch implements a simple but efficient form of job-control for vimscript,
enabling vim to cooperate with long-running processes without blocking the UI.

It is my second attempt to bring multitasking to vim, but unlike the
event-loop/message queue patch, this one does not rely on multithreading,
using only system functions available on most unixes. If required, it
could also be
ported to windows(I think it has a concept called IOCP which provides a
replacement to the 'select' system call).

Here's a simple demonstration:

```vimscript
" Start a background shell and save its id into the 'bgshell' variable.
"
" The first argument is the job name and will be matched with the JobActivity
" autocommand pattern.
" The second argument is the program name, with $PATH search rules applied
" The third argument is optional, but if provided it must be a list of program
" arguments
let g:bgshell = jobstart('background-shell', 'bash')

" This autocommand will be executed whenever jobs with the name maching
" 'background-shell'(eg: background-*) send data to std{out,err}.
" The v:job_data variable is a list, the first element being the job id
" and the second/third being the data read from stdout/stderr respectively.
" In this case, it simply echoes the shell stdout or stderr
au JobActivity background-shell call s:CtagsDone()
fun! s:CtagsDone()
if v:job_data[1] != ''
echohl StatusLine
echo v:job_data[1]
echohl None
else
echohl WarningMsg
echo v:job_data[2][:-2] " Cut the trailing newline from the error message
echohl None
endif
endfun

" Whenever we write a C file, tell the background shell to run ctags and report
" the result. The '\n' is needed to terminate the command in the shell.
au BufWrite *.c call jobwrite(g:bgshell, "ctags -R && echo -n ctags done!\n")
```

Another example, wake vim every 10 seconds:
```vimscript
let g:timer = jobstart('timer', 'sh', ['-c', 'while true; do sleep 10;
echo wake!; done'])
au JobActivity timer echo "Time's up!"
```

Jobs are destroyed when vim exits, but it can also be done at any time by
calling the `jobstop` function with the job id as argument.

Internally, polling is done in a function that replaces `ui_inchar` and
interrupts the blocking wait every 100 milliseconds to check for job activity.
When a job sends some data, it returns a special key code to the calling loop,
which will trigger the autocommand, similar to how the CursorHold event is
implemented. As far as most of vim's code is concerned, its just another key
being pressed, which is certainly more stable than messing with the main loops
like I did on the first versions of the event loop patch.

This approach to multitasking doesn't come with threading pitfalls which are
beyond our control, as exposed by @ashleyh here:
https://github.com/tarruda/vim/issues/5

The best of this feature, is that it makes vim extensible by any scripting
language(v8/node.js, python with threading...) without requiring the
engine to be linked into the
executable. All that's required is some vimscript code to communicate with it
via the standard streams using some protocol.

I hope someone enjoys this!

Thiago.

---
Filelist | 2 +
runtime/doc/eval.txt | 2 +
runtime/doc/various.txt | 1 +
src/Makefile | 5 +
src/config.h.in | 3 +
src/configure.in | 10 +-
src/edit.c | 6 +
src/eval.c | 172 ++++++++++++++++-
src/ex_getln.c | 5 +
src/fileio.c | 6 +-
src/getchar.c | 10 +-
src/globals.h | 6 +
src/job.c | 485 ++++++++++++++++++++++++++++++++++++++++++++++++
src/keymap.h | 3 +
src/macros.h | 8 +
src/main.c | 5 +-
src/misc1.c | 2 +-
src/normal.c | 19 ++
src/os_unix.c | 2 +-
src/os_win32.c | 2 +-
src/proto.h | 4 +
src/proto/eval.pro | 1 +
src/proto/job.pro | 8 +
src/version.c | 5 +
src/vim.h | 4 +-
25 files changed, 762 insertions(+), 14 deletions(-)
create mode 100644 src/job.c
create mode 100644 src/proto/job.pro

[Christian Wellenbrock]

Thank you for your ongoing effort to improve Vim's multithreading capabilities! I like your simple approach and hope we can get the discussion started.

[Paul Moore]

On Tuesday, 21 January 2014 20:24:27 UTC, Thiago Arruda wrote:
> This patch implements a simple but efficient form of job-control for
> vimscript, enabling vim to cooperate with long-running processes without
> blocking the UI.
>
> It is my second attempt to bring multitasking to vim, but unlike the
> event-loop/message queue patch, this one does not rely on multithreading,
> using only system functions available on most unixes. If required, it
> could also be ported to windows

Do you plan on doing the Windows port? Your example of having a background shell looks really useful, and I'd love to play with it, but the patch is useless to me unless it works on Windows.

Paul

[Jan Larres]

Thiago Padilha <tpadi...@gmail.com>:

> This patch implements a simple but efficient form of job-control for vimscript,
> enabling vim to cooperate with long-running processes without blocking the UI.
>
> It is my second attempt to bring multitasking to vim, but unlike the
> event-loop/message queue patch, this one does not rely on multithreading,
> using only system functions available on most unixes. If required, it
> could also be
> ported to windows(I think it has a concept called IOCP which provides a
> replacement to the 'select' system call).

This sounds great, I hope this approach will finally make it into Vim.
Bram, what do you think about it? It sounds like it has all of the
advantages without the disadvantages (except for the currently missing
Windows support). Merging this will make a whole lot of plugins much
simpler and allow for things not possible yet.

Jan

--
-[ OpenPGP key ID: 00A0FD5F ]-
Some cause happiness wherever they go, others whenever they go.
-- Oscar Wilde

[Thiago Padilha]

On Thu, Jan 23, 2014 at 8:24 AM, Paul Moore <p.f....@gmail.com> wrote:
>
> Do you plan on doing the Windows port? Your example of having a background shell looks really useful, and I'd love to play with it, but the patch is useless to me unless it works on Windows.

Windows API is very different and I have no plans on doing this now as
it would require me to read a lot of documentation. If you don't have
access to a unix machine maybe you can try cygwin(I think it
implements the unix system calls on top of windows)

[Anton Shemerey]

Thx for your great job

i've played with it and it looks like a charm, it's a great ability to do some work in a background,
i found few warnings during compilation, it's minor and i don't think it's a problem just for your information

Any way i am looking forward to this patch, great job thx again

---------

gcc -c -I. -Iproto -DHAVE_CONFIG_H -DMACOS_X_UNIX -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -DPYTHON_HOME='"/System/Library/Frameworks/Python.framework/Versions/2.7"' -fPIE -o objects/if_python.o if_python.c
In file included from if_python.c:809:
./if_py_both.h:2512:26: warning: format specifies type 'int' but the argument has type 'Py_ssize_t' (aka 'long') [-Wformat]
"to extended slice"), slicelen);
^~~~~~~~
./if_py_both.h:39:63: note: expanded from macro 'PyErr_FORMAT'
#define PyErr_FORMAT(exc, str, arg) PyErr_Format(exc, _(str), arg)
^
./if_py_both.h:2525:22: warning: format specifies type 'int' but the argument has type 'Py_ssize_t' (aka 'long') [-Wformat]
"of size %d"), i, slicelen);
^
./if_py_both.h:40:71: note: expanded from macro 'PyErr_FORMAT2'
#define PyErr_FORMAT2(exc, str, arg1, arg2) PyErr_Format(exc, _(str), arg1,arg2)
^
./if_py_both.h:2525:25: warning: format specifies type 'int' but the argument has type 'Py_ssize_t' (aka 'long') [-Wformat]
"of size %d"), i, slicelen);
~~ ^~~~~~~~
./if_py_both.h:40:76: note: expanded from macro 'PyErr_FORMAT2'
#define PyErr_FORMAT2(exc, str, arg1, arg2) PyErr_Format(exc, _(str), arg1,arg2)
^
3 warnings generated.

[王世东]

so,that is the community of vim_dev,no reply from Bram,that is why most of us switch to neovim.

[Christian Brabandt]

Am 2016-03-10 15:55, schrieb 王世东:
> so,that is the community of vim_dev,no reply from Bram,that is why
> most of us switch to neovim.

1) There were some replies (perhaps not in this thread)

2) Who are you, that you know, "most of us"?

3) please be polite, there is no reason to start a flame war here.


Thanks.

Best,
Christian