[PATCH] Non-blocking job control for vimscript

Showing 1-6 of 6 messages
[PATCH] Non-blocking job control for vimscript Thiago Arruda 1/21/14 12:24 PM
This patch implements a simple but efficient form of job-control for vimscript,
enabling vim to cooperate with long-running processes without blocking the UI.

It is my second attempt to bring multitasking to vim, but unlike the
event-loop/message queue patch, this one does not rely on multithreading,
using only system functions available on most unixes. If required, it
could also be
ported to windows(I think it has a concept called IOCP which provides a
replacement to the 'select' system call).

Here's a simple demonstration:

```vimscript
" Start a background shell and save its id into the 'bgshell' variable.
"
" The first argument is the job name and will be matched with the JobActivity
" autocommand pattern.
" The second argument is the program name, with $PATH search rules applied
" The third argument is optional, but if provided it must be a list of program
" arguments
let g:bgshell = jobstart('background-shell', 'bash')

" This autocommand will be executed whenever jobs with the name maching
" 'background-shell'(eg: background-*) send data to std{out,err}.
" The v:job_data variable is a list, the first element being the job id
" and the second/third being the data read from stdout/stderr respectively.
" In this case, it simply echoes the shell stdout or stderr
au JobActivity background-shell call s:CtagsDone()
fun! s:CtagsDone()
    if v:job_data[1] != ''
  echohl StatusLine
  echo v:job_data[1]
  echohl None
    else
  echohl WarningMsg
  echo v:job_data[2][:-2] " Cut the trailing newline from the error message
  echohl None
    endif
endfun

" Whenever we write a C file, tell the background shell to run ctags and report
" the result. The '\n' is needed to terminate the command in the shell.
au BufWrite *.c call jobwrite(g:bgshell, "ctags -R && echo -n ctags done!\n")
```

Another example, wake vim every 10 seconds:
```vimscript
let g:timer = jobstart('timer', 'sh', ['-c', 'while true; do sleep 10;
echo wake!; done'])
au JobActivity timer echo "Time's up!"
```

Jobs are destroyed when vim exits, but it can also be done at any time by
calling the `jobstop` function with the job id as argument.

Internally, polling is done in a function that replaces `ui_inchar` and
interrupts the blocking wait every 100 milliseconds to check for job activity.
When a job sends some data, it returns a special key code to the calling loop,
which will trigger the autocommand, similar to how the CursorHold event is
implemented. As far as most of vim's code is concerned, its just another key
being pressed, which is certainly more stable than messing with the main loops
like I did on the first versions of the event loop patch.

This approach to multitasking doesn't come with threading pitfalls which are
beyond our control, as exposed by @ashleyh here:
https://github.com/tarruda/vim/issues/5

The best of this feature, is that it makes vim extensible by any scripting
language(v8/node.js, python with threading...) without requiring the
engine to be linked into the
executable. All that's required is some vimscript code to communicate with it
via the standard streams using some protocol.

I hope someone enjoys this!

Thiago.

---
 Filelist                |   2 +
 runtime/doc/eval.txt    |   2 +
 runtime/doc/various.txt |   1 +
 src/Makefile            |   5 +
 src/config.h.in         |   3 +
 src/configure.in        |  10 +-
 src/edit.c              |   6 +
 src/eval.c              | 172 ++++++++++++++++-
 src/ex_getln.c          |   5 +
 src/fileio.c            |   6 +-
 src/getchar.c           |  10 +-
 src/globals.h           |   6 +
 src/job.c               | 485 ++++++++++++++++++++++++++++++++++++++++++++++++
 src/keymap.h            |   3 +
 src/macros.h            |   8 +
 src/main.c              |   5 +-
 src/misc1.c             |   2 +-
 src/normal.c            |  19 ++
 src/os_unix.c           |   2 +-
 src/os_win32.c          |   2 +-
 src/proto.h             |   4 +
 src/proto/eval.pro      |   1 +
 src/proto/job.pro       |   8 +
 src/version.c           |   5 +
 src/vim.h               |   4 +-
 25 files changed, 762 insertions(+), 14 deletions(-)
 create mode 100644 src/job.c
 create mode 100644 src/proto/job.pro
Re: [PATCH] Non-blocking job control for vimscript Christian Wellenbrock 1/22/14 1:06 PM

Thank you for your ongoing effort to improve Vim's multithreading capabilities! I like your simple approach and hope we can get the discussion started.

Re: [PATCH] Non-blocking job control for vimscript Paul Moore 1/23/14 3:24 AM
On Tuesday, 21 January 2014 20:24:27 UTC, Thiago Arruda  wrote:
> This patch implements a simple but efficient form of job-control for
> vimscript, enabling vim to cooperate with long-running processes without
> blocking the UI.
>
> It is my second attempt to bring multitasking to vim, but unlike the
> event-loop/message queue patch, this one does not rely on multithreading,
> using only system functions available on most unixes. If required, it
> could also be ported to windows

Do you plan on doing the Windows port? Your example of having a background shell looks really useful, and I'd love to play with it, but the patch is useless to me unless it works on Windows.

Paul

Re: [PATCH] Non-blocking job control for vimscript Jan Larres 1/24/14 2:43 AM
Thiago Padilha <tpadi...@gmail.com>:
> This patch implements a simple but efficient form of job-control for vimscript,
> enabling vim to cooperate with long-running processes without blocking the UI.
>
> It is my second attempt to bring multitasking to vim, but unlike the
> event-loop/message queue patch, this one does not rely on multithreading,
> using only system functions available on most unixes. If required, it
> could also be
> ported to windows(I think it has a concept called IOCP which provides a
> replacement to the 'select' system call).

This sounds great, I hope this approach will finally make it into Vim.
Bram, what do you think about it? It sounds like it has all of the
advantages without the disadvantages (except for the currently missing
Windows support). Merging this will make a whole lot of plugins much
simpler and allow for things not possible yet.

Jan

--
-[ OpenPGP key ID: 00A0FD5F ]-
Some cause happiness wherever they go, others whenever they go.
                -- Oscar Wilde
Re: [PATCH] Non-blocking job control for vimscript Thiago Arruda 1/24/14 7:30 AM
On Thu, Jan 23, 2014 at 8:24 AM, Paul  Moore <p.f....@gmail.com> wrote:
>
> Do you plan on doing the Windows port? Your example of having a background shell looks really useful, and I'd love to play with it, but the patch is useless to me unless it works on Windows.

Windows API is very different and I have no plans on doing this now as
it would require me to read a lot of documentation. If you don't have
access to a unix machine maybe you can try cygwin(I think it
implements the unix system calls on top of windows)
Re: [PATCH] Non-blocking job control for vimscript Anton Shemerey 1/24/14 8:48 AM

Thx for your great job

i've played with it and it looks like a charm, it's a great ability to do some work in a background,
i found few warnings during compilation, it's minor and i don't think it's a problem just for your information

Any way i am looking forward to this patch, great job thx again

---------

gcc -c -I. -Iproto -DHAVE_CONFIG_H   -DMACOS_X_UNIX  -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1       -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -DPYTHON_HOME='"/System/Library/Frameworks/Python.framework/Versions/2.7"' -fPIE  -o objects/if_python.o if_python.c
In file included from if_python.c:809:
./if_py_both.h:2512:26: warning: format specifies type 'int' but the argument has type 'Py_ssize_t' (aka 'long') [-Wformat]
                        "to extended slice"), slicelen);
                                              ^~~~~~~~
./if_py_both.h:39:63: note: expanded from macro 'PyErr_FORMAT'
#define PyErr_FORMAT(exc, str, arg) PyErr_Format(exc, _(str), arg)
                                                              ^
./if_py_both.h:2525:22: warning: format specifies type 'int' but the argument has type 'Py_ssize_t' (aka 'long') [-Wformat]
                    "of size %d"), i, slicelen);
                                   ^
./if_py_both.h:40:71: note: expanded from macro 'PyErr_FORMAT2'
#define PyErr_FORMAT2(exc, str, arg1, arg2) PyErr_Format(exc, _(str), arg1,arg2)
                                                                      ^
./if_py_both.h:2525:25: warning: format specifies type 'int' but the argument has type 'Py_ssize_t' (aka 'long') [-Wformat]
                    "of size %d"), i, slicelen);
                             ~~       ^~~~~~~~
./if_py_both.h:40:76: note: expanded from macro 'PyErr_FORMAT2'
#define PyErr_FORMAT2(exc, str, arg1, arg2) PyErr_Format(exc, _(str), arg1,arg2)
                                                                           ^
3 warnings generated.