Channel & job design

[Bram Moolenaar]

Thanks for the feedback. I think it's time to sketch the upcoming work.
It appears some thought that sockets was the end of it, that's not so.


For jobs the most useful are probably:
- A deamon, serving several Vim instances, connect by socket.
- One server working with one Vim instance, asynchronously.
- A server performing some work for a short time, asynchronously.
- Running a filter, synchronously (existing functionality)

A channel can communicate over:
- A socket (what ch_open() currently does)
- A pipe (only possible with an associated job) connected to
stdin/stdout/stderr.

A channel can communicate:
- With JSON messages
- With JS messages
- Raw messages
- NL-separated messages (this is what netbeans uses)

A channel can be associated with a job, but that job does not
necessarily need be under control of the current Vim (or any Vim)
instance.

It looks like all the functionality is already present in the Vim code
somewhere, we just have to make it available in the right way. The
missing part, a timeout on connect(), was recently added, this may still
need some work.

Raw messages have the problem that Vim doesn't know where the end of a
message is, thus it might invoke a callback with an incomplete message.
The callback will then have the task of parsing and queuing readahead.
Using line breaks is the most common alternative, we can support that.

Some combinations are likely to be most useful. I heard some people
mention using a server over pipes to write a message to and call a
callback when the response is received. I assume this would work best
with NL separated messages.

Using pipes only works when starting a job. Thus these are closely
coupled.

The deamon that serves multiple Vim instances would be most useful to
lookup code locations, for example. Probably best use JSON or JS for
that, as it makes it easier to add more features over time.

One step further would be to use protocol buffers. I think that it's a
step too far right now, maybe later. The current protocol buffer
support unfortunately doesn't support the JS format that I like (the
JSON format uses field names instead of numbers, an unfortunate choice,
as it makes version mismatches difficult).


Some parts that we still need that require some thougths:

- Communicating over a socket requires knowing the port number. For a
deamon this would be a known number. For a server started by Vim it's
best to let the server pick an available port. Then how does it tell
Vim what port it picked? In test_channel we write the port number in
a file. That's not ideal, but perhaps there is no better solution.
Having Vim pick a free port and pass it to the server has race conditions
(but it might still be an acceptable solution).

- When starting a job, where does it's stdin/stdout/stderr go?
Possibly the same terminal as Vim, a newly started terminal, a file or
/dev/null. No, running a terminal inside Vim is not an option. But
we could write stdout/stderr into a buffer.


For starting a deamon job we could have:
let job = job_start("cmd", {options})

We could make it a bit clever, only start the deamon when it's not
running yet. This would check whether connecting to the socket works,
with a zero timeout. Then we have the channel open as well.
let job = job_maystart("cmd", "address", {options})
let handle = job_getchannel(job)

Running one server (or more, doesn't really matter) and connecting over
a socket would work the same way, except that the job is always started
and it would default to be stopped when Vim exits. We don't really need
another function for this, just use two:
let job = job_start("cmd", {options})
let handle = ch_open("address", {options})
That's already working.

When using stdin/stdout/stderr we must start the job and open the
channel at the same time. We have more choices about where to connect
them. It's probably not useful to have separate functions, we can use
the {options}:
let job = job_filter("cmd", {options})
let handle = job_getchannel(job)

Once we have a channel it would be useful to be able to set callbacks:
call ch_setcallback(handle, "receive", "Callback")
call ch_setcallback(handle, "error", "Callback")
call ch_setcallback(handle, "close", "Callback")
And some other things, such as the timeout:
call ch_settimeout(handle, 100)
Using entries in {options} would also work. In fact, we could make it
more generic and just use:
call ch_setoptions(handle, {options})
Where {options} is the same as what's used when creating a channel
(minus the ones that only make sense during creation).

For {options} the main thing we're missing is what to do with
stdin/stdout/stderr:
"term": "pty" // use current terminal
"term": "open" // start a terminal
This would result in stdin/stdout/stderr to connect to the terminal,
unless specified otherwise. Except that reading from the current
terminal won't happen (Vim is reading that).

For a channel we have some options that are the same, such as the
callbacks. But these could be for a socket, thus we need generic
names. Would work by dropping the "std". So job options would be:

"in-io": "null" // read from nirvana
"in-io": "pipe" // read from channel
"in-io": "file" // read from file
"in-file": "/path/file" // file name

"out-io": "null" // write to nirvana
"out-io": "pipe" // write to channel
"out-cb": "Handler" // write handler
"out-io": "file" // write to file
"out-file": "/path/file" // file name
"out-io": "buffer" // write to buffer
"out-buffer": "name" // buffer name

"err-io": "out" // stderr writes to stdout
"err-io": "null"
// etc, like stdout

"kill": 1 // kill job on exit
"kill": 0 // don't kill job on exit

Channel options:
"out-cb": "Handler" // write handler
"err-cb": "Handler" // error handler
"close-cb": "Handler" // close handler
"timeout": msec

That's plenty already, but we may need a few more.

One problem that I haven't figured out yet: When starting a job using a
shell, we get the PID of the shell. So we can kill the shell, but not
the process that it started.
E.g. "/bin/sh python demoserver.py <here >there". A shell is also
useful for expanding wildcards, would be nice if we can make this work.

--
hundred-and-one symptoms of being an internet addict:
182. You may not know what is happening in the world, but you know
every bit of net-gossip there is.

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[tux.]

I have read your thoughts about msgpack vs. JSON and I just want to
take this chance and propose to use S-expressions instead, JSON
without the syntactic bloat.

[Thiago Arruda]

On Monday, February 8, 2016 at 6:08:52 PM UTC-3, Bram Moolenaar wrote:
> Thanks for the feedback. I think it's time to sketch the upcoming work.
> It appears some thought that sockets was the end of it, that's not so.

Bram

I suggest to only focus on the job control feature, at least for now. The reason is that job control as it currently exists in Neovim, is enough to cover any asynchronicity requirements by plugins. Let me give some examples using Neovim API.

```vimL

" Assume function named 's:JobCallback' exists in this script
let s:opts = {
\ 'on_stdout': 's:JobCallback',
\ 'on_stderr': 's:JobCallback',
\ 'on_exit': 's:JobCallback'
\ }

" First, the common use case which is to spawn a long running
program to process some data and return to Vim(in this case a syntax checking
command from gcc):
let job1 = jobstart(['gcc', '-c', '-gnats', 'x.adb'], s:opts)

" A socket connection to a daemon or remote server
let job2 = jobstart(['nc', '127.0.0.1', '1234'], s:opts)

" Download something with curl, saving to a file and notifying the user when
" complete(in neovim API, passing a string as first argument to `jobstart` will
" pass the program to the shell, just like `system()`)
let job3 = jobstart('curl -L -o - http://tarball.url.tgz > out.tgz', s:opts)

" Listen on a socket, waiting for rpc commands
let job4 = jobstart(['nc', '4567'], s:opts)

" Do something after 5 seconds
let job5 = jobstart('sleep 5', s:opts)

```

As you can see, by exposing a job control API that talks with processes through stdin/stdout is enough to cover anything plugins might require. Even if you want to spawn a program connected to a new pty, it would be possible with an external process that forwards data to Vim!

The main reason neovim directly implements a rpc protocol in C is to support the remote plugin host API (mainly to create language bindings without modifying the core) and remote UIs. This would be impossible to do in pure vimscript due to performance reasons.

But Vim does not have these requirements and IMO implementing the suggested
channel feature will only complicate maintenance when a simple job control could do anything users require(let external processes worry about system-specific differences since we only care about stdio!).

Another complication to keep in mind if implementing a json rpc: Vim buffers can contain binary data, what would happen if one evals `getline(lnum)` in such a buffer? The result would probably be invalid json. You'd have to walk through every string returned by the json rpc engine in order to find and escape binary data.

As I said in a previous thread, the old job control patch has almost everything required to make this work, it just needs to be adapted to the current code and for a callback-based API(the old implementation uses autocmd events). It will also have to be adapted for windows, but there seem to be quite a few competent windows developers that contribute to Vim(mattn) and could probably port this without trouble.

Also, I hope you see benefit of following Neovim API: There are already few
popular plugins out there that use it with a good amount of success(neomake,
vim-go, vim-R...).

Just my two cents.

[Olaf Dabrunz]

On 08-Feb-16, Bram Moolenaar wrote:
> Some parts that we still need that require some thougths:
>
> - Communicating over a socket requires knowing the port number. For a
> deamon this would be a known number. For a server started by Vim it's
> best to let the server pick an available port. Then how does it tell
> Vim what port it picked? In test_channel we write the port number in
> a file. That's not ideal, but perhaps there is no better solution.
> Having Vim pick a free port and pass it to the server has race conditions
> (but it might still be an acceptable solution).

To avoid the race, Vim could open the server's listening socket before
forking off the server, then close the socket descriptor on Vim's side.

> - When starting a job, where does it's stdin/stdout/stderr go?
> Possibly the same terminal as Vim, a newly started terminal, a file or
> /dev/null. No, running a terminal inside Vim is not an option. But
> we could write stdout/stderr into a buffer.

It seems most useful to open pipe()s in Vim that connect the job's
stdin/stdout/stderr back to Vim. This makes it easy to use simple
programs in filter jobs and server jobs with a single client.

This seems to be amongst the options you describe below. I do not come
up with an example where direct connection to the terminal is useful.

For multi-client daemon jobs, the stdout and stderr should probably go
to a log file, and stdin should be connected to /dev/null. This and
double-forking with setsid() in between also creates a daemon that is
not associated to a controlling terminal, so that user signals on a
terminal (Ctrl-C) are not accidentally sent to it.

http://stackoverflow.com/questions/3095566/linux-daemonize

Not sure what to do on Windows though.

> One problem that I haven't figured out yet: When starting a job using a
> shell, we get the PID of the shell. So we can kill the shell, but not
> the process that it started.

Put the shell in a new session with setsid(). Killing all processes in
the session is done by sending the signal to the session leader (here:
the shell).

Similar to daemon-izing a job.

http://www.win.tue.nl/~aeb/linux/lk/lk-10.html#ss10.3

--
Olaf Dabrunz (oda <at> fctrace.org)

[Bram Moolenaar]

Olaf Dabrunz wrote:

> On 08-Feb-16, Bram Moolenaar wrote:
> > Some parts that we still need that require some thougths:
> >
> > - Communicating over a socket requires knowing the port number. For a
> > deamon this would be a known number. For a server started by Vim it's
> > best to let the server pick an available port. Then how does it tell
> > Vim what port it picked? In test_channel we write the port number in
> > a file. That's not ideal, but perhaps there is no better solution.
> > Having Vim pick a free port and pass it to the server has race conditions
> > (but it might still be an acceptable solution).
>
> To avoid the race, Vim could open the server's listening socket before
> forking off the server, then close the socket descriptor on Vim's side.

I don't think that works. If Vim keeps the socket open the server can't
open it again, gives an "already in use" error.

> > - When starting a job, where does it's stdin/stdout/stderr go?
> > Possibly the same terminal as Vim, a newly started terminal, a file or
> > /dev/null. No, running a terminal inside Vim is not an option. But
> > we could write stdout/stderr into a buffer.
>
> It seems most useful to open pipe()s in Vim that connect the job's
> stdin/stdout/stderr back to Vim. This makes it easy to use simple
> programs in filter jobs and server jobs with a single client.
>
> This seems to be amongst the options you describe below. I do not come
> up with an example where direct connection to the terminal is useful.
>
> For multi-client daemon jobs, the stdout and stderr should probably go
> to a log file, and stdin should be connected to /dev/null. This and
> double-forking with setsid() in between also creates a daemon that is
> not associated to a controlling terminal, so that user signals on a
> terminal (Ctrl-C) are not accidentally sent to it.
>
> http://stackoverflow.com/questions/3095566/linux-daemonize
>
> Not sure what to do on Windows though.
>
> > One problem that I haven't figured out yet: When starting a job using a
> > shell, we get the PID of the shell. So we can kill the shell, but not
> > the process that it started.
>
> Put the shell in a new session with setsid(). Killing all processes in
> the session is done by sending the signal to the session leader (here:
> the shell).
>
> Similar to daemon-izing a job.
>
> http://www.win.tue.nl/~aeb/linux/lk/lk-10.html#ss10.3

Unfortunately that doesn't work. I used that in the test at first:

let job = job_start(['/bin/sh', '-c', 'python test_channel.py </dev/null >/dev/null'])
call getchar()
call job_stop(job)

While it's waiting for a character, check the processes. You should see
both the shell and the python process. After pressing Enter the shell
is gone but the python process keeps running. At least on my Ubuntu
system.

--
hundred-and-one symptoms of being an internet addict:

194. Your business cards contain your e-mail and home page address.

[James McCoy]

On Tue, Feb 09, 2016 at 01:57:32AM +0100, Bram Moolenaar wrote:
>
> Olaf Dabrunz wrote:
>
> > On 08-Feb-16, Bram Moolenaar wrote:
> > > Some parts that we still need that require some thougths:
> > >
> > > - Communicating over a socket requires knowing the port number. For a
> > > deamon this would be a known number. For a server started by Vim it's
> > > best to let the server pick an available port. Then how does it tell
> > > Vim what port it picked? In test_channel we write the port number in
> > > a file. That's not ideal, but perhaps there is no better solution.
> > > Having Vim pick a free port and pass it to the server has race conditions
> > > (but it might still be an acceptable solution).
> >
> > To avoid the race, Vim could open the server's listening socket before
> > forking off the server, then close the socket descriptor on Vim's side.
>
> I don't think that works. If Vim keeps the socket open the server can't
> open it again, gives an "already in use" error.

The whole point is that Vim doesn't keep it open. Vim opens the socket
and sets it up so the server that gets spawned inherits the descriptor.
The server can then just use the inherited descriptor and Vim closes its
descriptor.

Cheers,
--
James
GPG Key: 4096R/331BA3DB 2011-12-05 James McCoy <jame...@jamessan.com>

[Olaf Dabrunz]

On 09-Feb-16, Bram Moolenaar wrote:
>
> Olaf Dabrunz wrote:
>
> > On 08-Feb-16, Bram Moolenaar wrote:
> > > Some parts that we still need that require some thougths:
> > >
> > > - Communicating over a socket requires knowing the port number. For a
> > > deamon this would be a known number. For a server started by Vim it's
> > > best to let the server pick an available port. Then how does it tell
> > > Vim what port it picked? In test_channel we write the port number in
> > > a file. That's not ideal, but perhaps there is no better solution.
> > > Having Vim pick a free port and pass it to the server has race conditions
> > > (but it might still be an acceptable solution).
> >
> > To avoid the race, Vim could open the server's listening socket before
> > forking off the server, then close the socket descriptor on Vim's side.
>
> I don't think that works. If Vim keeps the socket open the server can't
> open it again, gives an "already in use" error.

James already answered it, more concisely.

My mid-air collision version, and longer...:

The socket descriptor opened in Vim will be inherited to the child
process, the server. The child process / server does not need to
re-open it, it simply uses the already open descriptor. (Trying to
re-open the socket would indeed give "already in use".)

The parent process (Vim) closes its copy of the socket descriptor after
the fork because it does not need the listening socket, and because the
socket should really shut down whenever the child (the server) closes it.

Inheriting the descriptor of the listening socket is done regularly in
pre-forked multiprocess servers.

The "prefork" bullet summarizes it:
http://freeprogrammersblog.vhex.net/post/linux-39-introduced-new-way-of-writing-socket-servers/2

> > > One problem that I haven't figured out yet: When starting a job using a
> > > shell, we get the PID of the shell. So we can kill the shell, but not
> > > the process that it started.
> >
> > Put the shell in a new session with setsid(). Killing all processes in
> > the session is done by sending the signal to the session leader (here:
> > the shell).
> >
> > Similar to daemon-izing a job.
> >
> > http://www.win.tue.nl/~aeb/linux/lk/lk-10.html#ss10.3
>
> Unfortunately that doesn't work. I used that in the test at first:
>
> let job = job_start(['/bin/sh', '-c', 'python test_channel.py </dev/null >/dev/null'])
> call getchar()
> call job_stop(job)
>
> While it's waiting for a character, check the processes. You should see
> both the shell and the python process. After pressing Enter the shell
> is gone but the python process keeps running. At least on my Ubuntu
> system.

Ah right, you already use this in 7.4.1274.

To correct myself: kill the negative of the PID of the session leader,
this sends the signal to all processes in its process group (!), see
kill(2):

mch_stop_job():

sig = atoi((char *)how);
else
return FAIL;
- kill(job->jv_pid, sig);
+ kill(-job->jv_pid, sig);
return OK;
}
#endif


This also triggers a sort of chain reaction that should bring down the
other process groups in the session (if any), unless they ignore SIGHUP:

When the session leader exits, all processes in the foreground
process receive a SIGHUP.

When the parent (typically the session leader) of a background
process group with at least one stopped process exits, all processes
in that background process group receive a SIGHUP. (And a SIGCONT,
which makes them continue running in the background in case they
ignore SIGHUP.)

See more details in
- http://www.win.tue.nl/~aeb/linux/lk/lk-10.html#ss10.3
- man 2 setpgid

[Gary Johnson]

On 2016-02-08, Bram Moolenaar wrote:

> One problem that I haven't figured out yet: When starting a job using a
> shell, we get the PID of the shell. So we can kill the shell, but not
> the process that it started.
> E.g. "/bin/sh python demoserver.py <here >there". A shell is also
> useful for expanding wildcards, would be nice if we can make this work.

Starting the process with exec should fix that. That will replace
the shell with the job, giving the job the same PID as the shell.
Your example would then become:

/bin/sh exec python demoserver.py <here >there

Regards,
Gary

[Olaf Dabrunz]

On 09-Feb-16, Olaf Dabrunz wrote:
> Ah right, you already use this in 7.4.1274.
>
> To correct myself: kill the negative of the PID of the session leader,
> this sends the signal to all processes in its process group (!), see
> kill(2):
>
> mch_stop_job():
>
> sig = atoi((char *)how);
> else
> return FAIL;
> - kill(job->jv_pid, sig);
> + kill(-job->jv_pid, sig);
> return OK;
> }
> #endif
>
>
> This also triggers a sort of chain reaction that should bring down the
> other process groups in the session (if any), unless they ignore SIGHUP:
>
> When the session leader exits, all processes in the foreground
> process receive a SIGHUP.
>
> When the parent (typically the session leader) of a background
> process group with at least one stopped process exits, all processes
> in that background process group receive a SIGHUP. (And a SIGCONT,
> which makes them continue running in the background in case they
> ignore SIGHUP.)

Processes that are not terminated by this chain reaction are
- in a background process group where all processes are running
(long running compute jobs)
$ break_encryption &
- background processes that choose to survive session termination,
even when temporarily stopped
$ nohup break_encryption &
$ kill -STOP %1 # give me some processing power now

Daemons detach even further, by leaving the session.

[Olaf Dabrunz]

The python process started from the shell in your example must have been
in its own process group and only blocked on reading from a socket, but
not stopped by a signal.

So you essentially started it as a long running compute job, which is
unaffected by session termination.

Gary's suggestion to use

/bin/sh -c "exec python demoserver.py <here >there"

should fix this, by replacing the session leader shell with the python
process executing demoserver.py, so it is not in a separate process
group and affected directly by either kill(job->jv_pid, sig) or
kill(-job->jv_pid, sig).

> - background processes that choose to survive session termination,
> even when temporarily stopped
> $ nohup break_encryption &
> $ kill -STOP %1 # give me some processing power now
>
> Daemons detach even further, by leaving the session.
>
> > See more details in
> > - http://www.win.tue.nl/~aeb/linux/lk/lk-10.html#ss10.3
> > - man 2 setpgid
>
>
> --
> Olaf Dabrunz (oda <at> fctrace.org)
>

> --
> --
> You received this message from the "vim_dev" maillist.
> Do not top-post! Type your reply below the text you are replying to.
> For more information, visit http://www.vim.org/maillist.php
>
> ---
> You received this message because you are subscribed to the Google Groups "vim_dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Olaf Dabrunz]

Hrm. An *interactive* shell terminates even *running* background
processes, so the "nohup" is needed (man bash):

The shell exits by default upon receipt of a SIGHUP. Before
exiting, an interactive shell resends the SIGHUP to all jobs,
running or stopped.

But when started from a non-interactive shell, a running background
process should not be terminated.

[Bram Moolenaar]

Olaf Dabrunz wrote:

> > > On 08-Feb-16, Bram Moolenaar wrote:
> > > > Some parts that we still need that require some thougths:
> > > >
> > > > - Communicating over a socket requires knowing the port number. For a
> > > > deamon this would be a known number. For a server started by Vim it's
> > > > best to let the server pick an available port. Then how does it tell
> > > > Vim what port it picked? In test_channel we write the port number in
> > > > a file. That's not ideal, but perhaps there is no better solution.
> > > > Having Vim pick a free port and pass it to the server has race conditions
> > > > (but it might still be an acceptable solution).
> > >
> > > To avoid the race, Vim could open the server's listening socket before
> > > forking off the server, then close the socket descriptor on Vim's side.
> >
> > I don't think that works. If Vim keeps the socket open the server can't
> > open it again, gives an "already in use" error.
>
> James already answered it, more concisely.
>
> My mid-air collision version, and longer...:
>
> The socket descriptor opened in Vim will be inherited to the child
> process, the server. The child process / server does not need to
> re-open it, it simply uses the already open descriptor. (Trying to
> re-open the socket would indeed give "already in use".)
>
> The parent process (Vim) closes its copy of the socket descriptor after
> the fork because it does not need the listening socket, and because the
> socket should really shut down whenever the child (the server) closes it.

Hmm, OK. But that only works for a process that was specifically
written to do this. Not something that takes a --port argument.

Also, how does the child process know which file descriptor has the
socket? For example, how would this work with the demo server?

Does this work for MS-Windows? Probably not, since it doesn't use
fork().

Ah, forgot about that way. It's actually in os_unix.c.

> This also triggers a sort of chain reaction that should bring down the
> other process groups in the session (if any), unless they ignore SIGHUP:
>
> When the session leader exits, all processes in the foreground
> process receive a SIGHUP.
>
> When the parent (typically the session leader) of a background
> process group with at least one stopped process exits, all processes
> in that background process group receive a SIGHUP. (And a SIGCONT,
> which makes them continue running in the background in case they
> ignore SIGHUP.)
>
> See more details in
> - http://www.win.tue.nl/~aeb/linux/lk/lk-10.html#ss10.3
> - man 2 setpgid

So, should we do this always or make it an option? I think we should
make this the default and have an option to only kill the process
itself. Although the child of the child could use setsid() if it
doesn't want to be killed.

--
% cat /usr/include/sys/errno.h
#define EPERM 1 /* Operation not permitted */
#define ENOENT 2 /* No such file or directory */
#define ESRCH 3 /* No such process */
[...]
#define EMACS 666 /* Too many macros */
%

[Olaf Dabrunz]

On 09-Feb-16, Bram Moolenaar wrote:
>

The server could take a --socket-fd <nr> argument. If there is an open
socket on that file descriptor, then it can use that.

struct stat statbuf;
int fd;

// get fd from argv

fstat(fd, &statbuf);
if (S_ISSOCK(statbuf.st_mode))
...;

Some tools already use a --<something>-fd <nr> option for file
descriptors, so there is some precedent here.

> Does this work for MS-Windows? Probably not, since it doesn't use
> fork().

Not an expert there. According to the information I gathered [1] this
used to work (across CreateProcess(), instead of fork()), and well
enough on WinNT/2000 [2], but cannot be relied upon, because MS broke it
[3, 1].

A solution could be to pass the socket *after* creating the child, by
using WSADuplicateSocket() [4], but this requires some other form of IPC
(e.g. a named pipe) between parent and child to pass the socket over it.

I think this could better be answered by someone who has experience with
socket passing on Windows.

[1] http://stackoverflow.com/questions/11847793/are-tcp-socket-handles-inheritable
[2] https://support.microsoft.com/en-us/kb/150523
[3] https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx
(look for "socket")
[4] https://msdn.microsoft.com/en-us/library/windows/desktop/ms741565(v=vs.85).aspx

I also think it should be the default. And probably it is overkill to
have an option that only kills the PID, because as you say, a child of a
child can use setsid() to avoid being killed.

But starting the server with "exec" when started from a shell should be
documented, as the server may otherwise not get killed (mentioned by
Gary and in one of my other mails).

[Christian Brabandt]

Hi Thiago!

On Mo, 08 Feb 2016, Thiago Arruda wrote:

> But Vim does not have these requirements and IMO implementing the
> suggested channel feature will only complicate maintenance when a
> simple job control could do anything users require(let external
> processes worry about system-specific differences since we only care
> about stdio!).
>
> Another complication to keep in mind if implementing a json rpc: Vim
> buffers can contain binary data, what would happen if one evals
> `getline(lnum)` in such a buffer? The result would probably be invalid
> json. You'd have to walk through every string returned by the json rpc
> engine in order to find and escape binary data.
>
> As I said in a previous thread, the old job control patch has almost
> everything required to make this work, it just needs to be adapted to
> the current code and for a callback-based API(the old implementation
> uses autocmd events). It will also have to be adapted for windows, but
> there seem to be quite a few competent windows developers that
> contribute to Vim(mattn) and could probably port this without trouble.
>
> Also, I hope you see benefit of following Neovim API: There are
> already few popular plugins out there that use it with a good amount
> of success(neomake, vim-go, vim-R...).
>
> Just my two cents.

Thanks for the constructive feedback. I appreciate it, that you as the
Neovim creator give feedback here. It's also appreciated, that certain
bugs, which have been found when developing Neovim have at least found
their way back to Vim itself (although, that communication could be
improved).

I can only speak for myself, but I think it would be appreciated by the
Vim/Neovim user community, if the API is at least standardized, so that
not every plugin writer needs to either decide whether he wants to
support Vim or Neovim or has to work around the particularities in both
applications.

Since you seem to know both interfaces, would you be willing to
contribute a patch, that will help develop a consistent interface as far
as this seems feasible? Bram, since you are actually working on that
part of the code, can we assume, that you would look into such a patch
and help the community to have at least some kind of consistent
interface for its users?

I really hope, that we can find a way, to work together and specify the
details of those interfaces which would gain the users a lot. So let's
not have personal prejudice get in our way! (I don't want to see more of
those angry discussions, as its happening currently at reddit, so let's
just make the best for all users).

Best,
Christian

[Olaf Dabrunz]

On 09-Feb-16, Christian Brabandt wrote:
> Thanks for the constructive feedback. I appreciate it, that you as the
> Neovim creator give feedback here. It's also appreciated, that certain
> bugs, which have been found when developing Neovim have at least found
> their way back to Vim itself (although, that communication could be
> improved).
>
> I can only speak for myself, but I think it would be appreciated by the
> Vim/Neovim user community, if the API is at least standardized, so that
> not every plugin writer needs to either decide whether he wants to
> support Vim or Neovim or has to work around the particularities in both
> applications.

+1

[LCD 47]

On 8 February 2016, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
> Thanks for the feedback. I think it's time to sketch the upcoming work.
> It appears some thought that sockets was the end of it, that's not so.
>
>
> For jobs the most useful are probably:
> - A deamon, serving several Vim instances, connect by socket.
> - One server working with one Vim instance, asynchronously.
> - A server performing some work for a short time, asynchronously.
> - Running a filter, synchronously (existing functionality)
>
> A channel can communicate over:

I don't really understand what problem do channels solve, or where
is all this heading. Thus I'll refer mostly to jobs below, and I'll
comment only on a few implementation details for channels.

> - A socket (what ch_open() currently does)
> - A pipe (only possible with an associated job) connected to
> stdin/stdout/stderr.

There are also UNIX domain sockets (a.k.a. named pipes), which you
get essentially for free once you have the code for INET sockets. They
are as efficient as (unnamed) pipes. The code is already there, but
it's mutually exclusive with INET sockets. I don't think there is any
reason why the two can't coexist as separate mechanisms.

[...]

> Some combinations are likely to be most useful. I heard some people
> mention using a server over pipes to write a message to and call a
> callback when the response is received. I assume this would work best
> with NL separated messages.
>
> Using pipes only works when starting a job. Thus these are closely
> coupled.

As you say, getting a program's output with callbacks over pipes
only works for jobs. The main applications to this are:

(1) running something like make in background
(2) running a slow linter in background.

With (1) we care to be notified when make is done, and its exit
code. We care about the output only if there have been errors (I'm
simplifying, of course). We don't really care what's going on before
make is finished.

With (2) we care to be notified about lint messages as soon as the
linter prints them, so that we can highlight syntax errors on the fly,
and perhaps also update a quickfix list. That's the main point of
having callbacks for the job's stdout and stderr. It's probably safe to
assume the output to consist of lines of text.

Both (1) and (2) are short-lived processes: they might be slow
anough to make running them in foreground annoying, but we expect them
to be finished (one way or another) when Vim exits.

Starting a (presumably long-lived) server from Vim on the other hand
is a very different animal. A server might print things like error
messages before it goes to background. After that it isn't supposed
to print anything or need output from stdin (doing that would rise a
signal). A server in background is supposed to communicate with other
processes only though a socket (INET, or UNIX domain).

> Some parts that we still need that require some thougths:
>
> - Communicating over a socket requires knowing the port number. For a
> deamon this would be a known number. For a server started by Vim it's
> best to let the server pick an available port. Then how does it tell
> Vim what port it picked? In test_channel we write the port number in
> a file. That's not ideal, but perhaps there is no better solution.
> Having Vim pick a free port and pass it to the server has race conditions
> (but it might still be an acceptable solution).

Communicating with a remote server requires a known address, and a
known port. Remote servers are not started by Vim though.

Servers started by Vim run on the local machine, and thus can use
UNIX domain sockets instead of INET. A path pointing to an unique named
pipe could be passed on to the server as a shared meeting point.

> - When starting a job, where does it's stdin/stdout/stderr go?
> Possibly the same terminal as Vim, a newly started terminal, a file or
> /dev/null. No, running a terminal inside Vim is not an option. But
> we could write stdout/stderr into a buffer.

The answer depends on whether the job is a daemon, or a short-lived
process. In the former case, as I said, we only need to catch messages
at startup (after that stdin, stdout, and stderr should be closed).

In the later case I think the output should go to the callback
functions. Default callback function could just send everything to
/dev/null. And if the user cares to print the output to the terminal
she could override the callbacks with echo or echomsg. No need to make
any special cases for it in Vim.

A further difference between long-lived and short-lived processes is
the way process groups are handled. For short-lived processes, the
sequence should be something like this:

- fork()
- in the child: exec()
- in the parent: install a SIGCHLD and waitpid()

For daemons the sequence should be:

- fork()
- in the child: setsid(), fork() again, and exit
- in the second child: exec()
- in the parent: blocking waitpid().

To kill the group of a daemon, you need to kill() the negated PID of
the _first_ child, because that's the session leader.

[...]

> For {options} the main thing we're missing is what to do with
> stdin/stdout/stderr:
> "term": "pty" // use current terminal
> "term": "open" // start a terminal
> This would result in stdin/stdout/stderr to connect to the terminal,
> unless specified otherwise. Except that reading from the current
> terminal won't happen (Vim is reading that).

[...]

Why is it important for the job to be able to access a pty?

/lcd

[LCD 47]

On 9 February 2016, LCD 47 <lcd...@gmail.com> wrote:
[...]

> Starting a (presumably long-lived) server from Vim on the other hand
> is a very different animal. A server might print things like error
> messages before it goes to background. After that it isn't supposed
> to print anything or need output from stdin (doing that would rise a

s/output/input/

> signal). A server in background is supposed to communicate with other
> processes only though a socket (INET, or UNIX domain).

[...]

> A further difference between long-lived and short-lived processes
> is the way process groups are handled. For short-lived processes, the
> sequence should be something like this:
>
> - fork()
> - in the child: exec()

Better: close stdin, then exec(). Dealing with stdin while we're
also reading output can run into a deadlock.

> - in the parent: install a SIGCHLD and waitpid()
>
> For daemons the sequence should be:
>
> - fork()
> - in the child: setsid(), fork() again, and exit

Better yet: initialize, print errors etc., then setsid(), fork()
again, and exit.

> - in the second child: exec()

Better: close fds 0-2, exec()

> - in the parent: blocking waitpid().
>
> To kill the group of a daemon, you need to kill() the negated PID
> of the _first_ child, because that's the session leader.
[...]

"Mixed" modes are possible too, but we really don't want to go
there.

/lcd

[Bram Moolenaar]

Lcd wrote:

> On 8 February 2016, Bram Moolenaar <Br...@moolenaar.net> wrote:
> >
> > Thanks for the feedback. I think it's time to sketch the upcoming work.
> > It appears some thought that sockets was the end of it, that's not so.
> >
> >
> > For jobs the most useful are probably:
> > - A deamon, serving several Vim instances, connect by socket.
> > - One server working with one Vim instance, asynchronously.
> > - A server performing some work for a short time, asynchronously.
> > - Running a filter, synchronously (existing functionality)
> >
> > A channel can communicate over:
>
> I don't really understand what problem do channels solve, or where
> is all this heading. Thus I'll refer mostly to jobs below, and I'll
> comment only on a few implementation details for channels.

Jobs are processes, channels are a way to communicate. I don't see how
you can have jobs and not communicate with them in a useful way.

> > - A socket (what ch_open() currently does)
> > - A pipe (only possible with an associated job) connected to
> > stdin/stdout/stderr.
>
> There are also UNIX domain sockets (a.k.a. named pipes), which you
> get essentially for free once you have the code for INET sockets. They
> are as efficient as (unnamed) pipes. The code is already there, but
> it's mutually exclusive with INET sockets. I don't think there is any
> reason why the two can't coexist as separate mechanisms.

We can support that later. As far as I know the main difference is how
you connect, after that it's basically the same. Since we are using the
netbeans code it uses sockets.

Sure, different kind of jobs, but communication will mostly work the
same way.

Not important. We do need the I/O to go somewhere. Main problem is
input, if the job tries to read from the same terminal Vim will stop
working. But output could go to the same place. Might mess up the
display. So, normally "pty" means you have to setup the callbacks and
everything to make it work, while "open" would have the job happily use
some terminal. That would be required if the job has to prompt for
anything, or display progress.

But like you say, many jobs would be completely silent, all output is
handled by handlers.

--
Spam seems to be something useful to novices. Later you realize that
it's a bunch of indigestable junk that only clogs your system.
Applies to both the food and the e-mail!

[Thiago Arruda]

On Tuesday, February 9, 2016 at 4:36:48 PM UTC-3, Christian Brabandt wrote:
> Thanks for the constructive feedback. I appreciate it, that you as the
> Neovim creator give feedback here.

Hi Christian, I also appreciate you taking notice.

> It's also appreciated, that certain
> bugs, which have been found when developing Neovim have at least found
> their way back to Vim itself (although, that communication could be
> improved).

I'd like to improve communication and hope that my experience in adding
asynchronous events to Neovim helps the Vim community to avoid some of the
mistakes I made. I also would welcome the Vim community to engage more in Neovim
development, even if it is only to use it as a "test bench" for new features
that could be integrated back into Vim.

> I can only speak for myself, but I think it would be appreciated by the
> Vim/Neovim user community, if the API is at least standardized, so that
> not every plugin writer needs to either decide whether he wants to
> support Vim or Neovim or has to work around the particularities in both
> applications.

My thoughts exactly. In a previous
[message](https://groups.google.com/d/msg/vim_dev/_SbMTGshzVc/XZEXxaxDDQAJ) I
said:

> I will be glad to add a compatibility layer to Vim's channel feature if plugin
> authors start to use it, but unfortunately I can't do the same for this job
> control API as it will only make things harder for users.

Tha doesn't mean we are unwilling to discuss changes in Neovim job API. On the
contrary, we are open to standardizing it, but we need to carefully examine the
strengths and weakness of each interface before making harsh decisions.

The reason I said we would not follow Vim is because I honestly felt that the
job proposal for Vim was a regression when compared to what we have in
Neovim(and I know many plugin developers that use it will agree with me), but
could be convinced otherwise with arguments and examples.

> Since you seem to know both interfaces, would you be willing to
> contribute a patch, that will help develop a consistent interface as far
> as this seems feasible? Bram, since you are actually working on that
> part of the code, can we assume, that you would look into such a patch
> and help the community to have at least some kind of consistent
> interface for its users?

As I said in the previous post, yes, I'm willing to update my old patch to Vim,
possibly making changes to the API(should we reach an agreement on it). But I
need to wait Bram's response, as so far I've had pratically no feedback from
him(and one can only have so much faith).

> I really hope, that we can find a way, to work together and specify the
> details of those interfaces which would gain the users a lot. So let's
> not have personal prejudice get in our way! (I don't want to see more of
> those angry discussions, as its happening currently at reddit, so let's
> just make the best for all users).

Same here! It is not unreasonable to think that Neovim had a major impact in
Bram's decision to implement the channel/job features, and that thought also
gave a bump into my motivation for making Neovim even better. This confirms that
competition is only good for everyone, but there's no reason we can't do it in a
friendly way, don't you think? So, instead of wasting our time with angry
discussions, let's join forces and focus on the "real enemy": Emacs!(jk hahaha)

Before I finish, let me quote something LCD said in a previous message:

> I don't really understand what problem do channels solve, or where
is all this heading.

This says everything: we should focus on solving the problem rather than simply
adding new features without a proper plan. Since you were so friendly, I will
later post another message that expands my reasoning as to why I think Vim
should focus on job control(at least for now).

Thanks again Chris!

Thiago.

[Marius Gedminas]

On Tue, Feb 09, 2016 at 10:24:55PM +0200, LCD 47 wrote:
> > - A socket (what ch_open() currently does)
> > - A pipe (only possible with an associated job) connected to
> > stdin/stdout/stderr.
>
> There are also UNIX domain sockets (a.k.a. named pipes), which you
> get essentially for free once you have the code for INET sockets. They
> are as efficient as (unnamed) pipes.

Nitpick: Unix domain sockets and named pipes (aka FIFOs) are two
different concepts.

http://linux.die.net/man/7/unix
http://linux.die.net/man/4/fifo

Marius Gedminas
--
IBM motto: "If you can't read our assembly language, you must be
borderline dyslexic, and we don't want you to mess with it anyway"
-- Linus Torvalds

[LCD 47]

On 9 February 2016, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
> Lcd wrote:
[...]

> > I don't really understand what problem do channels solve, or
> > where is all this heading. Thus I'll refer mostly to jobs below,
> > and I'll comment only on a few implementation details for channels.
>
> Jobs are processes, channels are a way to communicate. I don't see
> how you can have jobs and not communicate with them in a useful way.

I'm afraid I still don't get it. With jobs Vim can communicate
with processes that aren't specifically aware they are talking to Vim.
It can do that by opening pipes for the jobs' stdin and stdout. If
it needs to give them "wheels" (that is, communicate with processes
on another machine), this can be done with UCSPI. This is a solved
problem, and the common solutions such as UCSPI can do things like
connect over TLS (and actually get it right), and do fancy TCP window
scaling and error recovery.

On the other hand, there are daemons that do things like parse a
source tree in background and can be queried about it over a socket.
But these daemons have their own APIs, and are not specifically prepared
to talk to Vim. Making Vim able to talk to them would require some kind
of adapter, which might be a job started by Vim. But then the burden
of handling protocols, TCP, and so on would be on the adapter. Vim
could still talk to the adapter through pipes to the adapter's stdin
and stdout. But the adapter would have to be written specifically for
the particular daemon, and there's no way around that. If Vim is to
talk directly to the daemon, without the adapter, then JSON or protocol
buffers are either too much, or too little. I can't see how channels
(as they are now) can fit into this picture. _shrug_

[...]

> > There are also UNIX domain sockets (a.k.a. named pipes), which
> > you get essentially for free once you have the code for INET
> > sockets. They are as efficient as (unnamed) pipes. The code is
> > already there, but it's mutually exclusive with INET sockets. I
> > don't think there is any reason why the two can't coexist as
> > separate mechanisms.
>
> We can support that later. As far as I know the main difference is
> how you connect, after that it's basically the same. Since we are
> using the netbeans code it uses sockets.

The main advantage over INET is that with UNIX sockets can choose
the rendezvous point, and pass it to the daemon on the command line (or
whatever). With INET it's the daemon that gets to choose the port, and
it has to find a way to pass it back to Vim.

[...]
> > Why is it important for the job to be able to access a pty?
>
> Not important. We do need the I/O to go somewhere. Main problem is
> input, if the job tries to read from the same terminal Vim will stop
> working. But output could go to the same place. Might mess up the
> display. So, normally "pty" means you have to setup the callbacks and
> everything to make it work, while "open" would have the job happily
> use some terminal. That would be required if the job has to prompt
> for anything, or display progress.

[...]

I don't think a job's input can be allowed access to Vim's tty,
ever. There are two main cases here: programs that don't need any
input, and are happy with command line options; and programs that need
to read things from stdin. In the first case there's no problem to
solve. It's the second case that's interesting, and there Vim could
either use another callback for the job's stdin to feed it data as
the job requests, or simply let it read from a file. The callback is
problematic, because it can run into deadlocks with the callbacks for
output (this is a well-known hard problem). Reading from a file is
safe, but limits what the job can do.

/lcd

[LCD 47]

On 10 February 2016, Marius Gedminas <mar...@gedmin.as> wrote:
> On Tue, Feb 09, 2016 at 10:24:55PM +0200, LCD 47 wrote:
> > > - A socket (what ch_open() currently does)
> > > - A pipe (only possible with an associated job) connected to
> > > stdin/stdout/stderr.
> >
> > There are also UNIX domain sockets (a.k.a. named pipes), which
> > you get essentially for free once you have the code for INET
> > sockets. They are as efficient as (unnamed) pipes.
>
> Nitpick: Unix domain sockets and named pipes (aka FIFOs) are two
> different concepts.
>
> http://linux.die.net/man/7/unix
> http://linux.die.net/man/4/fifo

True, that. UNIX domain sockets are bidirectional, FIFOs are
unidirectional. Very different beasts. I was referring to UNIX domain
sockets.

/lcd

[LCD 47]

On 10 February 2016, LCD 47 <lcd...@gmail.com> wrote:
> On 9 February 2016, Bram Moolenaar <Br...@moolenaar.net> wrote:

[...]

> > We can support that later. As far as I know the main difference is
> > how you connect, after that it's basically the same. Since we are
> > using the netbeans code it uses sockets.
>
> The main advantage over INET is that with UNIX sockets can choose

s/can choose/Vim can choose/

> the rendezvous point, and pass it to the daemon on the command line (or
> whatever). With INET it's the daemon that gets to choose the port, and
> it has to find a way to pass it back to Vim.
[...]

/lcd

[Bram Moolenaar]

Marius Gedminas wrote:

> On Tue, Feb 09, 2016 at 10:24:55PM +0200, LCD 47 wrote:
> > > - A socket (what ch_open() currently does)
> > > - A pipe (only possible with an associated job) connected to
> > > stdin/stdout/stderr.
> >
> > There are also UNIX domain sockets (a.k.a. named pipes), which you
> > get essentially for free once you have the code for INET sockets. They
> > are as efficient as (unnamed) pipes.
>
> Nitpick: Unix domain sockets and named pipes (aka FIFOs) are two
> different concepts.
>
> http://linux.die.net/man/7/unix
> http://linux.die.net/man/4/fifo

Once opened don't they work practially the same? As in using select(),
read(), write() and things like that. Errors can be different, but that
is a minor thing.

--
hundred-and-one symptoms of being an internet addict:

208. Your goals for the future are obtaining an T1 connection and
a 130 gig hard drive.

[LCD 47]

On 10 February 2016, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
> Marius Gedminas wrote:
>
> > On Tue, Feb 09, 2016 at 10:24:55PM +0200, LCD 47 wrote:
> > > > - A socket (what ch_open() currently does)
> > > > - A pipe (only possible with an associated job) connected to
> > > > stdin/stdout/stderr.
> > >
> > > There are also UNIX domain sockets (a.k.a. named pipes), which
> > > you get essentially for free once you have the code for INET
> > > sockets. They are as efficient as (unnamed) pipes.
> >
> > Nitpick: Unix domain sockets and named pipes (aka FIFOs) are two
> > different concepts.
> >

> > http://linux.die.net/man/7/unix 4/fifo

>
> Once opened don't they work practially the same? As in using
> select(), read(), write() and things like that. Errors can be
> different, but that is a minor thing.

There are:

(1) INET sockets
(2) UNIX domain sockets
(3) FIFOs.

(1) and (2) are essentially the same, while (3) are something
different. Marius's objection is that "named pipes" refer to (3),
rather than (2), at least these days (there were conflicting names
between commercial UNIXen in the past).

/lcd

[Bram Moolenaar]

Thiago Arruda wrote:

[picking out individual points to avoid trying to discuss everything at
the same time]

> The reason I said we would not follow Vim is because I honestly felt
> that the job proposal for Vim was a regression when compared to what
> we have in Neovim(and I know many plugin developers that use it will
> agree with me), but could be convinced otherwise with arguments and
> examples.

I wonder if you read my proposal wrong, or perhaps I didn't explain
something properly. I read your response, and can only say that it's
all covered in what I proposed.

I think it may be caused by the notion that in NeoVim a job includes the
communication with the job (over pipes), while in my proposal I separate
them out. So you can have a channel in JSON mode communication to a job
with a pipe. I havent seen something like that in NeoVim. But then I
haven't spent much time trying to understand what NeoVim is doing. Can
NeoVim use msgpack on stdout?

> As I said in the previous post, yes, I'm willing to update my old
> patch to Vim, possibly making changes to the API(should we reach an
> agreement on it). But I need to wait Bram's response, as so far I've
> had pratically no feedback from him(and one can only have so much
> faith).

I don't think that old patch adds much, now we already have working
channels. Only the part with making channels work with
stdin/stdout/stderr still needs to be done. And that code already
exists in os_unix.c. MS-Windows is the most difficult (since sockets
there work differently from file handles) and I thought your patch
doesn't have that part.

> Before I finish, let me quote something LCD said in a previous message:
>
> > I don't really understand what problem do channels solve, or where
> > is all this heading.
>
> This says everything: we should focus on solving the problem rather
> than simply adding new features without a proper plan. Since you were
> so friendly, I will later post another message that expands my
> reasoning as to why I think Vim should focus on job control (at least
> for now).

The whole thing I propose was to support what plugin writers were asking
for. With future feature requests in mind, and combining with the
existing code for netbeans.

Problem with netbeans is that it uses a weird message format. I want to
take that in the right direction, don't want more people to use the
netbeans format. JSON appears to be the best choice. I know NeoVim
uses msgpack, but I just don't like it.

--
hundred-and-one symptoms of being an internet addict:

209. Your house stinks because you haven't cleaned it in a week.

[Bram Moolenaar]

It seems you are missing that stdin/stdout IS the channel. On top of it
you can define the protocol: raw, NL-separated, JSON, etc.

So if that deamon speaks language X, then you connect with a raw channel
and let the Vim plugin decode language X. Hopefully you can do
something more clever (raw messages suffer from the problem that you
never know when the message ends).

It's the good old layering: lower layer about how to connect and
send/receive (socket, pipe, whatever), layer above that about the
contents (message separation and parsing). Even higher about what
request belongs to what response (what is currently only supported for
JSON, using sequence numbers).

> > > Why is it important for the job to be able to access a pty?
> >
> > Not important. We do need the I/O to go somewhere. Main problem is
> > input, if the job tries to read from the same terminal Vim will stop
> > working. But output could go to the same place. Might mess up the
> > display. So, normally "pty" means you have to setup the callbacks and
> > everything to make it work, while "open" would have the job happily
> > use some terminal. That would be required if the job has to prompt
> > for anything, or display progress.
> [...]
>
> I don't think a job's input can be allowed access to Vim's tty,
> ever.

Unless it's a filter. Or a job that needs to prompt the user. It's
tricky though, as soon as two processes read from the tty things go
haywire. But as you mention above, using existing commands that weren't
specifically made to be used by Vim this way, it may be the only way to
make it work. I expect that the thing we might need to support is a job
prompting for something only right at the start. E.g. to enter a
password. Imagine connecting to a server to access a database. Better
would be if the plugin can prompt the user before starting the job, but
that might not always work.

> There are two main cases here: programs that don't need any
> input, and are happy with command line options; and programs that need
> to read things from stdin. In the first case there's no problem to
> solve. It's the second case that's interesting, and there Vim could
> either use another callback for the job's stdin to feed it data as
> the job requests, or simply let it read from a file. The callback is
> problematic, because it can run into deadlocks with the callbacks for
> output (this is a well-known hard problem). Reading from a file is
> safe, but limits what the job can do.

Right. We'll probably need to see a practical example. We can have an
input handler, but asynchronously prompting the user for input, a
dialog that pops up, is a completely new concept. For now I think we
can omit it.

--
"You mean there really is an answer?"
"Yes! But you're not going to like it!"
"Oh do please tell us!"
"You're really not going to like it!"
"but we MUST know - tell us"
"Alright, the answer is...."
"yes..."
"... is ..."
"yes... come on!"
"is 42!"
(Douglas Adams - The Hitchhiker's Guide to the Galaxy)

[Thiago Arruda]

Hi Bram

On Wednesday, February 10, 2016 at 1:14:21 PM UTC-3, Bram Moolenaar wrote:
> I wonder if you read my proposal wrong, or perhaps I didn't explain
> something properly. I read your response, and can only say that it's
> all covered in what I proposed.
>
> I think it may be caused by the notion that in NeoVim a job includes the
> communication with the job (over pipes), while in my proposal I separate
> them out.

I understood, but failed to see the advantage of the separate channel
abstraction in the context of job control. Please correct me if I missed
something, but this is how one spawns a job with a channel associated with
stdin/stdout:

```vim
let job = job_filter("cmd", {options})
let handle = job_getchannel(job)
call ch_setcallback(handle, "receive", "Callback")
call ch_setcallback(handle, "error", "Callback")
call ch_setcallback(handle, "close", "Callback")
" send some data to stdin
call ch_send(handle, "data")
```

Now compare with Neovim version:

```vim
let job = jobstart("cmd", {"on_stdout": "Callback", "on_exit": "Callback"})
" send some data to stdin
call jobsend(job, ["line1", "line2", ""])
```

As you can see, Neovim's version is shorter and covers the same use cases as the
one you proposed. It doesn't need a "close" or "error" event, the user can consider it was closed after the "on_exit" event, after which there will be no more "on_stdout" events. On top of that, I don't see how one can read stderr, but will
assume that you plan to add another function to get the stderr channel:

```vim
let job = job_filter("cmd", {options})
let handle = job_getchannel(job)
call ch_setcallback(handle, "receive", "Callback")
call ch_setcallback(handle, "error", "Callback")
call ch_setcallback(handle, "close", "Callback")
" send some data to stdin
call ch_send(handle, "data")
" listen for stderr
let stderr_handle = job_getstderrchannel(job)
call ch_setcallback(stderr_handle, "receive", "Callback")
call ch_setcallback(stderr_handle, "error", "Callback")
call ch_setcallback(stderr_handle, "close", "Callback")
```

That's even more verbose. Here's Neovim version of the same thing:

```vim
let job = jobstart("cmd", {"on_stdout": "Callback",
\ "on_stderr": "Callback", "on_exit": "Callback"})
" send some data to stdin
call jobsend(job, ["line1", "line2", ""])
```

Again, does the same with a much simpler API(note that omitting a callback from
the startup options simply redirects the handle to /dev/null).

I can see how the channel abstraction could be useful in a mainstream programming language with
support for custom types and libraries that operate in such types(eg: Node.js
Streams are everywhere in their echosystem) but in vimscript it only seems to be
an unnecessary complication.

Here are some related concerns:

- If the `job_getchannel` returns a shared handle for stdin/stdout, then what
happens if `ch_close` is called on that handle?(Will it have a separate
function for getting the stdin handle?)
- The API suggests that you get a job's channel after it was created, but
there's no way to redirect the stdio handles after the job is spawned. In this
case, you always have to connect pipes to the job's stdio and the user can
chose or not to attach callbacks, and there's no way the job will share stdout
with Vim(as it was suggested in a previous message).
- I see that channels can have an associated RPC protocol with them, but that
would not be useful for arbitrary processes that have no knowledge of Vim(eg:
compilers), which seem to be the main target of job control.

> So you can have a channel in JSON mode communication to a job
> with a pipe. I havent seen something like that in NeoVim. But then I
> haven't spent much time trying to understand what NeoVim is doing. Can
> NeoVim use msgpack on stdout?

Yes. In fact, Neovim also has the concept of channels, but it is unrelated to
job control only applies in the context of msgpack-rpc:

- `rpcrequest({channel}, {method}[, {args}...])` sends a msgpack-rpc request to
a channel and returns the response(the call only returns when the other end of
the channel sends a response)
- `rpcnotify({channel}, {event}[, {args}...])` sends a msgpack-rpc notification

These two functions can be called on any msgpack-rpc channel connected to
Neovim. At the moment, there are three ways a msgpack-rpc channel can be created
in Neovim:

- A process may connect to Neovim UNIX domain socket(or tcp address if it is
listening on one). The address is provided in the `$NVIM_LISTEN_ADRESS`
environment variable.
- Neovim may spawn a process that has a channel connected to its stdin/stdout.
This uses the job infrastructure internally, but the API exposed to vimscript
is separate(`rpcstart()`/`rpcstop()` are the msgpack-rpc equivalent to
`jobstart()`/`jobstop()`).
- Neovim may be spawned in embedding mode, where instead of sending terminal
escape codes to stdout(And parsing terminal input from stdin), it talks using
msgpack-rpc. This mode is used mainly external UIs and for its testing
infrastructure.

Currently we do not have a way to open a msgpack-rpc connection to an arbitrary
address, but it is only a matter of submitting a relatively small patch I made
a few months ago:
https://github.com/neovim/neovim/issues/3119#issuecomment-130906214

But this is not very relevant to the discussion, I have only talked about
msgpack-rpc to highlight how its API is completely independent from job
control(which can be seen as a more low-level/general-purpose asynchronicity
feature)

> I don't think that old patch adds much, now we already have working
> channels. Only the part with making channels work with
> stdin/stdout/stderr still needs to be done. And that code already
> exists in os_unix.c. MS-Windows is the most difficult (since sockets
> there work differently from file handles) and I thought your patch
> doesn't have that part.

Yes, I have very little Window programming experience, so I have only added an
UNIX version for review. But wasn't this the case for the channel and job
features? From what I could see in the mailing list patches, the initial
implementations of channel/jobs either partially worked on UNIX(with no Windows
support) or had quite a few bugs. The patch I sent was successfully tested
by a few people at the time.

I understand if you don't like my code and/or the patch, but please consider my
arguments and the possibility of following our API. As I said, there are already a
few popular plugins that successfully use it(some even helped us to develop/evolve the API).

> The whole thing I propose was to support what plugin writers were asking
> for. With future feature requests in mind, and combining with the
> existing code for netbeans.
>
> Problem with netbeans is that it uses a weird message format. I want to
> take that in the right direction, don't want more people to use the
> netbeans format. JSON appears to be the best choice. I know NeoVim
> uses msgpack, but I just don't like it.

I hope you understood that job control has nothing to do with msgpack in
Neovim(or in my old patch). It is just a simple way of implementing exactly what
plugin authors have been asking for quite some time: A simple and flexible way
to communicate with Vim asynchronously.

[LCD 47]

On 10 February 2016, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
> Lcd wrote:
>
[...]

> > I'm afraid I still don't get it. With jobs Vim can communicate
> > with processes that aren't specifically aware they are talking
> > to Vim. It can do that by opening pipes for the jobs' stdin and
> > stdout. If it needs to give them "wheels" (that is, communicate
> > with processes on another machine), this can be done with UCSPI.
> > This is a solved problem, and the common solutions such as UCSPI can
> > do things like connect over TLS (and actually get it right), and do
> > fancy TCP window scaling and error recovery.
> >
> > On the other hand, there are daemons that do things like parse a
> > source tree in background and can be queried about it over a socket.
> > But these daemons have their own APIs, and are not specifically
> > prepared to talk to Vim. Making Vim able to talk to them would
> > require some kind of adapter, which might be a job started by Vim.
> > But then the burden of handling protocols, TCP, and so on would be
> > on the adapter. Vim could still talk to the adapter through pipes
> > to the adapter's stdin and stdout. But the adapter would have to
> > be written specifically for the particular daemon, and there's no
> > way around that. If Vim is to talk directly to the daemon, without
> > the adapter, then JSON or protocol buffers are either too much, or
> > too little. I can't see how channels (as they are now) can fit into
> > this picture. _shrug_
>
> It seems you are missing that stdin/stdout IS the channel. On top of
> it you can define the protocol: raw, NL-separated, JSON, etc.

Yes, channels are supposed to provide a library of protocols and
codecs, I (think I) understand that.

> So if that deamon speaks language X, then you connect with a raw
> channel and let the Vim plugin decode language X. Hopefully you can
> do something more clever (raw messages suffer from the problem that
> you never know when the message ends).
>
> It's the good old layering: lower layer about how to connect and
> send/receive (socket, pipe, whatever), layer above that about the
> contents (message separation and parsing). Even higher about what
> request belongs to what response (what is currently only supported for
> JSON, using sequence numbers).

My point is Vim shouldn't try to handle the lowest layer in this
model. Protocol decoding is often the least problem in using an API.
Consider f.i. authentication: I presume nobody in his right mind is
going to write an OAuth 2.0 handler in VimL. So authentication has
to be delegated to a lower level daemon anyway, the thing I called
"adapter" above. Well, this adapter is likely to be a script written
in Python, or Perl, or Ruby, or some such. The interpreters for these
languages are a lot faster than Vim can hope to be, and they already
have plenty of libraries for all protocols, codecs, and everything else
remotely useful for dealing with any API. It's much easier to handle
all these details in an external script, and translate the results to a
minimal interface for Vim.

[...]
> >
> > I don't think a job's input can be allowed access to Vim's tty,
> > ever.
>
> Unless it's a filter. Or a job that needs to prompt the user. It's
> tricky though, as soon as two processes read from the tty things go
> haywire. But as you mention above, using existing commands that
> weren't specifically made to be used by Vim this way, it may be the
> only way to make it work. I expect that the thing we might need to
> support is a job prompting for something only right at the start.
> E.g. to enter a password. Imagine connecting to a server to access a
> database. Better would be if the plugin can prompt the user before
> starting the job, but that might not always work.

Interactive programs shouldn't run in background to begin with.
Things like passwords can usually be passed in environment variables, on
the command line, or in special files. But "true" interactive prompts
really complicate the problem exponentially.

> > There are two main cases here: programs that don't need any input,
> > and are happy with command line options; and programs that need to
> > read things from stdin. In the first case there's no problem to
> > solve. It's the second case that's interesting, and there Vim could
> > either use another callback for the job's stdin to feed it data as
> > the job requests, or simply let it read from a file. The callback
> > is problematic, because it can run into deadlocks with the callbacks
> > for output (this is a well-known hard problem). Reading from a file
> > is safe, but limits what the job can do.
>
> Right. We'll probably need to see a practical example. We can have
> an input handler, but asynchronously prompting the user for input, a
> dialog that pops up, is a completely new concept. For now I think we
> can omit it.

Vim is still a text editor. :) Being able to run in background
programs that don't care about stdin would be a huge step ahead already.

/lcd

[Bram Moolenaar]

Thiago Arruda wrote:

> On Wednesday, February 10, 2016 at 1:14:21 PM UTC-3, Bram Moolenaar wrote:
> > I wonder if you read my proposal wrong, or perhaps I didn't explain
> > something properly. I read your response, and can only say that it's
> > all covered in what I proposed.
> >
> > I think it may be caused by the notion that in NeoVim a job includes the
> > communication with the job (over pipes), while in my proposal I separate
> > them out.
>
> I understood, but failed to see the advantage of the separate channel
> abstraction in the context of job control. Please correct me if I missed
> something, but this is how one spawns a job with a channel associated with
> stdin/stdout:
>
> ```vim
> let job = job_filter("cmd", {options})
> let handle = job_getchannel(job)
> call ch_setcallback(handle, "receive", "Callback")
> call ch_setcallback(handle, "error", "Callback")
> call ch_setcallback(handle, "close", "Callback")
> " send some data to stdin
> call ch_send(handle, "data")
> ```

Not quite. I haven't mentioned defaults yet, but I would think it's
best that when a callback is not defined for stderr then the same one as
for stdout is called. Close is different, so the default would be no
callback.

The options can be set with job_filter() or on the channel, but it's of
course shorter with job_filter():

let job = job_filter("cmd", {'out-cb': 'Callback', 'close-cb': 'Callback'}
let handle = job_getchannel(job)
call ch_sendraw(handle, "data")

Actually, job_filter could return a tuple to make it simpler:

let [job, handle] = job_filter("cmd", {'out-cb': 'Callback', 'close-cb': 'Callback'}
call ch_sendraw(handle, "data")

> Now compare with Neovim version:
>
> ```vim
> let job = jobstart("cmd", {"on_stdout": "Callback", "on_exit": "Callback"})
> " send some data to stdin
> call jobsend(job, ["line1", "line2", ""])
> ```

I do not like jobsend(), it removes the abstraction layer of what is a
job and what is communication with a job. You would need several other
xxxsend() functions depending on how the communication was started.

Compare with a file descriptor: There are lots of ways to open a kind of
file, but once you have a file descriptor you can use read() and write()
on it no matter how it was opened.

> As you can see, Neovim's version is shorter and covers the same use
> cases as the one you proposed. It doesn't need a "close" or "error"
> event, the user can consider it was closed after the "on_exit" event,
> after which there will be no more "on_stdout" events. On top of that,
> I don't see how one can read stderr, but will
> assume that you plan to add another function to get the stderr channel:
>
> ```vim
> let job = job_filter("cmd", {options})
> let handle = job_getchannel(job)
> call ch_setcallback(handle, "receive", "Callback")
> call ch_setcallback(handle, "error", "Callback")
> call ch_setcallback(handle, "close", "Callback")
> " send some data to stdin
> call ch_send(handle, "data")
> " listen for stderr
> let stderr_handle = job_getstderrchannel(job)
> call ch_setcallback(stderr_handle, "receive", "Callback")
> call ch_setcallback(stderr_handle, "error", "Callback")
> call ch_setcallback(stderr_handle, "close", "Callback")
> ```

No, that's not what it would look like. The stderr is not a separate
handle. The handle includes stdin/stdout/stderr. Your example shows
why: there would be several close and error callbacks, even though
that's hardly ever useful. The current implementation has read and
write over the socket, thus putting stdin and stdout in the handle fits
well. stderr needs some special handling.

> That's even more verbose. Here's Neovim version of the same thing:
>
> ```vim
> let job = jobstart("cmd", {"on_stdout": "Callback",
> \ "on_stderr": "Callback", "on_exit": "Callback"})
> " send some data to stdin
> call jobsend(job, ["line1", "line2", ""])
> ```

I'm sorry but seems like you are picking things that are easy with
NeoVim and the comparing to the most verbose version of what you think
Vim would do. That's not a fair comparison.

Also, verbosity is one thing, what is much more important is that it's
easy to understand. In my opinion we should optimize for simplicity.

> Again, does the same with a much simpler API(note that omitting a
> callback from the startup options simply redirects the handle to
> /dev/null).
>
> I can see how the channel abstraction could be useful in a mainstream
> programming language with support for custom types and libraries that
> operate in such types(eg: Node.js Streams are everywhere in their
> echosystem) but in vimscript it only seems to be an unnecessary
> complication.

Every feature in Vim grows beyond its limits. That's what we learned from
history. There are several functions that gained more arguments and are
now harder to use than they should be. It's better to aim at covering
those features, even if not all of them would be implemented right away.
Otherwise we create a problem for adding features later.

> Here are some related concerns:
>
> - If the `job_getchannel` returns a shared handle for stdin/stdout, then what
> happens if `ch_close` is called on that handle?(Will it have a separate
> function for getting the stdin handle?)

Everything closes. Perhaps it's useful to close only one side, we'll
have to specify how to do that. Obviously the implemention is simple,
it's just a matter of defining how it works and having a good default.

> - The API suggests that you get a job's channel after it was created, but
> there's no way to redirect the stdio handles after the job is
> spawned. In this case, you always have to connect pipes to the job's
> stdio and the user can chose or not to attach callbacks, and there's
> no way the job will share stdout
> with Vim(as it was suggested in a previous message).

Please read my original message again, it's explained there. Obviously
connecting stdin/stdout/stderr to the channel happens when creating the
job, not later.

> - I see that channels can have an associated RPC protocol with them,
> but that would not be useful for arbitrary processes that have no
> knowledge of Vim(eg: compilers), which seem to be the main target of
> job control.

What you describe is an asynchronous pipe. Not sure if that is the main
target, but it should be easy to do. Except that it has the usual
problem of not knowing when a message is complete. Probably requires
checking for NL characters. So that would be the default message
protocol.

> > So you can have a channel in JSON mode communication to a job
> > with a pipe. I havent seen something like that in NeoVim. But then I
> > haven't spent much time trying to understand what NeoVim is doing. Can
> > NeoVim use msgpack on stdout?
>
> Yes. In fact, Neovim also has the concept of channels, but it is
> unrelated to job control only applies in the context of msgpack-rpc:
>
> - `rpcrequest({channel}, {method}[, {args}...])` sends a msgpack-rpc
> request to a channel and returns the response(the call only returns
> when the other end of the channel sends a response)
> - `rpcnotify({channel}, {event}[, {args}...])` sends a msgpack-rpc notification

How about communicating with some other protocol than msgpack? Looks
like you would need another set of functions.

> These two functions can be called on any msgpack-rpc channel connected
> to Neovim. At the moment, there are three ways a msgpack-rpc channel
> can be created in Neovim:
>
> - A process may connect to Neovim UNIX domain socket(or tcp address if it is
> listening on one). The address is provided in the `$NVIM_LISTEN_ADRESS`
> environment variable.

I hope you have thought of security, not allow just anybody to connect
to Vim.

> - Neovim may spawn a process that has a channel connected to its stdin/stdout.
> This uses the job infrastructure internally, but the API exposed to
> vimscript is separate(`rpcstart()`/`rpcstop()` are the msgpack-rpc
> equivalent to `jobstart()`/`jobstop()`).

Right, so you have combined process/job starting with rpc/pipe and end
up with four combinations. You haven't mentioned two yet, I suppose
it's also possible to use raw I/O over a socket?

> - Neovim may be spawned in embedding mode, where instead of sending terminal
> escape codes to stdout(And parsing terminal input from stdin), it
> talks using msgpack-rpc. This mode is used mainly external UIs and
> for its testing infrastructure.
>
> Currently we do not have a way to open a msgpack-rpc connection to an
> arbitrary address, but it is only a matter of submitting a relatively
> small patch I made a few months ago:
> https://github.com/neovim/neovim/issues/3119#issuecomment-130906214
>
> But this is not very relevant to the discussion, I have only talked about
> msgpack-rpc to highlight how its API is completely independent from job
> control(which can be seen as a more low-level/general-purpose asynchronicity
> feature)
>
> > I don't think that old patch adds much, now we already have working
> > channels. Only the part with making channels work with
> > stdin/stdout/stderr still needs to be done. And that code already
> > exists in os_unix.c. MS-Windows is the most difficult (since sockets
> > there work differently from file handles) and I thought your patch
> > doesn't have that part.
>
> Yes, I have very little Window programming experience, so I have only
> added an UNIX version for review. But wasn't this the case for the
> channel and job features? From what I could see in the mailing list
> patches, the initial implementations of channel/jobs either partially
> worked on UNIX(with no Windows support) or had quite a few bugs. The
> patch I sent was successfully tested by a few people at the time.

The difference is that the functionality already existed in Vim.
Therefore it was straigtforward to make it work on MS-Windows.

With these features I follow the "release early, release often"
strategy. That way I get feedback early. And fortunately there are
others that can then help adding the missing pieces.

> I understand if you don't like my code and/or the patch, but please
> consider my arguments and the possibility of following our API. As I
> said, there are already a few popular plugins that successfully use
> it(some even helped us to develop/evolve the API).

I do not plan to add msgpack encoding and NeoVim doesn't JSON, thus
there is not much overlap.

> > The whole thing I propose was to support what plugin writers were asking
> > for. With future feature requests in mind, and combining with the
> > existing code for netbeans.
> >
> > Problem with netbeans is that it uses a weird message format. I want to
> > take that in the right direction, don't want more people to use the
> > netbeans format. JSON appears to be the best choice. I know NeoVim
> > uses msgpack, but I just don't like it.
>
> I hope you understood that job control has nothing to do with msgpack
> in Neovim(or in my old patch). It is just a simple way of implementing
> exactly what plugin authors have been asking for quite some time: A
> simple and flexible way to communicate with Vim asynchronously.

Msgpack is not simple. I think that most plugin authors have never
heard of it before looking at NeoVim. JSON is widely supported and easy
to understand.

Does msgpack actually support message numbering, making it possible to
know which response belongs to which request?

There also is some naming issue. What you call job control I would call
an asynchronous pipe. Job control is about running / pausing /
cotinuing / scheduling jobs. There also are several job control
languages. Perhaps it's because of my background, but that's what I
associate it with.

--
Don't be humble ... you're not that great.
-- Golda Meir

[Christian Brabandt]

Hi Bram!

On Mo, 08 Feb 2016, Bram Moolenaar wrote:

> That's plenty already, but we may need a few more.

Can we make it work with vim commands? What comes to mind would be to
have syntax highlighting work asynchronously, so Vim would stay
responsive for languages that have known to be slow syntax highlighting.

Best,
Christian
--
Letzte Worte eines Chemikers:
"Ich habe es im Gefühl, wie lang 15 Sekunden sind."

[Bram Moolenaar]

Christian Brabandt wrote:

> On Mo, 08 Feb 2016, Bram Moolenaar wrote:
>
> > That's plenty already, but we may need a few more.
>
> Can we make it work with vim commands? What comes to mind would be to
> have syntax highlighting work asynchronously, so Vim would stay
> responsive for languages that have known to be slow syntax highlighting.

Hmm, that's something completely different. A simplistic version would
be to first redraw the screen in B&W, remembering what was put on the
screen, and then loop over those lines to add highlighting, bailing out
when a key is typed. Might actually work.

--
hundred-and-one symptoms of being an internet addict:

234. You started college as a chemistry major, and walk out four years
later as an Internet provider.

[Marcin Szamotulski]

Hello Bram!

That's a really great feature. I am trying to implemented a gulp plugin
(gulp is a node package used to build web apps) based on channels and
there are two things that I has a problem with:

1) when I use `job_start()` to start a server in the background and
immediately after I call `ch_open()` to open channel to that server I am
always refused: `E896: read from channel: Connection refused`.

However, if I check `job_status` in between (which is what
I should do anyway), then the following `ch_open` runs fine. There is
no way now to communicate from the job that started that what it runs is
ready to connect to.

2) when my server sends data back to vim only the first message is
received. This can be reproduced by the demoserver.py example. Just
double the lines in `ThreadedTCPRequestHandler.handle`:
print("sending {}".format(encoded))
self.request.sendall(encoded.encode('utf-8'))
print("sending {}".format(encoded))
self.request.sendall(encoded.encode('utf-8'))

And vim will only get the first message. It would be nice if the socket
was reading until explicitly closed by the server. It would be useful
for my case to send updates from the background process spawn by the server
that I wrote.

Best regards,
Marcin Szamotulski

On 22:08 Mon 08 Feb , Bram Moolenaar wrote:
>
> Thanks for the feedback. I think it's time to sketch the upcoming work.
> It appears some thought that sockets was the end of it, that's not so.
>
>
> For jobs the most useful are probably:
> - A deamon, serving several Vim instances, connect by socket.
> - One server working with one Vim instance, asynchronously.
> - A server performing some work for a short time, asynchronously.
> - Running a filter, synchronously (existing functionality)
>
> A channel can communicate over:

> - A socket (what ch_open() currently does)
> - A pipe (only possible with an associated job) connected to
> stdin/stdout/stderr.
>

> A channel can communicate:
> - With JSON messages
> - With JS messages
> - Raw messages
> - NL-separated messages (this is what netbeans uses)
>
> A channel can be associated with a job, but that job does not
> necessarily need be under control of the current Vim (or any Vim)
> instance.
>
> It looks like all the functionality is already present in the Vim code
> somewhere, we just have to make it available in the right way. The
> missing part, a timeout on connect(), was recently added, this may still
> need some work.
>
> Raw messages have the problem that Vim doesn't know where the end of a
> message is, thus it might invoke a callback with an incomplete message.
> The callback will then have the task of parsing and queuing readahead.
> Using line breaks is the most common alternative, we can support that.

>
> Some combinations are likely to be most useful. I heard some people
> mention using a server over pipes to write a message to and call a
> callback when the response is received. I assume this would work best
> with NL separated messages.
>
> Using pipes only works when starting a job. Thus these are closely
> coupled.
>

> The deamon that serves multiple Vim instances would be most useful to
> lookup code locations, for example. Probably best use JSON or JS for
> that, as it makes it easier to add more features over time.
>
> One step further would be to use protocol buffers. I think that it's a
> step too far right now, maybe later. The current protocol buffer
> support unfortunately doesn't support the JS format that I like (the
> JSON format uses field names instead of numbers, an unfortunate choice,
> as it makes version mismatches difficult).

>
>
> Some parts that we still need that require some thougths:
>
> - Communicating over a socket requires knowing the port number. For a
> deamon this would be a known number. For a server started by Vim it's
> best to let the server pick an available port. Then how does it tell
> Vim what port it picked? In test_channel we write the port number in
> a file. That's not ideal, but perhaps there is no better solution.
> Having Vim pick a free port and pass it to the server has race conditions
> (but it might still be an acceptable solution).
>

> - When starting a job, where does it's stdin/stdout/stderr go?
> Possibly the same terminal as Vim, a newly started terminal, a file or
> /dev/null. No, running a terminal inside Vim is not an option. But
> we could write stdout/stderr into a buffer.
>
>

> For starting a deamon job we could have:
> let job = job_start("cmd", {options})
>
> We could make it a bit clever, only start the deamon when it's not
> running yet. This would check whether connecting to the socket works,
> with a zero timeout. Then we have the channel open as well.
> let job = job_maystart("cmd", "address", {options})
> let handle = job_getchannel(job)
>
> Running one server (or more, doesn't really matter) and connecting over
> a socket would work the same way, except that the job is always started
> and it would default to be stopped when Vim exits. We don't really need
> another function for this, just use two:
> let job = job_start("cmd", {options})
> let handle = ch_open("address", {options})
> That's already working.
>
> When using stdin/stdout/stderr we must start the job and open the
> channel at the same time. We have more choices about where to connect
> them. It's probably not useful to have separate functions, we can use
> the {options}:

> let job = job_filter("cmd", {options})
> let handle = job_getchannel(job)
>

> Once we have a channel it would be useful to be able to set callbacks:

> call ch_setcallback(handle, "receive", "Callback")
> call ch_setcallback(handle, "error", "Callback")
> call ch_setcallback(handle, "close", "Callback")

> And some other things, such as the timeout:
> call ch_settimeout(handle, 100)
> Using entries in {options} would also work. In fact, we could make it
> more generic and just use:
> call ch_setoptions(handle, {options})
> Where {options} is the same as what's used when creating a channel
> (minus the ones that only make sense during creation).

>
> For {options} the main thing we're missing is what to do with
> stdin/stdout/stderr:
> "term": "pty" // use current terminal
> "term": "open" // start a terminal
> This would result in stdin/stdout/stderr to connect to the terminal,
> unless specified otherwise. Except that reading from the current
> terminal won't happen (Vim is reading that).
>

> For a channel we have some options that are the same, such as the
> callbacks. But these could be for a socket, thus we need generic
> names. Would work by dropping the "std". So job options would be:
>
> "in-io": "null" // read from nirvana
> "in-io": "pipe" // read from channel
> "in-io": "file" // read from file
> "in-file": "/path/file" // file name
>
> "out-io": "null" // write to nirvana
> "out-io": "pipe" // write to channel
> "out-cb": "Handler" // write handler
> "out-io": "file" // write to file
> "out-file": "/path/file" // file name
> "out-io": "buffer" // write to buffer
> "out-buffer": "name" // buffer name
>
> "err-io": "out" // stderr writes to stdout
> "err-io": "null"
> // etc, like stdout
>
> "kill": 1 // kill job on exit
> "kill": 0 // don't kill job on exit
>
> Channel options:
> "out-cb": "Handler" // write handler
> "err-cb": "Handler" // error handler
> "close-cb": "Handler" // close handler
> "timeout": msec

>
> That's plenty already, but we may need a few more.
>

> One problem that I haven't figured out yet: When starting a job using a
> shell, we get the PID of the shell. So we can kill the shell, but not
> the process that it started.

> E.g. "/bin/sh python demoserver.py <here >there". A shell is also

> useful for expanding wildcards, would be nice if we can make this work.

>
> --
> hundred-and-one symptoms of being an internet addict:

> 182. You may not know what is happening in the world, but you know
> every bit of net-gossip there is.

>
> /// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
> /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
> \\\ an exciting new programming language -- http://www.Zimbu.org ///
> \\\ help me help AIDS victims -- http://ICCF-Holland.org ///
>

[Bram Moolenaar]

Marcin Szamotulski wrote:

> That's a really great feature. I am trying to implemented a gulp plugin
> (gulp is a node package used to build web apps) based on channels and
> there are two things that I has a problem with:

Thanks for the feedback. Keep in mind that the feature is still being
developed. Some things may just be missing. But there can also be bugs
and things that don't work for you, so feedback is useful anyway.

> 1) when I use `job_start()` to start a server in the background and
> immediately after I call `ch_open()` to open channel to that server I am
> always refused: `E896: read from channel: Connection refused`.
>
> However, if I check `job_status` in between (which is what
> I should do anyway), then the following `ch_open` runs fine. There is
> no way now to communicate from the job that started that what it runs is
> ready to connect to.

It's normal for a process to have some initialization time before the
socket is ready for use. This is implemented, it already existed for
netbeans. Did you add a "waittime" argument? The default is zero.

> 2) when my server sends data back to vim only the first message is
> received. This can be reproduced by the demoserver.py example. Just
> double the lines in `ThreadedTCPRequestHandler.handle`:
> print("sending {}".format(encoded))
> self.request.sendall(encoded.encode('utf-8'))
> print("sending {}".format(encoded))
> self.request.sendall(encoded.encode('utf-8'))
>
> And vim will only get the first message. It would be nice if the socket
> was reading until explicitly closed by the server. It would be useful
> for my case to send updates from the background process spawn by the server
> that I wrote.

If you are using raw messages Vim has no clue where a message ends and
whether there is more coming. Vim should get the second message, but
possibly only in the next read. If it never arrives there can be a bug.
I just fixed one where raw messages were dropped when there is no
handler.

I just added the ch_logfile() function, that will be useful to find out
what is happening.

--
"Women marry men hoping they will change. Men marry women hoping
they will not. So each is inevitably disappointed."
- Einstein

[Bram Moolenaar]

Marcin Szamotulski wrote:

> > > That's a really great feature. I am trying to implemented a gulp plugin
> > > (gulp is a node package used to build web apps) based on channels and
> > > there are two things that I has a problem with:
> >
> > Thanks for the feedback. Keep in mind that the feature is still being
> > developed. Some things may just be missing. But there can also be bugs
> > and things that don't work for you, so feedback is useful anyway.
>

> Yes I understand that, the idea is to contribute to the development in
> some fruitful way.

Note that the type of the channel handle has changed. That mainly
matters for when checking whether ch_open() worked.

> > > 1) when I use `job_start()` to start a server in the background and
> > > immediately after I call `ch_open()` to open channel to that server I am
> > > always refused: `E896: read from channel: Connection refused`.
> > >
> > > However, if I check `job_status` in between (which is what
> > > I should do anyway), then the following `ch_open` runs fine. There is
> > > no way now to communicate from the job that started that what it runs is
> > > ready to connect to.
> >
> > It's normal for a process to have some initialization time before the
> > socket is ready for use. This is implemented, it already existed for
> > netbeans. Did you add a "waittime" argument? The default is zero.
>

> I just tried it:
> * adding waittime does not help
> * adding `sleep 1m` before calling `ch_open` solved the problem
>
> Which is not what I'd expect.

I assume we need to treat that error as a temporary failure and retry.
There was a similar problem on Mac, where a 1 msec waittime did work.

> > > 2) when my server sends data back to vim only the first message is
> > > received. This can be reproduced by the demoserver.py example. Just
> > > double the lines in `ThreadedTCPRequestHandler.handle`:
> > > print("sending {}".format(encoded))
> > > self.request.sendall(encoded.encode('utf-8'))
> > > print("sending {}".format(encoded))
> > > self.request.sendall(encoded.encode('utf-8'))
> > >
> > > And vim will only get the first message. It would be nice if the socket
> > > was reading until explicitly closed by the server. It would be useful
> > > for my case to send updates from the background process spawn by the server
> > > that I wrote.
> >
> > If you are using raw messages Vim has no clue where a message ends and
> > whether there is more coming. Vim should get the second message, but
> > possibly only in the next read. If it never arrives there can be a bug.
> > I just fixed one where raw messages were dropped when there is no
> > handler.
>

> I am using the default `json` messages. I found that this works fine,
> from the server:
>
> socket.write(JSON.stringify([0, ["ex", "call MyHandler('" + msg + "')"]);
>
> the problem with this approach is to encode msg properly so that it
> does not contain any `'` which would be misinterpreted by vim when
> evaluating the expression.

>
>
> > I just added the ch_logfile() function, that will be useful to find out
> > what is happening.
>

> Great, I will try and check what is going on.

--
hundred-and-one symptoms of being an internet addict:

246. You use up your free 1 Gbyte in two days.

[Marcin Szamotulski]

`sleep 100m` is more reliable, with just 1m wait I am getting errors
today,

`ch_logfile` is very helpful. I could see that when I attach a handler
with `ch_sendexpr` it is invoked only once on a message that has the
correct ID. I was sending from the server `[requestID, data]` and if
`requestID` is greater than 0 the callback is invoked only once and this
seems the intended behaviour of `may_invoke_callback` function in
channel.c. Reading the source I found that I should rather register the
callback as a channel callback using `ch_open`.

I think that the user expects that the callback will be called on every
message with that given `requestID`. The problem might be when to
un-register the callback. One could for example to check for the
returned value, if is is false or true and remove it when is it is true.
I guess this is all about freeing the memory and leaving this up to
a plugin author might not be a safe option, is that the reason for one
time callbacks?

The current implementation give the same expressive power, since I can
change some state variable in which is then used by the channel level
callback that will be invoked. One just has to be careful to send non
zero requestID to vim only once.

If you want to keep the current behaviour, I think the docs should
explain this in some detail. I can write a patch for that.


When my server crashes, with a `SyntaxError` or a `ReferenceError`
and the socket is not properly closed, I have to kill vim with SIGTERM,
since the cursor gets frozen. This is what `ch_log` reports:

on 0: Opening channel
on 1: Opening channel
: Connecting to localhost port 3746ERR on 1: channel_select_check(): Cannot read
RECV on 1: '"DETACH"
'
on 1: Closing channelERR on 0: channel_select_check(): Cannot read
RECV on 0: '"DETACH"
'
ERR on 0: channel_select_check(): Cannot read

Best regards,
Marcin

[Bram Moolenaar]

Thus increasing the waittime does not help, but a short sleep between
starting the job and ch_open() helps? I think we need to handle a
failing connect properly. There is code from Netbeans, but it's waiting
too long.

Correct.

> I think that the user expects that the callback will be called on every
> message with that given `requestID`. The problem might be when to
> un-register the callback. One could for example to check for the
> returned value, if is is false or true and remove it when is it is true.
> I guess this is all about freeing the memory and leaving this up to
> a plugin author might not be a safe option, is that the reason for one
> time callbacks?

The IDs are used to match the response to the request. There must not
be more than one response to the same request. They are considered
duplicates and dropped.

I don't know what you are doing, but if you have multiple responses than
you should probably use the channel callback, not the request callback.

> The current implementation give the same expressive power, since I can
> change some state variable in which is then used by the channel level
> callback that will be invoked. One just has to be careful to send non
> zero requestID to vim only once.

Correct.

> If you want to keep the current behaviour, I think the docs should
> explain this in some detail. I can write a patch for that.

I already made changes to channel.txt. I will add a remark now. Please
re-read the help later when the changes have settled down.

> When my server crashes, with a `SyntaxError` or a `ReferenceError`
> and the socket is not properly closed, I have to kill vim with SIGTERM,
> since the cursor gets frozen. This is what `ch_log` reports:
>
> on 0: Opening channel
> on 1: Opening channel
> : Connecting to localhost port 3746ERR on 1: channel_select_check(): Cannot read
> RECV on 1: '"DETACH"
> '
> on 1: Closing channelERR on 0: channel_select_check(): Cannot read
> RECV on 0: '"DETACH"
> '
> ERR on 0: channel_select_check(): Cannot read

Error handling is tricky. Perhaps Vim hangs in channel_close()?
Would be good if you can use a debugger to connect to the hanging Vim
and see where it's stuck.

--
hundred-and-one symptoms of being an internet addict:

252. You vote for foreign officials.

[赵启明赵]

I've tried these simple code for job control

let job = job_start('echo abc', {'waittime': 1})
let channel = job_getchannel(job)
let output = ch_readraw(channel)
call setline(1, output)

And I got NULL character inserted at the end of first line, I thought is should be trimmed by vim.

[Bram Moolenaar]

The NL format hasn't really been implemented yet. Just about to get on it
(the Netbeans code can be used).

Since the trailing NL isn't stripped from a RAW message and in Vim a NL
stands for NUL byte, that's what you get.

I suppose we can strip the NL when the channel is in NL format.

--
Edison's greatest achievement came in 1879, when he invented the
electric company. Edison's design was a brilliant adaptation of the
simple electrical circuit: the electric company sends electricity
through a wire to a customer, then immediately gets the electricity
back through another wire

[mattn]

My optinion is:

1. ch_readraw/ch_sendraw should block if callback is not specified
2. ch_readraw/ch_sendraw should return empty string if callback is specified

[mattn]

This is my suggestion.

https://gist.github.com/mattn/d6198b38d9b18edcb2aa

ch_readraw, ch_sendraw, ch_sendjson take arguments dict can be taken callback.
If callback is specified, those functions return empty string. And callback in later. If callback is not specified, those function will block while reading data.

Now below's script is working on windows.

----------------------------------
function! Callback(id, msg)
echo a:msg
endfunction
let g:job = job_start("tail -f logfile")
let handle = job_getchannel(job)
call ch_readraw(handle, {"callback": "Callback"})
----------------------------------

- mattn

[Bram Moolenaar]

Yasuhiro Matsumoto wrote:

> ------=_Part_725_18982456.1455794219476
> Content-Type: text/plain; charset=UTF-8

I don't quite get it. Looking at the code, this ch_readraw() doesn't
specify a callback for the one read, it specifies a new callback for the
channel. That's not what the user expects. Better do something like:

call ch_setoptions(channel, {"callback": "Callback"})

I find it unexpected that ch_readraw() doesn't actually read anything.

For ch_sendraw() and ch_sendexpr(), if a callback is specified the
functions don't wait for a response and let that specific callback
handle the response. When using JS or JSON the sequence number can be
used to know which response belongs to which request. With RAW and NL
we can only guess that the responses come in the same order as the
requests.

Queing up callbacks for RAW and NL responses isn't implemented yet,
perhaps that confused you.

This change seems to also support some other waiting for Win32, it's
hard to see from the diff. Perhaps you can adjust the patch for that,
or move it to a new patch?

--
TALL KNIGHT: We shall say Ni! again to you if you do not appease us.
ARTHUR: All right! What do you want?
TALL KNIGHT: We want ... a shrubbery!
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

[mattn]

On Friday, February 19, 2016 at 6:58:48 AM UTC+9, Bram Moolenaar wrote:
> I don't quite get it. Looking at the code, this ch_readraw() doesn't
> specify a callback for the one read, it specifies a new callback for the
> channel. That's not what the user expects. Better do something like:
>
> call ch_setoptions(channel, {"callback": "Callback"})
>
> I find it unexpected that ch_readraw() doesn't actually read anything.
>
> For ch_sendraw() and ch_sendexpr(), if a callback is specified the
> functions don't wait for a response and let that specific callback
> handle the response. When using JS or JSON the sequence number can be
> used to know which response belongs to which request. With RAW and NL
> we can only guess that the responses come in the same order as the
> requests.
>
> Queing up callbacks for RAW and NL responses isn't implemented yet,
> perhaps that confused you.
>
> This change seems to also support some other waiting for Win32, it's
> hard to see from the diff. Perhaps you can adjust the patch for that,
> or move it to a new patch?

I updated patch.

https://gist.github.com/mattn/d6198b38d9b18edcb2aa

[Bram Moolenaar]

Thanks. However, when I include the part of this change around
channel_wait() then the tests fail. It appears this is because of the
comment:
* Always returns OK for FEAT_GUI_W32.

is no longer true. I tried using select() also for the GUI but that
doesn't work, of course.

So I added an #ifdef to return OK for FEAT_GUI_W32 after checking the
pipes. I hope that works for now. At least the tests pass on my
system.

--
You got to work at a mill? Lucky! I got sent back to work in the
acid-mines for my daily crust of stale bread... which not even the
birds would eat.

[msz...@gmail.com]

Sorry I wasn't clear enough. The longer wait time does help.

>
> > When my server crashes, with a `SyntaxError` or a `ReferenceError`
> > and the socket is not properly closed, I have to kill vim with SIGTERM,
> > since the cursor gets frozen. This is what `ch_log` reports:
> >
> > on 0: Opening channel
> > on 1: Opening channel
> > : Connecting to localhost port 3746ERR on 1: channel_select_check(): Cannot read
> > RECV on 1: '"DETACH"
> > '
> > on 1: Closing channelERR on 0: channel_select_check(): Cannot read
> > RECV on 0: '"DETACH"
> > '
> > ERR on 0: channel_select_check(): Cannot read
>
> Error handling is tricky. Perhaps Vim hangs in channel_close()?
> Would be good if you can use a debugger to connect to the hanging Vim
> and see where it's stuck.

With the most recent version of vim this is gone, sorry I didn't had
enough time this week to look at it.

Best regards,
Marcin

[mattn]

On Saturday, February 20, 2016 at 5:02:17 AM UTC+9, Bram Moolenaar wrote:
> Thanks. However, when I include the part of this change around
> channel_wait() then the tests fail. It appears this is because of the
> comment:
> * Always returns OK for FEAT_GUI_W32.
>
> is no longer true. I tried using select() also for the GUI but that
> doesn't work, of course.
>
> So I added an #ifdef to return OK for FEAT_GUI_W32 after checking the
> pipes. I hope that works for now. At least the tests pass on my
> system.

Windows can't select file handle. So can't return OK always.

[Bram Moolenaar]

The tests are passing, but always returning OK doesn't look so good.
Also, the "loop" variable in channel_handle_events() looks a bit
strange.

I hope you find some time to further improve this. I hope to make
stderr work this weekend, perhaps some Win32 changes will be needed for
that as well.

--
PRINCE: He's come to rescue me, father.
LAUNCELOT: (embarrassed) Well, let's not jump to conclusions ...

"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

[Bram Moolenaar]

Please try the latest version. The waittime didn't actualy work on
Linux, the select() doesn't behave as expected. I made that work,
hopefully it also works for you now.

I also added timestamps to the log, so you can see how long something
takes.

--
FATHER: Who are you?
PRINCE: I'm ... your son ...
FATHER: Not you.
LAUNCELOT: I'm ... er ... Sir Launcelot, sir.

[Charles Campbell]

Bram Moolenaar wrote:

> Lcd wrote:
>
> There are also UNIX domain sockets (a.k.a. named pipes), which you get
> essentially for free once you have the code for INET sockets. They are

> as efficient as (unnamed) pipes. The code is already there, but it's
> mutually exclusive with INET sockets. I don't think there is any
> reason why the two can't coexist as separate mechanisms.
> We can support that later. As far as I know the main difference is how
> you connect, after that it's basically the same. Since we are using the
> netbeans code it uses sockets.
>
At least on Unix and variants, there also are message queues.

Regards,
Chip Campbell

[Charles Campbell]

The relevant functions: msgget(), msgctl(), and msgop(). These produce
a messaging channel that is two-way (but not via internet, of course --
for that you'll need INET sockets).

Regards,
Chip Campbell

[Christian Brabandt]

On Fr, 12 Feb 2016, Bram Moolenaar wrote:

> Christian Brabandt wrote:
>
> > Can we make it work with vim commands? What comes to mind would be to
> > have syntax highlighting work asynchronously, so Vim would stay
> > responsive for languages that have known to be slow syntax highlighting.
>
> Hmm, that's something completely different. A simplistic version would
> be to first redraw the screen in B&W, remembering what was put on the
> screen, and then loop over those lines to add highlighting, bailing out
> when a key is typed. Might actually work.

This hasn't been done yet, correct?

Also it would be neat, if we could e.g. in the background execute an
':Nread' command to download a file without blocking or uncompress an
archive so something similar.

Best,
Christian

[Bram Moolenaar]

Christian Brabandt wrote:

> > > Can we make it work with vim commands? What comes to mind would be to
> > > have syntax highlighting work asynchronously, so Vim would stay
> > > responsive for languages that have known to be slow syntax highlighting.
> >
> > Hmm, that's something completely different. A simplistic version would
> > be to first redraw the screen in B&W, remembering what was put on the
> > screen, and then loop over those lines to add highlighting, bailing out
> > when a key is typed. Might actually work.
>
> This hasn't been done yet, correct?

Nope. It could be done like we call timers while waiting for a
character to be typed. But storing the state of what screen lines still
need to be colored, and making sure the coloring is interrupted by a
typed character, is still some work. And when it takes long it also
requires timers to be invoked...

> Also it would be neat, if we could e.g. in the background execute an
> ':Nread' command to download a file without blocking or uncompress an
> archive so something similar.

That can be done with a separate process. job_start() has an option to
put the output in a buffer. We already use external commands for
downloading and decompression, thus it should be a metter of writing the
script.

--
ARTHUR: Well, I AM king...
DENNIS: Oh king, eh, very nice. An' how'd you get that, eh? By exploitin'
the workers -- by 'angin' on to outdated imperialist dogma which
perpetuates the economic an' social differences in our society! If
there's ever going to be any progress--
The Quest for the Holy Grail (Monty Python)