How to run a job asynchronously?

[bo...@airbladesoftware.com]

Hello!

I have been trying to run a job asynchronously and capture the output, but despite reading the docs many times I can't get it to work.

Ideally I want all the diff output in one go, rather than line by line, so I first tried using only an on_exit callback:

let cmd = 'diff A B'
let job = job_start(cmd, {'exit_cb': 'MyExitHandler'})

And then:

function! MyExitHandler(job, exit_status)
let data = ch_read(a:job)
echom data
endfunction

I would expect the handler to be called just about immediately but nothing happens. After 30s of waiting I get bored and move the cursor (if I'm in normal mode) / press <Escape> (if I'm in insert mode), at which point "E906: not an open channel" is echoed.

From this I deduce that the handler isn't called when the job exits, but instead some time later as and when vim checks up on the job.

So I tried using an on_stdout callback as well:

let job = job_start(cmd, {'on_stdout': 'MyOutHandler', 'exit_cb': 'MyExitHandler'})

And also:

function! MyOutHandler(channel, message)
echom a:message
endfunction

This handler gets called just about immediately, as expected, but a:message is always 'DETACH', not the diff output. And then a few seconds later the exit handler echoes its E906 error.

Please could somebody tell me what I'm doing wrong?

I'm using Vim v7.4.1770.

Many thanks in advance,

Andrew Stewart

[Bram Moolenaar]

Andrew Stewart wrote:

> I have been trying to run a job asynchronously and capture the output,
> but despite reading the docs many times I can't get it to work.
>
> Ideally I want all the diff output in one go, rather than line by
> line, so I first tried using only an on_exit callback:
>
> let cmd = 'diff A B'
> let job = job_start(cmd, {'exit_cb': 'MyExitHandler'})
>
> And then:
>
> function! MyExitHandler(job, exit_status)
> let data = ch_read(a:job)
> echom data
> endfunction
>
> I would expect the handler to be called just about immediately but
> nothing happens. After 30s of waiting I get bored and move the cursor
> (if I'm in normal mode) / press <Escape> (if I'm in insert mode), at
> which point "E906: not an open channel" is echoed.

The exit callback is only invoked once Vim detects the job is gone. By
then it's too late to read the output. This is documented.


There is also the close_cb for the channel, which gets called much
sooner than the exit_cb. Actually, it's called before the out_cb
function, should probably change that.

Perhaps, when there is a close_cb, the DETACH message should not be
added. That simplifies how it works.

> From this I deduce that the handler isn't called when the job exits,
> >but instead some time later as and when vim checks up on the job.
>
> So I tried using an on_stdout callback as well:
>
> let job = job_start(cmd, {'on_stdout': 'MyOutHandler', 'exit_cb':
> 'MyExitHandler'})
>
> And also:
>
> function! MyOutHandler(channel, message)
> echom a:message
> endfunction
>
> This handler gets called just about immediately, as expected, but
> a:message is always 'DETACH', not the diff output. And then a few
> seconds later the exit handler echoes its E906 error.

This works. But the script should use "out_cb":

func! OutHandler(channel, msg)
echomsg a:msg
endfunc
let cmd = 'diff -u /tmp/a /tmp/b'
let job = job_start(cmd, {'out_cb': 'OutHandler'})

The echo statements overwrite each other, "DETACH" is the last one. Use
":messages" to see what happened before this. Obviously you need to do
something else than echoing.

> Please could somebody tell me what I'm doing wrong?
>
> I'm using Vim v7.4.1770.

--
Two cows are standing together in a field. One asks the other:
"So what do you think about this Mad Cow Disease?"
The other replies: "That doesn't concern me. I'm a helicopter."

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[James McCoy]

On Mon, Apr 25, 2016 at 11:06:43PM +0200, Bram Moolenaar wrote:

>
> Andrew Stewart wrote:
>
> > This handler gets called just about immediately, as expected, but
> > a:message is always 'DETACH', not the diff output. And then a few
> > seconds later the exit handler echoes its E906 error.
>
> This works. But the script should use "out_cb":
>
> func! OutHandler(channel, msg)
> echomsg a:msg
> endfunc
> let cmd = 'diff -u /tmp/a /tmp/b'
> let job = job_start(cmd, {'out_cb': 'OutHandler'})
>
> The echo statements overwrite each other, "DETACH" is the last one. Use
> ":messages" to see what happened before this.

Why is Vim injecting "DETACH" into the output? The netbeans code uses
is because that's part of the established protocol, but why is that
being imposed on every other user of channels?

Also, the help implies that this is only done for channels representing
a socket, not ones that are used for a simple job_start() with a process:

On read error or ch_close(), when using a socket with RAW or NL mode, the
string "DETACH\n" is sent, if still possible. The channel will then be
inactive.

Cheers,
--
James
GPG Key: 4096R/331BA3DB 2011-12-05 James McCoy <jame...@jamessan.com>

[Andrew Stewart]

> On 25 Apr 2016, at 10:06 pm, Bram Moolenaar <Br...@Moolenaar.net> wrote:
> The exit callback is only invoked once Vim detects the job is gone. By
> then it's too late to read the output. This is documented.
>
> There is also the close_cb for the channel, which gets called much
> sooner than the exit_cb. Actually, it's called before the out_cb
> function, should probably change that.

Good to know, thanks.

> Perhaps, when there is a close_cb, the DETACH message should not be
> added. That simplifies how it works.

If I read a job's stdout with an "out_cb" handler, and the output has multiple lines, what's the right way to know that the job has finished and no more output is coming?

In NL mode I suppose I could keep accumulating the output via the handler, one line at a time, stopping when the message is "DETACH".

In RAW mode I get all the output at once – is that the recommended way for my situation?

Initially I tried to start a job without any handlers because I read this in the documentation:

Without the handler you need to read the output with ch_read() or ch_readraw().

Where would such a call go if it's not in a callback?

When I start two jobs with similar commands I only seem to get the output for the second. Here's a demo:

$ cat demo.vim
call ch_logfile('channellog', 'w')

function! OutHandler(channel, msg)
echom 'OutHandler'
echom ' '.a:channel
echom ' '.a:msg
endfunction

let job = job_start('ls', {'mode': 'raw', 'out_cb': 'OutHandler'})
let job = job_start('ls -l', {'mode': 'raw', 'out_cb': 'OutHandler'})

$ vim -u NONE -N -S demo.vim

$ cat channellog
==== start log session ====
0.000033 : Starting job: ls
0.000071 on 0: Created channel
0.000543 : Starting job: ls -l
0.000599 on 1: Created channel
0.001033 : looking for messages on channels
0.003977 RECV on 0: 'andanother
channellog
demo.vim
otherfile
somefile
'
0.003998 on 0: Closing channel
0.004103 on 0: Clearing channel
0.004113 on 0: Freeing channel
0.004120 : looking for messages on channels
0.005818 RECV on 1: 'total 16
-rw-r--r-- 1 andy staff 0 26 Apr 11:16 andanother
-rw-r--r-- 1 andy staff 204 26 Apr 11:17 channellog
-rw-r--r-- 1 andy staff 289 26 Apr 11:17 demo.vim
-rw-r--r-- 1 andy staff 0 26 Apr 11:16 otherfile
-rw-r--r-- 1 andy staff 0 26 Apr 11:16 somefile
'
0.005835 on 1: Invoking channel callback OutHandler
0.006085 ERR on 1: channel_select_check(): Cannot read from channel
0.006097 PUT on 1: 'DETACH
'
0.006100 on 1: Closing channel
0.006170 : looking for messages on channels
0.006177 on 1: Invoking channel callback OutHandler
1.338691 : looking for messages on channels

You can see the OutHandler callback is not invoked on channel 0.

However if the second job's command is, say, 'cal' instead of 'ls -l', the callback is invoked for both jobs as expected.

I also noticed that the job's command in sensitive to whitespace. A job command of ' ls' instead of 'ls' produces no output:

$ cat channellog
==== start log session ====
0.000032 : Starting job: ls
0.000070 on 0: Created channel
0.000600 : looking for messages on channels
0.001676 ERR on 0: channel_select_check(): Cannot read from channel
0.001685 PUT on 0: 'DETACH
'
0.001686 on 0: Closing channel
0.001715 : looking for messages on channels
0.001721 on 0: Invoking channel callback OutHandler
0.001829 : looking for messages on channels

I would expect leading / trailing whitespace not to affect anything.

[Bram Moolenaar]

James McCoy wrote:

> On Mon, Apr 25, 2016 at 11:06:43PM +0200, Bram Moolenaar wrote:
> >
> > Andrew Stewart wrote:
> >
> > > This handler gets called just about immediately, as expected, but
> > > a:message is always 'DETACH', not the diff output. And then a few
> > > seconds later the exit handler echoes its E906 error.
> >
> > This works. But the script should use "out_cb":
> >
> > func! OutHandler(channel, msg)
> > echomsg a:msg
> > endfunc
> > let cmd = 'diff -u /tmp/a /tmp/b'
> > let job = job_start(cmd, {'out_cb': 'OutHandler'})
> >
> > The echo statements overwrite each other, "DETACH" is the last one. Use
> > ":messages" to see what happened before this.
>
> Why is Vim injecting "DETACH" into the output? The netbeans code uses
> is because that's part of the established protocol, but why is that
> being imposed on every other user of channels?

Yeah, I was also wondering if we need it. At least when there is a
close callback it doesn't make sense.

I would think that normally the job will send some terminator.
Otherwise, one can add a close callback if one needs to know the channel
is done reading. Would there be any reason to add DETACH?

Previously the close callback didn't work properly. I fixed that now.

> Also, the help implies that this is only done for channels representing
> a socket, not ones that are used for a simple job_start() with a process:
>
> On read error or ch_close(), when using a socket with RAW or NL mode, the
> string "DETACH\n" is sent, if still possible. The channel will then be
> inactive.

The documentation is wrong.

--
The war between Emacs and Vi is over. Vi has won with 3 to 1.
http://m.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/030/3044/3044s1.html

[LCD 47]

On 26 April 2016, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
> James McCoy wrote:
>
> > On Mon, Apr 25, 2016 at 11:06:43PM +0200, Bram Moolenaar wrote:
> > >
> > > Andrew Stewart wrote:
> > >
> > > > This handler gets called just about immediately, as expected, but
> > > > a:message is always 'DETACH', not the diff output. And then a few
> > > > seconds later the exit handler echoes its E906 error.
> > >
> > > This works. But the script should use "out_cb":
> > >
> > > func! OutHandler(channel, msg)
> > > echomsg a:msg
> > > endfunc
> > > let cmd = 'diff -u /tmp/a /tmp/b'
> > > let job = job_start(cmd, {'out_cb': 'OutHandler'})
> > >
> > > The echo statements overwrite each other, "DETACH" is the last one. Use
> > > ":messages" to see what happened before this.
> >
> > Why is Vim injecting "DETACH" into the output? The netbeans code uses
> > is because that's part of the established protocol, but why is that
> > being imposed on every other user of channels?
>
> Yeah, I was also wondering if we need it.

Some way to test for EOF would be more useful, IMO. Not quite the
same thing as a close callback.

/lcd

[Bram Moolenaar]

Andrew Stewart wrote:

> > On 25 Apr 2016, at 10:06 pm, Bram Moolenaar <Br...@Moolenaar.net> wrote:
> > The exit callback is only invoked once Vim detects the job is gone. By
> > then it's too late to read the output. This is documented.
> >
> > There is also the close_cb for the channel, which gets called much
> > sooner than the exit_cb. Actually, it's called before the out_cb
> > function, should probably change that.
>
> Good to know, thanks.
>
> > Perhaps, when there is a close_cb, the DETACH message should not be
> > added. That simplifies how it works.
>
> If I read a job's stdout with an "out_cb" handler, and the output has
> multiple lines, what's the right way to know that the job has finished
> and no more output is coming?
>
> In NL mode I suppose I could keep accumulating the output via the
> handler, one line at a time, stopping when the message is "DETACH".
>
> In RAW mode I get all the output at once – is that the recommended way
> for my situation?
>
> Initially I tried to start a job without any handlers because I read
> this in the documentation:
>
> Without the handler you need to read the output with ch_read() or
> ch_readraw().
>
> Where would such a call go if it's not in a callback?

So far the scripts I have seen called ch_read() to wait for something to
read. That doesn't work asynchronously though.

It should be possible to use ch_read() in the close callback. Currently
that doesn't work, since the channel is considered already closed. And
the readahead is discarded, because there is no callback to pass it to.

I'll fix this. I'll also add another state to ch_status() to indicate
that the channel has readahead but is closed (can't send). Then this
works:

func! CloseHandler(channel)
while ch_status(a:channel) == 'buffered'
echomsg ch_read(a:channel)
endwhile
endfunc

call ch_logfile('channellog', 'w')

let cmd = 'diff -u /tmp/a /tmp/b'

let job = job_start(cmd, {'close_cb': 'CloseHandler'})

With the current Vim it works for me, but only when using "job1" and
"job2". Apparently overwriting the variable makes Vim think there is no
longer a reference to the job and discards the channel. I'll fix that.

> I also noticed that the job's command in sensitive to whitespace. A
> job command of ' ls' instead of 'ls' produces no output:
>
> $ cat channellog
> ==== start log session ====
> 0.000032 : Starting job: ls
> 0.000070 on 0: Created channel
> 0.000600 : looking for messages on channels
> 0.001676 ERR on 0: channel_select_check(): Cannot read from channel
> 0.001685 PUT on 0: 'DETACH
> '
> 0.001686 on 0: Closing channel
> 0.001715 : looking for messages on channels
> 0.001721 on 0: Invoking channel callback OutHandler
> 0.001829 : looking for messages on channels
>
> I would expect leading / trailing whitespace not to affect anything.

I can make it skip over leading white space.


--
TALL KNIGHT: When you have found the shrubbery, then you must cut down the
mightiest tree in the forest ... with a herring.
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

[Bram Moolenaar]

Lcd wrote:

> On 26 April 2016, Bram Moolenaar <Br...@moolenaar.net> wrote:
> >
> > James McCoy wrote:
> >
> > > On Mon, Apr 25, 2016 at 11:06:43PM +0200, Bram Moolenaar wrote:
> > > >
> > > > Andrew Stewart wrote:
> > > >
> > > > > This handler gets called just about immediately, as expected, but
> > > > > a:message is always 'DETACH', not the diff output. And then a few
> > > > > seconds later the exit handler echoes its E906 error.
> > > >
> > > > This works. But the script should use "out_cb":
> > > >
> > > > func! OutHandler(channel, msg)
> > > > echomsg a:msg
> > > > endfunc
> > > > let cmd = 'diff -u /tmp/a /tmp/b'
> > > > let job = job_start(cmd, {'out_cb': 'OutHandler'})
> > > >
> > > > The echo statements overwrite each other, "DETACH" is the last one. Use
> > > > ":messages" to see what happened before this.
> > >
> > > Why is Vim injecting "DETACH" into the output? The netbeans code uses
> > > is because that's part of the established protocol, but why is that
> > > being imposed on every other user of channels?
> >
> > Yeah, I was also wondering if we need it.
>
> Some way to test for EOF would be more useful, IMO. Not quite the
> same thing as a close callback.

You can now use ch_status() for that. If it returns "open" or
"buffered" there may be more to read. When "closed" then it's done.

I have added this example in the help:

func! CloseHandler(channel)
while ch_status(a:channel) == 'buffered'
echomsg ch_read(a:channel)
endwhile
endfunc

let job = job_start(command, {'close_cb': 'CloseHandler'})


--
TIM: Too late.
ARTHUR: What?
TIM: There he is!
[They all turn, and see a large white RABBIT lollop a few yards out of the
cave. Accompanied by terrifying chord and jarring metallic monster noise.]

[James McCoy]

On Tue, Apr 26, 2016 at 09:38:02PM +0200, Bram Moolenaar wrote:
>
> Lcd wrote:
>
> > On 26 April 2016, Bram Moolenaar <Br...@moolenaar.net> wrote:
> > >
> > > James McCoy wrote:
> > > > Why is Vim injecting "DETACH" into the output? The netbeans code uses
> > > > is because that's part of the established protocol, but why is that
> > > > being imposed on every other user of channels?
> > >
> > > Yeah, I was also wondering if we need it.
> >
> > Some way to test for EOF would be more useful, IMO. Not quite the
> > same thing as a close callback.

Why should that be needed? If you get to the point of exit_cb/close_cb
being called, all of the output should have been drained already, IMO.

> You can now use ch_status() for that. If it returns "open" or
> "buffered" there may be more to read. When "closed" then it's done.
>
> I have added this example in the help:
>
> func! CloseHandler(channel)
> while ch_status(a:channel) == 'buffered'
> echomsg ch_read(a:channel)
> endwhile
> endfunc
> let job = job_start(command, {'close_cb': 'CloseHandler'})

Seems like it'd be a lot easier to just define an out_cb instead.

[Andrew Stewart]

> On 26 Apr 2016, at 8:38 pm, Bram Moolenaar <Br...@Moolenaar.net> wrote:
>
> You can now use ch_status() for that. If it returns "open" or
> "buffered" there may be more to read. When "closed" then it's done.
>
> I have added this example in the help:
>
> func! CloseHandler(channel)
> while ch_status(a:channel) == 'buffered'
> echomsg ch_read(a:channel)
> endwhile
> endfunc
> let job = job_start(command, {'close_cb': 'CloseHandler'})

Should the condition be `while ch_status(a:channel) == 'buffered' || ch_status(a:channel) == 'open'` as per the description?

I upgraded to 7.4.1795 and tried out the example as is.

I couldn't find the example in the help.

Every few callbacks the screen needed a `:redraw!` to:

- display the correct signs (my handler updates them)
- display the cursor in the correct position (getpos('.') showed the right coordinates but the cursor was displayed elsewhere).

I haven't been able to reproduce this consistently. I tried adding a `redraw`/`redraw!` as the last line of my handler but it didn't make any difference.

Every few dozen callbacks `ch_read(a:channel)` threw an E906 error, though again I'm not able to reproduce it consistently. Perhaps there is a race condition where the channel is closed between checking its status and reading from it? I modified my handler to:

function! CloseHandler(channel)
let output = ''
try

while ch_status(a:channel) == 'buffered'

let line = ch_read(a:channel)
if line !=# 'DETACH'
let output .= line."\n"
endif
endwhile
echom output
endfunction

...which seemed to work well, at least until Vim crashed:

Vim: Caught deadly signal SEGV
Vim(85664,0x7fff7a952000) malloc: *** error for object 0x7ff2106739e8: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Vim: Finished.
Vim: Double signal, exiting
Abort trap: 6

I don't know how to set a breakpoint in malloc_error_break so I wasn't able to debug this, sorry. When it happened there was nothing unusual in the channel log.

I also implemented my job_start using an out_cb handler again instead of a close_cb handler. The occasional-missing-callback problem has disappeared and it all worked perfectly. Thanks!

[James McCoy]

On Wed, Apr 27, 2016 at 11:44:39AM +0100, Andrew Stewart wrote:
> ...which seemed to work well, at least until Vim crashed:
>
> Vim: Caught deadly signal SEGV
> Vim(85664,0x7fff7a952000) malloc: *** error for object 0x7ff2106739e8: incorrect checksum for freed object - object was probably modified after being freed.
> *** set a breakpoint in malloc_error_break to debug
> Vim: Finished.
> Vim: Double signal, exiting
> Abort trap: 6

That sounds like it might be related to the segfault I reported[0]
recently.

[0]: http://article.gmane.org/gmane.editors.vim.devel/58887

[bo...@airbladesoftware.com]

On Wednesday, 27 April 2016 11:44:48 UTC+1, Andrew Stewart wrote:
> I also implemented my job_start using an out_cb handler again instead of a close_cb handler. The occasional-missing-callback problem has disappeared and it all worked perfectly. Thanks!

I know this isn't the place for MacVim-specific discussion, but as it's related I thought it worth mentioning that MacVim never starts looking for messages on channels:

https://github.com/macvim-dev/macvim/issues/272

[Bram Moolenaar]

Andrew Stewart wrote:

> > On 26 Apr 2016, at 8:38 pm, Bram Moolenaar <Br...@Moolenaar.net> wrote:
> >
> > You can now use ch_status() for that. If it returns "open" or
> > "buffered" there may be more to read. When "closed" then it's done.
> >
> > I have added this example in the help:
> >
> > func! CloseHandler(channel)
> > while ch_status(a:channel) == 'buffered'
> > echomsg ch_read(a:channel)
> > endwhile
> > endfunc
> > let job = job_start(command, {'close_cb': 'CloseHandler'})
>
> Should the condition be `while ch_status(a:channel) == 'buffered' || ch_status(a:channel) == 'open'` as per the description?
>
> I upgraded to 7.4.1795 and tried out the example as is.
>
> I couldn't find the example in the help.
>
> Every few callbacks the screen needed a `:redraw!` to:
>
> - display the correct signs (my handler updates them)
> - display the cursor in the correct position (getpos('.') showed the right coordinates but the cursor was displayed elsewhere).

It's expected that the handler has to tell Vim what needs to be redrawn.
There is some redrawing after the callback returns, but things like
"echomsg" can mess up the screen, especially if it causes scrolling.

> I haven't been able to reproduce this consistently. I tried adding a
> `redraw`/`redraw!` as the last line of my handler but it didn't make
> any difference.
>
> Every few dozen callbacks `ch_read(a:channel)` threw an E906 error,
> though again I'm not able to reproduce it consistently. Perhaps there
> is a race condition where the channel is closed between checking its
> status and reading from it? I modified my handler to:
>
> function! CloseHandler(channel)
> let output = ''
> try
> while ch_status(a:channel) == 'buffered'
> let line = ch_read(a:channel)
> if line !=# 'DETACH'

Note that adding "DETACH" was removed, so you can drop this check.

> let output .= line."\n"
> endif
> endwhile
> echom output
> endfunction
>
> ...which seemed to work well, at least until Vim crashed:
>
> Vim: Caught deadly signal SEGV
> Vim(85664,0x7fff7a952000) malloc: *** error for object 0x7ff2106739e8: incorrect checksum for freed object - object was probably modified after being freed.
> *** set a breakpoint in malloc_error_break to debug
> Vim: Finished.
> Vim: Double signal, exiting
> Abort trap: 6
>
> I don't know how to set a breakpoint in malloc_error_break so I wasn't
> able to debug this, sorry. When it happened there was nothing unusual
> in the channel log.

A couple of problems have been fixed. Please give it another try and
report back if you still see a problem.

> I also implemented my job_start using an out_cb handler again instead
> of a close_cb handler. The occasional-missing-callback problem has
> disappeared and it all worked perfectly. Thanks!

That is the preferred way. Reading the output in the close callback
might only work if there isn't much to read.

--
Close your shells, or I'll kill -9 you
Tomorrow I'll quota you
Remember the disks'll always be full
And then while I'm away
I'll write ~ everyday
And I'll send-pr all my buggings to you.
[ CVS log "Beatles style" for FreeBSD ports/INDEX, Satoshi Asami ]

[bo...@airbladesoftware.com]

On Sunday, 1 May 2016 21:37:13 UTC+1, Bram Moolenaar wrote:

> Note that adding "DETACH" was removed, so you can drop this check.

[snip]

> A couple of problems have been fixed. Please give it another try and
> report back if you still see a problem.

[snip]

> > I also implemented my job_start using an out_cb handler again instead
> > of a close_cb handler. The occasional-missing-callback problem has
> > disappeared and it all worked perfectly. Thanks!
>
> That is the preferred way. Reading the output in the close callback
> might only work if there isn't much to read.

I upgraded from Vim 7.4.1795 to 7.4.1816 and found DETACH was no longer sent to the `out_cb` handler to demarcate the end of my job's output on stdout.

So I added a `close_cb` handler which does what my `out_cb` handler used to do on receiving DETACH: it gathers together all the preceding job output and then processes it.

When I edit a file my job runs. When I trigger a second run, Vim always crashes with a segfault:

Vim: Caught deadly signal SEGV

Vim: Finished.
Segmentation fault: 11

Here's my code:

let job = job_start([&shell, &shellcmdflag, a:cmd], {
\ 'out_cb': 'OutHandler',
\ 'close_cb': 'CloseHandler'
\ })

" Channel is in NL mode.
function! OutHandler(channel, line)
let channel_id = matchstr(a:channel, '\d\+')
call s:accumulate_job_output(channel_id, a:line)
endfunction

function! CloseHandler(channel)
let channel_id = matchstr(a:channel, '\d\+')
call ProcessOutput(s:job_output(channel_id))
call s:job_finished(channel_id)
" redraw!
endfunction

function! s:job_started(id)
let s:jobs[a:id] = 1
endfunction

function! s:is_job_started(id)
return has_key(s:jobs, a:id)
endfunction

function! s:accumulate_job_output(id, line)
if has_key(s:jobs, a:id)
let s:jobs[a:id] = add(s:jobs[a:id], a:line)
else
let s:jobs[a:id] = [a:line]
endif
endfunction

" Returns a string
function! s:job_output(id)
if has_key(s:jobs, a:id)
return join(s:jobs[a:id], "\n")."\n"
else
return ""
endif
endfunction

function! s:job_finished(id)
if has_key(s:jobs, a:id)
unlet s:jobs[a:id]
endif
endfunction

Regarding redrawing: adding a redraw! as the last line of my close handler doesn't redraw the screen. Should I be doing this somewhere else?

[Bram Moolenaar]

boss wrote:

> On Sunday, 1 May 2016 21:37:13 UTC+1, Bram Moolenaar wrote:
>
> > Note that adding "DETACH" was removed, so you can drop this check.
> [snip]
> > A couple of problems have been fixed. Please give it another try and
> > report back if you still see a problem.
> [snip]
> > > I also implemented my job_start using an out_cb handler again instead
> > > of a close_cb handler. The occasional-missing-callback problem has
> > > disappeared and it all worked perfectly. Thanks!
> >
> > That is the preferred way. Reading the output in the close callback
> > might only work if there isn't much to read.
>
> I upgraded from Vim 7.4.1795 to 7.4.1816 and found DETACH was no longer sent to the `out_cb` handler to demarcate the end of my job's output on stdout.
>
> So I added a `close_cb` handler which does what my `out_cb` handler used to do on receiving DETACH: it gathers together all the preceding job output and then processes it.
>
> When I edit a file my job runs. When I trigger a second run, Vim
> always crashes with a segfault:
>
> Vim: Caught deadly signal SEGV
> Vim: Finished.
> Segmentation fault: 11
>
> Here's my code:

Thanks for the description. I'll try to reproduce.

Hmm, does it redraw the moment you type something?
It looks like the function to redraw after a callback is not invoked
after using the close_cb. I'll make a patch for that.

--
No children may attend school with their breath smelling of "wild onions."
[real standing law in West Virginia, United States of America]

[Andrew Stewart]

> On 4 May 2016, at 8:49 pm, Bram Moolenaar <Br...@moolenaar.net> wrote:
>> Regarding redrawing: adding a redraw! as the last line of my close
>> handler doesn't redraw the screen. Should I be doing this somewhere
>> else?
>
> Hmm, does it redraw the moment you type something?

Yes it does. In my case the signs are suddenly displayed.

> It looks like the function to redraw after a callback is not invoked
> after using the close_cb. I'll make a patch for that.

Thanks. I upgraded to Vim 7.4.1817 and reinstated the redraw! as the last line of my close_cb handler. Unfortunately the problem remains.

[Andrew Stewart]

> On 4 May 2016, at 8:49 pm, Bram Moolenaar <Br...@moolenaar.net> wrote:
>

> boss wrote:
>> I upgraded from Vim 7.4.1795 to 7.4.1816 and found DETACH was no longer sent to the `out_cb` handler to demarcate the end of my job's output on stdout.
>>
>> So I added a `close_cb` handler which does what my `out_cb` handler used to do on receiving DETACH: it gathers together all the preceding job output and then processes it.
>>
>> When I edit a file my job runs. When I trigger a second run, Vim
>> always crashes with a segfault:
>>
>> Vim: Caught deadly signal SEGV
>> Vim: Finished.
>> Segmentation fault: 11
>>
>> Here's my code:
>
> Thanks for the description. I'll try to reproduce.

In case you can't reproduce with my code snippets, the actual code is here:

https://github.com/airblade/vim-gitgutter/tree/async

i.e. the "async" branch of the repo.

The callbacks are in this file:

https://github.com/airblade/vim-gitgutter/blob/async/autoload/gitgutter/async.vim

To reproduce:

1. Install the plugin from the repo's async branch.
2. Edit a file from a git repo and make some changes.
3. You should see diff signs in the sign column. They show up after &updatetime and/or when you write the buffer.
4. After one or two modifications Vim will segfault (it does for me).

To turn on logging use `let g:gitgutter_log = 1`. This will write a channel.log and a gitgutter.log to the plugin's directory.

Thanks for investigating all this!

[bo...@airbladesoftware.com]

On Thursday, 5 May 2016 10:25:19 UTC+1, Andrew Stewart wrote:
> >> When I edit a file my job runs. When I trigger a second run, Vim
> >> always crashes with a segfault:
> >>
> >> Vim: Caught deadly signal SEGV
> >> Vim: Finished.
> >> Segmentation fault: 11

I've just realised that every time a job runs, a new process is spawned as a child of vim's process, and they're all in the zombie state. They only disappear once I quit Vim or it crashes.

[Bram Moolenaar]

It looks like it works for me, with this test:


func! CloseCb(chan)
syn match Error "foo"

endfunc

call ch_logfile('channellog', 'w')

call job_start('echo nothing', {'close_cb': 'CloseCb'})


The "foo" gets highlighted right away.


--
INSPECTOR END OF FILM: Move along. There's nothing to see! Keep moving!
[Suddenly he notices the cameras.]
INSPECTOR END OF FILM: (to Camera) All right, put that away sonny.
[He walks over to it and puts his hand over the lens.]

"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

[Bram Moolenaar]

This must be a Mac specific problem, it doesn't call waitpid() to get
the status of the child. You can try using Vim as it's build on Unix,
should also work on Mac in a terminal.

--
"I love deadlines. I especially like the whooshing sound they
make as they go flying by."
-- Douglas Adams

[Andrew Stewart]

> On 5 May 2016, at 5:14 pm, Bram Moolenaar <Br...@moolenaar.net> wrote:

>
> Andrew Stewart wrote:
>>
>> I've just realised that every time a job runs, a new process is
>> spawned as a child of vim's process, and they're all in the zombie
>> state. They only disappear once I quit Vim or it crashes.
>
> This must be a Mac specific problem, it doesn't call waitpid() to get
> the status of the child. You can try using Vim as it's build on Unix,
> should also work on Mac in a terminal.

I get this behaviour using "pure" Vim (built with homebrew on OS X 10.11.4), not MacVim. v7.4.1817 this time.

I'm watching the processes using htop in tree view.

[Andrew Stewart]

> On 5 May 2016, at 5:14 pm, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
> Andrew Stewart wrote:
>
>>> On 4 May 2016, at 8:49 pm, Bram Moolenaar <Br...@moolenaar.net> wrote:
>>>> Regarding redrawing: adding a redraw! as the last line of my close
>>>> handler doesn't redraw the screen. Should I be doing this somewhere
>>>> else?
>>>
>>> Hmm, does it redraw the moment you type something?
>>
>> Yes it does. In my case the signs are suddenly displayed.
>>
>>> It looks like the function to redraw after a callback is not invoked
>>> after using the close_cb. I'll make a patch for that.
>>
>> Thanks. I upgraded to Vim 7.4.1817 and reinstated the redraw! as the
>> last line of my close_cb handler. Unfortunately the problem remains.
>
> It looks like it works for me, with this test:
>
> func! CloseCb(chan)
> syn match Error "foo"
> endfunc
>
> call ch_logfile('channellog', 'w')
>
> call job_start('echo nothing', {'close_cb': 'CloseCb'})
>
> The "foo" gets highlighted right away.

The "foo" gets highlighted right away for me too. It also works as expected if I place a sign instead.

So I don't know why the signs placed by the callback in my actual code aren't being displayed until I move the cursor. I also don't know what to do to investigate further.

[bo...@airbladesoftware.com]

While a callback runs, does Vim block everything else? Or do callbacks run at the same time as other things are executed?

In other words does Vimscript code with callbacks have to be thread-safe?

[Bram Moolenaar]

Callbacks are only run while waiting for the user to type a character.
This avoids one script interrupting another script at any point.
Vim itself is not multi-threaded. You can get that with Python.
But it's tricky.

--
SUPERIMPOSE "England AD 787". After a few more seconds we hear hoofbeats in
the distance. They come slowly closer. Then out of the mist comes KING
ARTHUR followed by a SERVANT who is banging two half coconuts together.

"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

[Bram Moolenaar]

Patch 7.4.1826 should fix this. Let me know if you still see a problem.

--
The problem with political jokes is that they get elected.

[Andrew Stewart]

It does fix it, and it also fixes my redraw-not-working problem. In fact I don't need the redraw at all.

Everything's working perfectly for me now.

Thank you very much!