ch_sendraw with large data (+65536 bytes) throws E630

[Matteo Landi]

Hello everyone,

ch_sendraw seems to fail when sending large chunks of data to a job
started in non-blocking mode. Is this expected?

Steps to reproduce:

- Copy the following into a file, and source it from vim

let args = ['cat']
let opts = {
\ 'noblock': 1
\}

let job = job_start(args, opts)
let data = repeat('X', 65537)

call ch_sendraw(job, data)
call ch_close_in(job)

Expected behavior:

- Nothing much (i.e. no errors)

Actual bevarior:

- E630: channel_write_input(): write while not connected

If on the other hand, I initialized data with 65536 bytes instead of
65537, then no error would be thrown.

In case you needed it, here is the output of `vim --version`

> vim --version
VIM - Vi IMproved 8.1 (2018 May 18, compiled Aug 16 2019 02:18:16)
macOS version
Included patches: 1-1850
Compiled by Homebrew
Huge version without GUI. Features included (+) or not (-):
+acl -farsi -mouse_sysmouse -tag_any_white
+arabic +file_in_path +mouse_urxvt -tcl
+autocmd +find_in_path +mouse_xterm +termguicolors
+autochdir +float +multi_byte +terminal
-autoservername +folding +multi_lang +terminfo
-balloon_eval -footer -mzscheme +termresponse
+balloon_eval_term +fork() +netbeans_intg +textobjects
-browse +gettext +num64 +textprop
++builtin_terms -hangul_input +packages +timers
+byte_offset +iconv +path_extra +title
+channel +insert_expand +perl -toolbar
+cindent +job +persistent_undo +user_commands
-clientserver +jumplist +postscript +vartabs
+clipboard +keymap +printer +vertsplit
+cmdline_compl +lambda +profile +virtualedit
+cmdline_hist +langmap -python +visual
+cmdline_info +libcall +python3 +visualextra
+comments +linebreak +quickfix +viminfo
+conceal +lispindent +reltime +vreplace
+cryptv +listcmds +rightleft +wildignore
+cscope +localmap +ruby +wildmenu
+cursorbind +lua +scrollbind +windows
+cursorshape +menu +signs +writebackup
+dialog_con +mksession +smartindent -X11
+diff +modify_fname -sound -xfontset
+digraphs +mouse +spell -xim
-dnd -mouseshape +startuptime -xpm
-ebcdic +mouse_dec +statusline -xsmp
+emacs_tags -mouse_gpm -sun_workshop -xterm_clipboard
+eval -mouse_jsbterm +syntax -xterm_save
+ex_extra +mouse_netterm +tag_binary
+extra_search +mouse_sgr -tag_old_static
system vimrc file: "$VIM/vimrc"
user vimrc file: "$HOME/.vimrc"
2nd user vimrc file: "~/.vim/vimrc"
user exrc file: "$HOME/.exrc"
defaults file: "$VIMRUNTIME/defaults.vim"
fall-back for $VIM: "/usr/local/share/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H -DMACOS_X -DMACOS_X_DARWIN -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: clang -L. -fstack-protector-strong -L/usr/local/lib -L/usr/local/opt/libyaml/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -L/usr/local/lib -o vim -lncurses -liconv -lintl -framework AppKit -L/usr/local/opt/lua/lib -llua5.3 -mmacosx-version-min=10.14 -fstack-protector-strong -L/usr/local/lib -L/usr/local/Cellar/perl/5.30.0/lib/perl5/5.30.0/darwin-thread-multi-2level/CORE -lperl -lm -lutil -lc -L/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin -lpython3.7m -framework CoreFoundation -lruby.2.6

Thanks,

--
Matteo Landi
https://matteolandi.net

[Bram Moolenaar]

> ch_sendraw seems to fail when sending large chunks of data to a job
> started in non-blocking mode. Is this expected?
>
> Steps to reproduce:
>
> - Copy the following into a file, and source it from vim
>
> let args = ['cat']
> let opts = {
> \ 'noblock': 1
> \}
>
> let job = job_start(args, opts)
> let data = repeat('X', 65537)
>
> call ch_sendraw(job, data)
> call ch_close_in(job)
>
> Expected behavior:
>
> - Nothing much (i.e. no errors)
>
> Actual bevarior:
>
> - E630: channel_write_input(): write while not connected
>
> If on the other hand, I initialized data with 65536 bytes instead of
> 65537, then no error would be thrown.

You are using "noblock", which means the write may happen a bit later,
but then you use ch_close_in(), closing the channel before all the
writes are done. If you put a sleep there it works.

--
From "know your smileys":
:.-( Crying

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Matteo Landi]

On 08/22, Bram Moolenaar wrote:
>
>> ch_sendraw seems to fail when sending large chunks of data to a job
>> started in non-blocking mode. Is this expected?
>>
>> Steps to reproduce:
>>
>> - Copy the following into a file, and source it from vim
>>
>> let args = ['cat']
>> let opts = {
>> \ 'noblock': 1
>> \}
>>
>> let job = job_start(args, opts)
>> let data = repeat('X', 65537)
>>
>> call ch_sendraw(job, data)
>> call ch_close_in(job)
>>
>> Expected behavior:
>>
>> - Nothing much (i.e. no errors)
>>
>> Actual bevarior:
>>
>> - E630: channel_write_input(): write while not connected
>>
>> If on the other hand, I initialized data with 65536 bytes instead of
>> 65537, then no error would be thrown.
>
>You are using "noblock", which means the write may happen a bit later,
>but then you use ch_close_in(), closing the channel before all the
>writes are done. If you put a sleep there it works.

Oh, I thought ch_close_in() would wait for any pending write to
complete, but I guess that _special mode_ on `:help channel-close-in`
did refer to non blocking channels in the end, didn't it?! My bad!

Then the next question is, how could I wait for any pending write
operation to complete before calling ch_close_in()? Should I include a
EOF in the last message, instead of calling ch_close_in()?

[Bram Moolenaar]

I don't think we have a mechanism for that. There are callbacks for
reading from input and error, but not for writing. I suppose it would
be possible to add an in_cb function, called after the write buffer is
empty. Or just after every finished write.

--
Common sense is what tells you that the world is flat.

[Matteo Landi]

I like the idea of letting users know when the buffer is empty; do you
want me to file a bug/enhancement request on Github?

Just to give you a bit more context about what I am trying to do. I am
trying to integrate with a small utility I created, `cb`, which acts as
*remote* `pbcopy`: you pipe some text into it while inside a ssh session
on a remote host, and it would send that back on the local host and copy
on the OS clipboard. Sometimes, copying big selections of text is not
always as fast as one would hope, so I decided to run this using Vim
jobs:

- Create a job where I invoke `cb`
- Write text selection into it -- stdin
- Close the job stdin -- `cb` would terminate on its own after all the
data has been sent over to the next end of the session

I hope this gives you a better understanding of what I am trying to
achieve, and why. Anyway, let me know what do you want me to do with
this request.

Thanks in advance.