capture() function to get output of command

[mattn]

Hi, list.

I wrote a patch to add new function 'capture()'.
When you want to get a list of all commands/functions/mappings with using vim script, probably, you'll do like follow.

---------------------------------
let funcs = ''
redir => funcs
silent! function
redir END
---------------------------------

I guess, most of vim script programmers want to get the result as expression. not command, want to get more easily.
If vim have 'capture()' which allow to getting those command line 'function', I guess this is very useful.

For example, getting :Command list as array.
---------------------------------
:echo map(split(capture("command"), "\n")[1:], 'split(v:val[4:])[0]')
---------------------------------

One more thing. vim doesn't support nest of :redir.
---------------------------------
function! s:foo()
let a = ''
redir => a
silent echo "foo"
redir END
return a
endfunction

function! s:bar()
let a = ''
redir => a
call s:foo()
redir END
return a
endfunction

echo s:bar()
---------------------------------

This 'echo s:bar()' doesn't show anything. But capture() can do it.
---------------------------------
echo capture("s:bar()")

foo
---------------------------------

capture() is working on sandbox. so you can't use this in modeline.

Below is a patch. How about this new feature?

https://gist.github.com/mattn/5449425

- Yasuhiro Matsumoto

[Andy Wokula]

Probably a nice addition, but one can already do all of this in vimscript,
see attachment.

(the script is not in release shape ... just what I use at the moment).

In your example, replace capture() with StrRedir().

see also (my older version, didn't get any feedback on it):
https://groups.google.com/d/msg/vim_use/Ea3IW7JH5MM/p41COFlyMO0J

--
Andy

[mattn]

Yes, I know.

> Probably a nice addition, but one can already do all of this in vimscript, see attachment.

> In your example, replace capture() with StrRedir().

No, it can't. It depend on your vim script. We want to get the result without side effect.

---------------------------------

function! s:bar()
let a = ''
redir => a

call thirdparty#library#function()

redir END
return a
endfunction

echo s:bar()
---------------------------------

Do you know the wheather thirdparty#library#function() call redir? or not?
I don't think all of vim script like the thirdparty#library#function() call your Redir function.

[Andy Wokula]

Am 01.05.2013 14:11, schrieb mattn:
> Yes, I know.
>
>> Probably a nice addition, but one can already do all of this in
>> vimscript, see attachment. In your example, replace capture() with
>> StrRedir().
>
> No, it can't. It depend on your vim script. We want to get the result
> without side effect.
>
> ---------------------------------
> function! s:bar()
> let a = ''
> redir => a
> call thirdparty#library#function()
> redir END
> return a
> endfunction
>
> echo s:bar()
> ---------------------------------
>
> Do you know the wheather thirdparty#library#function() call redir? or
> not? I don't think all of vim script like the
> thirdparty#library#function() call your Redir function.

Yep, all code is required to use StrRedir(). There could be an autoload
script shipped with Vim which does the same as StrRedir(), and the help
for :redir could contain a recommendation to use it. But this still
doesn't help existing code.

+1 for capture(), just wanted to point out an existing solution.

--
Andy

[Ingo Karkat]

While :redir has its uses in the interactive command-line, its
restriction of not allowing nesting is very bad for plugins, and I've
already been affected by this. (I had to send a patch for another plugin
so that its :redir failed gracefully when it ran inside of my plugin.)

You've shown that capture() handles a nested :redir. Just to be clear,
does it also handle a nested capture() inside a capture() execution?! (I
would expect that the outer capture gets the results of the inner
capture as well.)

If that's the case, +1 from me for inclusion.

In addition to taking commands and Lists of commands, it might also be
handy to take Funcrefs.

-- regards, ingo

[Christian Brabandt]

Hi mattn!

Perhaps we should first fix redir before we introduce another similar
function?

regads,
Christian
--
Zerstreuete Gedanken lieset man wieder zerstreuet und bl�ttert in
ihnen herum.
-- Jean Paul

[Ingo Karkat]

That would work, too. On the other hand, here's an idea from a
practitioner that might justify a separate function: Almost always, the
captured text is split() into a list of lines (mattn's example does
this, too). Why not have capture() return a list of lines in the first
place?! We obviously cannot change the original :redir interface, but a
new function could offer this and spare all plugins from doing that.

-- regards, ingo

[Christian Brabandt]

Hi Ingo!

The patch throws out Warnings:

eval.c: In function 'f_capture':
eval.c:9305:16: warning: assignment from incompatible pointer type [enabled by default]
gcc -c -I. -Iproto -DHAVE_CONFIG_H -I/usr/local/include -g -DDEBUG -Wall -Wshadow -Wmissing-prototypes -Wmaybe-uninitialized -o objects/main.o main.c
In file included from vim.h:1965:0,
from main.c:11:
globals.h:1098:18: warning: 'capture_ga' initialized and declared 'extern' [enabled by default]


This patch seems to fix those warnings:
iff --git a/src/eval.c b/src/eval.c
--- a/src/eval.c
+++ b/src/eval.c
@@ -9302,7 +9302,7 @@
if (check_secure())
return;

- capture_ga = alloc(sizeof(garray_T));
+ capture_ga = (garray_T *) alloc(sizeof(garray_T));
ga_init2(capture_ga, (int)sizeof(char), 80);

++msg_silent;
diff --git a/src/globals.h b/src/globals.h
--- a/src/globals.h
+++ b/src/globals.h
@@ -1095,7 +1095,11 @@
EXTERN int redir_reg INIT(= 0); /* message redirection register */
EXTERN int redir_vname INIT(= 0); /* message redirection variable */
#endif
-extern garray_T *capture_ga INIT(= NULL); /* capture() buffer */
+EXTERN garray_T *capture_ga /* capture() buffer */
+# ifdef DO_INIT
+ = NULL
+# endif
+ ;

I haven't played yet much with the capture() function, so I am not sure,
this works recursively.

It looks like a useful addition, but I am not sure, whether a new
function is the right way to go, instead of fixing redir.

Mit freundlichen Gr��en
Christian
--
Auf der Verpackung steht: Win95, NT oder besser, folglich ist Linux
eine unterst�tzte Plattform.

[Ingo Karkat]

On 02-May-2013 16:29 +0200, Christian Brabandt wrote:

> [100 lines deleted]

>
> It looks like a useful addition, but I am not sure, whether a new
> function is the right way to go, instead of fixing redir.

I fully agree, it would be better if :redir could be fixed; even with my
proposed inclusion of split()ing into lines, the additional benefit is
low, and it will take a long time until plugins can fully benefit from
that function without a compatibility wrapper.

Because I'm still mostly ignorant of the Vim source code, I had assumed
the patch author has found fixing :redir too difficult (as it also /
mostly handles file and register targets where there indeed will be
issues with overriding); I hope mattn can shed some light on that.

-- regards, ingo

[mattn]

> Mit freundlichen Gr��en

>
> Christian
>
> --
>
> Auf der Verpackung steht: Win95, NT oder besser, folglich ist Linux
>

> eine unterst�tzte Plattform.

Ah, thank you.

[mattn]

> I fully agree, it would be better if :redir could be fixed; even with my
> proposed inclusion of split()ing into lines, the additional benefit is
> low, and it will take a long time until plugins can fully benefit from
> that function without a compatibility wrapper.
>
> Because I'm still mostly ignorant of the Vim source code, I had assumed
> the patch author has found fixing :redir too difficult (as it also /
> mostly handles file and register targets where there indeed will be
> issues with overriding); I hope mattn can shed some light on that.

The functions in message.c write message with color attributes. So if splitting per line, it must get whole contents at the first.
I guess someone want to get the contents as text which contains \n, but someone want to get the contents as array of lines. So I want to keep the contents as text.

Thanks.

[Ingo Karkat]

On 07-May-2013 05:57 +0200, mattn wrote:

>> I fully agree, it would be better if :redir could be fixed; even with my
>> proposed inclusion of split()ing into lines, the additional benefit is
>> low, and it will take a long time until plugins can fully benefit from
>> that function without a compatibility wrapper.
>>
>> Because I'm still mostly ignorant of the Vim source code, I had assumed
>> the patch author has found fixing :redir too difficult (as it also /
>> mostly handles file and register targets where there indeed will be
>> issues with overriding); I hope mattn can shed some light on that.
>
> The functions in message.c write message with color attributes. So if
> splitting per line, it must get whole contents at the first.

Well, okay; right now, if split() is called separately, that works out
the same.

> I guess someone want to get the contents as text which contains \n,
> but someone want to get the contents as array of lines. So I want to
> keep the contents as text.

My point is that I've *never* seen that use case so far in plugins; they
all want to parse the captured output into a data structure. Do you have
any counterexample? (And for that rare use case, they could be join()'ed
back together easily.)

Any way, that built-in splitting would only be one weak counterargument
in favor of a separate capture() function. Could you make :redir work
with nested invocations instead?!

-- regards, ingo

[ZyX]

> The functions in message.c write message with color attributes. So if splitting per line, it must get whole contents at the first.
> I guess someone want to get the contents as text which contains \n, but someone want to get the contents as array of lines. So I want to keep the contents as text.
>
> Thanks.

Almost never used not splitted messages (they were only needed by me in case there is only one line of output; and only because splitting one line does not make much sense; code with proposed by @Ingo Karkat implementation of `capture()` is not a tiny bit harder to write in this case). I have another proposal:

1. Fix :redir to allow nesting.
2. Fix :redir to hold reference to dictionary containing variable (like l:, g:, s: or such) to workaround problem I mentioned in https://groups.google.com/forum/#!searchin/vim_dev/redir/vim_dev/1M3QS46SrAk/a13SqNpPWywJ. Obviously only in case redirecting to variable.
3. Make :redir automatically end when function ends if it holds reference to l:. In any case I doubt anybody needs to start redir in one function and end in another and do redirection to function-local variable at the time. This also must workaround stale :redir's left due to an exception (plugin authors almost always capture to function-local variables).
4. (optional) Add capture() function, but it should return a list of lists of dictionaries:

[
[{'hl': "Error", 'text': "Error on first line"}, {'hl': "OtherHL", 'text': "Other highlight in first line (possible when using :echon)"}],
[{'hl': "Normal", 'text': "Regular message on second line"}],
]

. It is useful in some cases (like any use of :redir this cases happen mainly because of problems in API: distinguishing between `<LT>Space>` and `<Space>` in :map output (current code I use assumes nobody needs to define mapping like `<LT>F1>`; assumption has not proved false by the time, but having to use it bugs me); distinguishing between regular text and special characters in messages (don’t remember when I needed this, but there was some case)).
5. (optional) Extend syntax of :redir to capture to a list of lines.

[Ingo Karkat]

On 07-May-2013 21:44 +0200, ZyX wrote:

>> The functions in message.c write message with color attributes. So if splitting per line, it must get whole contents at the first.
>> I guess someone want to get the contents as text which contains \n, but someone want to get the contents as array of lines. So I want to keep the contents as text.
>>
>> Thanks.
>
> Almost never used not splitted messages (they were only needed by me in case there is only one line of output; and only because splitting one line does not make much sense; code with proposed by @Ingo Karkat implementation of `capture()` is not a tiny bit harder to write in this case).

Yes, that's what I thought. Thanks for this data point!

> I have another proposal:

You bring up very good points; I fully agree.

> 1. Fix :redir to allow nesting.

I think everyone agrees with this, and it should be first priority.

> 2. Fix :redir to hold reference to dictionary containing variable (like l:, g:, s: or such) to workaround problem I mentioned in https://groups.google.com/forum/#!searchin/vim_dev/redir/vim_dev/1M3QS46SrAk/a13SqNpPWywJ. Obviously only in case redirecting to variable.

I don't fully understand, but as long as it's backwards compatible,
additional syntax, why not!?

> 3. Make :redir automatically end when function ends if it holds reference to l:. In any case I doubt anybody needs to start redir in one function and end in another and do redirection to function-local variable at the time. This also must workaround stale :redir's left due to an exception (plugin authors almost always capture to function-local variables).

This would alleviate a common programming error. Currently, all :redir
commands have to be wrapped in try..finally to be safe, but few people
do this.

> 4. (optional) Add capture() function, but it should return a list of lists of dictionaries:
>
> [
> [{'hl': "Error", 'text': "Error on first line"}, {'hl': "OtherHL", 'text': "Other highlight in first line (possible when using :echon)"}],
> [{'hl': "Normal", 'text': "Regular message on second line"}],
> ]
>

> . It is useful in some cases (like any use of :redir this cases happen mainly because of problems in API: distinguishing between `<LT>Space>` and `<Space>` in :map output (current code I use assumes nobody needs to define mapping like `<LT>F1>`; assumption has not proved false by the time, but having to use it bugs me); distinguishing between regular text and special characters in messages (don�t remember when I needed this, but there was some case)).

Good idea. I had this need once with the :jumps output: Without the
highlighting information, I had to implement a 99%-successful heuristic
to decide whether the line refers to the current buffer or another file.
It would be even better if the (mostly column-based) information issued
by most commands came pre-parsed, but that probably would mean modifying
all command implementations.

> 5. (optional) Extend syntax of :redir to capture to a list of lines.

Since this only applies for redirecting into a variable (not files or
registers), I'd rather not further complicate the existing :redir
command with this. Instead, if your proposal 4. capture() is
implemented, have it take an argument {includeHl}, and just return a
list of lines (without the Dictionary stuff) if 0 is passed. Most uses
don't need the highlighting; this would make the function applicable to
more use cases.


I'd love to have all of those good enhancements, but I know that unless
someone now starts working on a patch, this great discussion will again
fizzle out, like so many before.

-- regards, ingo

[ZyX]

> > 1. Fix :redir to allow nesting.
>
> I think everyone agrees with this, and it should be first priority.
>
> > 2. Fix :redir to hold reference to dictionary containing variable (like l:, g:, s: or such) to workaround problem I mentioned in https://groups.google.com/forum/#!searchin/vim_dev/redir/vim_dev/1M3QS46SrAk/a13SqNpPWywJ. Obviously only in case redirecting to variable.
>
> I don't fully understand, but as long as it's backwards compatible,
> additional syntax, why not!?

It is backwards compatible, but there is no additional syntax. When you call :redir with variable name it seems (I did not look at the code) to create a variable, save variable name (but not the typval_T reference) and when writing to it use usual ways to get variable and write to the structure. Thus if :redir END is called when created variable is not in (script-)local scope of this call it errors out with “variable does not exist” or like. If :redir instead saved reference to the dict_T structure and then used usual methods of assigning dictionary key that error won’t appear. Alternatively this saved reference can be used to check whether :redir END is called in the same scope and do nothing (or give different error message) in case it is not. With nested :redir’ections I have nothing against error, without you will have to wrap `:redir =>` into a separate :try..:catch (with empty :catch block, and with pattern argument in this command) to be safe. In addition to surrounding :try..:finally of course.

I am going to reduce the need for :redir for python users: further extension to python interface providing:

- vim.maps/vim.buffer.maps (*iterable* and with output similar to maparg(lhs, mode, abbr, dict=True) (but as python object and .noremap with three values in place of two)): removes the need for parsing :redir|:map|:redir END;
- vim.commands/vim.buffer.commands (iterable, after finished will be the only way to get command definition using something other then :redir);
- vim.autocommands (iterable; same as above);
- vim.signs (:sign [un]define/:sign list), vim.buffer.signs (:sign [un]place);
- vim.registers (there is no way to get NUL out of the registers except for pasting register in a scratch buffer: any string manipulation transforms internal representation of NUL to "\n" and :redir|:reg truncates output in addition to having to guess (or much more complex: compare with `getreg()` output) whether ^@ meant NUL or somebody was typing ^ then @ like I did here).

: this serves two purposes: make plugin writers prefer python (thus I am not talking about VimL API) and create normal API for the things that lack one. If you have other candidates tell me (I guess I already have one from you: :jumps). I will open a RFC here when API is fully designed; though I probably won’t have much time to write an implementation in next one or two months.

Maybe I even won’t forget about the discussion after both API and implementation are finished. The thing I won’t add to python though is vim.capture: usage of :redir only means some API is missing and that is the problem that should be fixed.

[Ingo Karkat]

On 08-May-2013 19:57 +0200, ZyX wrote:

>>> 1. Fix :redir to allow nesting.
>>
>> I think everyone agrees with this, and it should be first priority.
>>
>>> 2. Fix :redir to hold reference to dictionary containing variable (like l:, g:, s: or such) to workaround problem I mentioned in https://groups.google.com/forum/#!searchin/vim_dev/redir/vim_dev/1M3QS46SrAk/a13SqNpPWywJ. Obviously only in case redirecting to variable.
>>
>> I don't fully understand, but as long as it's backwards compatible,
>> additional syntax, why not!?
>

> It is backwards compatible, but there is no additional syntax. When you call :redir with variable name it seems (I did not look at the code) to create a variable, save variable name (but not the typval_T reference) and when writing to it use usual ways to get variable and write to the structure. Thus if :redir END is called when created variable is not in (script-)local scope of this call it errors out with �variable does not exist� or like. If :redir instead saved reference to the dict_T structure and then used usual methods of assigning dictionary key that error won�t appear. Alternatively this saved reference can be used to check whether :redir END is called in the same scope and do nothing (or give different error message) in case it is not. With nested :redir�ections I have nothing against error, without you will have to wrap `:redir =>` into a separate :try..:catch (with empty :catch block, and with pattern argument in this command) to be safe. In addition to surrounding :tr
y..:finally of course.

Ah, thanks for the clarification; I think I understand now. I've never
done :redir END in a different scope, but it would be nice if this
corner case bug could be fixed.

> : this serves two purposes: make plugin writers prefer python (thus I am not talking about VimL API) and create normal API for the things that lack one. If you have other candidates tell me (I guess I already have one from you: :jumps). I will open a RFC here when API is fully designed; though I probably won�t have much time to write an implementation in next one or two months.

I've seen your many recent Python patches, and it's certainly good to
have more things exposed in a native language way. However, I'm a bit
concerned if (but I know too little about your patches to be sure) you
have to write all the plumbing from Vim's internal structures to the
Python interface in a Python-specific way. I would prefer if most of the
plumbing were done in Vim core, so that Vimscript as well as other
language integrations could easily benefit. I imagine that the object
model of Vimscript, Python, Perl, Ruby, Lua, etc. are similar enough
that if all that access functionality existed in Vim core, each
integration could get at that with a very thin language-dependent bridge.

To put it in another way: Christian Brabandt recently implemented a
Vimscript getsign() function and extended exists() to be able to access
the sign information. I would hope that your vim.signs implementation
could leverage most of that from Vim core. It would be bad if your
implementation is a completely separate access of the core Vim
structures, because of:
- duplication = code bloat
- other language integrations will eventually want to have the same
access; each has to tediously reinvent the wheel, and will probably do
so inconsistently
- since maintenance is separate for each language, we end up with Python
abilities for A, B, and C, Vimscript access for A & B (and use of :redir
for C), and e.g. Ruby access only for C.

> Maybe I even won�t forget about the discussion after both API and implementation are finished. The thing I won�t add to python though is vim.capture: usage of :redir only means some API is missing and that is the problem that should be fixed.

Don't get me wrong; I appreciate your efforts to improve the Python
interface. I'm just a bit afraid that in the future, people will use
Python just because there's no equally convenient way to access the
information from Vimscript. To me, that would be bad. The Vimscript API
should remain the first-class, full interface (that doesn't preclude
other languages from having an equally rich interface), so that other
languages are chosen based on their merits (mostly familiarity and
external APIs that not available inside Vim), not based on need.

-- regards, ingo

[ZyX]

> I've seen your many recent Python patches, and it's certainly good to
> have more things exposed in a native language way. However, I'm a bit
> concerned if (but I know too little about your patches to be sure) you
> have to write all the plumbing from Vim's internal structures to the
> Python interface in a Python-specific way. I would prefer if most of the
> plumbing were done in Vim core, so that Vimscript as well as other
> language integrations could easily benefit. I imagine that the object
> model of Vimscript, Python, Perl, Ruby, Lua, etc. are similar enough
> that if all that access functionality existed in Vim core, each
> integration could get at that with a very thin language-dependent bridge.

You can review the patches. Most code is converting arguments and result to/from PyObject* and error-checking, there is not much that can be moved to core. I can’t generalize things like WindowAttr for e.g. lua: it cannot be used because WindowAttr returns PyObject* and in lua it would “return” void and push to the stack. Not saying I do not know other APIs. Things that make sense to put in core I put in core, but there are not much: in that sequence I only added functions for finding windows, temporary switching to specific window+tabpage and stricter get_option_value.

> To put it in another way: Christian Brabandt recently implemented a
> Vimscript getsign() function and extended exists() to be able to access
> the sign information. I would hope that your vim.signs implementation
> could leverage most of that from Vim core. It would be bad if your
> implementation is a completely separate access of the core Vim
> structures, because of:
> - duplication = code bloat

As I said I add to core what makes sense to be added there. Most of code in if_py_both is data conversions from C types to PyObject* and back, error handling or other python-specific stuff; I see no candidates for moving to core (obviously does not mean there are no). Code is mostly trivial, but hard to write (for a high-level language user) due to its amount and amount of things that need to be cared about (exceptions, failed memory allocation, decref/incref, rarely alloc/free, etc).

> - other language integrations will eventually want to have the same
> access; each has to tediously reinvent the wheel, and will probably do
> so inconsistently

Other languages’ integrations should be consistent with other languages’ best practices, not with python or VimL. Python users would not be happy with vim.lock(dict) in place of setattr(dict, 'locked', True) (dict.locked = True), but VimL uses first variant (with a command though). dict['abc']['def'] may be expected to throw an exception for absent 'abc' or 'def', or return undef and create a dictionary for key 'abc' (and even do this depending on options; I am talking about perl here, though dunno whether you can create a vim.dictionary there).

> - since maintenance is separate for each language, we end up with Python
> abilities for A, B, and C, Vimscript access for A & B (and use of :redir
> for C), and e.g. Ruby access only for C.

Yup. It would be good if there was a ruby user+developer to port my changes to ruby though. As I said most code is service one, lua won’t benefit from code that transforms to PyObject* handling errors in the middle, so you can expect extending lua being just as hard (or harder/easier depending on its C API) as extending python, with a small help from a small amount of functions that are general enough to have them in core.

> > Maybe I even won�t forget about the discussion after both API and implementation are finished. The thing I won�t add to python though is vim.capture: usage of :redir only means some API is missing and that is the problem that should be fixed.
>
> Don't get me wrong; I appreciate your efforts to improve the Python
> interface. I'm just a bit afraid that in the future, people will use
> Python just because there's no equally convenient way to access the
> information from Vimscript. To me, that would be bad. The Vimscript API
> should remain the first-class, full interface (that doesn't preclude
> other languages from having an equally rich interface), so that other
> languages are chosen based on their merits (mostly familiarity and
> external APIs that not available inside Vim), not based on need.

To choose other languages for their familiarity/*external* APIs somebody needs to find another person to write all the things I am implementing for Python and this would be good. I don’t think I will live long enough to see when you could choose language for vim plugin without taking vim<->language API into account. Also don’t forget that A and B that are not in ruby can be accessed through VimL layer.

VimL being a first-class is not a good thing because of a big number of issues not fixable due to either backwards compatibility or too big codebase changes that require to fix them; if I was sure vim users are required to have some language but VimL I would immediately switch to it. Though you have a point about another languages: things have to be added to VimL because this way they automatically “appear” in other supported languages.

Note though that VimL additions I can design will much likely smell python, but python has iterators to walk through vim.mappings without allocating a big list by a small chunks (you can’t do one alloc for all list items in C code and expect it not to crash as a consequence) and tp_as_mapping->mp_subscript to not allocate a big nested dictionary just for querying for autocommands['Powerline']['VimEnter']['*']. When you will see my VimL APIs in RFC please be captious.

[Ingo Karkat]

On 08-May-2013 22:10 +0200, ZyX wrote:

>> I've seen your many recent Python patches, and it's certainly good to
>> have more things exposed in a native language way. However, I'm a bit
>> concerned if (but I know too little about your patches to be sure) you
>> have to write all the plumbing from Vim's internal structures to the
>> Python interface in a Python-specific way. I would prefer if most of the
>> plumbing were done in Vim core, so that Vimscript as well as other
>> language integrations could easily benefit. I imagine that the object
>> model of Vimscript, Python, Perl, Ruby, Lua, etc. are similar enough
>> that if all that access functionality existed in Vim core, each
>> integration could get at that with a very thin language-dependent bridge.
>

> You can review the patches. Most code is converting arguments and result to/from PyObject* and error-checking, there is not much that can be moved to core. I can�t generalize things like WindowAttr for e.g. lua: it cannot be used because WindowAttr returns PyObject* and in lua it would �return� void and push to the stack. Not saying I do not know other APIs. Things that make sense to put in core I put in core, but there are not much: in that sequence I only added functions for finding windows, temporary switching to specific window+tabpage and stricter get_option_value.

Good, I'm relieved.

>> To put it in another way: Christian Brabandt recently implemented a
>> Vimscript getsign() function and extended exists() to be able to access
>> the sign information. I would hope that your vim.signs implementation
>> could leverage most of that from Vim core. It would be bad if your
>> implementation is a completely separate access of the core Vim
>> structures, because of:
>> - duplication = code bloat
>
> As I said I add to core what makes sense to be added there. Most of code in if_py_both is data conversions from C types to PyObject* and back, error handling or other python-specific stuff; I see no candidates for moving to core (obviously does not mean there are no). Code is mostly trivial, but hard to write (for a high-level language user) due to its amount and amount of things that need to be cared about (exceptions, failed memory allocation, decref/incref, rarely alloc/free, etc).

Yes, due to the mismatch between the C code in core and the mostly
object-oriented APIs of other languages, I had assumed that there is
quite some boilerplate code. If any of that could be wrapped for reuse
[by other language integrations], either through macros or helper
functions, that would be desirable.

>> - other language integrations will eventually want to have the same
>> access; each has to tediously reinvent the wheel, and will probably do
>> so inconsistently
>

> Other languages� integrations should be consistent with other languages� best practices, not with python or VimL. Python users would not be happy with vim.lock(dict) in place of setattr(dict, 'locked', True) (dict.locked = True), but VimL uses first variant (with a command though). dict['abc']['def'] may be expected to throw an exception for absent 'abc' or 'def', or return undef and create a dictionary for key 'abc' (and even do this depending on options; I am talking about perl here, though dunno whether you can create a vim.dictionary there).

>
>> - since maintenance is separate for each language, we end up with Python
>> abilities for A, B, and C, Vimscript access for A & B (and use of :redir
>> for C), and e.g. Ruby access only for C.
>

> Yup. It would be good if there was a ruby user+developer to port my changes to ruby though. As I said most code is service one, lua won�t benefit from code that transforms to PyObject* handling errors in the middle, so you can expect extending lua being just as hard (or harder/easier depending on its C API) as extending python, with a small help from a small amount of functions that are general enough to have them in core.

>
>>> Maybe I even won�t forget about the discussion after both API and implementation are finished. The thing I won�t add to python though is vim.capture: usage of :redir only means some API is missing and that is the problem that should be fixed.
>>
>> Don't get me wrong; I appreciate your efforts to improve the Python
>> interface. I'm just a bit afraid that in the future, people will use
>> Python just because there's no equally convenient way to access the
>> information from Vimscript. To me, that would be bad. The Vimscript API
>> should remain the first-class, full interface (that doesn't preclude
>> other languages from having an equally rich interface), so that other
>> languages are chosen based on their merits (mostly familiarity and
>> external APIs that not available inside Vim), not based on need.
>

> To choose other languages for their familiarity/*external* APIs somebody needs to find another person to write all the things I am implementing for Python and this would be good. I don�t think I will live long enough to see when you could choose language for vim plugin without taking vim<->language API into account. Also don�t forget that A and B that are not in ruby can be accessed through VimL layer.
>
> VimL being a first-class is not a good thing because of a big number of issues not fixable due to either backwards compatibility or too big codebase changes that require to fix them; if I was sure vim users are required to have some language but VimL I would immediately switch to it. Though you have a point about another languages: things have to be added to VimL because this way they automatically �appear� in other supported languages.

I've heard the dislike of Vimscript, but much of that is probably due to
the developers' relative unfamiliarity with it. For extensions that are
closely related to text-editing, I found it quite nice and powerful, and
there is no mismatch between the Vim API and another language to bridge.
But I agree that other stuff (e.g. any interfacing with external systems
or networking stuff) is extremely hard (though people have written
hashing functions or JSON parsers in Vimscript).

> Note though that VimL additions I can design will much likely smell python, but python has iterators to walk through vim.mappings without allocating a big list by a small chunks (you can�t do one alloc for all list items in C code and expect it not to crash as a consequence) and tp_as_mapping->mp_subscript to not allocate a big nested dictionary just for querying for autocommands['Powerline']['VimEnter']['*']. When you will see my VimL APIs in RFC please be captious.

Certainly, Vimscript won't win any performance contest; other language
interpreters are probably way more optimized.


Thanks for all your explanations; you indeed have very deep knowledge in
this area. I'm looking forward to your patches and improved Python
integration! Good luck!

-- regards, ingo

[ZyX]

> > As I said I add to core what makes sense to be added there. Most of code in if_py_both is data conversions from C types to PyObject* and back, error handling or other python-specific stuff; I see no candidates for moving to core (obviously does not mean there are no). Code is mostly trivial, but hard to write (for a high-level language user) due to its amount and amount of things that need to be cared about (exceptions, failed memory allocation, decref/incref, rarely alloc/free, etc).
>
> Yes, due to the mismatch between the C code in core and the mostly
> object-oriented APIs of other languages, I had assumed that there is
> quite some boilerplate code. If any of that could be wrapped for reuse
> [by other language integrations], either through macros or helper
> functions, that would be desirable.

To do this I need to know other languages APIs. And have to create a “header” file if_interp.h really containing C code like if_py_both.h (never added any files to vim except for tests and runtime ones). And have to write something that uses such functions in e.g. if_ruby.c: there is no other way I can prove this generalization necessary. Thus if some ruby developer will want to extend ruby in a similar way it would be he who should write generalization: I just do not have enough knowledge and time to do this.

> I've heard the dislike of Vimscript, but much of that is probably due to
> the developers' relative unfamiliarity with it. For extensions that are
> closely related to text-editing, I found it quite nice and powerful, and
> there is no mismatch between the Vim API and another language to bridge.
> But I agree that other stuff (e.g. any interfacing with external systems
> or networking stuff) is extremely hard (though people have written
> hashing functions or JSON parsers in Vimscript).

This is not a problem with other stuff. I have to care about a big bunch of options like &ignorecase. I have to always remember about strange defaults (:map without nore while best practices tell nore should be default, :command without -bar while most commands will be more convenient to use with it and so on). I have to remember about escaping, and correct escaping (three common escaping variants for different cases). I have to forget about writing some neat code because it does not work reliably (e.g. :!python %). I have to generate a big number of lines just to take care of NUL case when in python it is just a regular symbol. Even in zsh it looks like a regular symbol though zsh uses just the same NUL-terminated strings. I have to know about some inconsistencies which will never be fixed due to backwards compatibility. Do you know that there is no option for filename completion in user-defined commands except for `custom…`? I have to hope user have sane configuration. Did you ever hear python `subprocess` module refusing to work because user has wrong configuration options? Also try to write down rules for dividing a string into a list of statements for python and for VimL and compare it. There is no such thing like a bunch of variants of handling newline depending on the bunch of variants of non-immediately-surrounding context† in python.

By the way, I have written both (adler32 hash (math was easier then base64 encoder/decoder which I used to write too) and both YAML (superset of JSON) and JSON parsers). And even a DSL for describing (and validating and completing) function and command arguments.

Developers do not like VimL not because they do not know it. They dislike it because they do. I cannot say there are no things that bug me in python, zsh or perl (high-level languages I know good enough to judge, best known first), but VimL is the only one that bugs me that much. I have to say though that when I was writing some simple things it seemed good, but “the farther into the forest, the thicker the trees”. I just did not know all the cases where my simple things can break and did not care about them.

† (NL in file, NL in file followed by spaces and backslash, NL in execute after commands not taking | as an argument, NL in execute after other commands, NL in execute after (built-in!) command taking expression (:echo) and unclosed quote, NL in execute after :function): main questions to answer are whether we are in a file, in a Ex prompt (BTW, have not explored this), in an :execute or in a function inside :execute. And what is the command we are computing arguments for.

> > Note though that VimL additions I can design will much likely smell python, but python has iterators to walk through vim.mappings without allocating a big list by a small chunks (you can�t do one alloc for all list items in C code and expect it not to crash as a consequence) and tp_as_mapping->mp_subscript to not allocate a big nested dictionary just for querying for autocommands['Powerline']['VimEnter']['*']. When you will see my VimL APIs in RFC please be captious.
>
> Certainly, Vimscript won't win any performance contest; other language
> interpreters are probably way more optimized.

They are. Problem is not with optimizations, problem is some set of features like concurrent processing and a number of places where globals are used.

[mattn]

On Thursday, May 9, 2013 7:54:38 AM UTC+9, ZyX wrote:
<snip>

I don't hope that capture() contains highlight attributes. It should be plain text. And I guess fixing redir nesting has problems.

1. There are plugins that depend on original behavior. This break compatible.
2. If skipped 'redir END' with exceptions, we can't close the nesting.

For about 2, one of solution is adding bang like 'redir! END'. but this will close all of redir nesting. So it's become un-expected behavior.

Thanks.
- Yasuhiro Matsumoto

[ZyX]

пятница, 10 мая 2013 г., 6:09:15 UTC+4 пользователь mattn написал:

> On Thursday, May 9, 2013 7:54:38 AM UTC+9, ZyX wrote:
> <snip>
>
> I don't hope that capture() contains highlight attributes. It should be plain text. And I guess fixing redir nesting has problems.
>
> 1. There are plugins that depend on original behavior. This break compatible.

Show me them then. It breaks only with nested redirections and it breaks only in the fashion that you get more expected output in place of error and empty string in variable. There is only one case when nested :redir breaks in other fashion: a sequence of :redir's in one function without :redir ENDs except for last one. I have never seen such a strange thing, but there is a way to fix this case: make :redir update variable immediately each time new :redir is called without waiting for :redir END (in any case you need something to store text redirected at this level, why not make it store in variable/register/file? I do not think anybody benefits from “The variable will remain empty until redirection ends.”). Unneeded :redirs will likely be GCed when function ends (third suggestion).

> 2. If skipped 'redir END' with exceptions, we can't close the nesting.

This is why I have 3-rd suggestion. Most of the time :redir is used to redirect to variables.

>
> For about 2, one of solution is adding bang like 'redir! END'. but this will close all of redir nesting. So it's become un-expected behavior.

You don’t need :redir! END. Just don’t leave all redirection options to user besides for redirection to variable inside a function. Or use :try..:finally.

If somebody has left stale redirections currently you will see unexpected behavior (error) with regular :redir, refer to the link in my first message. In addition to the text logged somewhere (but only to one place) until next :redir is called.

If somebody has left stale redirections with new proposal you will just have text logged somewhere (but only to one place as well) until session ends. It is very rare to see stale redirections: I have never seen them so far and never written something that leaves such redirections; and logging to some place with usual amount of messages is not too harmful. Thus advantages are greater then disadvantages.

To diagnose the issue :redir (without arguments) should print the stack of places where redirected text will go in order; it will only be actually written to the one last position.

>
> Thanks.
> - Yasuhiro Matsumoto

[Ingo Karkat]

On 10-May-2013 04:09 +0200, mattn wrote:

I beg your pardon, but I find your entire argument weak and unconstructive.

> On Thursday, May 9, 2013 7:54:38 AM UTC+9, ZyX wrote:
> <snip>
>
> I don't hope that capture() contains highlight attributes. It should
> be plain text.

Both ZyX's and my recent messages have already shown cases where this
additional information is essential. If you don't need it, ignore the
information / don't request it (my suggestion makes this optional via a
passed flag).

> And I guess fixing redir nesting has problems.

Your intuitions and guesses aren't really helpful in this otherwise very
factual discussion.

> 1. There are plugins that depend on original behavior. This break compatible.

I cannot think of a plugin that depends on having a nested :redir throw
an error. That's just bad coding practice, and it could easily be
corrected in the plugin. Do you have an actual example? As I've shown,
contrarily there are plugins that (when working together) are indeed
broken due to what I would call an implementation deficiency.

> [3 lines deleted]

Please take no offense, I value your opinion like every other, but
please support them with facts and concrete examples as much as
possible. Thank you.

-- regards, ingo

[Ingo Karkat]

On 09-May-2013 00:54 +0200, ZyX wrote:

> > > [10 sentences deleted]

>
>> I've heard the dislike of Vimscript, but much of that is probably due to
>> the developers' relative unfamiliarity with it. For extensions that are
>> closely related to text-editing, I found it quite nice and powerful, and
>> there is no mismatch between the Vim API and another language to bridge.
>> But I agree that other stuff (e.g. any interfacing with external systems
>> or networking stuff) is extremely hard (though people have written
>> hashing functions or JSON parsers in Vimscript).
>
> This is not a problem with other stuff. I have to care about a big bunch of options like &ignorecase. I have to always remember about strange defaults (:map without nore while best practices tell nore should be default, :command without -bar while most commands will be more convenient to use with it and so on). I have to remember about escaping, and correct escaping (three common escaping variants for different cases). I have to forget about writing some neat code because it does not work reliably (e.g. :!python %). I have to generate a big number of lines just to take care of NUL case when in python it is just a regular symbol. Even in zsh it looks like a regular symbol though zsh uses just the same NUL-terminated strings. I have to know about some inconsistencies which will never be fixed due to backwards compatibility. Do you know that there is no option for filename completion in user-defined commands except for `custom…`? I have to hope user have sane configuration. Did you
ever hear python `subprocess` module refusing to work because user has wrong configuration options? Also try to write down rules for dividing a string into a list of statements for python and for VimL and compare it. There is no such thing like a bunch of variants of handling newline depending on the bunch of variants of non-immediately-surrounding context† in python.

Good explanation of the pain points. My take on this is: Vimscript is
made primarily for interactive use, and therefore it is difficult to
write watertight plugin code that works under all circumstances.

It's great to be able to write :! python % when you know the Python in
your PATH is the right one, and your file doesn't contain spaces, but in
a plugin, you have to pull together configuration variables,
executable(), fnamescape(), etc., and it gets messy quickly. Likewise,
:map is often right for ad-hoc mappings, but all plugins should use
:noremap. Same for 'ignorecase'.

> [2 sentences deleted]

>
> Developers do not like VimL not because they do not know it. They dislike it because they do. I cannot say there are no things that bug me in python, zsh or perl (high-level languages I know good enough to judge, best known first), but VimL is the only one that bugs me that much. I have to say though that when I was writing some simple things it seemed good, but “the farther into the forest, the thicker the trees”. I just did not know all the cases where my simple things can break and did not care about them.

For me, it's again the trade off (or historical accident) that was made
in favor of easy migration from ad-hoc macros to mappings to
full-fledged personal plugins to published plugins. The great thing is
that it's Vimscript all down that path, the downside is that what was a
simple one-liner has to be rewritten to a complex dozen-line function
(with stuff that has to be tediously learned) in order to be fully
robust under most users' customizations.

On the other hand, maybe we're holding ourselves to too high ideals.
Most plugins I've tried and reviewed (even from venerable long-time
contributors like Tim Pope or Dr. Chip, though admittedly much less so)
have certain deficiencies or break when used with obscure settings or
environments, but most people are happy to use them, anyway. So I tend
to defend Vimscript on its "worse is better" philosophy when language
purists try to bash it too much. But I would be happy if a different
approach (like your planned much-extended Python interface) turns out to
be as productive without so many pitfalls.

> [11 sentences deleted]

-- regards, ingo

[ZyX]

> I cannot think of a plugin that depends on having a nested :redir throw
> an error.

Refer to my message. Nested :redir throws an error only when context is switched, it is possible to have

redir => commands
command
redir => mappings
map
redir => imappings
imap
" and so on
redir END

> That's just bad coding practice, and it could easily be
> corrected in the plugin.

I completely agree there. Also refer to my message, I explained there how this situation may easily be worked around (just making it always write to target when any :redir is issued is enough for plugins to work; finishing redirections to local variables after function finishes will delete stale redirs in all cases that make sense to write from my point of view and even not finishing should be rare enough and not result in much harm: at maximum vim killed by OOM killer or a few MiBs long file). I have never actually seen plugins with the above code though.

[mattn]

I'm thinking capture() is not replacement of redir. So I'm thinking this issue should be separated with redir's issue.

1. The redir have a problem currently, But capture() doesn't.
2. Reasonable to implement.
3. Easy to use. Possible to write per one line.
4. I know there are some users who want capture().

I'm thinking it's good idea to solve some problem.

Thanks.

[Shougo]

2013年6月17日月曜日 11時49分10秒 UTC+9 mattn:

Yes. I think so, too.

1. I can use capture() easily than :redir.

Example:
let foo = capture(":mes")

redir => foo
mes
redir END

Which is the easy to understand?

2. And, the capture() is used for function arguments.

Example:
echo remote_expr("GVIM1", "capture(':mes')")

I think function is better than command to use it in Vim script.

3. I think the patch works well(because I tested it).
But to fix redir is no patches, it is too far to use it in my plugins.