[RFC] Some more python and VimL interfaces

[ZyX]

This is a description of proposed new python interfaces.

==================
New python objects
==================

vim.mappings/{buffer}.mappings:
Dictionary-like object mapping mode characters to vim.modemappings objects
below. Unlike vim.buffers iteration is being done for keys.

modemappings:
Dictionary-like object mapping expanded (see vim.special_expand() below)
strings to vim.Mapping objects. Assigning Mapping objects to some key works
like :*map. `sid` attribute is ignored when assigning. Unlike vim.buffers
iteration is being done for keys.

mapping:
Namedtuple-like object with the following attributes:
rhs Rhs of the mapping.
sid Script ID.
silent True or False, tells whether mapping is silent.
expr True or False, tells whether it is <expr> mapping.
noremap Depends on the numeric constant:
vim.MAPPING_REMAP False (zero) constant, mapping is
remappable
vim.MAPPING_NOREMAP True (non-zero), mapping is
noremappable
vim.MAPPING_SCRIPT True (non-zero), mapping is <script>
one
No .mode or .buffer attributes: they are expressed in using appropriate
vim.mappings[mode] (buffer.mappings[mode]) objects.

vim.commands/{buffer}.commands
Mapping-like object mapping command names to vim.command objects. Assigning
vim.Command object to the key acts like :command, `sid` attribute is
ignored. Unlike vim.buffers iteration is being done for keys.

command:
Namedtuple-like object with the following attributes:
command Command string.
sid Script ID.
nargs One of constants:
vim.COMMAND_ARGS_NO
vim.COMMAND_ARGS_ONE
vim.COMMAND_ARGS_ANY
vim.COMMAND_ARGS_ONE_OR_NO
vim.COMMAND_ARGS_SOME
complete String or None.
range Number or None. Number is either vim.COMMAND_RANGE_CURRENT,
vim.COMMAND_RANGE_WHOLE or non-negative integer. All
constants are negative integers. None is used when neither
range nor count is allowed.
count True or False. Determines whether .range attribute really
describes -count argument.
bang, bar, register
True or False.

vim.autocommands:
Mapping-like object mapping autocommand groups to vim.Augroup objects.
Iteration is done for keys. Autocommands without groups are autocommands
with group None.

augroup:
Mapping-like object mapping event names to list of vim.Autocommand objects.
Iteration is done for keys. Methods:
.append_autocmd(event, autocommand)
.del_autocmds(event, pattern=None)
.clear(event=None)
As usual, `sid` attribute is ignored when using above methods. Item
assignment works by calling first .clear(event=key) and then a sequence of
.append_autocmd(event=key, autocmd=item).

autocommand:
Namedtuple-like object with the following attributes:
command Command string.
sid Script ID.
nested True or False.
pattern Glob.

vim.signs:
Mapping-like object mapping sign names to vim.Sign attributes. Iterates over
keys, supports item assignment.

sign:
Namedtuple-like object with the following attributes:
icon None or file name.
linehl None or string.
text String.
texthl None or string.

{buffer}.signs:
Mapping-like object mapping place ids to signplace objects (note: they is no
constructor for this). Iterates over signplace objects in order they are
shown in the buffer. Methods:
place(line, name, id=None)
Place given sign on the given line. Returns id which is equal to id
argument if it was given or is a new unique id.
unplace(line=None, id=None)
Unplace sign from the given line or id or all signs if no arguments
were given. If both line and id are present then it verifies whether
sign with given id is placed on the given line and either unplaces
it or raises ValueError.

signplace:
Object with the following attributes:
id Identifier.
line Line number.
name Names of signs: keys in vim.signs.
. Also has method .jump().

vim.scripts:
Sequence-like object that removes the need of parsing :scriptnames. Maps
script numbers (subtracted by one) to file names.

vim.jumps:
Sequence-like object that removes the need of parsing :jumps. Contains jump
objects.

{buffer}.changes:
Like vim.jumps, but for :changes in the given buffer.

jump:
Objects with the following attributes:
line Line number.
col Column number.
file None or file name.
buffer None (file not loaded) or vim.buffer object.
. Also has method .jump().

vim.current.jump:
Index of the current jump in vim.jumps.

Notes:
- “Namedtuple-like” means that all attributes are RO and that there are no
methods.
- “Ignored when assigning” means “set to sid which is current when assigning”.

==================
New VimL functions
==================

maplist([{mode}[, {bufnr}]])
Returns a list of strings suitable for maparg().

cmdarg({cmdname}[, {bufnr}])
Returns a dictionary with the following keys, works with partial commands:
name Full {cmdname}.
command Command string.
sid Script ID.
nargs String constant or empty string.
complete String, possibly empty.
range String constant or empty string, stands only for -nargs.
count String constant or empty string, stands only for -count.
bang, bar, register, buffer
One or zero.

cmdlist([{bufnr}])
Returns a list of command names suitable for cmdarg().

aulist({event}[, {augroup}[, {pattern}]])
Returns a list of dictionaries with the following keys:
event {event}
augroup Autocmd group.
patterns List of globs.
command Command string.
nested One or zero.
sid Script ID.

auglist()
Returns a list of autocmd groups.

signarg({name})
Returns a dictionary with the following keys:
name {name}
icon Icon file or empty string.
linehl String or empty string.
texthl String or empty string.
text String.

signlist()
Returns a list of sign names suitable for signarg().

signplacelist({bufnr})
Returns a list of signs placed in the given buffer: dictionaries with keys
name Name suitable for signarg().
id Sign ID.
line Line number.

getjumps()
Returns a list of dictionaries with the following keys:
line Line number.
col Column number.
file File name.

getchanges({bufnr})
Returns a list of lists, latter lists are suitable for cursor(): look like
([{line}, {col}]).

Note: {bufnr} is {expr} argument from bufnr() function.

=======================================
Python classes that are made accessible
=======================================

vim.Mapping(rhs, noremap=vim.MAPPING_NOREMAP, silent=False, expr=False)
Returns mapping object described above. New mappings that are created using
vim.Mapping are then to be assigned to appropriate
`{vim,buffer}.mappings[mode][lhs]` thus there are no attributes like `mode`,
`buffer` or `lhs`: they are determined by appropriate keys and objects when
assigning. `sid` attribute is created automatically (though in any case it
is ignored when assigning).

vim.Command(command, nargs=vim.COMMAND_ARGS_NO, complete=None, range=None,
count=False, bang=False, bar=True, register=False)
Returns command object described above. New commands that are created using
vim.Command are then to be assigned to `{vim,buffer}.commands[cmd]` thus
there are no attributes like `name` or `buffer`. `sid` attribute is created
automatically (though in any case it is ignored when assigning).

vim.Autocommand(command, pattern, nested=True)
Returns new autocmd group as described above.

vim.Sign(text, icon=None, linehl=None, texthl=None)
Returns new sign as described above.

vim.Dictionary(dict)
vim.Dictionary(iterable)
vim.Dictionary(**kwargs)
Returns a vim.dictionary object. Is subclassable.

vim.List(iterable)
Returns a vim.list object. Is subclassable. Main reason for creating this and
previous classes is not subclassability (it is highly optional) or ability
to define new lists without using `vim.bindeval`, but ability to use
`isinstance(vim.bindeval("var"), vim.Dictionary)`.


Note: here are different (more convenient) defaults:
Mapping(noremap=vim.MAPPING_NOREMAP) in place of vim.MAPPING_REMAP
Command(bar=True) in place of False
Autocommand(nested=True) in place of False

=========================
Auxilary/helper functions
=========================

vim.special_expand() + special_expand() :: string -> string
Expands special characters like `<C-a>` in the string. Currently it is done
in VimL by using `eval('"'.escape(str, '\<"').'"')` which is rather hacky.

vim.match_glob() :: glob, string -> boolean
match_glob() :: string, glob -> boolean
Match string against glob expression. Order is different in python and VimL:
in python it is taken from `re.match(pattern, string)`, in VimL from
`matchstr({expr}, {pat})`.

vim.find_buffer() :: string -> vim.buffer
Like vim.buffers[vim.bindeval("bufnr(string)")].

=================================
Changes to existing functionality
=================================

- Iterators for *.windows and vim.tabpages that allow

for window in vim.windows:
window.close()

like it is allowed for buffer list.

- {buffer}.wipe(), {window}.close() and {tabpage}.close() methods.
Alternative: `del vim.buffers[number]`, `del {windowlist}[number]` and
`del vim.tabpages[number]`.
- {dictionary}.items(), {dictionary}.values(), {dictionary}.pop() and
{dictionary}.update() (last is most useful).
- Doing `item in {dictionary}` without doing `{dictionary}[item]`.
- `iter({dictionary})` iterating over keys.
- {function}.sid and {function}.code attributes.
- {buffer}.valid, {window}.valid, {tabpage}.valid, {windowlist}.valid and
{function}.valid attributes.
- {window}.tabpage attribute and {window}.number attribute working for
non-current tab pages.
- vim.eval(expr, number_strings=True): sometimes python objects are more desired
then what vim.bindeval returns, but numbers represented as strings spoil it
all.

[Bram Moolenaar]

ZyX wrote:

> This is a description of proposed new python interfaces.

Well, that's a very long list. Do we really need all of this?
Let's at least order by usefulness. First add things that help
plugin writers most. I'm assuming that very few users will type
Python commands, I expect them to be used in scripts.
We could lower priority for things that are already possible but
perhaps not in a very nice way.


--
Not too long ago, unzipping in public was illegal...

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[ZyX]

> Well, that's a very long list. Do we really need all of this?
> Let's at least order by usefulness. First add things that help
> plugin writers most. I'm assuming that very few users will type
> Python commands, I expect them to be used in scripts.
> We could lower priority for things that are already possible but
> perhaps not in a very nice way.

They are all possible (except for some edge cases like `<script>` mappings (maparg(dict=1) does not distinguish between `nore` and `<script>` mappings), `<LT>F1>` vs `<F1>` in lhs when using `:redir` to get list of mappings, determining whether some column in :jumps output is file name or line in the buffer, determining function name used for custom/customlist completion and so on). And all help plugin writers. The least useful for me would be jumps thing, but other plugin writers would not tell the same. Like all python interface improvements this reduces the need for `vim.command` and `vim.[bind]eval`. Main target is to reduce VimL in scripts that require python support to just one line `:python import foo; foo.setup()` call and have nice python code. Then python-enabled plugins are easily installable by pip with the single addition of that line to vimrc. Additionally python-only plugins will be able to be written in python and not in python+VimL. Currently people are not doing the former because this still requires a bunch of things written in VimL and wrapping this into `vim.command('''{multi-line}''')` does not look good and the latter is just not possible. There are a few more things to be added for this to work:

vim.functions:
Mapping-like object mapping user function names to vim.Function objects.
Accepts s: and <SID> in keys, but always expands it when creating objects.

function:
Callable object with the following attributes:
name Function name
code Function code
sid Script ID
is_dictionary True or False.
dict For convenience vim.Dictionary can be assigned to this
attribute in order to forget about specifying self= arg.
Supports assigning any mapping-like object that can be
converted.

vim.Function(args, code, name=None, is_dictionary=False, dict=None)
Creates function object like above (name=None means anonymous function, name
is automatically generated). Code is a iterable of strings.

vim._gen_function(callable, args=())
Generates vim.Function object from python callable object. At this point
can only do something like the following (but not exactly this; and also
written in C):

id = vim._callables.save(callable)
return vim.Function(args, ('python import vim', 'return pyeval("vim._callables[{0}]({1})")'.format(id, ','.join((arg+'=vim.bindeval("a:'+arg+'")' for arg in args)))), dict={})
# Rather hacky, but standard solution is preferred over 100500 ways
# plugin writers can reimplement it just to call python function
# from various places where it is needed.

. Is not supposed to be used directly: it is for
vim.Command(command=callable), vim.Autocommand(command=callable) and
vim.Mapping(rhs=callable, expr=True).

In vim.Autocommand(command=callable) callable receives buf=N (for <abuf>) and
match=string (for <amatch>) keyword arguments.

In vim.Command(command=callable) callable automatically receives a list of
arguments in args keyword argument and additionally count=N or range=(N, M),
bang=True/False and reg=char keyword arguments if appropriate.

In vim.Command(complete=callable) callable receives three arguments as usual
completion function. It is the only callable that receives non-keyword
arguments.

In vim.Mapping(rhs=callable, expr=True) callable is called without arguments.

vim.options, {buffer}.options and {window}.options should also support
assigning python functions when it makes sense:
&statusline: will use something like %! and receive vim.window argument.
&indentexpr, &formatexpr, &balloonexpr, &foldexpr, &diffexpr, &includeexpr,
&patchexpr, &printexpr
&omnifunc, &completefunc, &operatorfunc
. As when deleting commands, mappings, autocommands and resetting options no
GC is triggered this will use pyeval() directly as long as possible (i.e. no
need to make GC’ing functions also GC objects from vim._callables as functions
will not be garbage-collected). vim._callables will have .remove() method in
case somebody will need to GC things manually. vim._callables won’t contain two
references to same object.

[Bram Moolenaar]

If most things are already possible, then it should be possible to write
Python functions that work as a nice interface for the functionality.
Thus not change the C implementation of the Python interface.

We also need to keep in mind that all the Vim functionality is nicely
documented and there are examples, while if we add a different way to do
this in Python this requires documenting how that works. Thus a script
writer will have two sets of functionality to learn to use. It will be
good to have "how it's done in Vim" to be very similar to "how it's done
in Python". Otherwise it gets very complicated.

> vim.functions:
> Mapping-like object mapping user function names to vim.Function objects.
> Accepts s: and <SID> in keys, but always expands it when creating objects.

This would be a nicer way than using vim.eval(), right?

For the "expr" options this would currently be done by using pyeval(),
right?

> . As when deleting commands, mappings, autocommands and resetting options no
> GC is triggered this will use pyeval() directly as long as possible
> (i.e. no need to make GC�ing functions also GC objects from
> vim._callables as functions will not be garbage-collected).
> vim._callables will have .remove() method in case somebody will need
> to GC things manually. vim._callables won�t contain two references
> to same object.

--
GUARD #1: Where'd you get the coconut?
ARTHUR: We found them.
GUARD #1: Found them? In Mercea? The coconut's tropical!
ARTHUR: What do you mean?
GUARD #1: Well, this is a temperate zone.
The Quest for the Holy Grail (Monty Python)

[Andy Kittner]

On Sat, May 11, 2013 at 03:50:50PM +0200, Bram Moolenaar wrote:
> [...]

>If most things are already possible, then it should be possible to write
>Python functions that work as a nice interface for the functionality.
>Thus not change the C implementation of the Python interface.

Where it is feasible to implement it directly in python that sounds like
a good idea to me since it would probably ease the future maintenance
and allow more people to contribute to it.

Though for some of the suggested additions I imagine that it will
negatively impact the performance and/ or be difficult to implement (e.g.
getting the escaping right for eval(), reliably parsing the output of
commands etc.). But that's something where ZyX probably can give more
qualified comments than I could.

>
>We also need to keep in mind that all the Vim functionality is nicely
>documented and there are examples, while if we add a different way to do
>this in Python this requires documenting how that works. Thus a script
>writer will have two sets of functionality to learn to use. It will be
>good to have "how it's done in Vim" to be very similar to "how it's done
>in Python". Otherwise it gets very complicated.

I see that differently, currently people will have to learn quite a bit of
VimL to get anything useful done with the vim interface. For someone who is
new to vim, but already knows python, a "pythonic" interface is much
more useful then having to do eval("vimL_code()") all the time.
Some of the people I converted to vim users will probably never touch
VimL, but if it where possible to write plugins purely in python with a
pythonic interface they could write plugins for themselves as well
(since I also converted them to python ;))

So, from that point of view the additions proposed by ZyX sound very
good to me (whether implement in C or python, as long as it works "out
of the box").

Regards,
Andy
--
On the sixth day God created man
On the seventh day, man returned the favor.

[ZyX]

> We also need to keep in mind that all the Vim functionality is nicely
> documented and there are examples, while if we add a different way to do
> this in Python this requires documenting how that works. Thus a script
> writer will have two sets of functionality to learn to use. It will be
> good to have "how it's done in Vim" to be very similar to "how it's done
> in Python". Otherwise it gets very complicated.

Nope. It is now when script writer has to know “how it is done in Vim” in addition to knowing vim<-->python interface. Target of the new interfaces is that plugin writer has to know only if_pyth.txt for all things he may need to do. RFC is here to discuss how implementing my proposal completes this target and what other things need to be included.

And adding python interface that is the same as VimL is
1. throwing away some of the benefits python provides
2. intentionally creating interfaces with design flaws
3. intentionally ignoring python principles like “explicit is better then implicit” (for 2. and 3. refer to my explanation of why *.options are written the way they are written).

Not saying that I can’t imagine how to write VimL in python.

> For the "expr" options this would currently be done by using pyeval(),
> right?

For *expr options, and map-<expr> this will be done using pyeval() even after the patch; for :command it is now and will after be done using :python. What the patch will add is

a) Standard interface.
b) No need in knowing and using VimL. Exact implementation will be hidden.
c) When core vim features will allow this, implementation will change.

About c): I still have not forgot about suggestion I have written earlier about function references: making them more generic by embedding struct func_S pointer in place of char* with function name. There are two additions since then: no s: in function references, reference counting for all (not only anonymous) functions (implies holding normal user functions in a regular dict_T like g: one, but hidden(?)), reference counting in struct func_S and normal names for anonymous functions (they will not be added to dict_T as with reference count and all other data in a structure attached to typval_T* unique names and holding anonymous functions with other user functions are no longer needed).

[ZyX]

> Where it is feasible to implement it directly in python that sounds like
> a good idea to me since it would probably ease the future maintenance
> and allow more people to contribute to it.
>
> Though for some of the suggested additions I imagine that it will
> negatively impact the performance and/ or be difficult to implement (e.g.
> getting the escaping right for eval(), reliably parsing the output of
> commands etc.). But that's something where ZyX probably can give more
> qualified comments than I could.

As I said you can write all of this with current interfaces. There are some corner cases fallback implementation cannot handle (I have written down some of them in some of the previous messages) (because VimL itself cannot handle them), but it is possible. More, VimL plugin + python module with the fallback implementation is on my TODO list.

Negative impact to the performance will be there, but nothing as critical as fallback implementation to vim.bindeval would show.

> I see that differently, currently people will have to learn quite a bit of
> VimL to get anything useful done with the vim interface. For someone who is
> new to vim, but already knows python, a "pythonic" interface is much
> more useful then having to do eval("vimL_code()") all the time.
> Some of the people I converted to vim users will probably never touch
> VimL, but if it where possible to write plugins purely in python with a
> pythonic interface they could write plugins for themselves as well
> (since I also converted them to python ;))

Yes, this is the target.

[ZyX]

> Negative impact to the performance will be there, but nothing as critical as fallback implementation for vim.bindeval would show.

* Negative impact will not be that great only in absolute measures and because I don’t need to pass about a hundred KiBs with these new interfaces like I needed for vim.bindeval. In relative measures it will likely be greater due to the use of :redir and parsing output. I am talking here about querying information. Difference in speed of creation of commands and mappings using new interfaces will be insignificant and much less in both absolute in relative measures.

I just mentioned a thing I have forgot: interface to highlight groups.

[ZyX]

Created a gist with suggestions: https://gist.github.com/ZyX-I/5561409#file-rfc-rst.

[Bram Moolenaar]

ZyX wrote:

> > We also need to keep in mind that all the Vim functionality is nicely
> > documented and there are examples, while if we add a different way to do
> > this in Python this requires documenting how that works. Thus a script
> > writer will have two sets of functionality to learn to use. It will be
> > good to have "how it's done in Vim" to be very similar to "how it's done
> > in Python". Otherwise it gets very complicated.
>
> Nope. It is now when script writer has to know �how it is done in
> Vim� in addition to knowing vim<-->python interface. Target of the
> new interfaces is that plugin writer has to know only if_pyth.txt for
> all things he may need to do. RFC is here to discuss how implementing
> my proposal completes this target and what other things need to be
> included.

That is impossible. You may skip some of the list, dict and other Vim
language features, but when it comes to options, regexp, autocommands
and many, many other things you need to know how Vim works. Making a
Python interface for them won't change how Vim works.

For example, you can make some easy way in Python to set the 'tabstop'
option. To know what the effect is you still need to know about the Vim
option. Now, since that is a number option it's easy, but when you get
to the more complicated options you are just doing string manipulation.

Also keep in mind that the user will still have to type most Vim
commands. E.g., making a Python way to set an option that is a list of
names using a Python list might be nice, but the user still sees the
result as a string and has to type the string when he wants to change it
manually. Thus you won't be able to just omit this part of Vim
commands.

We really need to concentrate on the basic, generic mechanisms. Not try
to make a Python equivalent for every single Vim item. I mean, we don't
want to replicate every Ex command in Python, right? But we could add a
Python command that makes it easier to execute Ex commands, e.g. by
passing the arguments as a list instead of as a string, and indicate
whether special chars need to be escaped.

That's only indireclty related to the Python interface. Funcion
references are a bit of a hack.

--
GUARD #2: It could be carried by an African swallow!
GUARD #1: Oh, yeah, an African swallow maybe, but not a European swallow,
that's my point.
GUARD #2: Oh, yeah, I agree with that...

[ZyX]

>That is impossible. You may skip some of the list, dict and other Vim
>language features, but when it comes to options, regexp, autocommands
>and many, many other things you need to know how Vim works. Making a
>Python interface for them won't change how Vim works.
>
>For example, you can make some easy way in Python to set the 'tabstop'
>option. To know what the effect is you still need to know about the Vim
>option. Now, since that is a number option it's easy, but when you get
>to the more complicated options you are just doing string manipulation.

Excluding eval.txt and most common commands is still a big achievement. Also
note that there are no normal VimL interfaces for what I suggest: I guess after
cmdarg() somebody will like to create cmdadd() and cmddelete(): nobody likes
:execute (I am referring to one of the latest suggested patches).

>Also keep in mind that the user will still have to type most Vim
>commands. E.g., making a Python way to set an option that is a list of
>names using a Python list might be nice, but the user still sees the
>result as a string and has to type the string when he wants to change it
>manually. Thus you won't be able to just omit this part of Vim
>commands.

I am completely sure I can hide magic done when using python callables as
mapping rhs or action in a command. Passing sequences and using sequence-, map-
or namedtuple-like objects in options is also a good idea. Developers are smart
enough to guess why after setting &rtp to ('/usr/share/vim/vim73', '~/.vim') it
appears as '/usr/share/vim/vim73,~/.vim'. And they won’t have to deal with
manual escaping.

I can’t omit this part of Vim commands. But I can hide it. Also note that we are
not talking about user. There is exactly no difference for user between
modifying option initally set by
import re
import json
lcs = vim.eval('&lcs')
lcs = re.sub(',?eol:.,?', lcs, ',')
lcs += ',eol:$'
lcs = lcs.strip(',').replace(',,', ',')
vim.command("let &lcs="+json.dumps(lcs))
,
vim.command("set lcs+=eol:$")
,
# Same as first, but with two modifications:
…
lcs = vim.options['lcs']
…
vim.options['lcs']=lcs
and
vim.options['lcs']['eol']='$'
. Neither there is a difference in the result. But there is difference for
developers as only the last interface is most convenient one.

>We really need to concentrate on the basic, generic mechanisms. Not try
>to make a Python equivalent for every single Vim item. I mean, we don't
>want to replicate every Ex command in Python, right? But we could add a
>Python command that makes it easier to execute Ex commands, e.g. by
>passing the arguments as a list instead of as a string, and indicate
>whether special chars need to be escaped.

I do not want to replicate every Ex command. I do not want to replicate Ex
commands at all: I am trying to create an interface which is more convenient
then Ex commands. To achieve e.g. suggested mapadd() you need to write about 15
lines of code. More if you are writing in python and want to support callables.
My suggestion limits this to only one line by providing similar interface
out-of-the-box.

I am also unsure what are “basic, generic mechanisms”. There are four things
that Vim lacks from my point of view (more significant first): threading and/or
multiprocessing, non-regex syntax, full support for whatever keys you can find
on your keyboard and normal API. First three are too hard for me currently thus
I am targeting the last one.

By creating “a Python command that makes it easier to execute Ex commands” we
will just add another source of strange bugs that all have a fix that looks like
“add another `else if strcmp(…)`”: vim commands are not consistent at all about
how should “argument” look and what, where and how a string should be escaped.
With `vim.command()` we at least can peep at VimL code. With my interface you
will be using C functions directly and thus have no problems with escaping in
addition to having pythonic interface. New function (python has no commands)
will trade benefit for being error-prone (less benefit over `vim.command` - less
bugs, more benefit - more bugs) and still be non-pythonic and not very useful
compared to `vim.command`.

[ZyX]

I just explained what I mean in c). It should be relatively easy though: every
or almost every usage of function references is marked with VAR_FUNC macros.

[ZyX]

>I can’t omit this part of Vim commands. But I can hide it. Also note that we
>are not talking about user. There is exactly no difference for user between
>modifying option initally set by
> import re
> import json
> lcs = vim.eval('&lcs')
> lcs = re.sub(',?eol:.,?', lcs, ',')
> lcs += ',eol:$'
> lcs = lcs.strip(',').replace(',,', ',')
> vim.command("let &lcs="+json.dumps(lcs))
>,
> vim.command("set lcs+=eol:$")
>,
> # Same as first, but with two modifications:
> …
> lcs = vim.options['lcs']
> …
> vim.options['lcs']=lcs
>and
> vim.options['lcs']['eol']='$'
>. Neither there is a difference in the result. But there is difference for
>developers as only the last interface is most convenient one.

This is too hard to implement in C though and will break backwards compatibility
since my patch for *.options was already included. Should be written in python
as a module shipped with vim.

[Bram Moolenaar]

It appears you agree with my earlier reply. We need some discussion,
but I think eventually we want the same thing.

--
CUSTOMER: You're not fooling anyone y'know. Look, isn't there something
you can do?
DEAD PERSON: I feel happy... I feel happy.
[whop]
CUSTOMER: Ah, thanks very much.
MORTICIAN: Not at all. See you on Thursday.
CUSTOMER: Right.

[Bram Moolenaar]

ZyX wrote:

> >That is impossible. You may skip some of the list, dict and other Vim
> >language features, but when it comes to options, regexp, autocommands
> >and many, many other things you need to know how Vim works. Making a
> >Python interface for them won't change how Vim works.
> >
> >For example, you can make some easy way in Python to set the 'tabstop'
> >option. To know what the effect is you still need to know about the Vim
> >option. Now, since that is a number option it's easy, but when you get
> >to the more complicated options you are just doing string manipulation.
>
> Excluding eval.txt and most common commands is still a big
> achievement. Also note that there are no normal VimL interfaces for
> what I suggest: I guess after cmdarg() somebody will like to create
> cmdadd() and cmddelete(): nobody likes :execute (I am referring to one
> of the latest suggested patches).

Yes, trying to avoid :execute is good.

Yes, that looks nicer. At the same time it goes against your original
intend: replace Vim stuff with standard Python. The first alternative
uses standard Python functions, except for vim.eval() and vim.command().
The second one requires the developer to lookup what vim.options
actually does and apparently there is some magic that parses an option
so that a part of a string can be easiliy replaced. So you added Vim
magic inside the Python interface.

I rather add some Python functions that handle options. Then
vim.options returns the raw number/string, you use the Python functions
to manipulate them and then set the option with vim.options again.

The knowledge that something is a comma separated string then lies with
the developer, and he already knows that. He does need to learn the
Python function for manipulating option strings, but no Vim stuff.

The problem with vim.command is that it's not always easy to escape
special characters. At least for commands that take a file name
argument this can me made easier. But it can't be done in general,
since Ex commands are inherently difficult to parse, and thus difficult
to produce.

The best way to discover what we need in the Python API is probably to
go over existing implementations and find out what functions they define
to interact with Vim. Those are pieces of complexity that are repeated
often enough that the writer decided to make a function for it. Instead
of having every developer write their own functions, those should be
provided by the Python API.

--
CUSTOMER: Well, can you hang around a couple of minutes? He won't be
long.
MORTICIAN: Naaah, I got to go on to Robinson's -- they've lost nine today.
CUSTOMER: Well, when is your next round?
MORTICIAN: Thursday.
DEAD PERSON: I think I'll go for a walk.

[ZyX]

>> >That is impossible. You may skip some of the list, dict and other Vim
>> >language features, but when it comes to options, regexp, autocommands
>> >and many, many other things you need to know how Vim works. Making a
>> >Python interface for them won't change how Vim works.
>> >
>> >For example, you can make some easy way in Python to set the 'tabstop'
>> >option. To know what the effect is you still need to know about the Vim
>> >option. Now, since that is a number option it's easy, but when you get
>> >to the more complicated options you are just doing string manipulation.
>>
>> Excluding eval.txt and most common commands is still a big
>> achievement. Also note that there are no normal VimL interfaces for
>> what I suggest: I guess after cmdarg() somebody will like to create
>> cmdadd() and cmddelete(): nobody likes :execute (I am referring to one
>> of the latest suggested patches).
>
>Yes, trying to avoid :execute is good.
>
>> >Also keep in mind that the user will still have to type most Vim
>> >commands. E.g., making a Python way to set an option that is a list of
>> >names using a Python list might be nice, but the user still sees the
>> >result as a string and has to type the string when he wants to change it

>> >manually. Thus you won't be able to just omit this part of Vim
>> >commands.
>>

>> I am completely sure I can hide magic done when using python callables
>> as mapping rhs or action in a command. Passing sequences and using
>> sequence-, map- or namedtuple-like objects in options is also a good
>> idea. Developers are smart enough to guess why after setting &rtp to
>> ('/usr/share/vim/vim73', '~/.vim') it appears as
>> '/usr/share/vim/vim73,~/.vim'. And they won’t have to deal with
>> manual escaping.
>>

>> I can’t omit this part of Vim commands. But I can hide it. Also
>> note that we are not talking about user. There is exactly no
>> difference for user between modifying option initally set by
>> import re
>> import json
>> lcs = vim.eval('&lcs')
>> lcs = re.sub(',?eol:.,?', lcs, ',')
>> lcs += ',eol:$'
>> lcs = lcs.strip(',').replace(',,', ',')
>> vim.command("let &lcs="+json.dumps(lcs))
>> ,
>> vim.command("set lcs+=eol:$")
>> ,
>> # Same as first, but with two modifications:
>> …
>> lcs = vim.options['lcs']
>> …
>> vim.options['lcs']=lcs
>> and
>> vim.options['lcs']['eol']='$'
>> . Neither there is a difference in the result. But there is difference for
>> developers as only the last interface is most convenient one.
>

>Yes, that looks nicer. At the same time it goes against your original
>intend: replace Vim stuff with standard Python. The first alternative
>uses standard Python functions, except for vim.eval() and vim.command().
>The second one requires the developer to lookup what vim.options
>actually does and apparently there is some magic that parses an option
>so that a part of a string can be easiliy replaced. So you added Vim
>magic inside the Python interface.

Original intend will stop with *.options: in any case nobody can write
`vim.options['lcs']['eol']` without first looking at `:h 'lcs'`. This should be
a wrapper.

>I rather add some Python functions that handle options. Then
>vim.options returns the raw number/string, you use the Python functions
>to manipulate them and then set the option with vim.options again.

Functions cannot achieve `vim_ext.options['lcs']['eol']='$'`, and I really see
this more convenient then `vim_ext.set_listchars(eol='$')` or whatever. But
python classes can.

Not really the best. As I said, (almost) nobody is defining mappings, commands,
functions and autocommands in python code: because now it is easier to write
*.vim file in addition to a python module. If you try to write such things in
a python module it will result in having *.vim file embedded as a bunch of
strings in python.

I can say that in aurum most used thing is vim_export function that sends data
to caller VimL function. But it was written two years ago, now I would have
written a generator function generating pyeval- or fallback :py-based wrappers
around python callables.

In powerline *.options would be one of the most useful things (after interfaces
existed before my patches), getbufvar is used very commonly. Second is bindeval
again. *.vars will likely be the third. But if there was fast(!) vim.hlgroups it
would be on the second position: we had to replace checking whether some group
is a) defined and b) defined with some exact properties with assuming highlight
is no longer defined after some events because this is too slow even with
vim.bindeval. This is enough to be done once on each statusline redraw, but even
in this case using `hlexists()` and a bunch of synIDattr calls is too slow.

Conque would get some (not much) benefit from bindeval and *.options (if it used
it) and vim.hlgroups, but much more then that Conque needs non-regex-based :syn
and more async options.

I mean, for existing code bindeval, *.options and *.vars (and maybe not written
vim.current.vars (l: or g:) and vim.current.avars (and for completeness
vim.current.svars)) is enough. You should look at what was left coded in VimL
and think why it was coded there. The only plugin intentionally avoiding VimL as
much as it makes sense is powerline. It has some python initialization (not
needed if installed with pip), checks what python version should be used (cannot
obviously be coded in python; but with pip user will already know python
version), adds workarounds for old Vims without pyeval() and has 44 lines with
function, option and autocommand definitions. With new interfaces I can replace
half of them with nicely looking python code leaving only autocommand
manipulations.

On ruby side there is Command-T. It has only one *.vim file containing command
definitions, mapping definitions, a bunch of wrapper functions and ruby
initialization (this would not be needed if using gem I guess).

I.e. to get rid of *.vim files it is enough to create convenient interfaces for
mappings, commands and autocommands; hlgroups is here for convenience and speed
reasons; others are for convenience: avoiding both :redir and :execute (i.e.
vim.command). Second problem is non-regex syntax: with conceal feature both
Conque and Command-T started to use meaningless markers to define syntax
regions, before Command-T seems to not highlight matches at all (at least now it
does not without conceal), Conque uses `\%Nl\%>Nc\%<Nc` unless configured to use
conceal (for two reasons: with concealing it is much faster, but adds garbage
when trying to copy/paste).

Pyinteractive has usual commands, autocommands, mappings (and menus which
I missed), wrapper functions for completion and usual python initialization
required when using usual way of distributing through vim.org: nothing special.
But there is one thing I also missed: requirement to use vim.command to get
colored highlighting. It also calls input() through vim.eval, nothing more
interesting.

Python-mode-klen also needs colored highlighting and user interaction (now
getchar). It defines some interfaces in question (mappings, commands and
autocommands) using vim.command though. Additionally there are buffer and window
manipulations. Some things that are easy to write in python were written in
VimL. It also has hacks for doing something repetiavely.

Thus fine buffer and window manipulation options would be fine and should be
added, at first just vim.command+vim.fnameescape. Same for user interaction not
via callbacks in mappings.

Colored highlighting should be added; and additionally it would be fine to have
(both in VimL and in python) thing like :echon that puts the result in
:messages; I used to miss this when writing colored :! system()-based
replacement that had to use :echon for this very reason.

[Jan Pobrislo]

On Thu, 9 May 2013 08:30:00 -0700 (PDT), ZyX <zyx...@gmail.com> wrote:
> This is a description of proposed new python interfaces.

...
> =========================
> Auxilary/helper functions
> =========================

Feature request here if I may.

vim.fnameescape() and vim.evalescape() :: string -> string
Escapes string so it can be used as argument to command that takes a
filename or in a vim function respectively.

This could be further extended so evalescape (or perhaps just escape?)
handles further types that can be converted to vim literals, eg. lists and
dicts.

[ZyX]

> Feature request here if I may.
>
> vim.fnameescape() and vim.evalescape() :: string -> string
> Escapes string so it can be used as argument to command that takes a
> filename or in a vim function respectively.
>
> This could be further extended so evalescape (or perhaps just escape?)
> handles further types that can be converted to vim literals, eg. lists and
> dicts.

vim.fnameescape() makes sense, vim.evalescape does not: use vim.bindeval('function("Func")')('arg1', {'arg2_key': 'arg2_value'}, ...).

[Andy Kittner]

On Fri, May 10, 2013 at 03:49:45AM +0200, Bram Moolenaar wrote:
>
>ZyX wrote:
>
>> This is a description of proposed new python interfaces.
>
>Well, that's a very long list. Do we really need all of this?
>Let's at least order by usefulness.

I took some time today to look more closely at the RFC and the other
resonses in this thread. I also played around a bit with creating a
"meta-plugin" that would allow writing python-only plugins, just to
get a feel of things. So, with all that in mind here is my priorized
wish-list:

0. We all seem to agree that a complementary pure python module is a
good idea. Where it make sense to do so parts of the proposed
improvements cab be added there, and it will also give us a nice home
to add more convenience functions (e.g. for dealing with some ex.
commands) later.

1. Add the ability to create a FuncRef for arbitrary python callables:
The extra book keeping required to do this now feels unnecessarily
complicated and I think performance-wise the indirection
(vim_function -> id2callable registry -> python function) will
probably be noticeable for many repeated calls (e.g. when used in
a sub-replace-expression in a big file)

2. Ability to create / remove autocommands, maps and user commands:
The fully introspectable mappings mappings as proposed by ZyX would
be awesome, but at a bare minimum a convenient API for creation and
removal should be available

3. vim.functions:
Again, the introspection is not strictly necessary, but a convenience
wrapper that pulls the FuncRefs will make code look much nicer.
In that regard I'd also offer attribute access for builtin/ global
functions in addition to the getitem syntax. That way you can do
things like:
curbuf = vim.functions.bufnr("%")

4. $thing.valid for buffer, window, etc. objects
Being able to tell whether a reference to one of those is still
usable seems quite useful to me

The above are the most useful I think,


5. vim.signs & related

6. vim.jumps & related

7. vim.hlgroups % related

8. vim.find_buffer
This one would make more sense as a method on vim.buffers for me

9. anything else ;) it is an extensive RFC after all (Thanks very much
for writing it up and for working on improving the Python bindings in
general btw.)

Regards,
Andy
--
Happiness is just an illusion, filled with sadness and confusion.

[ZyX]

Thanks for the reply.

>I took some time today to look more closely at the RFC and the other resonses
>in this thread. I also played around a bit with creating a "meta-plugin" that
>would allow writing python-only plugins, just to get a feel of things. So, with
>all that in mind here is my priorized wish-list:
>
>0. We all seem to agree that a complementary pure python module is a
> good idea. Where it make sense to do so parts of the proposed
> improvements cab be added there, and it will also give us a nice home
> to add more convenience functions (e.g. for dealing with some ex.
> commands) later.
>
>1. Add the ability to create a FuncRef for arbitrary python callables:
> The extra book keeping required to do this now feels unnecessarily
> complicated and I think performance-wise the indirection
> (vim_function -> id2callable registry -> python function) will
> probably be noticeable for many repeated calls (e.g. when used in
> a sub-replace-expression in a big file)

This is one of the ideas behind

>About c): I still have not forgot about suggestion I have written earlier about
>function references: making them more generic by embedding struct func_S
>pointer in place of char* with function name. There are two additions since
>then: no s: in function references, reference counting for all (not only
>anonymous) functions (implies holding normal user functions in a regular dict_T
>like g: one, but hidden(?)), reference counting in struct func_S and normal
>names for anonymous functions (they will not be added to dict_T as with
>reference count and all other data in a structure attached to typval_T* unique
>names and holding anonymous functions with other user functions are no longer
>needed).

>2. Ability to create / remove autocommands, maps and user commands:

> The fully introspectable mappings mappings as proposed by ZyX would
> be awesome, but at a bare minimum a convenient API for creation and
> removal should be available
>
>3. vim.functions:
> Again, the introspection is not strictly necessary, but a convenience
> wrapper that pulls the FuncRefs will make code look much nicer.
> In that regard I'd also offer attribute access for builtin/ global
> functions in addition to the getitem syntax. That way you can do
> things like:
> curbuf = vim.functions.bufnr("%")

Good idea.

>4. $thing.valid for buffer, window, etc. objects
> Being able to tell whether a reference to one of those is still
> usable seems quite useful to me
>
>The above are the most useful I think,
>
>
>5. vim.signs & related
>
>6. vim.jumps & related
>
>7. vim.hlgroups % related
>
>8. vim.find_buffer
> This one would make more sense as a method on vim.buffers for me

And this as well. Will update the RFC at
https://gist.github.com/ZyX-I/5561409#file-rfc-rst.

[ZyX]

RFC was updated. In addition to @Andy Kittner’s suggestions I also added a few words about exceptions. Main proposal here: transform VimL errors/exceptions into python ones; this should be relatively easy as we already have :try whose implementation can be hacked.

[Bram Moolenaar]

ZyX wrote:

> >3. vim.functions:
> > Again, the introspection is not strictly necessary, but a convenience
> > wrapper that pulls the FuncRefs will make code look much nicer.
> > In that regard I'd also offer attribute access for builtin/ global
> > functions in addition to the getitem syntax. That way you can do
> > things like:
> > curbuf = vim.functions.bufnr("%")
>
> Good idea.

Would it be possible to have most Vim functions made callable from
Python this way? Obviously it's much better than using vim.eval().

An example from the help:

:py tagList = vim.eval('taglist("eval_expr")')

Would then be something like:

:py tagList = vim.functions.taglist(eval_expr)

With the difference that eval_expr would be a Python expression.

For the internals we would need to have a list of all funtions with
their argument types, so that the arguments can be converted from Python
to the Vim type. Could be added to the functions array in eval.c

A big advantage is that it's going to be much easier to add
functionality both to Vim and the Python interface.

--
ARTHUR: I command you as King of the Britons to stand aside!
BLACK KNIGHT: I move for no man.

[Andy Kittner]

On 05/15/13 13:06, Bram Moolenaar wrote:
>
> ZyX wrote:
>
>>> 3. vim.functions:
>>> Again, the introspection is not strictly necessary, but a convenience
>>> wrapper that pulls the FuncRefs will make code look much nicer.
>>> In that regard I'd also offer attribute access for builtin/ global
>>> functions in addition to the getitem syntax. That way you can do
>>> things like:
>>> curbuf = vim.functions.bufnr("%")
>>
>> Good idea.
>
> Would it be possible to have most Vim functions made callable from
> Python this way? Obviously it's much better than using vim.eval().

Yes, I think it should work with all builtins (and user functions as
well, basically anything where function("name") works).

>
> An example from the help:
>
> :py tagList = vim.eval('taglist("eval_expr")')
>
> Would then be something like:
>
> :py tagList = vim.functions.taglist(eval_expr)
>
> With the difference that eval_expr would be a Python expression.

Yes, thats exactly the way it would be used. It looks much nicer that
way and there is no need to escape "eval_expr" appropriately.

>
> For the internals we would need to have a list of all funtions with
> their argument types, so that the arguments can be converted from Python
> to the Vim type. Could be added to the functions array in eval.c

In my experiments on the weekend I made a simple wrapper class that
returned vim.bindeval("function('{0}')".format(attribute_name)).
I assumed that the FuncRef takes care of the appropriate
argument conversion when called (but I have not tested it extensively).
If it does, this should also be doable purely in python with the
exception of enumerating the available functions.

Regards,
Andy

[ZyX]

>Would it be possible to have most Vim functions made callable from
>Python this way? Obviously it's much better than using vim.eval().

I assumed it be a convenience wrapper replacing

vim.bindeval('function("bufnr")')('%')

(obviously, you can save `vim.bindeval('function("bufnr")')` result in
a variable) with a nicer syntax. I believe author of proposal assumed the same.

>An example from the help:
>
> :py tagList = vim.eval('taglist("eval_expr")')
>
>Would then be something like:
>
> :py tagList = vim.functions.taglist(eval_expr)
>
>With the difference that eval_expr would be a Python expression.

You mean, python regex? I already thought about it (but not applied to
`vim.functions.*`). It is not too hard (if written in python), but I never heard
about functions working this way (i.e. when you need regexes you take `re`
module, take generator/list expression, for cycle or whatever and do what you
want). Unlike Perl here regexes are not the part of the syntax and even not very
significant part of the language thus I assumed this behavior will not be
expected if expr is string and not necessary if you have iterators.

>For the internals we would need to have a list of all funtions with
>their argument types, so that the arguments can be converted from Python
>to the Vim type. Could be added to the functions array in eval.c
>
>A big advantage is that it's going to be much easier to add
>functionality both to Vim and the Python interface.

For types that are already in VimL you need nothing: they are converted
automatically. To use regexes, expressions and such things that are “types” only
for humans, but not for VimL function, from python much more job needs to be
done. From my point of view the best you can do is provide good API for such
things. E.g. for tags this would be sequence-like `{buffer}.tags` (or
`vim.current.tags` depending on what is there on the C level) with
namedtuple-like objects so you can filter tags using

def get_taglist(regex):
for tag in vim.current.tags:
if regex.match(tag.name):
yield tag

taglist = list(get_taglist(regex))

that usually narrows down to just

taglist = [tag for tag in vim.current.tags if regex.match(tag.name)]

. Note though that the following is completely possible with my first (one that
introduced `pyeval`) python patch:

vim_taglist = vim.bindeval('function("taglist")')

alltags = vim_taglist('.*')

. But for e.g. help buffers with `len(taglist('.*')) == 15550` this takes
9 seconds to complete in VimL and same in python (even if I forcefully convert
the result to a python list of python dicts: likely problem is in requirement to
allocate a very big bunch of small memory pieces using libc malloc: without
profiling I can’t imagine other explanation of why creating just the same number
of python objects adds insignificant amount of time). Thus you don’t want
vim.functions.taglist using taglist implementation from eval.c at all, you need
direct python → C interface to work with such amount of tags flawlessly.

Note that I know at least one case when I need *all* tags: to know what to look
at when converting help files to HTML using format.vim. taglist() function
listing only tags present in the file would do the job, but it is not there and
I do not expect existing one to work fine (i.e. with fine performance) if
I would e.g. continiously feed continiously truncated tail of the line into
`taglist()` to determine whether there is a tag.

This is another reason why it is better to have native python interfaces written
in C: while after adding bindeval I have reduced performance issues to the
lowest value possible without changing architecture of aurum for aurum, but
other plugins needing other parts of Vim will still suffer from them every time
they will use `vim.*eval` or `vim.command`.


By the way, thanks for pointing to another thing that need nice python
interface.

[ZyX]

>On 05/15/13 13:06, Bram Moolenaar wrote:
>>
>> ZyX wrote:
>>
>>>> 3. vim.functions:
>>>> Again, the introspection is not strictly necessary, but a convenience
>>>> wrapper that pulls the FuncRefs will make code look much nicer.
>>>> In that regard I'd also offer attribute access for builtin/ global
>>>> functions in addition to the getitem syntax. That way you can do
>>>> things like:
>>>> curbuf = vim.functions.bufnr("%")
>>>
>>> Good idea.
>>
>> Would it be possible to have most Vim functions made callable from
>> Python this way? Obviously it's much better than using vim.eval().
>Yes, I think it should work with all builtins (and user functions as
>well, basically anything where function("name") works).

It does.

>
>>
>> An example from the help:
>>
>> :py tagList = vim.eval('taglist("eval_expr")')
>>
>> Would then be something like:
>>
>> :py tagList = vim.functions.taglist(eval_expr)
>>
>> With the difference that eval_expr would be a Python expression.
>
>Yes, thats exactly the way it would be used. It looks much nicer that
>way and there is no need to escape "eval_expr" appropriately.

You already don’t need this, vim.function type works fine. `vim.functions` is
a convenience wrapper that also avoids `vim.bindeval` (and thus all that
expression parsing). I thought Bram is talking about using python regular
expressions here: otherwise there is no need to record argument types anywhere.

Though now another thought: automatic type conversion (e.g. when deciding
whether some object that can be converted to float, number or string should be
converted to one of this). Automatic type conversion is not pythonic: unlike Vim
trying to e.g. add number to string will result in TypeError, not in string
being converted to number. Making using existing types less strict (i.e. using
number protocol in place of strict checking for expected number being an
integer) though is a good idea, I will add it to RFC.

>>
>> For the internals we would need to have a list of all funtions with
>> their argument types, so that the arguments can be converted from Python
>> to the Vim type. Could be added to the functions array in eval.c
>In my experiments on the weekend I made a simple wrapper class that
>returned vim.bindeval("function('{0}')".format(attribute_name)).
>I assumed that the FuncRef takes care of the appropriate
>argument conversion when called (but I have not tested it extensively).
>If it does, this should also be doable purely in python with the
>exception of enumerating the available functions.

You can do this purely in python even without `vim.bindeval`. I bet I can create
`vim.bindeval` using only VimL, python own capabilities and `vim.eval`, but it
would be hackish and very slow.

[ZyX]

>Hi,
>
>I've written a couple of large vim plugins, I'm working with ZyX on
>Powerline, and I'd like to add my vote for the proposed functionality in
>this RFC.
>
>Having direct access to vim functionality with good bindings like
>suggested in this RFC would remove a huge hurdle with writing vim
>plugins, and I think it would boost the plugin ecosystem and modernize
>vim quite a bit.
>
>The most important bindings would in my opinion be *.options, *.augroup,
>the ability to call vim functions directly through *.functions.*, the
>ability to access variables directly and the ability to access and
>update hlgroups directly.
>
>I also think that if any functionality from this RFC should be
>implemented, it should be either all or nothing. I'm already sick of
>having to deal with different levels of Python support in the vim 7.3
>patchsets, and I'd prefer to not have to tell users to use certain
>patchsets to have access to certain parts of this RFC, or to have to add
>even more code to maintain support for different vim versions.

As I said I can write a fallback module. Note though that such fallback will
show even worse performance then if you used special fallbacks for your case
only.

>A concern I have is: would this RFC apply similarly to other supported
>languages like Ruby? I think it would be unfair to add this awesome
>functionality to Python only, and I can imagine that doing so would create
>a bit of anger among developers who prefer other languages.

To add this functionality to other languages somebody has to be interested and
take time to write it. The only interest I have there is looking at how things
are implemented in different languages and such interest is not enough to write
that much code. Also note that I can’t construct lua-ish interface without
previously writing much code in lua and writing pythonic interface for lua after
saying that VimL-like one is not acceptable in python because python is not VimL
is a good example of inconsistent words and deeds.

[Bram Moolenaar]

ZyX wrote:

[...]

> >Would it be possible to have most Vim functions made callable from
> >Python this way? Obviously it's much better than using vim.eval().
>
> I assumed it be a convenience wrapper replacing
>
> vim.bindeval('function("bufnr")')('%')
>
> (obviously, you can save `vim.bindeval('function("bufnr")')` result in
> a variable) with a nicer syntax. I believe author of proposal assumed
> the same.

Calling functions would be a very common thing. Thus I would keep the
name as short as possible: vim.f.bufnr('%')

> >An example from the help:
> >
> > :py tagList = vim.eval('taglist("eval_expr")')
> >
> >Would then be something like:
> >
> > :py tagList = vim.functions.taglist(eval_expr)
> >
> >With the difference that eval_expr would be a Python expression.
>
> You mean, python regex?

No, just the difference of passing the whole string to vim.eval vs.
processing the arguments in Python and passing the result of that to
Vim.

--
Mental Floss prevents moral decay!

[ZyX]

> Calling functions would be a very common thing. Thus I would keep the
> name as short as possible: vim.f.bufnr('%')

Adding cryptic names is not a good idea. Also note that calling built-in functions it is not so common and will not become more common later. Most used after examining sources of L9, Python-mode-klen, Gundo, clang_complete, ConqueShell, pyinteractive, threesomelib, powerline, ConqueGDB: get variable, set variable (obviously because it is the easiest way of interaction between python and VimL code aside from using bindeval), set option (but not get it; powerline is most extensive in getting options), call built-in function (bufnr and other buf*, feedkeys, expand, other), call external function (mostly via vim.command: second easiest way of interaction, used in case you don’t need reply from the function). Most used is calling random commands: :normal, buffer and window manipulations, other (setting variables and options was counted separately).

Subitem “bufnr and other buf*” should be normally replaced with buffer objects manipulations (some more attributes are to be added). Item “call external function” should not exist in pure-python plugin. Removing this moves calling random functions to one of the last places. And note that getting rid of necessity to use VimL functions was claimed as one of the goals.

> No, just the difference of passing the whole string to vim.eval vs.
> processing the arguments in Python and passing the result of that to
> Vim.

It is already there.

[Roland Eggner]

Hi Bram!

On 2013-05-15 Wednesday at 20:51 +0200 Bram Moolenaar wrote:
>
> ZyX wrote:
>
> [...]
>
> > >Would it be possible to have most Vim functions made callable from
> > >Python this way? Obviously it's much better than using vim.eval().
> >
> > I assumed it be a convenience wrapper replacing
> >
> > vim.bindeval('function("bufnr")')('%')
> >
> > (obviously, you can save `vim.bindeval('function("bufnr")')` result in
> > a variable) with a nicer syntax. I believe author of proposal assumed
> > the same.
>
> Calling functions would be a very common thing. Thus I would keep the
> name as short as possible: vim.f.bufnr('%')

Please keep in mind:

• Identifiers should tell for what purpose they have been created, otherwise
the source code becomes unmaintainable. E.g. in the Linux kernel XFS module
several function names are longer than 32 characters. Probably with reasons.
And if you object, no user needs to type this function names: think of debugger
sessions, think of grep commands in the source tree.

• The age of punch cards is over. Nowadays the common 80 columns limit is just
a habit without any technical reasons. Few people realized it, e.g. the Funtoo
Linux Project has a coding standard with line length up to 160 characters.

• Some people argue, short lines are convenient to read. Probably they don't
understand the consequences of “use it or loose it” for their memory.

--
Regards
Roland Eggner

[Ben Fritz]

On Thursday, May 16, 2013 7:20:13 AM UTC-5, Roland Eggner wrote:
> • The age of punch cards is over. Nowadays the common 80 columns limit is just
>
> a habit without any technical reasons.

I agree with using long self-documenting variable names. But 80 characters DOES have technical meaning still.

First, most terminal windows default to 80 columns wide. So using Vim in a terminal is much easier if text is short.

Second, with a rotated monitor, or with tiled/split windows, or just keeping a smaller resized window over the top of some reference material, 80 characters is actually a little larger than what can fit without horizontal scrolling.

And adding whitespace to keep lines at 80 characters (or pretty close) is not hard.

I know I would be very annoyed editing source code at 160 characters. With my rotated monitor, TagBar window, and DejaVu font, that would wrap at least once per line, or force me to scroll back and forth just to see the code.

[Bram Moolenaar]

Roland Eggner wrote:

> Hi Bram!
>
> On 2013-05-15 Wednesday at 20:51 +0200 Bram Moolenaar wrote:
> >
> > ZyX wrote:
> >
> > [...]
> >
> > > >Would it be possible to have most Vim functions made callable from
> > > >Python this way? Obviously it's much better than using vim.eval().
> > >
> > > I assumed it be a convenience wrapper replacing
> > >
> > > vim.bindeval('function("bufnr")')('%')
> > >
> > > (obviously, you can save `vim.bindeval('function("bufnr")')` result in
> > > a variable) with a nicer syntax. I believe author of proposal assumed
> > > the same.
> >
> > Calling functions would be a very common thing. Thus I would keep the
> > name as short as possible: vim.f.bufnr('%')
>
> Please keep in mind:
>
> • Identifiers should tell for what purpose they have been created,
> otherwise the source code becomes unmaintainable. E.g. in the Linux
> kernel XFS module several function names are longer than 32
> characters. Probably with reasons. And if you object, no user needs
> to type this function names: think of debugger sessions, think of
> grep commands in the source tree.

"f" means function in my dictionary, no confusion.
You can grep for "vim.f." easily.

> • The age of punch cards is over. Nowadays the common 80 columns
> limit is just a habit without any technical reasons. Few people
> realized it, e.g. the Funtoo Linux Project has a coding standard with
> line length up to 160 characters.

Most of my terminals are 80 columns wide. 100 when editing Java (which
has the problem of too long names). Main reason is that this way I can
see more windows side-by-side.

> • Some people argue, short lines are convenient to read. Probably
> they don't understand the consequences of “use it or loose it” for
> their memory.

It's not short that matters, it's quickly reading. Common words can be
read quickly even when they are long.

Anyway, it's more a personal preference than anything else.

In my opinion vim.function could mean something else than the leader for
all functions. Also because there is a function() function.
Perhaps it should be vim.functions instead of vim.function?
Considering we have vim.windows and vim.buffers. It's not the same
though, windows and buffers are dynamic, the list of functions is fixed.
Well, unless you include user functions.

--
Q: What's orange and sounds like a parrot?
A: A carrot

[ZyX]

> "f" means function in my dictionary, no confusion.
> You can grep for "vim.f." easily.

“f” is commonly used for file objects (*one* file object for each f). It may also refer to *one* function. We have more functions there.

> > • The age of punch cards is over. Nowadays the common 80 columns
> > limit is just a habit without any technical reasons. Few people
> > realized it, e.g. the Funtoo Linux Project has a coding standard with
> > line length up to 160 characters.
>
> Most of my terminals are 80 columns wide. 100 when editing Java (which
> has the problem of too long names). Main reason is that this way I can
> see more windows side-by-side.

I have slightly less then 120 columns for two windows (119 and 119 columns + 1 separator, 239 total). Thus I do not care when the limit is lesser then 119, but 160 is too much for my taste.

> It's not short that matters, it's quickly reading. Common words can be
> read quickly even when they are long.
>
> Anyway, it's more a personal preference than anything else.
>
> In my opinion vim.function could mean something else than the leader for
> all functions. Also because there is a function() function.
> Perhaps it should be vim.functions instead of vim.function?

Please refer to my RFC. I *never* named it vim.function, it is vim.functions what was written there. See commit history if you don’t believe me.

I also found my message where this data was duplicated. *There* is also `vim.functions`. Note that `vim.Function` is a function class, `vim.function` is the name of the type, they may be mentioned in a number of places.

> Considering we have vim.windows and vim.buffers. It's not the same
> though, windows and buffers are dynamic, the list of functions is fixed.
> Well, unless you include user functions.

Initial intention of `vim.functions` is to provide an interface *only* to user functions (main reasons: necessity to expand function names and intention of removing the need in functions and commands by adding python interfaces (obviously except for the cases when you need interaction with VimL plugins: this is why vim.functions appeared at all)). `vim.functions.bufnr` was suggested later by @Andy Kittner and I accepted this suggestion. Note: @Andy Kittner also wrote `vim.functions.bufnr`, not `vim.function.bufnr`.

[Jan Pobrislo]

On Thu, 16 May 2013 14:20:13 +0200, Roland Eggner <ed...@systemanalysen.net> wrote:
> • The age of punch cards is over. Nowadays the common 80 columns limit is just
> a habit without any technical reasons. Few people realized it, e.g. the Funtoo
> Linux Project has a coding standard with line length up to 160 characters.
>
> • Some people argue, short lines are convenient to read. Probably they don't
> understand the consequences of “use it or loose it” for their memory.

Let's bikeshed! :-)

This is clearly not about line length, but about information density. When
your code is littered with BridgeFactoryFooEnterpriseBeanAdaptors it might
be bit hard to spot the 'Foo' part.

But clearly there's no issue with users doing:
from vim import functions as f, commands as c
So I don't think being ambiguous by default is all that justifiable here.

[Marc Weber]

> vim.signs:
> Mapping-like object mapping sign names to vim.Sign attributes. Iterates over
> keys, supports item assignment.

If we are at it: The signs feature is broken by design, because its the
user setting "sign ids" - so multiple scripts don't know where to start.

What about adding a next_sign_id() function returning a new unused id ?
A simple counter should be enough. The perfect fix would be dropping the
old api, only allowing a new one place_sign() returning an id - but that
would break scripts.

Marc Weber

[ZyX]

> If we are at it: The signs feature is broken by design, because its the
> user setting "sign ids" - so multiple scripts don't know where to start.
>
> What about adding a next_sign_id() function returning a new unused id ?
> A simple counter should be enough. The perfect fix would be dropping the
> old api, only allowing a new one place_sign() returning an id - but that
> would break scripts.

Python is not VimL: proposed API is not good for python; and I have already written

> .place(line, name, id=None)
> Place given sign on the given line. Returns id which is equal to id argument ***if it was given or is a new unique id***.

* there is a typo here: should read “or a new unique id”, without “is”. Corrected, fix pushed.

. There is not something like this for sign names though.

Also note that simple counter could not be enough, unless you can make sure that no plugin will use sign id which will be returned by this counter.

Proposed “unique id”: concatenate buffer and line numbers and find the first gap in the vicinity of this id if it was already taken.

[ZyX]

Added tags interface, vim.environ, signs.find(linenr), suggestion for __enter__/__exit__:


``vim._tag_files``, ``{buffer}._tag_files``:
Iterators yielding tag file names. ``list(vim._tag_files)`` is an expanded
version of ``&tags`` option.

``vim.tags``, ``{buffer}.tags``:
Mapping-like object mapping tag names to ``vim.Tag`` objects. Does not
support assignment. Iterates over ``vim.Tag`` objects, not over tag names.
Iterator loads tags from files lazily, holding file handle while until its
destruction or until exhausting tags in that file.

Note: it would be good if parsed contents of tag files was cached and if
there added a possibility to define tags not present in files (like with zsh
``hash`` command: by adding tags to cache through existing API). Without
caching it would not be possible to solve performance problems with
``taglist()``.

``tag``:
Objects with the following attributes:

========= ===============
Attribute Description
========= ===============
name Name of the tag
filename Name of the file where tag is defined
lineno ``None`` or line number in that file
pattern ``None`` or pattern used to locate tag in that file
cmd ``None`` or Ex command used to locate tag; is present in case it is neither lineno nor pattern (should not really ever happen)
kind String, tag kind
static ``True`` or ``False``, specifies whether tag is specific to this file
========= ===============

Also has method ``.jump()``.


``vim.disabled_autocommands``:
Object with defined ``__enter__`` method setting ``&eventignore`` to ``all``
and ``__exit__`` method restoring previous value. Should be written in pure
python. Used to allow expressions like::

…
with vim.disabled_autocommands:
buffer = vim.buffers.new('aurum://file:…')
vim.current.buffer = buffer
…


---------------------------------------
Buffer- and window-manipulation methods
---------------------------------------

``vim.buffers.new(name=None, encoding={&enc}, binary=False, fileformat={…})``:
Create a new buffer. ``None`` in place of name means “create buffer with
empty name”. Returns ``vim.Buffer`` object.

``vim.windows.new(direction, buffer=None, size=0.5)``, also for ``vim.current.tabpage.windows``:
Split the window. ``size`` may be either an integer (absolute size) or
a floating-point value in the open interval ``(1.0, 0.0)`` (size relative to
the current window size or total vim size, depending on direction).
``buffer`` specifies what buffer will be opened in new window, defaults to
``vim.current.buffer``, must be a ``vim.Buffer`` object. ``direction`` may
be one of the following:

============== ===========
Direction Description
============== ===========
``horizontal`` Like ``:split``
``vertical`` Like ``:vsplit``
``diff`` Like ``:diffsplit``, but without setting ``diff*`` options
``above`` Like ``:above vsplit``
``below`` Like ``:below vsplit``
``left`` Like ``:lefta split``
``right`` Like ``:rightb split``
``top`` Like ``:top split``, uses &lines for relative sizes
``bottom`` Like ``:bot split``, uses &lines for relative sizes
``leftmost`` Like ``:topleft vsplit``, uses &columns for relative sizes
``rightmost`` Like ``:botright vsplit``, uses &columns for relative sizes
============== ===========

May use numeric constants in place of strings as a direction.

Returns ``vim.Window`` object.

* ``{buffer}``, ``{window}`` and ``{tabpage}`` ``__enter__`` and ``__exit__``
methods: temporary switch to given buffer/window/tabpage and switch back
afterwards.

[Marc Weber]

Excerpts from ZyX's message of Mon May 20 17:04:25 +0200 2013:

> Python is not VimL: proposed API is not good for python; and I have already written

Well - I just thought it would be a nice improvement for VimL, because
it will never happen that all plugins get rewritten using python.
So we need a fix for VimL, maybe its OT for the blue print, but related
to the new "release" and easy to fix.

I'm talking about this interface:
:sign place {id} line={lnum} name={name} file={fname}

I'd like to replace this (quoted from docs)

It's up to the user to make sure the {id} is used only once in

by

let my_sign_id = new_sign_id()
use 'exec sign place '.my_sign_id.'

Introducing such a function is simple, and would us all allow to fix
plugins eventually ..

Just a thought, maybe Bram finds time to add such trivial function.

Marc Weber

[Marc Weber]

Yet another suggestion - what about adding &rtp to sys.path somehow ?
(with or without python version site-packages or the like)

Then you could
- write .py files
- use them from python by

import module
do_stuff()

Yes - it can be trivially implemented in tools like vim-addon-manager,
but I'd prefer a global solution usable by everyone.

Marc Weber

[Marc Weber]

oh and it would be cool if python/*.py files could be loaded then, too ?
Did I also miss such feature proposal?

Marc Weber

[ZyX]

вторник, 21 мая 2013 г., 5:36:45 UTC+4 пользователь MarcWeber написал:

It is being discussed here: https://groups.google.com/d/msg/vim_dev/g5MTn7VOcnM/y95Ccjt1vh8J.

> oh and it would be cool if python/*.py files could be loaded then, too ?
> Did I also miss such feature proposal?

Loading files from python/ in a way other then `import {name}` is not a good idea unless this directory is not added to `sys.path`. And even if it is not added in is not a good idea either: existing plugins do not expect such behavior. And `import {name}` cannot work reliably: it may, e.g., import other module from `sys.path`. I would rather suggest using execfile on special names: maybe `{rtp}/python/**/{__vim_init__.py,__vim_init_*_.py}` (not so good as being added to `sys.path` `{rtp}/python` is expected to contain modules and not script files) or even just `{rtp}/__vim_init{_*_,}.py` or `{rtp}/plugin/__vim_init{_*_,}.py` or `{rtp}/python/__vim_init__/**/*.py`.

AFAIR there are some plugins keeping scripts (I mean executables) in `python/`, they would not be fine even with `import {name}` solution. Thus we have to use special name, it is very much unlikely __vim_init__ is used by existing plugins (I have seen no).

[Bram Moolenaar]

Yes, this also came up as a side effect of the os.chdir() solution.

It's especially useful if a plugin can be distributed as a Python module
plus some lines in the Vim plugin. It would have a similar effect as
using a Vim script under autoload.

--
A M00se once bit my sister ...
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD