REQUEST FOR COMMENTS: Enhanced Native Call Interface

[Francisco Lopes]

Some days ago I've created a topic about using libffi for an enhanced native call interface.

I've changed my mind all the way about how to build a native call interface for VIM, right

now I'm changing VIM sources to implement it as I'm thinking, some comments at this stage

would be good.

Goals:

    Make VIM able to call and be called, efficiently, from native libraries built as VIM plugins.

Non Goals:

    Make VIM able to call any native function.

    Make VIM automatically marshal VIM types to C types.

I started thinking of using libffi to do something as python ctypes for VIM but I thought it was not

a good idea because:

    Too much complexity for marshaling VIM types to C equivalents.

    Marshaling is not efficient.

    If there going be limitations at which functions are able to be called from VIM, why expend time on this.

    What I, myself, really want is build efficient plugins for VIM in C, not being able to call any function.

So this is about a interface to call VIM crafted plugins, exporting specific prototypes to be called from VIM

with not overhead.

How I'm building this:

    libload

        function to load a library from path/name and return a Libhandle.

    libfunc

        function to get a Libfunc from a Libhandle and function name.

    libcall / libcallex / call (not decided yet which one to use)

        function to call a Libfunc passing a raw VIM List of raw VIM types as arguments and returning a raw

        VIM type.

    libinit

        function initialize a VIM library from a Libhandle. This function will try to call an exported initialization

        function from the library:

            void vim( void *(*vim_get)(char *name) )

        If the function exist, the library should use the given vim_get function to ask for VIM functions and settings

        by name.

        The functions exposed by this function should be an API to allow the library to deal with raw VIM types

        without overhead and without internal implementation knowledge. Beyond dealing with VIM types

        it should export functions for VIM evaluation from strings and maybe other stuff.

        This API should be documented and may be extended over time.

To have libload and libfunc, I needed a way to represent library handles and function pointers. No VIM type

can represent that well, so I started by extending VIM variables to become yet more two types, Libhandle and

Libfunc. These types only show up in the context of these functions. Variables of these types can only be assigned,

compared (==, !=) and used in the empty function to check for nullity.

Right now I've extend VIM variables, I'm implementing the functions, the hard part will be to define and implement

what the vim_get callback will return.

Any comments please?

Regards

---

Francisco Lopes

[Tony Mechelynck]

On 13/09/12 23:58, Francisco Lopes wrote:
> Some days ago I've created a topic

> <https://groups.google.com/d/topic/vim_dev/F2z5I9tayKY/discussion> about

> *libload*
> function to load a library from path/name and return a *Libhandle*.
> *libfunc*
> function to get a *Libfunc* from a *Libhandle* and function name.
> *libcall* / *libcallex* / *call* (not decided yet which one to use)
> function to call a *Libfunc *passing a raw VIM List of raw VIM

> types as arguments and returning a raw
> VIM type.

> *libinit*
> function initialize a VIM library from a *Libhandle*. This

> function will try to call an exported initialization
> function from the library:
> void vim( void *(*vim_get)(char *name) )
> If the function exist, the library should use the given

> *vim_get* function to ask for VIM functions and settings

> by name.
> The functions exposed by this function should be an API to
> allow the library to deal with raw VIM types
> without overhead and without internal implementation knowledge.
> Beyond dealing with VIM types
> it should export functions for VIM evaluation from strings and
> maybe other stuff.
> This API should be documented and may be extended over time.
>

> To have *libload* and *libfunc*, I needed a way to represent library

> handles and function pointers. No VIM type
> can represent that well, so I started by extending VIM variables to

> become yet more two types, *Libhandle* and
> *Libfunc*. These types only show up in the context of these functions.

> Variables of these types can only be assigned,

> compared (==, !=) and used in the *empty* function to check for nullity.

>
> Right now I've extend VIM variables, I'm implementing the functions, the
> hard part will be to define and implement
> what the vim_get callback will return.
>
> Any comments please?
>
> Regards
>
> ---
>
> Francisco Lopes

See also
:help libcall()
:help libcallnr()

about already implemented possibilities.


Best regards,
Tony.
--
ARTHUR: Shut up! Will you shut up!
DENNIS: Ah, now we see the violence inherent in the system.
ARTHUR: Shut up!
DENNIS: Oh! Come and see the violence inherent in the system!
HELP! HELP! I'm being repressed!
The Quest for the Holy Grail (Monty
Python)

[Francisco Lopes]

On 13/09/2012 19h46min15s UTC-3, Tony Mechelynck wrote:
>
> See also
>
> :help libcall()
>
> :help libcallnr()
>
>
>
> about already implemented possibilities.
>

See my previous topic where I explain about the problem with these functions: https://groups.google.com/d/msg/vim_dev/F2z5I9tayKY/UNGZ1-kSvq4J

Regards

---

Francisco Lopes

[ZyX]

пятница, 14 сентября 2012 г., 1:59:05 UTC+4 пользователь Francisco Lopes написал:

If I were doing this thing I would change function references to have type

typedef struct funcref_S funcref_T;
struct funcref_S {
void *fv_data;
funcdef_T *fv_func_def;
int fv_refcount;
int fv_copyid;
funcref_T *fv_used_prev;
funcref_T *fv_used_next;
}
typedef struct {
int (*fd_call) __ARGS((void *data, typval_T *argvars, dict_T *selfdict, typval_T *rettv));
char_u *(*fd_string) __ARGS((void *data)); /* string() implementation */
void (*fd_print) __ARGS((void *data)); /* :function Funcref output
* Maybe it is worth being
* char ** (*fd_print)(void *)
*/
int (*fd_free) __ARGS((void *data)); /* Called when there are no more references left */
} funcdef_T;
There is a reason behind exactly this interface:
1. This avoids useless dictionary (user functions) or even binary (built-in functions) searches when using the funcref.
2. This makes it much easier to implement equivalent of
vim.bindeval('function("tr")')("a", "b", "c") # Python
vim.funcref("tr")("a", "b", "c") -- Lua, if I did not mistake anything
-- Requires recent patches from Luis Carvalho
for vimscript.
3. Being implemented properly this purges out the situations like

function s:ScriptLocal()
endfunction
let d={'f': function('s:ScriptLocal')}
call FunctionFromAnotherPlugin(d.f) " An error: there is no script-local
" function with this name
" in another plugin
and
function Func()
endfunction
let d={'f': function('Func')}
delfunction Func
call d.f() " The error: function no longer exists

Unfortunately I currently do not have time for such a change (more true, I do not want to invest it in what I do not believe will be accepted, comments from Bram are needed on the subject). With this interface I get the second (and hence will implement funcrefs calling python functions) and everybody get first and last as a benefits.

You then will have no need in one of new data types and remove the need in libcall/libcallex/call (the latter will do the trick for all types of functions). You can also remove a need in the second one if you do the following instead:
1. Add libfunc(path_to_lib, function_name). It should do the following:
1. Normalize path_to_lib and lookup it in special dictionary with library handles. If it is found export symbol from there and return.
2. If not, get library handle.
3. Then get symbol “vim_init” or whatever and use it to initialize.
4. Then get function_name, return.
In returned structure reference to the symbol, its name and a structure describing library are stored. It assumes library itself does not have any global state, all state is to be stored in arguments if needed. If you do want to support state skip this point (except the description of what is stored in void *fv_data) and go on with libload+libinit.
2. fd_call should just do the following:
int
fd_call (data, argvars, selfdict, rettv)
struct {
int (*func) (typval_T *, typval_T *, typval_T *);
char_u *name;
struct {
void *libhandle;
char_u *path;
/* GC stuff here */
} *libhandle;
} *data;
int (*data) (typval_T *, typval_T *, typval_T *);
typval_T *argvars;
typval_T *selfdict;
typval_T *rettv;
{
return data->func(argvars, selfdict, rettv);
}
3. fd_string emits “libfunc(path, name)”.
4. fd_print prints path to library and symbol name.
5. fd_free is freeing data and unreferencing opened library, unloading if needed. Unreferencing and unloading in both cases: whether you stick to internal dictionary with already opened libraries or use libload+…, it only makes a difference at the first point.

As you see here adding external libraries support is trivial. What is less trivial is changing every piece when funcrefs are involved.

[Francisco Lopes]

On 14/09/2012, at 01:54, ZyX <zyx...@gmail.com> wrote:

If I were doing this thing I would change function references to have type

   typedef struct funcref_S funcref_T;
   struct funcref_S {
       void      *fv_data;
       funcdef_T *fv_func_def;
       int        fv_refcount;
       int        fv_copyid;
       funcref_T *fv_used_prev;
       funcref_T *fv_used_next;
   }
   typedef struct {
       int     (*fd_call)   __ARGS((void *data, typval_T *argvars, dict_T *selfdict, typval_T *rettv));
       char_u *(*fd_string) __ARGS((void *data)); /* string() implementation */
       void    (*fd_print)  __ARGS((void *data)); /* :function Funcref output
                                                   * Maybe it is worth being
                                                   * char ** (*fd_print)(void *)
                                                   */
       int     (*fd_free)   __ARGS((void *data)); /* Called when there are no more references left */
   } funcdef_T;
There is a reason behind exactly this interface:
1. This avoids useless dictionary (user functions) or even binary (built-in functions) searches when using the funcref.

Just to be clear, here you are suggesting to change all the way how VIM implement VIM Script functions (user and internal)

beyond my aim of better native call support.

This seems a big and non isolated change compared to what I'm proposing, I really prefer do it the simpler way although if

implemented it could benefit overall in the end, but, anyway, adding the new kinds of variables was trivial, some may say

it may look ugly, but these new types only show up in the context of native plugin development, so I don't see it as a big

problem. I know I can apply this design, the data structure, reference counting (I though of that initially), etc, for the

native calls only, but it's much less trivial to implement =).

2. This makes it much easier to implement equivalent of
      vim.bindeval('function("tr")')("a", "b", "c") # Python
      vim.funcref("tr")("a", "b", "c")  -- Lua, if I did not mistake anything
                                        -- Requires recent patches from Luis Carvalho
  for vimscript.

Sounds neat although I see it still possible by <whatever>.bindeval(<whatever>) returning Libfunc vars (to call C functions to do job with

python).

3. Being implemented properly this purges out the situations like

     function s:ScriptLocal()
     endfunction
     let d={'f': function('s:ScriptLocal')}
     call FunctionFromAnotherPlugin(d.f) " An error: there is no script-local
                                         " function with this name
                                         " in another plugin

If FunctionFromAnotherPlugin is a Libfunc var (got after libload, libfunc), what should happen here (if native calls is implemented

through call) is just a simple call to the C function pointed by the var. A simple C call will be in the same context as the current line

of the script, if it receives a Funcref and is able to make calls to it, there should be no problem of undefinition.

Did I got the point here? because I'm just talking about a possibility of the implementation I described, not about the overall benefits

to VIM Script of your idea. I think you may be talking about a current problem in VIM Script, I'm not sure.

I like the Idea of having an special function for libcall's because it turns things clear and isolated. But too, having an uniform

call syntax looks good… but my primary aim is just the aim of the current libcall, just without the problems and limitations.

  and
     function Func()
     endfunction
     let d={'f': function('Func')}
     delfunction Func
     call d.f() " The error: function no longer exists

Is this a problem in VIM script? I really don't know, I've not played with it too much yet. And the truth is that I'm not minding with these current

weak implementation spots to just do the enhanced native call. I'm trying to be modest and objective with this propose because the current

libcall is just too limited and there's no good way to build C plugins.

Unfortunately I currently do not have time for such a change (more true, I do not want to invest it in what I do not believe will be accepted, comments from Bram are needed on the subject). With this interface I get the second (and hence will implement funcrefs calling python functions) and everybody get first and last as a benefits.

Yes, to me it sounds cool too, but I'm afraid if I'd gone that way it would be barred indeed, or at last, it would take much longer to be

accepted (and longer to implement).

I'm just afraid of being barred with my current propose. I hope it'll not.

5. fd_free is freeing data and unreferencing opened library, unloading if needed. Unreferencing and unloading in both cases: whether you stick to internal dictionary with already opened libraries or use libload+..., it only makes a difference at the first point.

Ok, so since I'm not going the funcref_S way, I know I could apply the effort of dictionaries, reference counting and vanish

with the two extra variable kinds. It looks a bit as (over?)engineering just to eliminate them, but, anyway, I may look further

at similar options, thanks.

As you see here adding external libraries support is trivial. What is less trivial is changing every piece when funcrefs are involved.

It's trivial to see the idea, but implementing is another history =)

Thanks for all the discussion, it brings a lot of things to light. Indeed, a lot I wasn't even remotely thinking about, but I feel it's

a lot more to cover.

Best Regards

---

Francisco

[ZyX]

No, just how funcrefs work. I bet it is easier then you think, but harder then
I do (I expect problems only in one point: deref_func_name; this can’t be really
true).

>This seems a big and non isolated change compared to what I'm proposing,
>I really prefer do it the simpler way although if implemented it could benefit
>overall in the end, but, anyway, adding the new kinds of variables was trivial,
>some may say it may look ugly, but these new types only show up in the context
>of native plugin development, so I don't see it as a big problem. I know I can
>apply this design, the data structure, reference counting (I though of that
>initially), etc, for the native calls only, but it's much less trivial to
>implement =).

It changes mostly in the same places where you want to change. Except
deref_func_name, there is no way how you propose touches this. But no matter
what will you do, please, make all C function arguments the same as in func_call
except the first one (first is function name which you obviously don’t need).
Then if I succeed implementation will be easy to change.

>> 2. This makes it much easier to implement equivalent of
>> vim.bindeval('function("tr")')("a", "b", "c") # Python
>> vim.funcref("tr")("a", "b", "c") -- Lua, if I did not mistake anything
>> -- Requires recent patches from Luis Carvalho
>> for vimscript.
>
>Sounds neat although I see it still possible by <whatever>.bindeval(<whatever>)
>returning Libfunc vars (to call C functions to do job with python).

I did not say it is impossible. I said it is easier. More, “easier” is wrong
word: with my proposal you won’t need to change *anything* in python code after
it was changed to use new funcrefs.

>> 3. Being implemented properly this purges out the situations like
>>
>> function s:ScriptLocal()
>> endfunction
>> let d={'f': function('s:ScriptLocal')}
>> call FunctionFromAnotherPlugin(d.f) " An error: there is no script-local
>> " function with this name
>> " in another plugin
>
>If FunctionFromAnotherPlugin is a Libfunc var (got after libload, libfunc),
>what should happen here (if native calls is implemented through call) is just
>a simple call to the C function pointed by the var. A simple C call will be in
>the same context as the current line of the script, if it receives a Funcref
>and is able to make calls to it, there should be no problem of undefinition.
>Did I got the point here? because I'm just talking about a possibility of the
>implementation I described, not about the overall benefits to VIM Script of
>your idea. I think you may be talking about a current problem in VIM Script,
>I'm not sure.

Yes, it is a problem (but this one is solvable): when you define s: function in
your script “s:” is translated to “<SNR>{snr}_”, but not when you create
a funcref. I was actually not precise here: it may call s:ScriptLocal function
*that is defined in another plugin*. If it is defined, of course.

>I like the Idea of having an special function for libcall's because it turns
>things clear and isolated. But too, having an uniform call syntax looks good…
>but my primary aim is just the aim of the current libcall, just without the
>problems and limitations.

If both suggestions are merged (your libcall with your original proposal and my
new function references) we will end up in having another two unconsistent
things. I am more convinced in investing time in new funcrefs.

>> and
>> function Func()
>> endfunction
>> let d={'f': function('Func')}
>> delfunction Func
>> call d.f() " The error: function no longer exists
>
>Is this a problem in VIM script? I really don't know, I've not played with it
>too much yet. And the truth is that I'm not minding with these current weak
>implementation spots to just do the enhanced native call. I'm trying to be
>modest and objective with this propose because the current libcall is just too
>limited and there's no good way to build C plugins.

There is: C plugins can be Python/ruby/lua modules :)
It is subjectable whether it is a bug: “delfunction” does its job and deletes
a function thus it is no more accessible. I don’t like the idea, I want to be
sure that if I received a funcref I am able to call it. *Like I do if I use
“anonymous” functions*. Anonymous functions can be deleted by just the same
command, but consequences of the deletion when I reported them were considered
as a bug. Not fixed yet as “delfunction” is not likely to be used for funcrefs
and I know where the problem lies but unsure what is the proper fix.

>> Unfortunately I currently do not have time for such a change (more true, I do
>> not want to invest it in what I do not believe will be accepted, comments
>> from Bram are needed on the subject). With this interface I get the second
>> (and hence will implement funcrefs calling python functions) and everybody
>> get first and last as a benefits.
>
>Yes, to me it sounds cool too, but I'm afraid if I'd gone that way it would be
>barred indeed, or at last, it would take much longer to be accepted (and longer
>to implement). I'm just afraid of being barred with my current propose. I hope
>it'll not.

That is why I suggest to start implementing things you have to do in any case
and wait for reply from Bram. These things are library for handling internal
types and fd_* functions: even if my proposal is not accepted they are still can
be used for implementing f_libcallex and others.

Lets wait for Bram before you proceed this way.
- Maybe (but, I think, unlikely) he accepts two new types.
- Maybe there will be just one type, similar to lua userdata.
- Maybe you will go on with holding everything in internal dictionaries and
using their keys for lib* functions.
- Or maybe my proposal is accepted.
In any case you will be able to make use of fd_* functions (except for fd_print
and, maybe, fd_string) and in any case you need typval library. So start with
them.

>> As you see here adding external libraries support is trivial. What is less
>> trivial is changing every piece when funcrefs are involved.
>
>It's trivial to see the idea, but implementing is another history =)
>
>Thanks for all the discussion, it brings a lot of things to light. Indeed,
>a lot I wasn't even remotely thinking about, but I feel it's a lot more to
>cover.
>
>Best Regards
>
>---
>
>Francisco
>

>--
>You received this message from the "vim_dev" maillist.
>Do not top-post! Type your reply below the text you are replying to.
>For more information, visit http://www.vim.org/maillist.php
>