[vim/vim] vim9: Make Autoload Functions Easier to Define and Call (Issue #9435)
rbong
Is your feature request about something that is currently impossible or hard to do? Please describe the problem.
As the language is being overhauled I believe we should make autoload functions nicer and easier to convert to other languages.
I've included some quotations from the vim 9 doc to support this idea.
we can also make Vim script easier to use. In other words: "less weird"
Compared with other languages, prefixing the file name on every function call is weird and awkward.
The only other option for public functions is the global namespace, which practically also requires some name prefix.
It should be possible to convert code from other languages to Vim script.
For a project with a public API, my options with vim9script function naming are g:SomeLongPrefix or some#long#prefix#.
Naming prefixes like this are not required in any other language.
This makes conversion more awkward than anything else with vim9script.
This becomes worse the more deeply nested autoload files are.
Using the global namespace is no better.
Describe the solution you'd like
Add a command to define the current autoload file and automatically prefix this name.
Allow "importing" functions directly from autoload files without pre-loading them.
Example:
In Vim9 script:
vim9script # This prefixes function names with "my_plugin#nested#" setautoload my_plugin#nested# def foo() echo "Foo" enddef # To avoid name conflicts, you can also manually define function names. def my_plugin#nested#bar() echo "Bar" enddef
This is equivalent to the following in legacy vimscript:
def my_plugin#nested#foo() echo "Foo" enddef def my_plugin#nested#bar() echo "Bar" enddef
You could also import this function:
vim9script # Replaces all calls to "foo" with "my_plugin#nested#foo" import { foo } from autoload "my_plugin#nested#" # Replaces all calls to "nested_foo" with "my_plugin#nested#foo" import { foo as nested_foo } from autoload "my_plugin#nested#"
Using this, functions still would not be loaded until they were actually called.
Describe alternatives you've considered
Autoloading could be automatic, or autoload names could automatically be prefixed.
—
Reply to this email directly, view it on GitHub.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you are subscribed to this thread.
Bram Moolenaar
> As the language is being overhauled I believe we should make autoload
> functions nicer and easier to convert to other languages.
- In the autoload script itself, all functions need to be prefixed with
path and directory name, such as "some#long#prefix".
- Repeating this prefix every time the function is called.
Generally the names are in the global namespace and the prefix is used
to locate the script. That can't be changed. We can look into how the
prefix is used.
Using a long name, including directory names, is not unusual. Most
languages have an "import" or "include" mechanism that uses this.
Usually then, after the import, only the last part is used, e.g. the
class name, and the path leading to it does not need to be repeated.
Some languages import a file and you never know what it defines. Such
as "include" in C and "import" in Dart. I definitely want to avoid
that, it makes debugging very hard: You never know where a symbol came
from.
Vim9 has an import/export mechanism. But so far this didn't support
auto-loading. The advantage of loading it right away is that any
imported items can be inspected, whether they actually exist and what
types they use. This also allows for compiling the function call.
We could try to add auto-loading to import/export somehow. First thing
is that the import only specifies the path below the runtime/autoload
directory. Thus "import... autoload 'some/long/prefix/script.vim'"
would search for the script 'some/long/prefix/script.vim' in all
"autoload" directories of entries in the 'runtime' option. This is
similar to how compilers specify a list of directories where library
headers can be found.
Otherwise it could work like existing Vim import commands, thus allowing
for importing a single item, multiple items, or everything with "* as
That".
In the autoload script we should explicitly export those things that it
wants to publish. The rest is script-local. That is consistent with
how Vim9 scripts work. Like in Java you export the class and nothing
else.
Using "export autoload ..." doesn't make sense, since the autoload
argument would be either on none or all "export" commands. It would be
needed to declare that the script is an autoload script once. And
fairly early.
We could use "vim9script autoload". We already require "vim9script" to
be used once in the script, and it specifies the type of script. Vim
can figure out the prefix from the path down from "runtime/autoload",
there is no need to specify it. You can then even move the file
elsewhere and rename it, the contents does not need to change. Still,
the new name must be used where it is imported.
I think that covers your suggestion. It is an addition to what we
already have, I don't see a conflict.
> Using this, functions still would not be loaded until they were
> actually called.
checked until the function is used. We do not want to load the whole
autoload script, since that would have a large performance penalty.
I was wondering if we can have some "header script" that defines the
types. But it's hard to keep in sync. It could perhaps be generated,
but that complicates things.
Also, just searching for the header file through directories already has
a performance penalty. It seems better to just the autoload-import to
remember what the script name is, and postpone looking for it until an
item is actually used.
Perhaps we can have a way to force loading and compiling an autoload
script the moment it is encountered, so that you can test for errors?
Perhaps with the test_override() function.
--
If you feel lonely, try schizophrenia.
/// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\
/// \\\
\\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
—
Reply to this email directly, view it on GitHub.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you are subscribed to this thread.
rbong
I think that covers your suggestion.
It does cover my suggestion, it's exactly what I'm looking for - a way not to type the autoload prefix every time.
But as you point out there are wider reaching problems as well.
That works, but does have a disadvantage: Existence and types are not
checked until the function is used.
I have some ideas on this that may or may not be practical.
You might not need to check autoload functions themselves every time Vim starts up, just when installing or updating them.
Maybe you can do something at install and update so that you can type-check calls to them as well.
I think ideally, Vim would have a way to install and update packages from different sources.
This would be consistent, I think, with making Vim script nicer to use.
Most modern languages have tooling like this.
This ties in with the next idea about this...
I was wondering if we can have some "header script" that defines the
types. But it's hard to keep in sync. It could perhaps be generated,
but that complicates things.
I was thinking ideally, Vim could cache compiled functions similar to __pycache__ in Python.
The cache could be invalidated whenever the script changes.
You could also auto-generate a "header file" while compiling the cache.
You could also cache errors, so that if an autoload function that compiled with errors and hasn't changed is imported, it gives a warning or error.
This would also solve the problem of recompiling autoload functions the first time they're used every time.
Like I said, you only generally need to check types when installing or updating scripts.
If you had package install/update commands, you could do the first compile/cache for scripts when you run these commands.
After that you could still recompile if the script changes.
You could also implement the cache before package management, and initially just have autoload functions compile for the first time when they are imported.
This could come with a one-time performance hit that is not as transparent as explicitly installing packages.
A real problem with this solution is that if you import autoload functions and the autoload script has changed, you can't check types.
That is, unless if you recompile the autoload functions with invalidated caches on import, which will cause a performance hit.
However autoload functions would still have the advantage that they don't need to be loaded into memory until they're actually called, and most won't need to be recompiled every time.
Another problem with this solution, similar to what you pointed out, is that header files will cause a performance hit.
However despite these performance hits, if the cache is used for everything, performance might overall be better.
Like you said, this complicates things, but this is my idealistic solution.
There are probably problems and practical considerations I'm not thinking of, but I think it's worth at least talking about better tooling and caching as long as Vim script is being overhauled.
—
Reply to this email directly, view it on GitHub.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you are subscribed to this thread.
Bram Moolenaar
[...]
> >That works, but does have a disadvantage: Existence and types are not
> >checked until the function is used.
>
> I have some ideas on this that may or may not be practical.
>
> You might not need to check autoload functions themselves every time
> Vim starts up, just when installing or updating them.
> Maybe you can do something at install and update so that you can
> type-check calls to them as well.
>
> I think ideally, Vim would have a way to install and update packages
> from different sources.
> This would be consistent, I think, with making Vim script nicer to use.
> Most modern languages have tooling like this.
least one of them is actually an autoload plugin, so that you only have
to get one file to make it work.
Vim provides some support for updating packages, but the actual fetching
is something that goes beyond Vim's core, it's better done in a plugin.
We could add some mechanism to load an autoload file as soon as it is
encountered, so that any errors are uncovered early, not at some
unexpected moment later. This could be used by a plugin developer who
wants to check the latest modifications. Or by a plugin manager that
just downloaded a new version.
[...]
> I was thinking ideally, Vim could cache compiled functions similar to `__pycache__` in Python.
> The cache could be invalidated whenever the script changes.
> You could also auto-generate a "header file" while compiling the cache.
> You could also cache errors, so that if an autoload function that
> compiled with errors and hasn't changed is imported, it gives a
> warning or error.
source of bugs. I would only go this way if we actually solve a big
problem. I don't think there is a performance problem big enough to
take on this extra ballast.
There are so many things to take into account besides normal use:
- restoring a backup
- copying the Vim files to another system
- having the files reside on a removable device
- the system time being changed (DST start and end)
- someone deleting a plugin, how do we clean up?
- using two Vim versions simultaniously
- etc.
--
A computer program does what you tell it to do, not what you want it to do.
/// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\
/// \\\
\\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
—
Reply to this email directly, view it on GitHub.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you are subscribed to this thread.
lacygoill
I've just noticed that my Vim startup time increased significantly. It's around 160ms. In the past, it was between 110ms and 120ms (while the system is mostly idle).
That's a big increase. It seems the culprits are all the import autoload statements.
For example, my slowest plugin contains these statements:
import autoload 'cmdline.vim'
import autoload 'cmdline/cL.vim'
import autoload 'cmdline/cycle/filter.vim' as FilterCycle
import autoload 'cmdline/cycle/generic.vim' as GenericCycle
import autoload 'cmdline/cycle/vimgrep.vim' as VimgrepCycle
import autoload 'cmdline/tab.vim'
import autoload 'cmdline/transform.vim'
import autoload 'cmdline/unexpand.vim'
Here is the average time when I write finish write above them:
0.028
And here is the average time when moving finish below the first import autoload, the 2nd one, the 3rd one, ...:
0.478
0.881
1.373
1.722
2.2
2.81
3.433
3.568
It seems each statement adds on average about 0.4 milliseconds. I have 116 import autoload across all of my plugins. If each of them adds 0.4ms, that makes 46.4ms in total; this matches the overall startup time increase which I noticed originally.
Did someone else notice that the Vim9 import autoload statement was slower than the legacy mechanism?
Is it working as intended, or has there been some recent regression?
Could the way Vim handles import autoload somehow be optimized?
Do I need to gather more information (possibly via gprof(1))?
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Bram Moolenaar
Do you perhaps have many entries in 'runtimepath'? The "import autoload" does not load the script, but it does find its location.
That means going over entries 'runtimepath' to search for "autoload/scriptname". This involves scanning directories, which might be slow.
A profile might help to locate any bottlenecks in this process. We could add some kind of caching for finding autoload scripts. But only when it really helps, since caching can cause trouble. It could even make it slower.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
lacygoill
Do you perhaps have many entries in 'runtimepath'?
I guess so: I have 104 entries in 'runtimepath'.
A profile might help to locate any bottlenecks in this process.
A profile of the C code, right? I tried to get one, here it is.
If that's not what is needed, I would need someone to tell me which commands to run to get the right data.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
lacygoill
A profile of the C code, right? I tried to get one, here it is.
To get it, I compiled Vim like this (in fish):
$ git reset --hard $(git rev-parse HEAD) \
; make clean \
; make distclean \
; sed -i '
/#PROFILE_CFLAGS = -pg -g -DWE_ARE_PROFILING/s/^#//
; /#PROFILE_LIBS = -pg -no-pie/s/^#//
' src/Makefile \
; make \
; tput bel
Then, I started Vim with my full config:
$ ./src/vim -Nu NONE -S /tmp/t.vim \
Finally, I asked gprof(1) to produce a log from the gmon.out binary dump:
$ gprof ./src/vim ./gmon.out
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
lacygoill
Sorry, the previous log was obtained while the system was not idle. Here is a new log (this time, the system was mostly idle).
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Bram Moolenaar
> > Do you perhaps have many entries in 'runtimepath'?
>
> I guess so: I have 104 entries in `'runtimepath'`.
> > A profile might help to locate any bottlenecks in this process.
>
> If that's not what is needed, I would need someone to tell me which
> commands to run to get the right data.
sense, the top being vim_strcat() with two calls. I guess the binary
and the symbol table are not matching.
It's best to compile without optimizing, start with "make clean" and use
the options mentioned in src/Makefile at PROFILING.
--
There can't be a crisis today, my schedule is already full.
/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
Bram Moolenaar
> Sorry, the previous log was obtained while the system was not idle.
> (this time, the system was mostly idle).
Central appears to be do_in_path(), which loops over the entries in
'runtimepath'. That confirms my suspicion. do_in_path() is called 333
times, and it calls copy_option_part() 17192 times, which suggests that
'runtimepath' has 51 entries (on average) that are looked at. There can
be more entries, since often only the first match is found.
The calls come half from ex_packadd(), I guess you are installing
many plugins. This actually may add an entry to 'runtimepath', thus
every later call has to go through more entries.
120 come from find_script_in_rtp(), which is called from ex_import().
I can't think of a way to avoid this with just code changes, the loop
through 'runtimepath' is needed for several reasons.
Is it perhaps true that these autoload imports actually use the autoload
directory relative to where the script is? Then we would not have to
look through 'runtimepath'. E.g.:
plugin/script.vim:
import autoload '../autoload/lib.vim'
autoload/lib.vim:
export def TheFunction()
...
enddef
This currently isn't supported, but it could be made to work.
Another advantage is that the autoload script will be found even when
another plugin uses the same script name. Perhaps we should not put it
in the "autoload" directory for that reason. However, that would be a
new mechanism and might cause problems somehow.
--
Did you ever stop to think... and forget to start again?
-- Steven Wright
/// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\
/// \\\
\\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
lacygoill
Is it perhaps true that these autoload imports actually use the autoload
directory relative to where the script is?
I suspect that yes, the vast majority are probably relative to the plugin script. IOW, most of the time, there should be no need to look into the runtimepath, provided that we support the ../autoload/script.vim syntax.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
bfrg
I think this issue can be closed since we have import autoload by now.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Bram Moolenaar
Closed #9435 as not planned.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.
Bram Moolenaar
Right, there haven't been comments or discussions on the current implementation recently. I take that as an indication that most script writers are OK with how it works now.
If someone does come up with a problem or idea for improvement, please open a new issue.
—
Reply to this email directly, view it on GitHub.
You are receiving this because you are subscribed to this thread.