So, how about Vim 9?

[Bram Moolenaar]

Vim 8.2 has been released, so what is next?


1. FASTER VIM SCRIPT

The third item on the poll results of 2018, after popup windows and text
properties, is faster Vim script. So how do we do that?

I have been throwing some ideas around, and soon came to the conclusion
that the current way functions are called and executed, with
dictionaries for the arguments and local variables, is never going to be
very fast. We're lucky if we can make it twice as fast. The overhead
of a function call and executing every line is just too high.

So what then? We can only make something fast by having a new way of
defining a function, with similar but different properties of the old
way:
- Arguments are only available by name, not through the a: dictionary or
the a:000 list.
- Local variables are not available in an l: dictionary.
- A few more things that slow us down, such as exception handling details.

I Implemented a "proof of concept" and measured the time to run a simple
for loop with an addition (Justin used this example in his presentation,
code is below):

Vim old function: 5.018541
Python: 0.369598
Lua: 0.078817
Vim new function: 0.073595

That looks very hopeful! It's just one example, but it shows how much
we can gain, and also that Vim script can be faster than builtin
interfaces.

How does this work? The function is first compiled into a sequence of
instructions. Each instruction has one or two parameters and a stack is
used to store intermediate results. Local variables are also on the
stack, space is reserved during compilation. This is a fairly normal
way of compilation into an intermediate format, specialized for Vim,
e.g. each stack item is a typeval_T. And one of the instructions is
"execute Ex command", for commands that are not compiled.


2. PHASING OUT INTERFACES

Attempts have been made to implement functionality with built-in script
languages such as Python, Perl, Lua, Tcl and Ruby. This never gained much
foothold, for various reasons.

Instead of using script language support in Vim:
- Encourage implementing external tools in any language and communicate
with them. The job and channel support already makes this possible.
Really any language can be used, also Java and Go, which are not
available built-in.
- Phase out the built-in language interfaces, make maintenance a bit easier
and executables easier to build. They will be kept for backwards
compatibility, no new features.
- Improve the Vim script language, so that it can be used when an
external tool is undesired.

All together this creates a clear situation: Vim with the +eval feature
will be sufficient for most plugins, while some plugins require
installing a tool that can be written in any language. No confusion
about having Vim but the plugin not working because some specific
language is missing. This is a good long term goal.

Rationale: Why is it easier to run a tool separately from Vim than using a
built-in interface and interpreter? Take for example something that is
written in Python:
- The built-in interface uses the embedded python interpreter. This is less
well maintained than the python command. Building Vim with it requires
installing developer packages. If loaded dynamically there can be a version
mismatch.
- When running the tool externally the standard python command can be used,
which is quite often available by default or can be easily installed.
- A .py file can be compiled into a .pyc file and execute much faster.
- Inside Vim multi-threading can cause problems, since the Vim core is single
threaded. In an external tool there are no such problems.
- The Vim part is written in .vim files, the Python part is in .py files, this
is nicely separated.
- Disadvantage: An interface needs to be made between Vim and Python.
JSON is available for this, and it's fairly easy to use. But it still
requires implementing asynchronous communication.


3. BETTER VIM SCRIPT

To make Vim faster a new way of defining a function needs to be added.
While we are doing that, since the lines in this function won't be fully
backwards compatible anyway, we can also make Vim script easier to use.
In other words: "less weird". Making it work more like modern
programming languages will help. No surprises.

A good example is how in a function the arguments are prefixed with
"a:". No other language I know does that, so let's drop it.

It should be possible to convert code from other languages to Vim
script. We can add functionality to make this easier. This still needs
to be discussed, but we can consider adding type checking and a simple
form of classes. If you look at JavaScript for example, it has gone
through these stages over time, adding real class support and now
Typescript adds type checking. But we'll have to see how much of that
we actually want to include in Vim script. Ideally a conversion tool
can take Python, Javascript or Typescript code and convert it to Vim
script, with only some things that cannot be converted.

Vim script won't work the same as any specific language, but we can use
mechanisms that are commonly known, ideally with the same syntax. One
thing I have been thinking of is assignments without ":let". I often
make that mistake (after writing JavaScript especially). I think it is
possible, if we make local variables shadow commands. That should be OK,
if you shadow a command you want to use, just rename the variable.
Using "let" and "const" to declare a variable, like in JavaScript and
Typescript, can work:

def MyFunction(arg)
let local = 1
const ADD = 88
while arg > 0
local += ADD
--arg
endwhile
return local
enddef

Just some ideas, this will take time to design, discuss and implement.
Eventually this will lead to Vim 9!


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Code for the timing measurements (build with -O2):

func VimOld()
let sum = 0
for i in range(1, 2999999)
let sum += i
endfor
return sum
endfunc

func Python()
py3 << END
sum = 0
for i in range(1, 3000000):
sum += i
END
return py3eval('sum')
endfunc

func Lua()
lua << END
sum = 0
for i = 1, 2999999 do
sum = sum + i
end
END
return luaeval('sum')
endfunc

def VimNew()
let sum = 0
for i in range(1, 2999999)
let sum += i
endfor
return sum
enddef

let start = reltime()
echo VimOld()
echo 'Vim old: ' .. reltimestr(reltime(start))

let start = reltime()
echo Python()
echo 'Python: ' .. reltimestr(reltime(start))

let start = reltime()
echo Lua()
echo 'Lua: ' .. reltimestr(reltime(start))

let start = reltime()
echo VimNew()
echo 'Vim new: ' .. reltimestr(reltime(start))

A couple of remarks:
- Python is picky about the indent, has to be left aligned.
- Lua doesn't support "sum += i".

--
Facepalm statement #6: "Estland is a fantasy place, just like Middle Earth and
Madagaskar"

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Christian Brabandt]

On Di, 17 Dez 2019, Bram Moolenaar wrote:

> Vim 8.2 has been released, so what is next?

Wow, respect for the well-thought roadmap. I like that approach, but
this sounds like an awful lot of work, but looking forward :)

Just one minor thought, have you thought about storing the intermediate
Vim script format into a new binary file (similar to how python uses .py
and .pyc)?

Best,
Christian
--
Wenn man es nur versucht, so geht's, das heißt, mitunter, doch nicht
stets.
-- Wilhelm Busch

[Bram Moolenaar]

Christian wrote:

> On Di, 17 Dez 2019, Bram Moolenaar wrote:
>
> > Vim 8.2 has been released, so what is next?
>
> Wow, respect for the well-thought roadmap. I like that approach, but
> this sounds like an awful lot of work, but looking forward :)

Yes, it's going to be a lot of work. But it's the kind of work I enjoy
doing.

> Just one minor thought, have you thought about storing the intermediate
> Vim script format into a new binary file (similar to how python uses .py
> and .pyc)?

It might help a bit, but only for large files. Quite often locating and
opening the file takes longer than parsing the contents: CPUs are very
fast these days. And then we need to read the original file anyway if
the compilation changed. Not sure if it is worth the hassle. It would
help more to avoid searching for files and to postpone work by using
autoload scripts.

--
Facepalm reply #9: "Speed up, you can drive 80 here" "Why, the cars behind us
are also driving 60"

[Prabir Shrestha]

Glad to see you are already thinking of vim 9 and it is good to see the rythm of having a new goal ever year. I was personally waiting to hear more people comment on the ECMAScript (https:/ta.github.io/WasmExplorer//github.com/vim/vim/pull/5198) support before I share more details but here it goes.

Just one minor thought, have you thought about storing the intermediate Vim script format into a new binary file (similar to how python uses .py and .pyc)?

This is exactly what WASM (WebAssembly) is. Why not make WASM a first class citizen in Vim 9 instead?

FASTER VIMSCRIPT wasm does a fabulous job here and was designed for this. Here is an exact quote from their github repo. https://github.com/WebAssembly/design

"WebAssembly is efficient and fast: Wasm bytecode is designed to be encoded in a size- and load-time-efficient binary format. WebAssembly aims to execute at native speed by taking advantage of common hardware capabilities available on a wide range of platforms."

Currently the new function prototype only makes function fast. There will be lot more work to make rest of the features fast. While I would love vimscript to be fast I'm not sure if it worth it. I would rather have vimscript compile to WASM and then execute. This way we only think about making the runtime fast. There are also lot of WASM runtimes currently some of which even supports JIT and native compilation and other optimizations. You can try this website and see what native assembly code it generates. https://mbebenita.github.io/WasmExplorer/

PHASING OUT INTERFACES

Lot of languages now supports compiling to WASM and this would solve the issue of having multiple language interfaces. Another option would be to use jobs and channels but currently someone still needs to write a small wrapper to take full advantage of this. Neovim has a remote RPC concept and official libraries for their languages and hence there is more abundance of the neovim plugins taking advantage of this and doing more complicated stuff then vim8 plugins. I personally don't use any of these interfaces but there are lot of people who use python plugins just so they can have the speed or use libraries that come with languages.

WASM fixes this issues about supporting someones best language.

BETTER VIMSCRIPT

This is the one I'm worried the most. I would personally be against breaking the language compatibility as it would be pain for us to support as a plugin author.

I don't think comparing javascript with vimscript on adding breaking languages is the right comparison. While most devs might write in the ES6 or newer they either compile it down to ES3 or ES5 so it works in older browsers or they compile it to both versions but give hints to browsers to only pick one of them so that it continues to work in older browsers.

Here is an example:

<script src="es5.js" nomodule"></script>
<script src="es6.js" type="module"></script>

Old browsers will only use es5.js because if type is not specified it automatically treats it as text/javascript and nomodule is not understood by them and they just ignore it. es6.js has type module so while it may download that file it doesn't know how to execute it so in terms of executing it becomes a noop. For mordern browsers, they understand nomodule so it doesn't download es5.js but understands type=module and treats it as javascript file and executes it. How would vimscript do this? Same for typescript, it compiles down to either ES5 or ES6 or some variant of javascript with mostly typings removed. Typing is also something I miss a lot in vimscript because as a vim plugin author I would most likely continue to use func instead of def even if def is faster if I wanted to support vimscript. If I really wanted it to be fast I would rather continue to use Python or lua interfaces or WASM. I personally prefer WASM>LUA>VimScript. Lua is a good alternative if it ships in vim primarily because the language it self is very much set and is very powerful with features such as co-routines allowing us to express async code in an easy way (this is actually my personal pain compared to speed of vimscript).

To me I think vim9 should all be about WASM. This solves everything mentioned eariler in the thread and in a better way without too much work.

Why?

* It is fast, really fast. This will only continue to be better due to the energy the current industry is betting on. Create a compiler to compile vimscript to wasm

* I can write in whatever language I want solving the language interface problem. WASM is most likely also officially supported by your favorite language.

* WASI Interface https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/ Think of this as standard c library which is portable.

* WASM threads https://github.com/WebAssembly/threads ex: using immutable data structures to fuzzy search effeciently by sharing between threads.

* WASM webgl https://rustwasm.github.io/wasm-bindgen/examples/webgl.html Similar to popup window but imagine getting a raw canvas to draw in gui vim ex: rendering markdown files. Who knows might be one day we will get 3d games running inside vim :)

* WASM c api https://github.com/WebAssembly/wasm-c-api Once this takes off I'm hoping it will be easy to swap out runtimes into a faster runtime.

[Yegappan Lakshmanan]

Hi Bram,

On Tue, Dec 17, 2019 at 8:49 AM Bram Moolenaar <Br...@moolenaar.net> wrote:
>
>
> Vim 8.2 has been released, so what is next?
>

It is good to see the long term plan for the evolution of Vim.
In addition to all the good stuff you have mentioned below,
I would like to also see the pending items at the top of the
todo list for the popup window, text properties, prompt
buffer, terminal debugger and terminal window features
addressed.

Regards,
Yegappan

[tux.]

I really like the approach to unify the scripting interfaces, bringing Vim closer to Acme where any language can extend the editor by echoing plain text.

Thank you for the roadmap, I will definitely be a tester!

[Bram Moolenaar]

Prabir Shrestha wrote:

> Glad to see you are already thinking of vim 9 and it is good to see the
> rythm of having a new goal ever year. I was personally waiting to hear more
> people comment on the ECMAScript
> (https:/ta.github.io/WasmExplorer//github.com/vim/vim/pull/5198) support
> before I share more details but here it goes.
>
> > Just one minor thought, have you thought about storing the intermediate Vim
> > script format into a new binary file (similar to how python uses .py and
> > .pyc)?
>
> This is exactly what WASM (WebAssembly) is. Why not make WASM a first class
> citizen in Vim 9 instead?
>
> FASTER VIMSCRIPT wasm does a fabulous job here and was designed for this.
> Here is an exact quote from their github repo.
> https://github.com/WebAssembly/design

I have considered using something like WASM. WASM is a very low level
execution engine. It has no concept of higher level data structures,
such as lists and dictionaries. It would require a very complex
compiler to turn a Vim script into WASM instructions. Not something we
would include in the Vim binary. It would require an external WASM
compiler, probably with some intermediate format and a final step that
produces the binary code. And then Vim needs to provide a library to
interface WASM code with the rest of Vim. That's going to be very
complicated.

What I'm aiming for is using very high level instructions, such as
"create a list". It is executed by the C code, thus as fast as it can
be. The result is a variable of typval_T, which we already have. Thus
it integrates very well with existing code and can be passed to Vim
script lines that are not compiled. This is something I can implement
in a few months, it is feasible (I implemented that "create a list"
instruction in 10 minutes).

> > "WebAssembly is efficient and fast: Wasm bytecode is designed to be
> > encoded in a size- and load-time-efficient binary format. WebAssembly aims
> > to execute at native speed by taking advantage of common hardware
> > capabilities available on a wide range of platforms."
>
> Currently the new function prototype only makes function fast. There will
> be lot more work to make rest of the features fast. While I would love
> vimscript to be fast I'm not sure if it worth it.

As I mentioned, it only makes sense to compile code that is executed
more than once. That is functions, autocommands and user commands.
If you do have a loop at the script level, you can put it in a function
and then call that function. This nicely separates the old, backwards
compatible (and necessarily slow) Vim script from the new fast Vim
script.

> I would rather have
> vimscript compile to WASM and then execute. This way we only think about
> making the runtime fast. There are also lot of WASM runtimes currently some
> of which even supports JIT and native compilation and other optimizations.
> You can try this website and see what native assembly code it generates.
> https://mbebenita.github.io/WasmExplorer/

As mentioned above, no matter how nice WASM is, compiling Vim script
into it appears to be infeasible.

> PHASING OUT INTERFACES
> Lot of languages now supports compiling to WASM and this would solve the
> issue of having multiple language interfaces. Another option would be to
> use jobs and channels but currently someone still needs to write a small
> wrapper to take full advantage of this. Neovim has a remote RPC concept and
> official libraries for their languages and hence there is more abundance of
> the neovim plugins taking advantage of this and doing more complicated
> stuff then vim8 plugins. I personally don't use any of these interfaces but
> there are lot of people who use python plugins just so they can have the
> speed or use libraries that come with languages.
> WASM fixes this issues about supporting someones best language.

Not sure how this relates to WASM, you can write a tool in any language.
Neovim needs a complex RPC mechanism because they externalize things so
you can run a separate UI. Plugins can do with a much simpler
interface. You mention "I personally don't use any of these
interfaces", which I think is typical for plugin writers. When you
write a UI on top of Neovim then you would need that RPC mechanism.

> BETTER VIMSCRIPT
> This is the one I'm worried the most. I would personally be against
> breaking the language compatibility as it would be pain for us to support
> as a plugin author.

There is no such thing, old Vim scripts will just keep working.

> I don't think comparing javascript with vimscript on adding breaking
> languages is the right comparison. While most devs might write in the ES6
> or newer they either compile it down to ES3 or ES5 so it works in older
> browsers or they compile it to both versions but give hints to browsers to
> only pick one of them so that it continues to work in older browsers.
>
> Here is an example:
> <script src="es5.js" nomodule"></script>
> <script src="es6.js" type="module"></script>
>
>
> Old browsers will only use es5.js because if type is not specified it
> automatically treats it as text/javascript and nomodule is not understood
> by them and they just ignore it. es6.js has type module so while it may
> download that file it doesn't know how to execute it so in terms of
> executing it becomes a noop. For mordern browsers, they understand nomodule
> so it doesn't download es5.js but understands type=module and treats it as
> javascript file and executes it. How would vimscript do this? Same for
> typescript, it compiles down to either ES5 or ES6 or some variant of
> javascript with mostly typings removed.

It doesn't matter what your ES6 is translated into, what matters is what
you write. As it happens, the way variables are declared with "let" and
"const" matches with what Vim script does, thus we can use that concept.
Many other things from ES6 won't ever make it into Vim script, either
because we prefer a mechanism from another language or we don't need it.
We'll have to see what works best.

> Typing is also something I miss a lot in vimscript

I'm going to experiment with this to see how well it works. It's one of
those things that some people consider essential while others are
annoyed by it. If you want to discuss it, please join the Python
discussions on this topic...

> because as a vim plugin author I would most likely
> continue to use func instead of def even if def is faster if I wanted to
> support vimscript. If I really wanted it to be fast I would rather continue
> to use Python or lua interfaces or WASM. I personally prefer
> WASM>LUA>VimScript. Lua is a good alternative if it ships in vim primarily
> because the language it self is very much set and is very powerful with
> features such as co-routines allowing us to express async code in an easy
> way (this is actually my personal pain compared to speed of vimscript).

Lua is not a popular language. It doesn't even support "var += i", as I
found out from writing the example. Python is a lot more popular, but
the embedding doesn't work that great. And it's quite slow, as my
measurements also show. The embedded Lua also isn't that fast either,
you probably need to run Lua as a separate binary for that.

We just have to come to the conclusion that plugin writers don't use the
interfaces much, so let's phase them out.

Multi-threading and coroutines are very complex mechanisms that do not
fit well with the Vim core. Would be an awful lot of work to implement.
Adding a language interface doesn't solve that. I do miss it sometimes
for functionality that really is asynchronous. Maybe for Vim 10?

So write a tool in any language you like and communicate with it from
Vim. In Vim we'll just use Vim script.

> To me I think vim9 should all be about WASM. This solves everything
> mentioned eariler in the thread and in a better way without too much work.
>
> Why?
> * It is fast, really fast. This will only continue to be better due to the
> energy the current industry is betting on. Create a compiler to compile
> vimscript to wasm

How fast can WASM write a file? Or change a line in a buffer? It
doesn't do any of that, it only provides a way to run code in a portable
way. That compiler from Vim script to WASM would be very complex and
I think it would take too long to implement. And then still it would
have to run as a separate binary and require installing WASM support.

It makes much more sense for a plugin to write code in any language,
communicate with Vim using the existing communication mechanisms, and
run it as a separate tool. It can perhaps be compiled to WASM to make
the tool more portable, assuming one has a WASM runtime (is there a WRE
like there is JRE for Java?).

But when Vim script is much faster (think 100 times) it will be good
enough for lots of plugins. Something that now takes 3 seconds becomes
30 msec, goes from slow to unnoticeable.

> * I can write in whatever language I want solving the language interface
> problem. WASM is most likely also officially supported by your favorite
> language.
> * WASI Interface
> https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/
> Think of this as standard c library which is portable.
> * WASM threads https://github.com/WebAssembly/threads ex: using immutable
> data structures to fuzzy search effeciently by sharing between threads.
> * WASM webgl https://rustwasm.github.io/wasm-bindgen/examples/webgl.html
> Similar to popup window but imagine getting a raw canvas to draw in gui vim
> ex: rendering markdown files. Who knows might be one day we will get 3d
> games running inside vim :)
> * WASM c api https://github.com/WebAssembly/wasm-c-api Once this takes off
> I'm hoping it will be easy to swap out runtimes into a faster runtime.

All that doesn't seem useful to execute Vim script. You can use it for
the tool you are writing, but you still need to communicate with Vim.
Perhaps Vim can include a WASM runtime to execute that code in a
separate thread, but it won't be different from running it as a separate
process.

--
GALAHAD: No. Look, I can tackle this lot single-handed!
GIRLS: Yes, yes, let him Tackle us single-handed!
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

[Andy Massimino]

The timings are very encouraging.  Is the primary speedup in the function parsing or better argument passing?

We have lambdas to write filter and map statements but for performance critical applications it's still much faster to use an eval string instead due to the function call overhead.  (I.e. map(a, 'v:val > 5') is faster than map(a, {k,v->v>5}).

--
--
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vim_dev/201912181226.xBICQhsB002372%40masaka.moolenaar.net.

[Bram Moolenaar]

Andy Massimino wrote:

> The timings are very encouraging. Is the primary speedup in the function
> parsing or better argument passing?

The main speedup is that instead of parsing each line over and over
again, it is parsed once and turned into instructions which can be
executed quickly.

What also helps a lot is to not store local variables in a dictionary,
but put them on the stack, so they can be accessed directly.

On top of that, the loop to execute lines has necessary overhead, since
it has to take care of anything that might happen in that line, such as
throwing an exception.

> We have lambdas to write filter and map statements but for performance
> critical applications it's still much faster to use an eval string instead
> due to the function call overhead. (I.e. map(a, 'v:val > 5') is faster
> than map(a, {k,v->v>5}).

That is because of the overhead of calling a function. It has to create
the dictionaries for a: and l:, and several other things to handle
whatever may happen in the function. With the new way calling a
function will still have some overhead (creating a stack frame) but it's
much, much less.


--
### Hiroshima 45, Chernobyl 86, Windows 95 ###

[Gianmaria Bajo]

Pardon the question, but which are the situations right now where vimscript is the bottleneck in terms of speed? From my (maybe little) experience with the editor, if vim is sometimes slow/sluggish is because of a slow syntax highlighting script. What kind of problems would a faster vimscript actually solve?

[Wei Zhang]

Only one request, can we just replace all the endif endwhile endfor endfunction with a single end statement?

[Bram Moolenaar]

Wei Zhang wrote:

> Only one request, can we just replace all the endif endwhile endfor
> endfunction with a single end statement?

It would be even more readable if we use "}":

if cond
do something
}

for i in list
do something
}

{
let local = 0
...
}

"endfunction" should probably be kept, otherwise a missing "}" or "end"
is hard to locate.

--
FATHER: One day, lad, all this will be yours ...
PRINCE: What - the curtains?

"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

[Andy Massimino]

Please don't change endfunction/endif/endwhile... it makes it hard for plugins like match-up to make % work.  Also complicates things for very little benefit, at best saving a tiny amount of typing, at worst makes it easy to lose track of scopes.

--
--
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to the Google Groups "vim_dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vim_dev/201912182127.xBILR5q1006083%40masaka.moolenaar.net.

[Gary Johnson]

On 2019-12-18, Andy Massimino wrote:
> Please don't change endfunction/endif/endwhile... it makes it hard for plugins
> like match-up to make % work.  Also complicates things for very little benefit,
> at best saving a tiny amount of typing, at worst makes it easy to lose track of
> scopes.

+1

Regards,
Gary

[Tony Mechelynck]

+2. Replacing them all by a single "end" or "}" statement might make
it harder to spot a missing close statement, it would require a
complete rewrite of the matchit plugin, and all that for little
benefit. Do we have a common end bracket for (, [ and { ? No. So why
have one for if, for, while and func? OTOH, IMHO what we have now is
clearer than (for the sake of less typing) replacing them by fi, rof,
elihw and cnuf (or noitcnuf).

If a single end statement had been wanted, Vim could have used
begin..end statement brackets à la ALGOL and required the use of a
single imperative statement or begin..end block inside if, while, etc.
statements; but it didn't take that route and IMHO it's too late now.

Best regards,
Tony.

[skywind3000]

Please provide a standard way to locate project root directory.

Many famous plugins (including: CtrlP, LeaderF, Fzf, YouCompleteMe, Gutentags, ALE)

require to identify the project root of current file. And they don't have standard to follow,

So, some of plugins take current directory as project root, some of them take the nearest 

ancestor directory containing .git or .svn as the project root. some of them seek the nearest

Makefile / CMakeList.txt ...

what a mess ?

And most of them have project-wide configurations like:

1. ALE: project-wide linting configurations.

2. YouCompleteMe: ycm_extra_conf.py, used to generate clang compiling flags (include dir, ...) for sources 

3. vim-clang: compile_commands.json

4. local_vimrc: _vimrc_local.vim

5. gutentags: gtags / ctags parameter config file.

These config files pollute my source tree. and can easily get committed to svn/git by accident. 

Consider supporting vscode's `.vscode` folder ? it can be used to locate project root (nearest parent dir containing .vscode).

And can be used to store many project-wide configs.

Or with a different name like `.vim` ??

benefits:

1. all plugins can have a standard to locate project.

2. all plugins can have a place to store project-wide configs.

3. possible for plugins to share information and collaborate together.

[Gary Johnson]

On 2019-12-19, skywind3000 wrote:
> Please provide a standard way to locate project root directory.

This is an attractive idea, but there is no standard way to locate
a project root directory.

I have a plugin that attempts to do this. It has evolved over many
years and contains several different methods for identifying project
root directories. Each company I have worked for has had different
ideas of how project directories should be structured and where they
should be put. For company, open-source and personal projects, some
have VCS files in the project root and some don't. For a project
using git submodules, is a submodule a project or does the top-level
directory contain the project?

I want to be able to identify a project even if it is not my own
directory and the owner doesn't use vim, so putting a file such as
.vim_project in the root directory is out. Also, I really don't
want dot files cluttering every directory that I consider a project.
(My own plugin uses heuristics and a database of exceptions that
resides under ~/.vim.)

We can't impose a standard on the world's projects and personal
requirements and preferences vary a lot, so I think this problem is
best solved by using plugins, not by adding something to the Vim
executable.

Regards,
Gary

[skywind3000]

a `.vim_project` file to mark the project root ? It's fine, but give us a 

standard, or:

plugin A uses `.vim_project1`, plugin B uses `.vim_root`, plugin C

uses `.project_root` and plugin C uses `.root`.

And plugin D uses `.git`, and due to submodule, or some other reasons

You will get totally different project roots for the same file.

what a mess ?

在 2019年12月20日星期五 UTC+8上午5:24:09，Gary Johnson写道：

[Christian Brabandt]

On Do, 19 Dez 2019, skywind3000 wrote:

> a `.vim_project` file to mark the project root ? It's fine, but give us a 
> standard, or:
>
> plugin A uses `.vim_project1`, plugin B uses `.vim_root`, plugin C
> uses `.project_root` and plugin C uses `.root`.
>
> And plugin D uses `.git`, and due to submodule, or some other reasons
>
> You will get totally different project roots for the same file.
>
> what a mess ?

Agree, but I suppose it's not Vim that sets the standard, we should only
support whatever seems to have been established as standard. Perhaps
editorconfig could be used for that (see also the open issue:
https://github.com/vim/vim/issues/2286
)


Best,
Christian
--
Als Gott den Mann erschuf, übte sie nur.

[skywind3000]

Obviously, there are project related problems needs to be solved. At least we have consensus below:

1. use something which seems to have been established as standard.

2. which is easy to be implemented by plugins with minimal dependency.

editorconfig seems like a good choice, but only one thing, there can by multiple `.editorconfig` file

in one project.

And you need to parse the `.editorconfig` to check if there is a `root = true` in it, without actually

parsing one `.editorconfig` file, the project tree cannot be located. 

It is not simple to implemented by plugins without introducing new dependency (the `EditorConfig` plugin).

So, let's find some alternatives.

OK, here is a new one:

    The vscode's `.vscode` folder

1. it can be used for both locating project tree and storing project-specific configs, 

2. it is well known and nearly a standard.

3. it is easy to be implement by plugins respectively without any dependency and can also run in old vims.

how about this ??

--

在 2019年12月20日星期五 UTC+8下午8:30:03，Christian Brabandt写道：

[Gary Johnson]

On 2019-12-20, skywind3000 wrote:
> Obviously, there are project related problems needs to be solved. At least we
> have consensus below:
>
> 1. use something which seems to have been established as standard.
> 2. which is easy to be implemented by plugins with minimal dependency.
>
> editorconfig seems like a good choice, but only one thing, there can by
> multiple `.editorconfig` file
> in one project.
>
> And you need to parse the `.editorconfig` to check if there is a `root = true`
> in it, without actually
> parsing one `.editorconfig` file, the project tree cannot be located. 
>
> It is not simple to implemented by plugins without introducing new dependency
> (the `EditorConfig` plugin).
>
> So, let's find some alternatives.
>
> OK, here is a new one:
>
>     The vscode's `.vscode` folder
>
> 1. it can be used for both locating project tree and storing project-specific
> configs, 
> 2. it is well known and nearly a standard.
> 3. it is easy to be implement by plugins respectively without any dependency
> and can also run in old vims.
>
> how about this ??

Using .vscode would probably work great for users of Visual Studio
Code. I do almost all my development on Linux, never use Visual
Studio and had to look up what .vscode was. .vscode is not well
known by other than VSC users. This is a good example of why and
how plugins are a good solution for this. VSC users can use
a plugin that looks for .vscode directories; git users can use
a plugin that looks for .git directories; etc.

Another consideration is that some of us use Vim for a lot of little
editing tasks, not just long project editing sessions, and we
wouldn't want it to spend a lot of start-up time determining
whether or not it's in a project environment.

On a more positive note: A common idea here seems to be looking for
a certain file name in the parent directory of the current file or
above it in the directory hierarchy. If Vim can do such a search
internally quickly, it might work to have an option that lists the
file names to be searched for and by some notation such as an
appended '!' indicate whether Vim is to stop its search at the first
or last such name found and declare that directory to be the project
directory. Vim's findir() and findfile() functions already do most
of that work. That still leaves the problem of using that directory
name to find Vim's configuration for that project.

Someone could try to write a universal project-detection plugin
using these ideas and if it truly is universal, propose that it be
added to Vim's standard plugin set or incorporated into Vim's code
internally. Until we have such a working model, I think it's
premature to add anything to Vim's internal code.

Regards,
Gary

[skywind3000]

Great idea, a universal script to locate project.

Emacs has a plugin named `projectile` which is definitely the "universal script" for emacs,

It has been widely used and many other emacs plugins (helm / ivy / rails / ...) use it to 

retrive project location of the current file.

Here I present a simple one, `projectile.vim`:

" root markers used to locate project

if !exists('g:projectile#marker')

    let g:projectile#marker = '.svn,.git,.projectile'

endif

" returns nearest parent directory contains one of the markers

function! projectile#find_root(name)

    let name = fnamemodify((a:name != '')? a:name : bufname(), ':p')

    let finding = ''

    " iterate all markers

    for marker in split(g:projectile#marker, ',')

        if marker != ''

            " search as a file

            let x = findfile(marker, name . '/;')

            let x = (x == '')? '' : fnamemodify(x, ':p:h')

            " search as a directory

            let y = finddir(marker, name . '/;')

            let y = (y == '')? '' : fnamemodify(y, ':p:h:h')

            " which one is the nearest directory ?

            let z = (strchars(x) > strchars(y))? x : y

            " keep the nearest one in finding

            let finding = (strchars(z) > strchars(finding))? z : finding

        endif

    endfor

    return (finding == '')? '' : fnamemodify(finding, ':p')

endfunc

" returns nearest parent directory contains one of the markers

" if matching failed, returns the directory containing the file

function! projectile#find_home(name)

    let name = fnamemodify((a:name != '')? a:name : bufname(), ':p')

    let path = projectile#find_root(name)

    if path != ''

        return path

    else

        return isdirectory(name)? name : fnamemodify(name, ':p')

    endif

endfunc

Sorry I don't know how to handle `!` in your proposal, but it is simple enough. 

It can be used to perform a project-wide grep:

" project-wide grep

function! projectile#grep(what)

    let home = projectile#find_home('')   " get the project home of current file

    if home != ''

        exec "vimgrep /" . a:what . "/g " . fnameescape(home) . '**/*'

    endif

endfunc

or press F9 to return project home:

" goto project home

function! projectile#go_home()

    let home = projectile#find_home('')

    if home != ''

        exec "e " . fnameescape(home)

    endif

endfunc

noremap <F9> :call projectile#go_home()<cr>

here is the gist:

https://gist.github.com/skywind3000/690618557f22de7aaab9aa0226b7edef

For old vim compatibility, if a new plugin using projectile.vim want to work in 

old vims, then can just copy these two function into their own source, not hard.

Is it simple enough ? Could it be shipped with Vim now ?

在 2019年12月21日星期六 UTC+8上午2:53:24，Gary Johnson写道：

[Ben Jackson]

Bram, if you're going to be a doing a fair amount of work on the vimscript side of things, one thing I would like to request is to have a mind towards being able to support a full-on vimscript debugger. I realise there is a built-in debugging mode, but it would be exceptionally cool to be able to implement a Debug Adapter Protocol server within vim, so that something like Vimspector could be used to debug plugin code externally.

Obviously doesn't have to be done right away, but having it as a medium term goal would be great. As always, happy to contribute.