Vim9 script: first steps

[Bram Moolenaar]

I have created a repository for the Vim9 script experiments:
https://github.com/brammool/vim9

I did another measurement for a more realistic example, re-indenting a
large number of lines. In old Vim script it would be:

let totallen = 0
for i in range(1, 100000)
call setline(i, ' ' .. getline(i))
let totallen += len(getline(i))
endfor

The timing differences are much smaller than for the computational
example, but Vim9 script is clearly the fastest:

Vim old: 0.853752
Python: 0.304584
Lua: 0.286573
Vim new: 0.190276

Compared to legacy Vim script it is a 4 times faster.

If you want to look at the instructions that are used internally, the
":disassemble" command shows what's the compilation result. For
example, this function:

def VimNew(): number
let totallen = 0
for i in range(1, 100000)
setline(i, ' ' .. getline(i))
totallen += len(getline(i))
}
return totallen
enddef

Results in this:

let totallen = 0
0 STORE 0 in $0

for i in range(1, 100000)
1 STORE -1 in $1
2 PUSHNR 1
3 PUSHNR 100000
4 BCALL range(argc 2)
5 FOR $1 -> 21
6 STORE $2

setline(i, ' ' .. getline(i))
7 LOAD $2
8 PUSHS " "
9 LOAD $2
10 BCALL getline(argc 1)
11 CONCAT
12 BCALL setline(argc 2)
13 DROP

totallen += len(getline(i))
14 LOAD $0
15 LOAD $2
16 BCALL getline(argc 1)
17 BCALL len(argc 1)
18 ADDNR
19 STORE $0

}
20 JUMP -> 5
21 DROP

return totallen
22 LOAD $0
23 RETURN


Obviously there is still an awful lot of work to be done. Fortunately,
the numbers show it's worth it.

--
The Law of VIM:
For each member b of the possible behaviour space B of program P, there exists
a finite time t before which at least one user u in the total user space U of
program P will request b becomes a member of the allowed behaviour space B'
(B' <= B).
In other words: Sooner or later everyone wants everything as an option.
-- Vince Negri

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[mattn]

Why you don't use { ?

[Prabir Shrestha]

It is great to see the numbers. @Bram What about something like this so it is 100% backwards compatible. The best part is that it works in current vim8 without any errors.

function! s:greet(name) abort

  "use strict"

  echom a:name

endfunction

call s:greet('Vim9')

Javascript does this too. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Strict_mode. Here is their exact quotes from the website.

Strict mode makes several changes to normal JavaScript semantics:

1. Eliminates some JavaScript silent errors by changing them to throw errors.

1. Fixes mistakes that make it difficult for JavaScript engines to perform optimizations: strict mode code can sometimes be made to run faster than identical code that's not strict mode.

1. Prohibits some syntax likely to be defined in future versions of ECMAScript.

Basically in strict mode it would still use a:name as the variable name but it is no longer a dictionary. This way all new vimscript is 100% compatible with old vimscript < 9

This one didn't work.

function! s:greet(name) strict

  "use strict"

  echom a:name

endfunction

call s:greet('Vim9')

The reason first one works without errors is because we are using " which is a start of a comment.

Another option would be to do something like this so you don't have to peek to the next line.

function! s:greet(name) abort " use strict

  echom a:name

endfunction

call s:greet('Vim9')

We can come up with some magic comment to opt in to strict mode.

As for a:000 compatibility, that list can only exist in strict mode when using ...

function! s:greet(name, ...) abort " use strict

  echom a:name . json_encode(a:000)

endfunction

We could do the same with l:variables. In strict mode it is no longer a dictionary but a direct ref.

To me backwards compatibility would be very important.

[Bram Moolenaar]

Prabir Shrestha wrote:

> It is great to see the numbers. @Bram What about something like this so it
> is 100% backwards compatible. The best part is that it works in current
> vim8 without any errors.
>
> function! s:greet(name) abort
> "use strict"
> echom a:name
> endfunction
>
> call s:greet('Vim9')
>
> Javascript does this too.
> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Strict_mode.
> Here is their exact quotes from the website.
>
> > Strict mode makes several changes to normal JavaScript semantics:
> >

> > 1. Eliminates some JavaScript silent errors by changing them to throw
> > errors.
> >
> >
> > 1. Fixes mistakes that make it difficult for JavaScript engines to

> > perform optimizations: strict mode code can sometimes be made to run faster
> > than identical code that's not strict mode.
> >
> >

> > 1. Prohibits some syntax likely to be defined in future versions of

> > ECMAScript.
>
> Basically in strict mode it would still use a:name as the variable name but
> it is no longer a dictionary. This way all new vimscript is 100% compatible
> with old vimscript < 9

I don't see that. If a: is no longer available as a dictionary it won't
be 100% compatible. There are other subtle assumptions, e.g. about how
error messages are handled. It will work like "abort" was present.

> This one didn't work.
>
> function! s:greet(name) strict
> "use strict"
> echom a:name
> endfunction
>
> call s:greet('Vim9')
>
> The reason first one works without errors is because we are using " which
> is a start of a comment.
>
> Another option would be to do something like this so you don't have to peek
> to the next line.
>
> function! s:greet(name) abort " use strict
> echom a:name
> endfunction
>
> call s:greet('Vim9')
>
> We can come up with some magic comment to opt in to strict mode.

An even easier way would be to try compile the function and if it fails
then assume it's not compatible. Thus it is 100% backwards compatible,
and it will be faster without changing it to a "def" function. It will
not be as fast as a "def" function, because types have to be checked at
runtime. But it should be a lot faster than not compiling at all.

I'm not sure this is possible. It will require very accurate checking
of the syntax. We can give this a try at some point. I first want to
make the "def" function compile, that is still quite a bit of work.

> As for a:000 compatibility, that list can only exist in strict mode when
> using ...
>
> function! s:greet(name, ...) abort " use strict
> echom a:name . json_encode(a:000)
> endfunction
>
>
> We could do the same with l:variables. In strict mode it is no longer a
> dictionary but a direct ref.
>
> To me backwards compatibility would be very important.

It sounds more like forward compatibility: old code works better.

--
If cars evolved at the same rate as computers have, they'd cost five euro,
run for a year on a couple of liters of petrol, and explode once a day.

[Bram Moolenaar]

Yasuhiro Matsumoto wrote:

> Why you don't use { ?

I know this will trigger a discussion. Nearly all languages use blocks
ending with "}". It's much easier to see where a block ends that way
than using "endif", "endwhile", etc., it stands out.

Since we already have the start of the block without extra punctuation,
there is no need to add a "{" there. It would be pointless to add it.

The only place where "{" will appear is a block by itself, used to
declare a variable:

{
let tmp = 0
...
}
" tmp not available here

If you don't like using "}" you can use "endif", "endfor", etc. as before.

--
Why isn't there mouse-flavored cat food?

[Ken Takata]

Hi Bram,

2020/1/3 Fri 19:02:13 UTC+9 Bram Moolenaar wrote:

Yasuhiro Matsumoto wrote:

> Why you don't use { ?

I know this will trigger a discussion.  Nearly all languages use blocks
ending with "}".  It's much easier to see where a block ends that way
than using "endif", "endwhile", etc., it stands out.

Since we already have the start of the block without extra punctuation,
there is no need to add a "{" there.  It would be pointless to add it.

Text objects like "a{" or "i{" don't work well without the starting "{".

Also "%" may not work well.

 

 

The only place where "{" will appear is a block by itself, used to
declare a variable:

        {
           let tmp = 0
           ...
        }
        " tmp not available here

If you don't like using "}" you can use "endif", "endfor", etc. as before.

Regards,

Ken Takata

[Bram Moolenaar]

Ken Takata wrote:

> 2020/1/3 Fri 19:02:13 UTC+9 Bram Moolenaar wrote:
> >
> >
> > Yasuhiro Matsumoto wrote:
> >
> > > Why you don't use { ?
> >
> > I know this will trigger a discussion. Nearly all languages use blocks
> > ending with "}". It's much easier to see where a block ends that way
> > than using "endif", "endwhile", etc., it stands out.
> >
> > Since we already have the start of the block without extra punctuation,
> > there is no need to add a "{" there. It would be pointless to add it.
> >
>
> Text objects like "a{" or "i{" don't work well without the starting "{".

That never worked for "for" / "endfor", right?

> Also "%" may not work well.

With some adjustments to b:match_words it works.

And when "%" works you can select a block from "if"/"while"/"for" until
the matching "}" with "v%".

--
TIM: That is not an ordinary rabbit ... 'tis the most foul cruel and
bad-tempered thing you ever set eyes on.
ROBIN: You tit. I soiled my armour I was so scared!
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

[mattn]

If you are thinking vim9 accept to use lambda in the expression in while/for, it will be confusable.

while {ｰ>1}()
...
}

[Dominique Pellé]

That's confusing indeed. I'd rather keep { … } for blocks.

Now that's a matter of personal preference, but I find
the asymmetry of having a closing } without opening {
a bit ugly, and I suspect I won't be alone there. It also
annoyingly breaks pressing % to jump to a matching
curly.

I'd rather keep the new vim script as similar as the old
vim script language as possible, unless there is a very
good reason (e.g. performance boost). Cleaning
up the syntax for sometimes arbitrary personal
preference should be avoided. So I'd rather:
* keep opening curlies { ... }
* keep a: for argument prefixes
* keep let x = … instead of x = …

Many vim users never really learned vim script.
If there are 2 different vim scripts flavors for
unwarranted reasons, developers will be even
less likely to want to learn it.

Python2->Python3 and Perl5->Perl6 have shown that
trying to improve the syntax of the language can lead
to many problems lasting for years.

Regards
Dominique

[Andy Massimino]

On 1/3/20 8:17 AM, Bram Moolenaar wrote:
> Ken Takata wrote:
>
>> 2020/1/3 Fri 19:02:13 UTC+9 Bram Moolenaar wrote:
>>>
>>> Yasuhiro Matsumoto wrote:
>>>
>>>> Why you don't use { ?
>>> I know this will trigger a discussion. Nearly all languages use blocks
>>> ending with "}". It's much easier to see where a block ends that way
>>> than using "endif", "endwhile", etc., it stands out.
>>>
>>> Since we already have the start of the block without extra punctuation,
>>> there is no need to add a "{" there. It would be pointless to add it.
>>>
>> Text objects like "a{" or "i{" don't work well without the starting "{".
> That never worked for "for" / "endfor", right?
>
>> Also "%" may not work well.
> With some adjustments to b:match_words it works.

b:match_words sequences between if/else/endif and
function/return/endfunction tuples not just if/endif
function/endfunction pairs.  Adjusting b:match_words to handle '}' ends
breaks this.  There is no way to handle this with b:match_words (do you
know of one?)

Consider:

function A() | if x | return | endif | endfunction

vs:

function A() | if x | return | } | }

One of the nicer things about vimscript (with respect to %) is 'return'
can only match with 'endfunction', 'break' with 'endwhile'/'endfor',
'else' with 'endif', etc.

[mattn]

Simply poor readability for me.

And it will be difficult to support Vim script syntax highlighting in other text editors or viewer (like GitHub).

[Prabir Shrestha]

-1 for }. That is the most absurd syntax I have seen in any language that only uses it for closing.

So I went and re-read the poll https://github.com/vim/vim/issues/3573. And here is what I can think of reading it again in Jan 2020.

Can we get more info on what does make vim-script faster mean?

Second comment there says about syntax highlighting. So making vimscript faster here won't help much because [tree-sitter](https://tree-sitter.github.io/tree-sitter/) will be lot faster than vimscript, which means syntax highlighting in neovim will also be more accurate and faster then vim.

Instead of picking up non real world examples to demonstrate perf gains can we pick some proper goals.

Here are some of the examples I can think of:

* Faster fuzzy finder. Currently there is 0 plugins that is fast in pure vimscript. I would like to see an example of FuzzyFinder in vim9 that is fast as https://github.com/liuchengxu/vim-clap and https://github.com/junegunn/fzf.vim. Few requirements here: Searching large number of nested directories and files, highlighting matches. Some of these I elaborated in this comment https://github.com/vim/vim/issues/3573#issuecomment-433730939. 

* vim9 syntax highlighting vs neovim tree-sitter syntax highlighting. We should count speed as well as accuracy.

* Fuzzy search for auto complete popup menu. 

I still think WASM is the way to go and not vimscript 9. This way I can code in what ever language suits me and just ship the bytecode. 

For example: will vimscript 9 support generics/async,await/pattern matching/threading or greenthreads/immutability? This list will keep going on. Making it fast is just one issue it solves. A language gets attractive not just for perf but many other features that come around it.

Here is a new WASM runtime since I last mentioned which claims to be 15x faster then other c wasm runtimes. https://github.com/wasm3/wasm3. 

As a plugin author I can then code in any languages I want that suits the needed. For example a python dev might write in python, a JS dev might right in Javascript or Typescript if they need typing. And they can still write in go/rust/zig or even c if they need performance. And most languages these days already support compiling to WASM.

[Tony Mechelynck]

Hm.

- Vim must keep support of Vim script for compatibility reasons

- WASM means "web assembly", right? Well, Vim must go on working even when no web connection is available, and I'm not going to code my vimrc in assembly language, which is the least compatible source language there is. (x86 even has two of them, MASM and AT&T assembler).

Just my 0.02 €.

Best regards,

Tony.

[Prabir Shrestha]

My goal is not to remove VimScript. It should remain there. But instead of adding if_my_favorite_langauge. We should add if_wasm and call stop adding any other languages.

"Web" in WASM is misleading, it has nothing to do with "Web" as in internet or browser similar to Java in Javacsript has nothing to Java programming.

While you can write in WASM assembly. You will most likely never write it. You write in a higher language such as c, rust, javascript which then compiles to WASM binaries or WASM assemblies.I'm more interested in WASM binaries as they can be compact and are portable across various machine architecture and OS.

You can try here: https://mbebenita.github.io/WasmExplorer/

Here is a simple example:

You write in your favorite language that can compile to WASM.

I'm using C here:

int add(int a, int b) {

  return a+b;

}

int main() {

   int result = add(1,2);

}

Then it converts to WASM assembly or WASM binary which is portable:

WASM assembly looks like this. WASM binaries are portable and lot more efficient at running then manually parsing this ASM and executing.

(module

 (table 0 anyfunc)

 (memory $0 1)

 (export "memory" (memory $0))

 (export "_Z3addii" (func $_Z3addii))

 (export "main" (func $main))

 (func $_Z3addii (; 0 ;) (param $0 i32) (param $1 i32) (result i32)

  (i32.add

   (get_local $1)

   (get_local $0)

  )

 )

 (func $main (; 1 ;) (result i32)

  (i32.const 0)

 )

)

Based on the WASM VM you choose it can do other optimizations.

And here is the WASM generating x86 ASM code:

wasm-function[0]:

  sub rsp, 8                            ; 0x000000 48 83 ec 08

  mov ecx, esi                          ; 0x000004 8b ce

  mov eax, ecx                          ; 0x000006 8b c1

  add eax, edi                          ; 0x000008 03 c7

  nop                                   ; 0x00000a 66 90

  add rsp, 8                            ; 0x00000c 48 83 c4 08

  ret                                   ; 0x000010 c3

wasm-function[1]:

  sub rsp, 8                            ; 0x000000 48 83 ec 08

  xor eax, eax                          ; 0x000004 33 c0

  nop                                   ; 0x000006 66 90

  add rsp, 8                            ; 0x000008 48 83 c4 08

  ret                                   ; 0x00000c c3

As you can see there is nothing about internet or browser in this code. 

As you can see, WASM has first class notion of types (i32) allowing WASM VMs to optimize natively. We don't need vimscript to implement a type.

This mean I can use the entire ecosystem of that particular language which includes libraries that the language uses as well as others have written.

So my thought here is let us first get a WASM VM running natively in vim as first class citizen and then we can worry about writing a better VM optimizing it to be faster.

WASM also have other extensions point some of which are official spec and some which can be own.

* WASI - WebAssembly System Interface, think of this like the C stdlib. This allows WASM to make system calls similar to read/write in c.

* WASM threads - Allows to use threading, locks. 

* and many more

[Ingo Karkat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 05-Jan-2020 01:22 +0100, Prabir Shrestha wrote:

> -1 for }. That is the most absurd syntax I have seen in any
> language that only uses it for closing.
>
> So I went and re-read the poll
> https://github.com/vim/vim/issues/3573. And here is what I can
> think of reading it again in Jan 2020.
>
> Can we get more info on what does make vim-script faster mean?
>

> Screen Shot 2020-01-04 at 3.53.13 PM.png
>

> Screen Shot 2020-01-04 at 4.04.27 PM.png
>

I agree; most performance issues I encounter are in syntax
highlighting with large, often deeply nested structures (like big XML
or JSON files). Poor Vimscript performance hardly is an issue for the
kind of plugins that I write; and it's always been possible to use a
language binding and reimplement the core parts, or call out to an
external program (now even easier with the jobs feature). If WASM is
the future (and you present enticing arguments here), deprecating the
many existing language bindings in favor of WASM as the single
go-forward integration would indeed contribute to a simplified code
base, avoid fragmented plugin requirements, and therefore reduce
compatibility issues in the diverse user base.

Still, if Bram has ideas about eliminating some inefficiencies of
Vimscript, so that Vimscript writers can easily opt-in (ideally, in a
backwards-compatible way and without rewriting entire functions or
plugins), why not? Vim, with its great backwards compatibility and
platform support, has been a pleasure to use and extend for many many
years; I'll happily let Bram indulge in such pet projects (and the
accompanying bike shedding about syntax details :-) if that's what
keeps him happy and committed to the project!

- -- regards, ingo
- --
-- Ingo Karkat -- /^-- /^-- /^-- /^-- /^-- http://ingo-karkat.de/ --
-- https://www.vim.org/account/profile.php?user_id=9713 --
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJeEa+8AAoJEA7ziXlAzQ/vX3wH/35PUWts94SNz8IGT2PCG24V
Cs7x14Zj4774qjL6ZWi+rn/7pdzUAzTqapsBoLdBz8qZM3bjV/J0ydwg8D8xSYzo
tCS3LwHL6wGOYpYmg8KuB7+joAkn5TCfOH69f5aXwAje6fL0g91aigYGRmR35Bze
IX0sQwfLesH/zArL1qk7G8pqjrRmmSpG63zyABOeYaTPbKB6rwJ/Dy9vsRdjDAEM
Y/vzqnC6aGHCjGhD/kVzlBaFZTV4YD2TxDdHSXRd+47j5LRr0J4QKLYJj91wgzCE
je5vZn1OTETlAUXzI9+B1u12/XVYEFba4PFQr5EfAY81cZXoF0aWhJ6rwPj1Xj8=
=nFRe
-----END PGP SIGNATURE-----

[Bram Moolenaar]

Ingo Karkat wrote:

> I agree; most performance issues I encounter are in syntax
> highlighting with large, often deeply nested structures (like big XML
> or JSON files). Poor Vimscript performance hardly is an issue for the
> kind of plugins that I write; and it's always been possible to use a
> language binding and reimplement the core parts, or call out to an
> external program (now even easier with the jobs feature). If WASM is
> the future (and you present enticing arguments here), deprecating the
> many existing language bindings in favor of WASM as the single
> go-forward integration would indeed contribute to a simplified code
> base, avoid fragmented plugin requirements, and therefore reduce
> compatibility issues in the diverse user base.

I don't think WASM helps at all with syntax highlighting. The good old
Vim syntax highlighting depends on regexp patterns. I have already
optimized the speed of that, I don't think it can be improved much more.

The main alternative now is to use an external parser or language server
and communicate with it. Such as the govim plugin does. You an now use
text properties to highlight the text.

Perhaps tree sitter can be used. It's still new, makes some promises,
but we would need to see how it can integrate with Vim.

> Still, if Bram has ideas about eliminating some inefficiencies of
> Vimscript, so that Vimscript writers can easily opt-in (ideally, in a
> backwards-compatible way and without rewriting entire functions or
> plugins), why not? Vim, with its great backwards compatibility and
> platform support, has been a pleasure to use and extend for many many
> years; I'll happily let Bram indulge in such pet projects (and the
> accompanying bike shedding about syntax details :-) if that's what
> keeps him happy and committed to the project!

--

[Another hideous roar.]
BEDEVERE: That's it!
ARTHUR: What?
BEDEVERE: It's The Legendary Black Beast of Aaaaarrrrrrggghhh!

[Bram Moolenaar]

Prabir Shrestha wrote:

> -1 for }. That is the most absurd syntax I have seen in any language that
> only uses it for closing.

OK, it appears nobody likes it. The whole idea of experimenting is to
try out alternatives.

> So I went and re-read the poll https://github.com/vim/vim/issues/3573. And
> here is what I can think of reading it again in Jan 2020.
>
> Can we get more info on what does make vim-script faster mean?
>
> [image: Screen Shot 2020-01-04 at 3.53.13 PM.png]
>
> [image: Screen Shot 2020-01-04 at 4.04.27 PM.png]
>
> [image: Screen Shot 2020-01-04 at 4.05.36 PM.png]
>
>
>
> Second comment there says about syntax highlighting. So making vimscript
> faster here won't help much because [tree-sitter](
> https://tree-sitter.github.io/tree-sitter/) will be lot faster than
> vimscript, which means syntax highlighting in neovim will also be more
> accurate and faster then vim.

As I mentioned before, using an external tool is the main alternative.
With channel support and text properties the pieces are already
available today.

> Instead of picking up non real world examples to demonstrate perf gains can
> we pick some proper goals.
>
> Here are some of the examples I can think of:
>
> * Faster fuzzy finder. Currently there is 0 plugins that is fast in pure
> vimscript. I would like to see an example of FuzzyFinder in vim9 that is
> fast as https://github.com/liuchengxu/vim-clap and
> https://github.com/junegunn/fzf.vim. Few requirements here: Searching large
> number of nested directories and files, highlighting matches. Some of these
> I elaborated in this comment
> https://github.com/vim/vim/issues/3573#issuecomment-433730939.

It's impossible to measure the speed of a whole plugin without
implementing 99% of the compiling and execution. We'll have to start
with simple examples to check if we are going in the right direction.

Feel free to come up with some part that needs to be fast and is simple
enough we could implement that part of the compilation/execution.

> * vim9 syntax highlighting vs neovim tree-sitter syntax highlighting. We
> should count speed as well as accuracy.

As mentioned above, you can already do that. With Vim9 script the part
inside Vim will be faster. By how much is hard to predict.

> * Fuzzy search for auto complete popup menu.
>
>
> I still think WASM is the way to go and not vimscript 9. This way I can
> code in what ever language suits me and just ship the bytecode.

I already explained that using an external tool allows you to do this.

> For example: will vimscript 9 support generics/async,await/pattern
> matching/threading or greenthreads/immutability? This list will keep going
> on. Making it fast is just one issue it solves. A language gets attractive
> not just for perf but many other features that come around it.

We can add some more in Vim9, but it's not what plugin authors have been
asking for.

> Here is a new WASM runtime since I last mentioned which claims to be 15x
> faster then other c wasm runtimes. https://github.com/wasm3/wasm3.

I'm sure other languages like C and Go will be even faster.

> As a plugin author I can then code in any languages I want that suits the
> needed. For example a python dev might write in python, a JS dev might
> right in Javascript or Typescript if they need typing. And they can still
> write in go/rust/zig or even c if they need performance. And most languages
> these days already support compiling to WASM.

The point I made was that you can use any language you like, just run it
as an external tool and communicate with Vim. I don't see how
integrating any language interface inside Vim would work better. We do
have enough evidence that the exising ones are not popular, adding
another one is unlikely going to change that.

--
"When I die, I want a tombstone that says "GAME OVER" - Ton Richters

[Ben Jackson]

Hi Bram, thanks for sharing. Looks really promising.

Regarding https://github.com/brammool/vim9#3-better-vim-script, I have a few ideas for your consideration:

* Execution often relies on user configuration (ignorecase, magic, etc.) and there is common boilerplate required in many scripts to set/restore cpo. Could we say that when `scriptversion` is `vim9`, we always execute as if `set cpo&vim` ?

* Line continuation. I know this is probably hard to change, but could we look at a line-continuation scheme that's more familiar ? As we don't have line termination, Perhaps something more like python's e.g. if a line ends with an unbalanced parenthesis (or a `<backslash><newline>`), then assume the next line is a continuation? I think the `<newline><backslash>` syntax is unfamiliar to a lot of people.

* Scoped functions or lambdas containing ex-commands (or better, def function bodies). When combined with try/finally, this would allow constructs that take a "block" of code, such as "WithCurrentBuffer( buf, { => block of code/commands, maybe split across lines } )" or something like this when working with external tools `SendMessageToTool( message, then = { => some block of ex commands } ). This can currently be done with Funcref, but I think that you have to do 'function ...' and then 'delfiunc' as all functions are currently global. Maybe the new scoping rules could be extended to allow this.

I already mentioned elsewhere that the other barrier to entry for vim scripting is tooling and debugging. Hopefully with a these changes we can consider how to:

* extract the parser so that an external tool can parse and do code comprehension, go-to, refactor, etc., think Vim Language Server (there is one, but it's not canonical)

* provide a interface to a debug adapter to allow debugging of vimscript executing in a Vim instance (i.e. Vim Debug Adapter)

* provide better source/line information for stack traces (so that we can reliably jump to location of assertions e.g. test failures using cnext). I saw a recent patch which improves sourcing_lineno and uses a proper stack. I was planning to work on a patch to store source/line info when debugging is enabled, but might hold off while vim 9 shakes down, as this is likely required for the above debugger interface (assuming that's a goal).

Just a few ideas to throw in to the mix!

Thanks again, 

Ben

[Christian Brabandt]

On So, 05 Jan 2020, Bram Moolenaar wrote:

> Perhaps tree sitter can be used. It's still new, makes some promises,
> but we would need to see how it can integrate with Vim.

We could have a look at the Neovim implementation. It has been included
there as well (and as far as I remember, @bfredl mentioned that his work
can also be included here).

Best,
Christian
--
Die Politik ist das Paradies zungenfertiger Schwätzer.
-- George Bernard Shaw

[Bram Moolenaar]

Christian wrote:

> On So, 05 Jan 2020, Bram Moolenaar wrote:
>
> > Perhaps tree sitter can be used. It's still new, makes some promises,
> > but we would need to see how it can integrate with Vim.
>
> We could have a look at the Neovim implementation. It has been included
> there as well (and as far as I remember, @bfredl mentioned that his work
> can also be included here).

I believe it's still under construction. I haven't yet been able to
find documentation, only some Lua interface (which looks complicated).

--
Sorry, no fortune today.

[Bram Moolenaar]

Ben Jackson wrote:

> Hi Bram, thanks for sharing. Looks really promising.
>
> Regarding https://github.com/brammool/vim9#3-better-vim-script, I have a
> few ideas for your consideration:
>
> * Execution often relies on user configuration (ignorecase, magic, etc.)
> and there is common boilerplate required in many scripts to set/restore
> cpo. Could we say that when `scriptversion` is `vim9`, we always execute as
> if `set cpo&vim` ?

For :def functions those will be parsed as if 'nocompatible' is set.

For 'ignorecase' and 'magic' we can ignore those for the =~ and !~
operators. What else?

Not sure yet what will happen at the script level. No changes are
needed for speed. If needed the file can be put inside :def/:enddef
and then calling that function. Not ideal, but at least it's possible.

> * Line continuation. I know this is probably hard to change, but could we
> look at a line-continuation scheme that's more familiar ? As we don't have
> line termination, Perhaps something more like python's e.g. if a line ends
> with an unbalanced parenthesis (or a `<backslash><newline>`), then assume
> the next line is a continuation? I think the `<newline><backslash>` syntax
> is unfamiliar to a lot of people.

It will be difficult to implement, and we'll still need the old way of
line continuation in some places. Putting the backslash at the end of
the line isn't nice anyway, too easy to make mistakes, we should
probably not add it. Mixing the two is probably worse.
If we can have some statements continue on the next line without a
backslash remains to be seen.

> * Scoped functions or lambdas containing ex-commands (or better, def
> function bodies). When combined with try/finally, this would allow
> constructs that take a "block" of code, such as "WithCurrentBuffer( buf, {
> => block of code/commands, maybe split across lines } )" or something like
> this when working with external tools `SendMessageToTool( message, then = {
> => some block of ex commands } ). This can currently be done with Funcref,
> but I think that you have to do 'function ...' and then 'delfiunc' as all
> functions are currently global. Maybe the new scoping rules could be
> extended to allow this.

That a function declared locally in a function becomes a global function
is indeed unexpected. We can at least make them local inside a :def
function, unless prefixed with g:, just like with variables.

Defining a function and creating a funcref in one step would also be
useful. Not sure about the syntax. TypeScript does something like
this:
let ref = function(arg: type): rettype { ... body };

Perhaps we can do:
let ref = def(arg: type): rettype
... body
enddef

> I already mentioned elsewhere that the other barrier to entry for vim
> scripting is tooling and debugging. Hopefully with a these changes we can
> consider how to:
>
> * extract the parser so that an external tool can parse and do code
> comprehension, go-to, refactor, etc., think Vim Language Server (there is
> one, but it's not canonical)

It's a lot easier to make an independent parser. Trying to share a
parser will make things very complicated.

> * provide a interface to a debug adapter to allow debugging of vimscript
> executing in a Vim instance (i.e. Vim Debug Adapter)

Sure.

> * provide better source/line information for stack traces (so that we can
> reliably jump to location of assertions e.g. test failures using cnext). I
> saw a recent patch which improves sourcing_lineno and uses a proper stack.
> I was planning to work on a patch to store source/line info when debugging
> is enabled, but might hold off while vim 9 shakes down, as this is likely
> required for the above debugger interface (assuming that's a goal).

Yes, I started improving the execution stack. Currently it only tracks a
sequence of function calls, it does not continue with a :source command
or executing autocommands, and it's a string without file/line info.
This can be greatly improved.

--
ROBIN: (warily) And if you get a question wrong?
ARTHUR: You are cast into the Gorge of Eternal Peril.
ROBIN: Oh ... wacho!

"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

[Prabir Shrestha]

> * provide a interface to a debug adapter to allow debugging of vimscript
> executing in a Vim instance (i.e. Vim Debug Adapter)
Sure.

+1 for this. This is always the first function I write in a plugin. It would be good to either support chrome devtools debugg protocol or language server debug protocol. https://code.visualstudio.com/blogs/2018/08/07/debug-adapter-protocol-website

But I'm not in hurry to have this.

function! asyncomplete#log(...) abort

    if !empty(g:asyncomplete_log_file)

        call writefile([json_encode(a:000)], g:asyncomplete_log_file, 'a')

    endif

endfunction

* Scoped functions or lambdas containing ex-commands (or better, def function bodies). When combined with try/finally, this would allow constructs that take a "block" of code, such as "WithCurrentBuffer( buf, { => block of code/commands, maybe split across lines } )" or something like this when working with external tools `SendMessageToTool( message, then = { => some block of ex commands } ). This can currently be done with Funcref, but I think that you have to do 'function ...' and then 'delfiunc' as all functions are currently global. Maybe the new scoping rules could be extended to allow this.

This was was something I mentioned before in a different post about vim9. It would really be good to allow multiple lines in lambda or functions.Frankly this is one of the biggest pain point I have with vimscript currently when dealing with async code.

call job_start({

    \ 'out_cb': {job, data->

    \     if data != ''

    \          echom data

    \     endif

    \ }

    \ })

call job_start({

    \ 'out_cb': function(job, data) abort

    \     if data != ''

    \          echom data

    \     endif

    \ endfunction

    \ })

I don't think WASM helps at all with syntax highlighting.  The good old
Vim syntax highlighting depends on regexp patterns.  I have already
optimized the speed of that, I don't think it can be improved much more.
The main alternative now is to use an external parser or language server
and communicate with it. Such as the govim plugin does.  You an now use
text properties to highlight the text.

I was hoping for vim to support WASM + WASM threads, so we can do most of the expensive functions in a thread. The biggest problem with having external jobs is that how do the users get the external binary that works in their OS. Languages such as rust and go having a single static binary has made it easy but this still suffers from compilation in some OS and the IT department allowing the random binary. Or might be what we need is "vim --server --script=server.vim" which can read and write to stdio/stdout/stderr or sockets. This way we can do heavy lifting in vimscript in a different process.

https://github.com/prabirshrestha/vim-lsp/issues/633 - Semantic highlight slows down editing

In vim-lsp we implemented semantic highlighting if langserver supports it using text_props, but then the protocol currently returns the entire document highlighting which we need to compute in vimscript, making it very slow. For now we have disabled this feature by default.

The only way I can think of improving this is by using python threads or different process.

FUNCTIONS SORTED ON SELF TIME
count  total (s)   self (s)  function
10004              2.192760  lsp#utils#base64_decode()
10004   2.202340   1.375972  <SNR>87_add_highlight()
10004   3.162274   0.504508  <SNR>87_tokens_to_hl_info()
30024              0.465006  <SNR>87_octets_to_number()
10005   0.351520   0.286610  lsp#capabilities#has_semantic_highlight()
100053              0.251802  <SNR>87_get_textprop_name()
    1   5.561511   0.196170  lsp#ui#vim#semantic#handle_semantic()
10005   0.574658   0.169267  lsp#ui#vim#semantic#get_scopes()
20011              0.118792  lsp#get_server_capabilities()

> Basically in strict mode it would still use a:name as the variable name but
> it is no longer a dictionary. This way all new vimscript is 100% compatible
> with old vimscript < 9
I don't see that.  If a: is no longer available as a dictionary it won't
be 100% compatible.  There are other subtle assumptions, e.g. about how
error messages are handled.  It will work like "abort" was present.

This is why it is an explicit opt in using " use strict. By default it treats a:0 as a dictionary but if "use strict is turned on, it treats a:0 as a variable and not a dict.

[Ben Jackson]

> On 5 Jan 2020, at 18:55, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
>
> Ben Jackson wrote:
>
>> Hi Bram, thanks for sharing. Looks really promising.
>>
>> Regarding https://github.com/brammool/vim9#3-better-vim-script, I have a
>> few ideas for your consideration:
>>
>> * Execution often relies on user configuration (ignorecase, magic, etc.)
>> and there is common boilerplate required in many scripts to set/restore
>> cpo. Could we say that when `scriptversion` is `vim9`, we always execute as
>> if `set cpo&vim` ?
>
> For :def functions those will be parsed as if 'nocompatible' is set.
>
> For 'ignorecase' and 'magic' we can ignore those for the =~ and !~
> operators. What else?

The “usual" expression operators behave differently (when not suffixed with # or ?) depending on `ignorecase`, right. As in: if x == “CaseSensitiveMatch?” Depends on ignorecase.

Examples from :help expr4:

"abc" ==# "Abc" evaluates to 0
"abc" ==? "Abc" evaluates to 1
"abc" == "Abc" evaluates to 1 if 'ignorecase' is set, 0 otherwise

I think this is one of those things you just have to know (like using === in javascirpt). Would be fewer accidental bugs if `==` worked consistently I guess.

>
> Not sure yet what will happen at the script level. No changes are
> needed for speed. If needed the file can be put inside :def/:enddef
> and then calling that function. Not ideal, but at least it's possible.

I guess the javascript idiom here is to have an anonymous function that’s immediately called:

( def ( args ) : return type
Code
enddef )()

But that’s hardly art.

>
>
>> * Line continuation. I know this is probably hard to change, but could we
>> look at a line-continuation scheme that's more familiar ? As we don't have
>> line termination, Perhaps something more like python's e.g. if a line ends
>> with an unbalanced parenthesis (or a `<backslash><newline>`), then assume
>> the next line is a continuation? I think the `<newline><backslash>` syntax
>> is unfamiliar to a lot of people.
>
> It will be difficult to implement, and we'll still need the old way of
> line continuation in some places. Putting the backslash at the end of
> the line isn't nice anyway, too easy to make mistakes, we should
> probably not add it. Mixing the two is probably worse.
> If we can have some statements continue on the next line without a
> backslash remains to be seen.

Sure. I figured this would be difficult.

>
>> * Scoped functions or lambdas containing ex-commands (or better, def
>> function bodies). When combined with try/finally, this would allow
>> constructs that take a "block" of code, such as "WithCurrentBuffer( buf, {
>> => block of code/commands, maybe split across lines } )" or something like
>> this when working with external tools `SendMessageToTool( message, then = {
>> => some block of ex commands } ). This can currently be done with Funcref,
>> but I think that you have to do 'function ...' and then 'delfiunc' as all
>> functions are currently global. Maybe the new scoping rules could be
>> extended to allow this.
>
> That a function declared locally in a function becomes a global function
> is indeed unexpected. We can at least make them local inside a :def
> function, unless prefixed with g:, just like with variables.
>
> Defining a function and creating a funcref in one step would also be
> useful. Not sure about the syntax. TypeScript does something like
> this:
> let ref = function(arg: type): rettype { ... body };
>
> Perhaps we can do:
> let ref = def(arg: type): rettype
> ... body
> enddef
>

I personally like the python approach: a def inside a function is a local variable of function type. As in:

let myFunc = function( … ) { … body }

Would be written simply as

def myFunc( … ):
… body

In vimscript we could do the same, having a nested ‘def’ always declare a local Funcref by name, but have a ‘real’ name that’s hidden from the user (like a lambda). In the above example, that would create a local variable ‘myFunc’ which is a Funcref to an (unnamed?) function with ( … ) args and … body. Again, not sure how practical that is, but it _feels_ smiler to how lambdas currently work.

>> I already mentioned elsewhere that the other barrier to entry for vim
>> scripting is tooling and debugging. Hopefully with a these changes we can
>> consider how to:
>>
>> * extract the parser so that an external tool can parse and do code
>> comprehension, go-to, refactor, etc., think Vim Language Server (there is
>> one, but it's not canonical)
>
> It's a lot easier to make an independent parser. Trying to share a
> parser will make things very complicated.

Fair enough.

>
>> * provide a interface to a debug adapter to allow debugging of vimscript
>> executing in a Vim instance (i.e. Vim Debug Adapter)
>
> Sure.

Woo \o/. Debugging vimscript with vimspector would be very kickass. I’m more than willing to put in hard graft to make this happen.

>
>> * provide better source/line information for stack traces (so that we can
>> reliably jump to location of assertions e.g. test failures using cnext). I
>> saw a recent patch which improves sourcing_lineno and uses a proper stack.
>> I was planning to work on a patch to store source/line info when debugging
>> is enabled, but might hold off while vim 9 shakes down, as this is likely
>> required for the above debugger interface (assuming that's a goal).
>
> Yes, I started improving the execution stack. Currently it only tracks a
> sequence of function calls, it does not continue with a :source command
> or executing autocommands, and it's a string without file/line info.
> This can be greatly improved.

I’ll keep researching. I have some ideas about how to do it, so I’ll maybe put a prototype together over the next few months.

[Bram Moolenaar]

Ben Jackson wrote:

> >> Hi Bram, thanks for sharing. Looks really promising.
> >>
> >> Regarding https://github.com/brammool/vim9#3-better-vim-script, I have a
> >> few ideas for your consideration:
> >>
> >> * Execution often relies on user configuration (ignorecase, magic, etc.)
> >> and there is common boilerplate required in many scripts to set/restore
> >> cpo. Could we say that when `scriptversion` is `vim9`, we always execute as
> >> if `set cpo&vim` ?
> >
> > For :def functions those will be parsed as if 'nocompatible' is set.
> >
> > For 'ignorecase' and 'magic' we can ignore those for the =~ and !~
> > operators. What else?
>
> The “usual" expression operators behave differently (when not suffixed
> with # or ?) depending on `ignorecase`, right. As in: if x ==
> “CaseSensitiveMatch?” Depends on ignorecase.
>
> Examples from :help expr4:
>
> "abc" ==# "Abc" evaluates to 0
> "abc" ==? "Abc" evaluates to 1
> "abc" == "Abc" evaluates to 1 if 'ignorecase' is set, 0 otherwise
>
> I think this is one of those things you just have to know (like using
> === in javascirpt). Would be fewer accidental bugs if `==` worked
> consistently I guess.

I have already made it work like that, 'ignorecase' is not used.

> > Not sure yet what will happen at the script level. No changes are
> > needed for speed. If needed the file can be put inside :def/:enddef
> > and then calling that function. Not ideal, but at least it's possible.
>
> I guess the javascript idiom here is to have an anonymous function that’s immediately called:
>
> ( def ( args ) : return type
> Code
> enddef )()
>
> But that’s hardly art.

I like to take the good parts of JavaScript, not the weird parts :-).
It's very easy to miss the "()" at the end, especially if the function
takes up most of the file.

It can also made to work so that the function is defined normally, and
when using that name without () it will be a funcref, as if using
"function('FuncName')". Then it also works for global functions, with
"g:" prepended.

> >> * provide a interface to a debug adapter to allow debugging of vimscript
> >> executing in a Vim instance (i.e. Vim Debug Adapter)
> >
> > Sure.
>
> Woo \o/. Debugging vimscript with vimspector would be very kickass.
> I’m more than willing to put in hard graft to make this happen.

It's not really related to Vim9 script, could be an independend project.

--
BRIDGEKEEPER: What is your favorite colour?
GAWAIN: Blue ... No yelloooooww!