vim9 Benchmark

[Ni Va]

Hi All,

Do you know if the stats displayed in https://github.com/brammool/vim9 are updated ?

Thank you

NiVa

[Bram Moolenaar]

> Do you know if the stats displayed in https://github.com/brammool/vim9 are
> updated ?

I have not touched them for a long time. I expect Vim9 might be a bit
slower now, with the extra checks for errors that were added.
I might spend some time on finding bottlenecks, but priority is still
with finishing the specification.

--
From "know your smileys":
;-0 Can't find shift key
,-9 Kann Umschalttaste nicht finden

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// \\\
\\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Ni Va]

Understood.

Thank you Bram.

[Ni Va]

Maybe interesting for benchmark

https://github.com/tjdevries/vim9jit

[aro...@vex.net]

> Maybe interesting for benchmark
> https://github.com/tjdevries/vim9jit
>

If learning Rust isn't the whole purpose of the project, you might want to
look at Raku, specifically "grammars". It can be a language for generating
languages.

[Dominique Pellé]

In the page https://github.com/tjdevries/vim9jit  there is something not right IMO:
the section about performance uses the "sum" performance test from
https://github.com/vim/vim/blob/master/README_VIM9.md where luajit
is indeed much faster than vim9:

Sum benchmark:

| how     | time in sec |
| --------| -------- |
| Vim old | 5.018541 |                                                                                                                
| Python  | 0.369598 |
| Lua     | 0.078817 |
| LuaJit  | 0.004245 |
| Vim new | 0.073595 |

However, https://github.com/tjdevries/vim9jit omits the fact that in the

other benchmark called "indent", vim9 is faster than luajit. According

to the same README_VIM9.md the second script might be more

representative of what vim scripts typically do.

Indent benchmark:

| how     | time in sec |
| --------| -------- |
| Vim old | 0.578598 |
| Python  | 0.152040 |
| Lua     | 0.164917 |
| LuaJit  | 0.128400 |
| Vim new | 0.079692 |

In any case, it's nice to see that neovim developers are considering

supporting Vim9 script.

Regards

Dominique

[Ni Va]

I'm not a neovim user, nor am I learning rust :)

I just wanted confirmation that vim9 vimnew wasn't that far behind LuaJit.

And it's a good news for 2022 and futur! :)

Best wishes for Vim9 and Bram too.

NiVa

[Gianmaria Bajo]

By the way, in that indent benchmark doesn't use lua as it would be done in neovim.

For example (neovim 0.6.0)

local getlines = vim.api.nvim_buf_get_lines
local setlines = vim.api.nvim_buf_set_lines
local exec = vim.cmd

function LuaTest()
  local totallen = 0
  local start = vim.fn.reltime()
  exec 'new'
  exec 'call setline(1, range(100000))'
  local bnr = vim.fn.bufnr()
  local lines = getlines(bnr, 0, -1, false)
  for i = 1, 100000 do
    lines[i] = '    ' .. lines[i]
    totallen = totallen + lines[i]:len()
  end
  setlines(bnr, 0, -1, false, lines)
  exec ('echom ' .. totallen)
  exec ('echom ' .. vim.fn.reltimestr(vim.fn.reltime(start)))
  exec 'bwipe!'
end

gVim 8.2 (patch 4058)

vim9script

def g:VimTest(): void
    var totallen = 0
    var start = reltime()
    new
    call setline(1, range(100000))
    for i in range(1, 100000)
        setline(i, '    ' .. getline(i))
        totallen += len(getline(i))
    endfor
    echom totallen
    echom reltimestr(reltime(start))
    bwipe!
enddef

With nvim I get:

:lua LuaTest()
888890
0.197217

With gVim I get:

:call VimTest()
888890
  0.308830

[Bram Moolenaar]

The Lua version calls getlines() once, manipulates the array and then
has one call to setlines(), the Vim version calls getline() and
setline() for each line separately. That is not a good comparison.

The idea of the benchmark is to manipulate the text in the buffer, using
one setlines() call is not representing that. Perhaps it should be
changing only every second line.

--
Error:015 - Unable to exit Windows. Try the door.

[Gianmaria Bajo]

Even getting/setting lines in one go (as in the lua version) doesn't improve it for me:

vim9script

def g:VimTest(): void
    var totallen = 0
    var start = reltime()
    new

    setline(1, range(100000))
    var lines = getline(1, 100000)
    for i in range(100000)

        lines[i] = '    ' .. lines[i]

        totallen += len(lines[i])
    endfor
    setline(1, lines)

    echom totallen
    echom reltimestr(reltime(start))
    bwipe!
enddef

Result:

:call VimTest()
888890
  0.330822

[Christian Brabandt]

On Di, 11 Jan 2022, Gianmaria Bajo wrote:

> Even getting/setting lines in one go (as in the lua version) doesn't improve it for me:
>
> vim9script
>
> def g:VimTest(): void
> var totallen = 0
> var start = reltime()
> new
> setline(1, range(100000))
> var lines = getline(1, 100000)
> for i in range(100000)
> lines[i] = ' ' .. lines[i]
> totallen += len(lines[i])
> endfor
> setline(1, lines)
> echom totallen
> echom reltimestr(reltime(start))
> bwipe!
> enddef
>
> Result:
>
> :call VimTest()
> 888890
> 0.330822

That is even slower?



Mit freundlichen Grüßen
Christian
--
Mit welcher Geschwindigkeit breitet sich das Dunkel aus?

[Gianmaria Bajo]

I repeated those tests multiple times. It seems to be a bit slower, yes.

vim9script

def VimTest1(): void

    var totallen = 0
    var start = reltime()
    new

    call setline(1, range(100000))
    for i in range(1, 100000)
        setline(i, '    ' .. getline(i))
        totallen += len(getline(i))
    endfor

    echom 'VimTest1'

    echom totallen
    echom reltimestr(reltime(start))
    bwipe!
enddef

def VimTest2(): void

    var totallen = 0
    var start = reltime()
    new
    setline(1, range(100000))
    var lines = getline(1, 100000)
    for i in range(100000)
        lines[i] = '    ' .. lines[i]
        totallen += len(lines[i])
    endfor
    setline(1, lines)

    echom 'VimTest2'

    echom totallen
    echom reltimestr(reltime(start))
    bwipe!
enddef

def g:VimTests(): void
    VimTest1()
    VimTest2()
enddef

[Gianmaria Bajo]

But there might other things that come into play. In neovim setline() seems to be much faster (when it's setting multiple lines).

For example:

======================================================

neovim lua

======================================================

local getlines = vim.api.nvim_buf_get_lines
local setlines = vim.api.nvim_buf_set_lines
local exec = vim.cmd

function LuaTest()
  local totallen = 0
  local start = vim.fn.reltime()
  exec 'new'
  exec 'call setline(1, range(100000))'

  exec ('echom "After setting lines to range: "' .. vim.fn.reltimestr(vim.fn.reltime(start)))

  local bnr = vim.fn.bufnr()
  local lines = getlines(bnr, 0, -1, false)
  for i = 1, 100000 do

    lines[i] = '    ' .. lines[i]

    totallen = totallen + lines[i]:len()
  end
  setlines(bnr, 0, -1, false, lines)
  exec ('echom ' .. totallen)
  exec ('echom ' .. vim.fn.reltimestr(vim.fn.reltime(start)))
  exec 'bwipe!'
end

======================================================

Result:

After setting lines to range:  0.062205
888890
0.200782

======================================================

gVim 8.2

======================================================

vim9script

def VimTest1(): void
    echom 'VimTest1'

    var totallen = 0
    var start = reltime()
    new
    setline(1, range(100000))

    echom 'After setting lines to range: ' .. reltimestr(reltime(start))

    for i in range(1, 100000)
        setline(i, '    ' .. getline(i))
        totallen += len(getline(i))
    endfor

    echom totallen
    echom 'After processing loop: ' .. reltimestr(reltime(start))

    bwipe!
enddef

def VimTest2(): void

    echom 'VimTest2'

    var totallen = 0
    var start = reltime()
    new
    setline(1, range(100000))

    echom 'After setting lines to range: ' .. reltimestr(reltime(start))

    var lines = getline(1, 100000)
    for i in range(100000)
        lines[i] = '    ' .. lines[i]
        totallen += len(lines[i])
    endfor

    echom 'After processing loop: ' .. reltimestr(reltime(start))
    setline(1, lines)
    echom 'After setting modified lines: ' .. reltimestr(reltime(start))

    echom totallen
    echom reltimestr(reltime(start))
    bwipe!
enddef

def g:VimTests(): void
    VimTest1()
    VimTest2()
enddef

======================================================

Result:

VimTest1
After setting lines to range:   0.092458
888890
After processing loop:   0.310901
VimTest2
After setting lines to range:   0.110934
After processing loop:   0.241299
After setting modified lines:   0.365874
888890
  0.375667

[Ni Va]

Oh. Okay, i will look at. Thank you ! 

--
--
You received this message from the "vim_use" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

---
You received this message because you are subscribed to a topic in the Google Groups "vim_use" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vim_use/flsehwecstQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vim_use+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vim_use/6f4ecd356633b4314964b19257a05489.squirrel%40webmail.vybenetworks.com.

[Bram Moolenaar]

Gianmaria Bajo wrote:

> Even getting/setting lines in one go (as in the lua version) doesn't=20

> improve it for me:
>
> vim9script
>
> def g:VimTest(): void
> var totallen = 0
> var start = reltime()
> new
> setline(1, range(100000))
> var lines = getline(1, 100000)
> for i in range(100000)
> lines[i] = ' ' .. lines[i]
> totallen += len(lines[i])
> endfor
> setline(1, lines)
> echom totallen
> echom reltimestr(reltime(start))
> bwipe!
> enddef
>
> Result:
>
> :call VimTest()
> 888890
> 0.330822

You are taking it in the wrong direction. The benchmark is for a plugin
that would go over the text and make changes here and there, e.g. adjust
the indent. That would only change some lines, but to get it do more
work for a more reliable measurement the benchmark changes many lines.
Changing this to take all lines at once, change them, then set all lines
at once, is changing what it is trying to measure.

It is no surprise Lua would be faster with list manipulations. Thus if
you shift work from interacting with the buffer lines to string
manipulations it can be expected that using Lua is faster. In practice
plugins do have to manipulate buffer lines, so I rather do more of that
to have a representative benchmark.

Note that even when Vim script is a bit slower than Lua I'm still
satisfied. I expect Lua was already tweaked for performance, Vim9 is
still young, there probably is a performance gain if we put some effort
into it. If Vim script was ten times slower I would wonder if we are
doing something wrong.

--
The software said it requires Windows 95 or better, so I installed Linux.

[Dominique Pellé]

The 2 benchmarks in README_VIM9.md (sum and indent) have
loops with many iterations. So they only benchmark execution time
of compiled code as opposed to measuring the time to compile
the script.

I suppose that often, some functions are executed only
a few times (sometimes only once) and so speed of compilation
could also matter for large scripts. It's harder to compare
compilation time between Vim9 and LuaJIT though.

Dominique

[Gianmaria Bajo]

> so speed of compilation could also matter for large scripts.

Yes, first time scripts are run they're slower, but I discarded those.

> Note that even when Vim script is a bit slower than Lua I'm still
satisfied.

I wrote my statusline in both lua and vim9, lua is 1.5x-2x faster there. Not a huge plugin, but still.

So I expect luajit to be at least a bit faster in most situations.

Anyway, repeating tests in linux I get very different results, btw (very close execution times for vim9 and luajit).

neovim:

After setting lines to range:   0.020408
After setting modified lines:   0.094530
888890

----------------------------------------

vim:

----------------------------------------
VimTest1
----------------------------------------
After setting lines to range:   0.024622
After processing loop:          0.097223
888890
----------------------------------------
VimTest2
----------------------------------------
After setting lines to range:   0.027620
After processing loop:          0.055139
After setting modified lines:   0.101768
888890