[patch] if_lua support linking with LuaJIT on Unix

[Hiroshi Shirosaki]

Hi,

Recently some vim plugins use if_lua for performance issues because Vim script is not so fast. For example Shougo's neocomplete.vim[1] shows better performance thanks to if_lua.

According to some benchmark results LuaJIT looks much faster than Lua and is fully compatible to Lua 5.1.[2][3]
So that we would get more performance advantage by using LuaJIT instead of Lua.

On Windows just replacing Lua's dll with LuaJIT's dll in Vim directory seems to work as expected without rebuilding vim in the case of dynamic link. Now LuaJIT bundled vim binary for Windows is distributed by kaoriya.[4]

However linking vim with LuaJIT on Unix like OS requires some works in configure and build process because library names and directory path of LuaJIT are slightly different from Lua. And LuaJIT requires special link options on Mac OS X. It might not easy for beginners.

It would be nice if configure would prepare for LuaJIT automatically so that users who want to use LuaJIT could easily build Vim with LuaJIT.

For this purpose I added a new configure option `--with-luajit`.
Here is a patch. This changes only configure.

https://gist.github.com/shirosaki/5663617/

When adding `--enable-luainterp --with-luajit` as configure option, configure tries to use LuaJIT instead of Lua .
Just with `--enable-luainterp` (without --with-luajit) configure tries to use Lua as before.

In addition I fixed an issue that build with `--enable-luainterp=dynamic` doesn't work properly with Lua on Mac OS X since shared library extension on Mac OS X is *.dylib while *.so on Linux.

Some vim users in Japan are interested in this fix and testing.[5]
I've tested this on Mac OS X, Ubuntu 13.04 and CentOS 6.4 with latest vim 7.3 and it seems to work.

Install instructions with LuaJIT:

Ubuntu:
$ sudo apt-get install luajit libluajit-5.1
$ cd vim_source_directory
$ curl https://gist.github.com/shirosaki/5663617/raw | patch -p1
$ ./configure --with-features=huge --enable-luainterp --with-luajit
$ make
$ sudo make install

Mac OS X with Homebrew:
$ brew install luajit
$ cd vim_source_directory
$ curl https://gist.github.com/shirosaki/5663617/raw | patch -p1
$ ./configure --with-features=huge --enable-luainterp --with-luajit --with-lua-prefix=/usr/local
$ make
$ sudo make install

[1] https://github.com/Shougo/neocomplete.vim
[2] http://luajit.org/luajit.html
[3] https://github.com/vim-jp/issues/issues/48#issuecomment-15355516
[4] http://www.kaoriya.net/news/2013/05/19/
[5] https://github.com/vim-jp/issues/issues/348

Thanks,
Hiroshi Shirosaki

[Bram Moolenaar]

Hiroshi Shirosaki wrote:

> Recently some vim plugins use if_lua for performance issues because
> Vim script is not so fast. For example Shougo's neocomplete.vim[1]
> shows better performance thanks to if_lua.
>
> According to some benchmark results LuaJIT looks much faster than Lua
> and is fully compatible to Lua 5.1.[2][3]
> So that we would get more performance advantage by using LuaJIT
> instead of Lua.
>
> On Windows just replacing Lua's dll with LuaJIT's dll in Vim directory
> seems to work as expected without rebuilding vim in the case of
> dynamic link. Now LuaJIT bundled vim binary for Windows is distributed
> by kaoriya.[4]

Does this require renaming the .dll, thus afterwards you may forget that
the file is actually the JIT one?

> However linking vim with LuaJIT on Unix like OS requires some works in
> configure and build process because library names and directory path
> of LuaJIT are slightly different from Lua. And LuaJIT requires special
> link options on Mac OS X. It might not easy for beginners.
>
> It would be nice if configure would prepare for LuaJIT automatically
> so that users who want to use LuaJIT could easily build Vim with
> LuaJIT.
>
> For this purpose I added a new configure option `--with-luajit`.
> Here is a patch. This changes only configure.
>
> https://gist.github.com/shirosaki/5663617/
>
> When adding `--enable-luainterp --with-luajit` as configure option,
> configure tries to use LuaJIT instead of Lua .
> Just with `--enable-luainterp` (without --with-luajit) configure tries
> to use Lua as before.

What you say hints that the JIT one is always better. So why not use
luajit when it's available, falling back to non-JIT when it's not?
Then it would work well for people who don't even know about the option.
Right?

The option might still be useful to force using one or the other.
Perhaps the JIT version is released a bit later?

--
hundred-and-one symptoms of being an internet addict:
180. You maintain more than six e-mail addresses.

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Hiroshi Shirosaki]

On Thu, Jun 13, 2013 at 10:35 PM, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
> Hiroshi Shirosaki wrote:
>
>> Recently some vim plugins use if_lua for performance issues because
>> Vim script is not so fast. For example Shougo's neocomplete.vim[1]
>> shows better performance thanks to if_lua.
>>
>> According to some benchmark results LuaJIT looks much faster than Lua
>> and is fully compatible to Lua 5.1.[2][3]
>> So that we would get more performance advantage by using LuaJIT
>> instead of Lua.
>>
>> On Windows just replacing Lua's dll with LuaJIT's dll in Vim directory
>> seems to work as expected without rebuilding vim in the case of
>> dynamic link. Now LuaJIT bundled vim binary for Windows is distributed
>> by kaoriya.[4]
>
> Does this require renaming the .dll, thus afterwards you may forget that
> the file is actually the JIT one?
>

Maybe yes. But now I can confirm which dll is linked by running a
benchmark script since performance is 10x different.

>> However linking vim with LuaJIT on Unix like OS requires some works in
>> configure and build process because library names and directory path
>> of LuaJIT are slightly different from Lua. And LuaJIT requires special
>> link options on Mac OS X. It might not easy for beginners.
>>
>> It would be nice if configure would prepare for LuaJIT automatically
>> so that users who want to use LuaJIT could easily build Vim with
>> LuaJIT.
>>
>> For this purpose I added a new configure option `--with-luajit`.
>> Here is a patch. This changes only configure.
>>
>> https://gist.github.com/shirosaki/5663617/
>>
>> When adding `--enable-luainterp --with-luajit` as configure option,
>> configure tries to use LuaJIT instead of Lua .
>> Just with `--enable-luainterp` (without --with-luajit) configure tries
>> to use Lua as before.
>
> What you say hints that the JIT one is always better. So why not use
> luajit when it's available, falling back to non-JIT when it's not?
> Then it would work well for people who don't even know about the option.
> Right?
>
> The option might still be useful to force using one or the other.
> Perhaps the JIT version is released a bit later?
>

That was discussed before at https://github.com/vim-jp/issues/issues/348.

If both Lua and LuaJIT are installed on an environment, it may be good
to be selectable Lua or LuaJIT for plugin developers to test both
versions.
Now Lua 5.2 is released. But LuaJIT version is still Lua 5.1
compatible. 5.2 is not compatible with 5.1
If a user want to use Lua 5.2 and both Lua and LuaJIT are installed,
it is better to be selectable.

Indeed I can change configure to give high priority to LuaJIT if is's OK.

Possible way is:
--enable-luainterp: use LuaJIT, but fallback Lua if LuaJIT is not available
--enable-luainterp --with-lua: force to use Lua

Any ideas?

--
Hiroshi Shirosaki

[Luis Carvalho]

<snip>

> > What you say hints that the JIT one is always better. So why not use
> > luajit when it's available, falling back to non-JIT when it's not?
> > Then it would work well for people who don't even know about the option.
> > Right?
> >
> > The option might still be useful to force using one or the other.
> > Perhaps the JIT version is released a bit later?
> >
>
> That was discussed before at https://github.com/vim-jp/issues/issues/348.
>
> If both Lua and LuaJIT are installed on an environment, it may be good
> to be selectable Lua or LuaJIT for plugin developers to test both
> versions.
> Now Lua 5.2 is released. But LuaJIT version is still Lua 5.1
> compatible. 5.2 is not compatible with 5.1
> If a user want to use Lua 5.2 and both Lua and LuaJIT are installed,
> it is better to be selectable.
>
> Indeed I can change configure to give high priority to LuaJIT if is's OK.
>
> Possible way is:
> --enable-luainterp: use LuaJIT, but fallback Lua if LuaJIT is not available
> --enable-luainterp --with-lua: force to use Lua
>
> Any ideas?

Since Lua 5.2 is the official Lua release, I'd suggest to stick to it as the
default and have a --with-luajit option as before.

Cheers,
Luis

--
Computers are useless. They can only give you answers.
-- Pablo Picasso

--
Luis Carvalho (Kozure)
lua -e 'print((("lexca...@NO.gmail.SPAM.com"):gsub("(%u+%.)","")))'

[Hiroshi Shirosaki]

On Thu, Jun 13, 2013 at 10:35 PM, Bram Moolenaar <Br...@moolenaar.net> wrote:
>
> Hiroshi Shirosaki wrote:

>> On Windows just replacing Lua's dll with LuaJIT's dll in Vim directory
>> seems to work as expected without rebuilding vim in the case of
>> dynamic link. Now LuaJIT bundled vim binary for Windows is distributed
>> by kaoriya.[4]
>
> Does this require renaming the .dll, thus afterwards you may forget that
> the file is actually the JIT one?
>

Taro MURAOKA said that dll file name of both Lua and JIT versions is same
lua51.dll so we don't need to rename it on Windows and cannot know if it is
actually JIT one by the file name. LuaJIT has `jit` built-in module so that we
can see the linking dll version of LuaJIT.

:lua print(jit.version)

shows the LuaJIT version string.
http://luajit.org/ext_jit.html

On Fri, Jun 14, 2013 at 12:47 PM, Luis Carvalho <lexca...@gmail.com> wrote:
> Since Lua 5.2 is the official Lua release, I'd suggest to stick to it as the
> default and have a --with-luajit option as before.

Then would adding document for `--with-luajit` be better?
I'm not sure where is the right place for the document. src/INSTALL?

I wrote a document patch for it.
https://gist.github.com/5780295

--
Hiroshi Shirosaki

[Hiroshi Shirosaki]

I've updated the patch since the following issue is reported.

If lua executable is not exist in the PATH and configure cannot find liblua.so, strange library name `liblua.so.` (end with .) is used in the case of dynamic loading. It occurs even if I build Vim without luajit patch.

$ ./configure --with-features=huge --enable-luainterp=dynamic --with-lua-prefix=/usr/local
$ make
(snip)
gcc -c -I. -Iproto -DHAVE_CONFIG_H -DMACOS_X_UNIX -no-cpp-precomp -g -O2
-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
-DDYNAMIC_LUA_DLL=\"liblua.so.\" -I/usr/local/include -o objects/if_lua.o if_lua.c

I added some fixes for the case where lua or luajit executable is not exist in the PATH.
configure can fail or use default name with dynamic if a valid library name cannot be found when static or dynamic linking.
I've added some document in Makefile.

Here is updated patch.
https://gist.github.com/shirosaki/5663617

[Hiroshi Shirosaki]

On Monday, July 15, 2013 7:47:42 AM UTC+9, Hiroshi Shirosaki wrote:
> I've updated the patch since the following issue is reported.
>
> If lua executable is not exist in the PATH and configure cannot find liblua.so, strange library name `liblua.so.` (end with .) is used in the case of dynamic loading. It occurs even if I build Vim without luajit patch.
>
> $ ./configure --with-features=huge --enable-luainterp=dynamic --with-lua-prefix=/usr/local
> $ make
> (snip)
> gcc -c -I. -Iproto -DHAVE_CONFIG_H -DMACOS_X_UNIX -no-cpp-precomp -g -O2
> -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
> -DDYNAMIC_LUA_DLL=\"liblua.so.\" -I/usr/local/include -o objects/if_lua.o if_lua.c
>
>

Hi Bram,

As I said before current Vim 7.4a has two bugs of if_lua configure with dynamic loading:
1) *.so is used as library name on Mac OS X whille *.dylib is correct on Mac OS X
2) -DDYNAMIC_LUA_DLL=\"liblua.so.\" is used (end with . ) If lua executable is not exist in the PATH and configure cannot find liblua.so

Should these bugs be fixed with Vim 7.4a?

If we will create only bug fix patch without LuaJIT support, then the bug fix patch will be accepted before 7.4 release?

Thank you.
--
Hiroshi Shirosaki

[Bram Moolenaar]

This is a big change and I can't properly test it myself.
Can a few people confirm that this patch works well?

It does look like adding luajit support is mixed with the rest. It
might be better to separate them to reduce any risks.

--
Far back in the mists of ancient time, in the great and glorious days of the
former Galactic Empire, life was wild, rich and largely tax free.
Mighty starships plied their way between exotic suns, seeking adventure and
reward among the furthest reaches of Galactic space. In those days, spirits
were brave, the stakes were high, men were real men, women were real women
and small furry creatures from Alpha Centauri were real small furry creatures
from Alpha Centauri. And all dared to brave unknown terrors, to do mighty
deeds, to boldly split infinitives that no man had split before -- and thus
was the Empire forged.
-- Douglas Adams, "The Hitchhiker's Guide to the Galaxy"

[lilydjwg]

On Thu, Jul 25, 2013 at 10:03:21PM +0200, Bram Moolenaar wrote:
>
> Hiroshi Shirosaki wrote:
>
> > On Monday, July 15, 2013 7:47:42 AM UTC+9, Hiroshi Shirosaki wrote:
> > > I've updated the patch since the following issue is reported.
> > >
> > > If lua executable is not exist in the PATH and configure cannot find liblua.so, strange library name `liblua.so.` (end with .) is used in the case of dynamic loading. It occurs even if I build Vim without luajit patch.
> > >
> > > $ ./configure --with-features=huge --enable-luainterp=dynamic --with-lua-prefix=/usr/local
> > > $ make
> > > (snip)
> > > gcc -c -I. -Iproto -DHAVE_CONFIG_H -DMACOS_X_UNIX -no-cpp-precomp -g -O2
> > > -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
> > > -DDYNAMIC_LUA_DLL=\"liblua.so.\" -I/usr/local/include -o objects/if_lua.o if_lua.c
> > >
> > >
> >
> > Hi Bram,
> >
> > As I said before current Vim 7.4a has two bugs of if_lua configure with dynamic loading:
> > 1) *.so is used as library name on Mac OS X whille *.dylib is correct on Mac OS X
> > 2) -DDYNAMIC_LUA_DLL=\"liblua.so.\" is used (end with . ) If lua executable is not exist in the PATH and configure cannot find liblua.so
> >
> > Should these bugs be fixed with Vim 7.4a?
> >
> > If we will create only bug fix patch without LuaJIT support, then the
> > bug fix patch will be accepted before 7.4 release?
>
> This is a big change and I can't properly test it myself.
> Can a few people confirm that this patch works well?

The patch works well with my configure options, both with Lua and
LuaJIT, default path, on Arch Linux 64bit.

> It does look like adding luajit support is mixed with the rest. It
> might be better to separate them to reduce any risks.

--

Best regards,
lilydjwg

Linux Vim Python 我的博客:
http://lilydjwg.is-programmer.com/
--
A: Because it obfuscates the reading.
Q: Why is top posting so bad?