Re: LuaJIT 2.0.3 vs 2.1

[Yichun Zhang (agentzh)]

Hello!

On Mon, Dec 8, 2014 at 2:43 AM, Andrei Belov wrote:
> I noticed that openresty bundle has LuaJIT 2.1 (development branch) inside.
> Also I found this repository: https://github.com/openresty/luajit2
>

Yes.

> It seems like it is being updated from original LuaJIT author’s git repo,
> and the branch “agentzh” is used for tagging purposes only, so the code
> in openresty bundle comes from this branch - am I right? :)
>

Actually OpenResty uses the tags from the "v2.1-agentzh" branch in
that luajit2 github repos instead of the "agentzh" branch.

> May I ask you why you have chosen to use 2.1 instead of “stable” 2.0.3?
> What are the main differences between 2.0.3 and 2.1?

Well, the v2.1 branch is not "unstable". Mike Pall and I have tried
very hard to ensure v2.1 is always production ready.

OpenResty uses v2.1 for various practical reasons, mostly for
performance. For example,

1. LuaJIT v2.1's JIT compiler is way more powerful and featureful than
2.0.x's. With the "stable" branch, only a very small portion of our
real-world Lua code can be JIT compiled.

2. LuaJIT v2.1 supports the new table.new() and table.clear() Lua
builtins while the stable branch does not. These two builtins can be
JIT compiled and are important for the performance of table-intensive
Lua apps.

3. My employer has been sponsoring Mike Pall to do new optimizations
in LuaJIT according to OpenResty's requirements (and further
prioritized by CloudFlare's business requirements). The sponsored
items are mostly in 1) and 2) above. And new development can only
happen in the v2.1 branch because the 2.0.x branch has been feature
frozen for long.

4. We always run the latest LuaJIT v2.1 in production across
CloudFlare's global network. We've been working closely with Mike Pall
in the past year and successfully nailed down quite some (a dozen?)
very deep and very obscure bugs LuaJIT's JIT compiler that had been
hidden for years even in the stable branch (with the immense help of
our opensource gdb tools [1]). And Mike Pall keeps backporting these
fixes to the stable branch. So you can see that v2.1 is actually
pushing the stable branch to become more "stable".

5. Our latest advanced (online and offline) debugging tools [1] based
on gdb and systemtap are always based on the latest LuaJIT v2.1 code
base. Because these tools have to know the LuaJIT internals, and it's
always a burden to maintain older LuaJIT versions.

BTW, in OpenResty's branch of LuaJIT v2.1, we also make some minor
changes to make it fit OpenResty's needs better. For example, the
experimental "trace stitching" feature is known to have issues and we
haven't nailed down the real cause yet so OpenResty's branch turns it
off by default while v2.1 enables it by default. We'll consider
turning it on by default once we fix all the issues.

Well, it's true that we are more likely to hit some deep bugs in the
JIT compiler with the v2.1 branch than the master branch because with
v2.1, a *lot* more real-world Lua code can be actually JIT compiled
while with the stable branch almost all the Lua code is always
interpreted and the JIT compiler is seldom used. The interpreter is
dead solid (in both 2.0.x and v2.1), so if you really paranoid about
extreme stability, then you can always disable the JIT compiler by
calling jit.off() in the entry point of your Lua app. But that also
means all of your Lua code will always be interpreted (well, it's
still much faster than the standard Lua 5.1 interpreter though).

BTW, it's recommended to join the openresty-en mailing list and post
such questions there. Please see http://openresty.org/#Community for
more details. Thanks for your cooperation.

Best regards,
-agentzh

P.S. I'm cc'ing the list.

[1] https://github.com/openresty/nginx-gdb-utils#readme ,
https://github.com/openresty/nginx-systemtap-toolkit#readme , and
https://github.com/openresty/stapxx#samples

[Aapo Talvensaari]

Are those table.new and table.clear documented somewhere? Does it really work if you respect the tables be certain predefined size?

local t = table.new(1, 1)
t[1] = "a"
t.a = "a"
-- everthing fine so far? but what happens next?
t[2] = "b"
t.b = "b"

So what is the magic that table.new does behind the scenes? Or have I understood this incorrectly (does narr represent number of array elements in table and nrec the number of keys in table?)

[Andrei Belov]

Great to know it, thanks a lot for such a detailed explanation.

I also spent some time surfing through the luajit mailing list archives,

but haven't found anything related to the current state of the v2.1 branch,

i.e. when it suppose to became the new "stable" officially?

 

5. Our latest advanced (online and offline) debugging tools [1] based
on gdb and systemtap are always based on the latest LuaJIT v2.1 code
base. Because these tools have to know the LuaJIT internals, and it's
always a burden to maintain older LuaJIT versions.

BTW, in OpenResty's branch of LuaJIT v2.1, we also make some minor
changes to make it fit OpenResty's needs better. For example, the
experimental "trace stitching" feature is known to have issues and we
haven't nailed down the real cause yet so OpenResty's branch turns it
off by default while v2.1 enables it by default. We'll consider
turning it on by default once we fix all the issues.

Well, it's true that we are more likely to hit some deep bugs in the
JIT compiler with the v2.1 branch than the master branch because with
v2.1, a *lot* more real-world Lua code can be actually JIT compiled
while with the stable branch almost all the Lua code is always
interpreted and the JIT compiler is seldom used. The interpreter is
dead solid (in both 2.0.x and v2.1), so if you really paranoid about
extreme stability, then you can always disable the JIT compiler by
calling jit.off() in the entry point of your Lua app. But that also
means all of your Lua code will always be interpreted (well, it's
still much faster than the standard Lua 5.1 interpreter though).

That's interesting to know - thanks again!

[Aapo Talvensaari]

On Tuesday, 9 December 2014 12:10:44 UTC+2, Aapo Talvensaari wrote:

Are those table.new and table.clear documented somewhere?

So it seems:

http://repo.or.cz/w/luajit-2.0.git/blob_plain/v2.1:/doc/extensions.html#table_new

http://pgl.yoyo.org/luai/i/lua_createtable

So, this allocated a predefined sized table. It's a little bit strange that Lua has method to allocate x number of array elements and y number of non-array elements, but there is no quick way to ask how many of them there are (you have to iterate, and optionally take metatables in consideration).

 

[Yichun Zhang (agentzh)]

Hello!

On Tue, Dec 9, 2014 at 3:01 AM, Andrei Belov wrote:
> I also spent some time surfing through the luajit mailing list archives,
> but haven't found anything related to the current state of the v2.1 branch,
> i.e. when it suppose to became the new "stable" officially?
>

Regarding the ETA, you'll have to ask Mike Pall instead :) I don't
really care about that date because being "stable" also means "feature
frozen", which is not really interesting to me ;)

Regards,
-agentzh