Compiling Lua scripts to binary works. Use Lua instead of AngelScript?

[Sinan - info@sinsoft.com]

For anyone wondering, if you compile Lua scripts to bytecode with luac (5.1) like this:

luac -o new_file.lua original_source_file.lua

and give the binary file new_file.lua to Urho3D, it works. So Urho3D can process source and bytecode Lua scripts.

For that reason I'd like to move to Lua, because it's a bit similar to Python and Python is what I use for creating tools within Blender, and so I like to keep the same mindset when I script.

My question is, is the Lua API on par with the AngelScript API?

Looking at the API for Lua, it looks like it matches the AngelScript API.

Also given that ARM is the same endianness as x86, if I compile Lua scripts on x86, will the scripts work on mobile (Android/iOS) as well?

Does anyone recommend not to use Lua and use AngelScript? If so, why?

[Lasse Öörni]

There's 3 things which might scare you off from using Lua yet, but if they're not a problem for you, sure, go ahead.

- There is an outstanding issue (ie. not implemented yet) to be able to attach Lua script objects to a scene node similarly how the ScriptInstance component works with AngelScript.
- The Lua subsystem is not really documented yet, though it is rather straightforward to use.
- The AngelScript subsystem has a few years more of testing under its belt.

[Sinan - info@sinsoft.com]

- There is an outstanding issue (ie. not implemented yet) to be able to attach Lua script objects to a scene node similarly how the ScriptInstance component works with AngelScript.

Hmm, didn't know this feature was missing.

- The Lua subsystem is not really documented yet, though it is rather straightforward to use.

Yeah, I have no issues with this. Using Urho3D Lua is very straightforward for me, simply by looking at /Source/Extras/LuaScript/pkgs I have access to all the API available. But yes, AngelScript has a list of all APIs as well.

- The AngelScript subsystem has a few years more of testing under its belt.

 Well if I end up using Lua, I can report back any issues I might find.

[Chris Friesen]

you can't just drop in compiled lua code unless you are using luajit.  i would love to see luajit make its way into urho.  someone has a mostly working cmake -- it doesn't supprot 64 bit mode.  https://github.com/LuaDist/luajit/blob/master/CMakeLists.txt

another option would be to make dynamically link to lua and then the user can swap in the luajit dll

i would like to get the opinion of the urho maintainers about getting luajit in place.

i do prefer the syntax and brevity of lua.  especially when coupled with moonscript. http://moonscript.org/

it is however incredible hard to argue with how awesome angel script integrates 2 ways with the urho code.  it is masterful.

i have ran into issues where some of my lua scripts weren't being optimized well and was suffering pretty large performance penalties :(.

i havn't been using lua much in urho.  I have mostly been working on getting a workflow setup for my wife and i to get assets in the engine and improving the editor in little ways.

[Lasse Öörni]

Whether to integrate luajit is up to Aster Jian, the Lua subsystem maintainer. I don't have anything against it, though the use of assembly may make the integration rather involved, considering all the supported platforms.

[Sinan - info@sinsoft.com]

Yeah, if luajit complicates the engine, please don't add it in.

How about compiling luac from source to ARM? And include luac (ARM) with Urho3D. Would that work?

If not, then it may be best for me to just stick to AngelScript.

[Sinan - info@sinsoft.com]

The way Lua is integrated into Urho3D is pretty elegant, and easy to add new API if needed. I don't want something complicated to replace that.

[Chris Friesen]

Maybe we can add a flag to cmake to dynamic link lua so we can drop the luajit dll in?  Luajit and Lua have the same interface.  That way it is unobtrusive to urho except a flag to dynamic link a library.

[Sinan - info@sinsoft.com]

It turns out I have looked into this issue in the past, but never actually tried it or tested it.

If you compile luac for ARM and run luac on an ARM machine, it should compile Lua scripts to bytecode that will work on ARM devices.

I just need a chance to test it. But if you Google this issue, that's exactly what's recommended. Also, something like QEMU could be used to run an ARM emulator, and inside the emulator you could run luac (ARM) to compile a set of scripts.

This approach I think would be better (for me at least), without making major changes to the engine.

So I will use Lua.

[Sinan - info@sinsoft.com]

The 'Android emulator' is actually an ARM emulator, and it works on my system and can launch "Android Virtual Devices":

http://developer.android.com/tools/help/emulator.html

So there you go, just run luac (ARM) on the Android emulator whenever you need to compile Lua scripts to ARM bytecode, and then extract the scripts from it.

You can also use the same emulator to compile luac to ARM in the first place.

[Alex Fuller]

Hi all,

I actually really like using AngelScript just for the fact that it matches very closely to Urho3D's C++ code. It looks very trivial to convert code by hand to C++ for any added speed boost that is needed (most likely negligible for most things), and the documentation shows the differences between the two to assist in this.

I get the impression AS is a very thin layer over the C++ parts underneath anyway, most all of the 'standard library' that it uses is a wrapped form of Urho's containers and math. If you need to create custom objects or use the threaded tasking system for speed you'll be going into C++ anyway. Another example, when you specify the StringHash by a string name of something in script I think the string gets 'compiled away' once it hits bytecode to its hash representation which would make things faster for some operations (Lasse will have to clarify if that's accurate).

I was working on Python bindings purely so I could import it into Maya (or Blender) but I underestimated the time needed to do this, and I'm now investigating just to do something simple like send angelscript commands over a basic socket or wrap the network module for usage in Maya (and hot reload works for a primitive inter-op itself too).

My 2c :)

-Alex

[Sinan - info@sinsoft.com]

Actually what about AngelScript?

If you compile AngelScript to binary bytecode on an x86 machine, will that same binary bytecode run on ARM mobile machines?

If that's the case, then that's a big advantage. Also as mentioned, AngelScript code can be converted to C++ a lot easier than Lua.

Having to compile Lua scripts on an ARM emulator or actual ARM device will be cumbersome.

[Sinan - info@sinsoft.com]

There is also this method to cross compile to Lua (ARM) binary using x86 host:

http://code.google.com/p/llvm-lua/wiki/CrossCompile

No need for compiling on an ARM emulator or an actual ARM device.

[Sinan - info@sinsoft.com]

It looks like you can use luajit to compile to different architectures:

So if I do:

sudo apt-get install luajit

I can compile to arm using:

luajit -ba arm A.lua A.out

And compile to x86 using:

luajit -ba x86 A.lua A.out

but when I rename A.out to A.lua and give that to Urho3D, it doesn't work. I wonder if we can get luajit to output bytecode that can be used by regular Lua VM?

[Sinan - info@sinsoft.com]

Actually, luajit bytecode is incompatible with lua bytecode.

However, luajit bytecode (with -t raw flag) is multi-platform.

So yeah, I see now why integrating luajit is the best option (multi-platform bytecode, better performance).

[Alex Fuller]

Ok,

Bytecode is different from machine code. Bytecode is source code parsed to the script engine's virtual machine to a format that is more optimal to load in, but shouldn't be faster only at the initialising/loading phase (as all script is turned into bytecode at runtime at the end of the day.

JIT code is turned into machine code and that is CPU-dependent. It is also an issue on some platforms like iOS which doesn't allow JIT for security reasons.

Lua-llvm is the way to go as I think it generates symbols which can be linked and not loaded into the runtime like bytecode. This is called AheadOfTime (AOT) and this is how Unity loads in mono code for iOS plaforms. I've never done this before so YMMV...

I wonder if there's an Angelscript llvm project somewhere...

-Alex

[Lasse Öörni]

AngelScript says this about precompiled bytecode:

http://www.angelcode.com/angelscript/sdk/docs/manual/doc_adv_precompile.html

"The saved bytecode is almost fully platform independent. The following is properly handled, pointer size, size of registered types, offsets of registered properties, CPU endianess, etc. Some things that are not handled include difference in size of primitive types (e.g. bool on older Mac PPC platforms) and CPU floating point representation (e.g. IEEE 754 versus non-IEEE 754)."

[Sinan - info@sinsoft.com]

Ok, I'm still leaning towards Lua.

Some final thoughts:

luajit should be intergrated into Urho3D (if possible). It provides multi-platform bytecode and better performance than regular Lua bytecode

luajit disables "Just in Time" for iOS anyhow (due to App Store policy), so you can't have dynamic code in-game, or if you really need that, you must have it in non-bytecode form (i.e. source)

So if someone wants to use Lua with Urho3D in the current form and wants to compile source to bytecode, here is my process:

on PC, Linux and Mac, use luac. This gives x86 bytecode

on Android, iOS:

1) Install an Android Virtual Machine

2) Compile Lua for Android using NDK (lua (ARM) + luac (ARM))

3) Transfer luac (ARM) for Android to Android Virtual Machine, plus your Lua scripts

4) SSH to Android Virtual Machine, and run luac (ARM) on your scripts to get a bytecode version that works on ARM. Those bytecode ARM files should work on iOS as well.

If you don't like the above method, you can get an ARM Linux distro and run it on you x86 machine using QEMU.

[Sinan - info@sinsoft.com]

OR use llvm_lua. Sorry forgot about that, I guess I'll try llvm_lua at some point as well.

[Sinan - info@sinsoft.com]

Actually,

the "best" method to do this is to download a Linux ARM distro and compile lua and luac for ARM, and then compile your Lua scripts for ARM.

Arch Linux has good ARM linux distros:

http://archlinuxarm.org/

You can run an ARM Linux distro on x86 Linux using QEMU.

Of course, integrating luajit into Urho3D is a better solution. But the above solution I think is the safest, i.e. it's best to compile Lua script on an actual ARM platform.

[Sinan - info@sinsoft.com]

no luck getting Arch Linux ARM distro to work with QEMU :(

So I decided to compile lua (ARM) and luac (ARM) using Android NDK. So I suggest the Android Virtual Machine method I outline in my previous post.

Here are the source files (Attached)

lua (ARM) for Android NDK

luac (ARM) for Android NDK

Simply run ndk-build in each root directory. Must have Android SDK / NDK installed.

[Sinan - info@sinsoft.com]

After further thought, the best option for me is to use AngelScript.

Main reason:

Regardless of whether I want to use Lua or not, the fact is that AngelScript can be converted to C++ code a lot faster than Lua. This is a major advantage.

So I will go with AngelScript.

[Sinan - info@sinsoft.com]

With recent updates, it looks like Lua is now in feature parity with AngelScript? That is, ScriptInstance component can be done with Lua?

Either way, I'm finding AngelScript very efficient to work with and I prefer it over Lua.

What's the plan going forward? Support two scripting languages? What will be Urho3D's main scripting language in the future? AngelScript or Lua?

AngelScript byte code is multi-platform (from what I read so far), while with Lua, you would have to use luac for ARM like mentioned above.

[Lasse Öörni]

Yes, Aster Jian has done very good work with the Lua script system and we're now finalizing all the samples to work in Lua also. The samples already utilize ScriptInstance heavily.

Lua support naturally will stay and continue to develop, but AngelScript is still the default scripting language, considering how it supports direct exposing of functions & classes without a wrapper library, and its ref counting is an exact match with Urho classes' ref counting and memory management, so you can be sure objects can safely pass between script & C++ and be constructed / destructed on either side. In Lua there are still problems with eg. functions that return a shared pointer (like ResourceCache::GetFile()) and those are not yet exposed. It's also possible to crash the system if an object created on Lua side is prematurely freed (needs investigation.)

[Sinan - info@sinsoft.com]

That's nice to know. Yes, AngelScript is very nice indeed, and I'm finding that it's very efficient to work with.