Updates to LuaJIT Port
72 views
Skip to first unread message
Reuben
Sep 28, 2026, 8:35:18 PMSep 28
to Shen
I made some recent updates to shen-lua, focused on the LuaJIT runtime.
you can also install w/ `luarocks --lua-version=5.1 install shen 0.11.1-1`
I’ve worked on a number of different ports, but I’ve found this one particularly interesting because LuaJIT is such a different host. It has led to a distinctive architecture and set of trade-offs.
First, KLambda is compiled to Lua source. LuaJIT initially runs that code in its VM; when a path becomes “hot,” its tracer records the operations taken and compiles that path to machine code. It optimizes frequently executed paths, rather than simply compiling whole functions because they’ve been called often. The shape of the generated Lua matters, but so does what the Shen program actually does at runtime.
Closures were one of the most interesting parts of the port. Shen’s Prolog compiler and typechecker produce continuation-heavy KLambda. A naïve translation of every freeze into a nested Lua function literal would make some Shen 42 programs exceed LuaJIT’s parser limits. Instead, our compiler hoists continuation bodies into a flat table of functions created at load time. At each freeze site, a small thunk snapshots the free-variable values and refers to the appropriate body. This preserves the delayed computation without nesting another function definition at the use site.
LuaJIT’s FFI offers another way to reduce allocation in the reasoning hot path. When a query takes the native path, the Prolog/type-inference engine stores terms in compact integer arrays and represents continuations with integer handles and a capture buffer, rather than allocating an ordinary Lua closure for every continuation. The compiled-KLambda engine remains available as a fallback. This changes the execution substrate, not Shen’s inference rules.
Other accommodations grew out of the same approach. Large KLambda literal trees can become flat construction blueprints rather than expressions nested beyond LuaJIT’s parser limit. Pure self-tail recursion can become a loop, while list builders shaped like [X | (f ...)] can use tail recursion modulo cons to build long lists without growing the Lua call stack.
@Bruno, as you look at compiling Shen directly to native code, I thought this might be of interest. It’s a rather different way to approach the runtime execution of Shen code.
you can also install w/ `luarocks --lua-version=5.1 install shen 0.11.1-1`
I’ve worked on a number of different ports, but I’ve found this one particularly interesting because LuaJIT is such a different host. It has led to a distinctive architecture and set of trade-offs.
First, KLambda is compiled to Lua source. LuaJIT initially runs that code in its VM; when a path becomes “hot,” its tracer records the operations taken and compiles that path to machine code. It optimizes frequently executed paths, rather than simply compiling whole functions because they’ve been called often. The shape of the generated Lua matters, but so does what the Shen program actually does at runtime.
Closures were one of the most interesting parts of the port. Shen’s Prolog compiler and typechecker produce continuation-heavy KLambda. A naïve translation of every freeze into a nested Lua function literal would make some Shen 42 programs exceed LuaJIT’s parser limits. Instead, our compiler hoists continuation bodies into a flat table of functions created at load time. At each freeze site, a small thunk snapshots the free-variable values and refers to the appropriate body. This preserves the delayed computation without nesting another function definition at the use site.
LuaJIT’s FFI offers another way to reduce allocation in the reasoning hot path. When a query takes the native path, the Prolog/type-inference engine stores terms in compact integer arrays and represents continuations with integer handles and a capture buffer, rather than allocating an ordinary Lua closure for every continuation. The compiled-KLambda engine remains available as a fallback. This changes the execution substrate, not Shen’s inference rules.
Other accommodations grew out of the same approach. Large KLambda literal trees can become flat construction blueprints rather than expressions nested beyond LuaJIT’s parser limit. Pure self-tail recursion can become a loop, while list builders shaped like [X | (f ...)] can use tail recursion modulo cons to build long lists without growing the Lua call stack.
@Bruno, as you look at compiling Shen directly to native code, I thought this might be of interest. It’s a rather different way to approach the runtime execution of Shen code.
Bruno Deferrari
Sep 30, 2026, 7:33:58 PMSep 30
to qil...@googlegroups.com
Cool, it never really occurred to me to apply TRMC to KLambda. A somewhat related case (but quite different actually) would be to figure out how to make @s and @v deconstruction and construction efficient. By default they are quite wasteful because lots of short-lived intermediary sub-copies are created. If the compiler could detect that, it could preallocate the space and also avoid the recursive call. In the case of @s it could use sub-slices and share the original memory (strings are immutable) every time, for @v it would need to detect that there is no mutation.
Regarding the tracing JIT, I briefly considered it initially, but I think for Shen the benefit is not as great as what you get on Lua or Typescript (which have objects and dynamic dispatch is far more common). In Shen (except for everything surrounding `eval`) code is far more static, and datatype shapes too. On the other hand, Shen Prolog can probably benefit from it. I didn't discard the option, just didn't consider it a good start in my case, but worth experimenting with later.
--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/qilang/e49669f5-c000-40dd-9f06-f3955cd56f0bn%40googlegroups.com.
BD
Bruno Deferrari
Oct 2, 2026, 12:10:25 PM (12 days ago) Oct 2
to qil...@googlegroups.com
On Wed, Sep 30, 2026 at 8:33 PM Bruno Deferrari <uti...@gmail.com> wrote:
On Mon, Sep 28, 2026 at 9:35 PM Reuben <reub....@gmail.com> wrote:I made some recent updates to shen-lua, focused on the LuaJIT runtime.
you can also install w/ `luarocks --lua-version=5.1 install shen 0.11.1-1`
I’ve worked on a number of different ports, but I’ve found this one particularly interesting because LuaJIT is such a different host. It has led to a distinctive architecture and set of trade-offs.
First, KLambda is compiled to Lua source. LuaJIT initially runs that code in its VM; when a path becomes “hot,” its tracer records the operations taken and compiles that path to machine code. It optimizes frequently executed paths, rather than simply compiling whole functions because they’ve been called often. The shape of the generated Lua matters, but so does what the Shen program actually does at runtime.
Closures were one of the most interesting parts of the port. Shen’s Prolog compiler and typechecker produce continuation-heavy KLambda. A naïve translation of every freeze into a nested Lua function literal would make some Shen 42 programs exceed LuaJIT’s parser limits. Instead, our compiler hoists continuation bodies into a flat table of functions created at load time. At each freeze site, a small thunk snapshots the free-variable values and refers to the appropriate body. This preserves the delayed computation without nesting another function definition at the use site.
LuaJIT’s FFI offers another way to reduce allocation in the reasoning hot path. When a query takes the native path, the Prolog/type-inference engine stores terms in compact integer arrays and represents continuations with integer handles and a capture buffer, rather than allocating an ordinary Lua closure for every continuation. The compiled-KLambda engine remains available as a fallback. This changes the execution substrate, not Shen’s inference rules.
Other accommodations grew out of the same approach. Large KLambda literal trees can become flat construction blueprints rather than expressions nested beyond LuaJIT’s parser limit. Pure self-tail recursion can become a loop, while list builders shaped like [X | (f ...)] can use tail recursion modulo cons to build long lists without growing the Lua call stack.
@Bruno, as you look at compiling Shen directly to native code, I thought this might be of interest. It’s a rather different way to approach the runtime execution of Shen code.Cool, it never really occurred to me to apply TRMC to KLambda. A somewhat related case (but quite different actually) would be to figure out how to make @s and @v deconstruction and construction efficient. By default they are quite wasteful because lots of short-lived intermediary sub-copies are created. If the compiler could detect that, it could preallocate the space and also avoid the recursive call. In the case of @s it could use sub-slices and share the original memory (strings are immutable) every time, for @v it would need to detect that there is no mutation.
Btw are you familiar with Koka? There is interesting research there related to those kinds of optimizations.
Regarding the tracing JIT, I briefly considered it initially, but I think for Shen the benefit is not as great as what you get on Lua or Typescript (which have objects and dynamic dispatch is far more common). In Shen (except for everything surrounding `eval`) code is far more static, and datatype shapes too. On the other hand, Shen Prolog can probably benefit from it. I didn't discard the option, just didn't consider it a good start in my case, but worth experimenting with later.
--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/qilang/e49669f5-c000-40dd-9f06-f3955cd56f0bn%40googlegroups.com.
--BD
BD
Reuben
Oct 2, 2026, 7:04:28 PM (12 days ago) Oct 2
to Shen
Koka is fascinating -- had no idea this existed. Don't have a CS background but I'm constantly discovering curious things like this... TY.
I wonder if developing a Persius-inspired "P-lambda" might be useful for making more performant ports simple to implement? Shen -> Klambda -> Plambda -> {generic port}
The sort of idea that I'd never have gotten around to before AI but might just be worth spiking w/ AI just to try it out...
I wonder if developing a Persius-inspired "P-lambda" might be useful for making more performant ports simple to implement? Shen -> Klambda -> Plambda -> {generic port}
The sort of idea that I'd never have gotten around to before AI but might just be worth spiking w/ AI just to try it out...
Reply all
Reply to author
Forward
0 new messages