[ANN] Shen/JIT

140 views
Skip to first unread message

Bruno Deferrari

unread,
Nov 5, 2026, 11:25:57 AMNov 5
to qil...@googlegroups.com
Just released Shen/JIT, an implementation of Shen that compiles to native code.

Runs faster than Shen/Scheme and ships as a single 1 MB executable. Unlike Shen/Scheme, it has no native bignums (integers are 62-bit).

Shen/JIT was built over two months, using Codex and Claude. It is the third implementation: a first prototype took about 3 weeks and a second about 5, each informing the next. This final one took less than 5 days, and is faster and much smaller than both.

Highlights:

- Backtraces
- Built-in profiler
- C FFI
- DSL for writing and generating your own native code at run time, in Shen/JIT’s instructions.
- Saving the heap as an image, or as a standalone executable
- Self-bootstrapping

Guide: https://tizoc.github.io/shen-jit/
Source: https://github.com/tizoc/shen-jit
Releases: https://github.com/tizoc/shen-jit/releases

--
BD

Joshua Scholar

unread,
Nov 5, 2026, 1:33:29 PMNov 5
to qil...@googlegroups.com
That's amazing. How long have you been working on this?  
I've been wanting to write something similar (but more a general purpose backend than Shen) for a long time, but I was planning on using the lowest level parts from LLVM (machine language, no not their IR and not their optimizers).
But you did it all yourself, that's huge!
I would have been afraid to try that because just getting the assembly language correct on one cpu is a huge job and you did two!  Did you adapt code from chez?

And how did you do the optimization?  Is it nan-coded pointers?  That would explain getting a speed up over chez maybe, which I think tends to box floats. Did you adapt from nanopass?

happy_for_you_meme.jpg



--
You received this message because you are subscribed to the Google Groups "Shen" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qilang+un...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/qilang/CACnPB4oM%2BpuWGX_btMPWdFoNu3SOFSBfUS1_%3DTv4oL4BikpD8g%40mail.gmail.com.

dr.mt...@gmail.com

unread,
Nov 5, 2026, 2:12:50 PMNov 5
to Shen
A great achievement.  I've tested this port on Schubert's Steamroller;
it clocks 36MLips vs Chez/Scheme51's 20MLips.   Extremely well
done.  My highest commendation.

M.

nha...@gmail.com

unread,
Nov 5, 2026, 3:03:22 PMNov 5
to Shen
Really impressive, well done.

Reuben

unread,
Nov 5, 2026, 4:09:59 PMNov 5
to Shen
Really awesome stuff. I ran some initial benchmarks, tracks everything you said. I'd started working on a shen-llvm port but my performance numbers don't even sniff at shen-scheme, let alone shen-jit.

image_2026-11-05_145219845.png

Echo Josh's questions on methodology -- did you end up using any of the nanopass-style approach?

Bruno Deferrari

unread,
Nov 5, 2026, 4:17:16 PMNov 5
to qil...@googlegroups.com
About two months I think, with two prototypes before this one, and using the learning of each prototype to improve the next one (and with Shen/Scheme as the main reference and testing target).

First prototype was purely (informed) vibecoded, resulting in lots of C code, about ~50kloc, quite messy.
Second prototype was quite Chez-like. More of it was written in Shen, but was still too big (~28kloc, but cleaner).
This third version was made with the explicit goal to not go over 10kloc, clean, while also being fast (it is the fastest of the three), and kinda following the Niklaus Wirth principle that compiler optimizations must pay for themselves.

One reason the second prototype got so big was that the GC was quite a bit more advanced than what I have on this version, which also required lots of machinery in the compiler itself to make sure the generated code and the GC behaved correctly. But doing multiple passes turned out to make it bigger too.

It is different from Chez (in-memory representation of values is somewhat similar to what Chez and OCaml do tho): compiler is single-pass (so, not at all like nanopass), and floats are boxed like in Chez and OCaml, no NaN-coding.

As for how it can be faster:
- First, it is faster at the benchmarks I used, and general idiomatic Shen code, but that doesn't mean it produces better code for every workload, at least not yet. It is faster at Prolog for sure because I optimized specifically for that by doing something I could not do in Shen/Scheme, which is give a native implementation for pvars (which are just an integer with a specific tag in Shen/JIT, while in Shen/Scheme they are a small vector with a tag and an integer).
- Second, Chez is optimized for Scheme, Shen/JIT is optimized for KLambda and Shen. They may be similar, but not exactly the same.
- Third, I have one optimization to more efficiently support re-definable functions compared to Chez (but "sealed" mode is also supported for a little extra performance). If you enable sealing in Shen/Scheme (so that Chez compiles the code with the assumption that redefinitions will not happen), you get an extra boost. In Shen/Scheme the kernel code is already compiled sealed, but user code is not.
- Also Shen/JIT usually consumes more memory because the GC is more basic (it is not bad, but it is not as advanced as Chez's or OCaml's)




--
BD

Bruno Deferrari

unread,
Nov 5, 2026, 4:19:24 PMNov 5
to qil...@googlegroups.com
LLVM is overkill, just fork Shen/JIT and start playing with modifications, there is still space for optimizations.

Now my plan is to just take care of the long hanging fruit, the idea is that it must be as fast as is reasonable without making it too big, then take advantage of the native code generation DSL to take care of the code that really needs to be as fast as possible (there is a basic DSL there already but I am still working on it).



--
BD

Bruno Deferrari

unread,
Nov 5, 2026, 4:35:35 PMNov 5
to qil...@googlegroups.com
Btw on which processor are these numbers? Right now the ARM target is more optimized than the Intel one (mainly because I am doing all this primarily on a macbook, and then from time to time I work on a linux machine with an AMD processor to benchmark and improve the generated code there).

On Thu, Nov 5, 2026 at 6:09 PM Reuben <reub....@gmail.com> wrote:


--
BD

nha...@gmail.com

unread,
Nov 5, 2026, 4:59:56 PMNov 5
to Shen

I guess Naive-reverse loses to Chez due to GC?

Also I wonder if it’s better to just have a DSL for numeric computing than 
 to try to bend over backwards trying to optimise floating point code in lisp like languages.

Bruno Deferrari

unread,
Nov 5, 2026, 5:12:02 PMNov 5
to qil...@googlegroups.com
Probably not, but I am not sure what that benchmark was (prolog? plain shen?) or on which kind of CPU (some things are currently slower on x64).

Something at which Chez is more sophisticated is at the use of registers. Second prototype was similar in that sense but I decided it was not worth it.



--
BD

Bruno Deferrari

unread,
Nov 5, 2026, 5:13:54 PMNov 5
to qil...@googlegroups.com
On Thu, Nov 5, 2026 at 6:59 PM nha...@gmail.com <nha...@gmail.com> wrote:
Also I wonder if it’s better to just have a DSL for numeric computing than 
 to try to bend over backwards trying to optimise floating point code in lisp like languages.

Ah, I missed this part. IMO yes, especially with agents being able to do the bulk of the work (testing all the options, which native code performs better, and even coming up for a higher level DSL that you could use to describe the type of programs to which you are trying to optimize for).

--
BD

Reuben

unread,
Nov 5, 2026, 5:32:02 PMNov 5
to Shen
Mac M5. No guarantee they were accurate as not a quiet machine but confident the order is right.

Bruno Deferrari

unread,
Nov 6, 2026, 6:21:47 AM (13 days ago) Nov 6
to qil...@googlegroups.com
Next Shen/Scheme release is likely going to get faster btw (maybe beating the current version of Shen/JIT again, except the Prolog parts).
Having Shen/JIT as an alternative to compare against allowed me to find parts to improve in Shen/Scheme.
--
BD

Joshua Scholar

unread,
Nov 6, 2026, 7:23:15 AM (13 days ago) Nov 6
to qil...@googlegroups.com
" LLVM is overkill,"

I'm only going to use the MC (machine code) parts of LLVM, I don't think I even have to link the rest.  The point was to get assemblers, both in-memory and linking and maybe get some debugging support more or less for free.  Also I'm going to have a prolog engine and maybe even a term rewriting engine before I consider building a shen that uses them.
Hopefully it will be a fast prolog with a lot of optimizations stolen from BProlog, a jit, rational tree unification and tabling so it can handle proofs that SLD resolution can't. 

Bruno Deferrari

unread,
Nov 6, 2026, 7:34:51 AM (13 days ago) Nov 6
to qil...@googlegroups.com
Fair enough. I didn't even know about LLVM-MC until I read your reply btw. My comment was about the whole LLVM stack with its IR, optimization pipeline and everything.




--
BD

Reuben

unread,
Nov 6, 2026, 5:43:51 PM (13 days ago) Nov 6
to Shen
Agree, I've also found that having shen-jit for fuel has driven significant improvements in `shen-lua` and `shen-go`, the main ones I've looked at.

Relatedly, inspired by this from Anthropic, I've been taking some steps towards exploring how to formally verify a shen implementation. I was working from shen-scheme but have pivoted to shen-jit since it landed. 

https://github.com/pyrex41/buri

I've also updated bifrost and yggdrasil to support shen-jit.
Reply all
Reply to author
Forward
0 new messages