About two months I think, with two prototypes before this one, and using the learning of each prototype to improve the next one (and with Shen/Scheme as the main reference and testing target).
First prototype was purely (informed) vibecoded, resulting in lots of C code, about ~50kloc, quite messy.
Second prototype was quite Chez-like. More of it was written in Shen, but was still too big (~28kloc, but cleaner).
This third version was made with the explicit goal to not go over 10kloc, clean, while also being fast (it is the fastest of the three), and kinda following the Niklaus Wirth principle that compiler optimizations must pay for themselves.
One reason the second prototype got so big was that the GC was quite a bit more advanced than what I have on this version, which also required lots of machinery in the compiler itself to make sure the generated code and the GC behaved correctly. But doing multiple passes turned out to make it bigger too.
It is different from Chez (in-memory representation of values is somewhat similar to what Chez and OCaml do tho): c
ompiler is single-pass (so, not at all like nanopass), and floats are boxed like in Chez and OCaml, no NaN-coding.
As for how it can be faster:
- First, it is faster at the benchmarks I used, and general idiomatic Shen code, but that doesn't mean it produces better code for every workload, at least not yet. It is faster at Prolog for sure because I optimized specifically for that by doing something I could not do in Shen/Scheme, which is give a native implementation for pvars (which are just an integer with a specific tag in Shen/JIT, while in Shen/Scheme they are a small vector with a tag and an integer).
- Second, Chez is optimized for Scheme, Shen/JIT is optimized for KLambda and Shen. They may be similar, but not exactly the same.
- Third, I have one optimization to more efficiently support re-definable functions compared to Chez (but "sealed" mode is also supported for a little extra performance). If you enable sealing in Shen/Scheme (so that Chez compiles the code with the assumption that redefinitions will not happen), you get an extra boost. In Shen/Scheme the kernel code is already compiled sealed, but user code is not.
- Also Shen/JIT usually consumes more memory because the GC is more basic (it is not bad, but it is not as advanced as Chez's or OCaml's)