Hi v8-dev,
After catching up with Leszek about a month ago to test the waters on a new lexer/parser design, I've finally finished the design doc. I've also done a lot of exploratory work inside V8 to check that the integration points can deliver the benefits described, and to flush out any obvious issues.
The design comes from more than 2,000 hours of research in this area across a few years. Its main benefit is parse speed: in V8, a first estimate puts the scanner swap alone at 1.2x to 1.5x, and about 1.9x with parser changes (projected, coarsely measured).
The design centers on "unfusing" lexing and parsing. The scanner becomes SIMD-driven, heavily inspired by simdjson, although the algorithm for JS differs a lot. The doc links a prototype and a full reference implementation. These are references, not the intended V8 implementation, which will differ in deliberate ways.
The doc suggests doing this in Rust over FFI, though any language with SIMD access would work.
I'm an external contributor, so I'd also appreciate guidance on who should act as TL for the LGTMs. The doc lists Leszek, Marja, Toon and v8-security.
Thanks,
Shanon Jackson