Fwd: IDL compiler (V8 bindings generator) switched to Python | Nils Barth | 2/27/14 9:53 PM | We rewrote the IDL compiler in Python (from Perl)! The new Python IDL compiler is much more readable and maintainable. (opened May 10, 2013, over 9 months ago). As of r168029, the venerable CodeGeneratorV8.pm is gone, replaced by idl_compiler.py and modules thereof.
What you need to do Blink developers (non-bindings):
What happened? The Tokyo bindings team rewrote the Perl IDL compiler in Python. This was a ground-up rewrite, and included significant cleanups of a hoary code base; it significantly reduces the technical debt in the bindings and should make future velocity much higher.
I (Nils) wrote the code, and haraken tirelessly reviewed it and suggested revisions.
Koji (ex) did initial work on the code generator (and chose the Jinja template processor), while Kouhei's rewrite of the SVG bindings (308818) significantly simplified the rewrite, meaning SVG handling was 1 day of work and a few dozen lines of code, not a month of work and thousands of line.
The rewrite was developed incrementally on-trunk via test-driven development, and switched when the two compilers had identical output (on tests and all actual code). There was virtually no impact on bindings developers, as I synced all Perl changes to Python, with a few developers hacking on the Python themselves even before the switch.
Why? To improve hackability (ease of modification and understanding of code), primarily.
The Perl compiler was unmaintainable, due to organic growth over the years (since 2005, starting with KDE, then in WebKit). Also, we now use Python, not Perl, and this was the last significant piece of Perl in Blink (347866). Thus it needed to be rewritten in Python, and we took the opportunity to re-engineer it. This was easier once V8 was the only back end, hence why it was done when it was.
Impact The new code is significantly more hackable, particularly in the code generator (the main component and complexity). This will allow faster and more correct development.
This is due to:
The new code has a design, and a design document: IDL compiler.
The new code is faster (roughly 25-30% faster in computation time, twice as fast in real time in ninja), and can be optimized further, thanks to its modularity. I expect to speed it up further in the next week (by optimized caching of the lexer-parser).
Followup work
|
Re: [blink-dev] Fwd: IDL compiler (V8 bindings generator) switched to Python | Kentaro Hara | 2/27/14 9:55 PM | Congratulations! You removed 9000 lines of Perl :) This was a 9-month work with over 400 CLs. I’d like to emphasize that this is not just a trivial rewrite but a rewrite with a massive amount of clean-ups:
- The new Python compiler removes all the complexity that has existed in the old Perl compiler. - The new Python compiler removed all special casing for SVG. - The new Python compiler significantly improves build performance by guaranteeing that only necessary IDL files are rebuilt (Previously we were rebuilding all IDL files when any IDL file is touched).
This significantly improves maintainability of our binding infrastructure. Thanks Nils and Kouhei! (Nit: But I love Perl.) --
Kentaro Hara, Tokyo, Japan |
[blink-dev] Fwd: IDL compiler (V8 bindings generator) switched to Python | Adam Barth | 2/27/14 10:06 PM | Nils, Thank you for driving this project to completion. I'm very happy that we've freed ourselves from the old Perl code generator. That thing was a monster. Adam |
Re: [blink-dev] Fwd: IDL compiler (V8 bindings generator) switched to Python | Yoshifumi Inoue | 2/27/14 10:17 PM | Configurations! I would like to be fast customer for hacking new IDL parser to generate Closure compiler compatible externs. -yosi P.S. I was serious Perl programmer before. I wrote over 200K perl code for Amazon.com ;-)
|
Re: [blink-dev] Fwd: IDL compiler (V8 bindings generator) switched to Python | Philip Jägenstedt | 2/27/14 10:30 PM | I have had the occasional run-in with the Perl IDL compiler, so this
makes me very happy. Thank you Nils for all your work on this! Philip > To unsubscribe from this group and stop receiving emails from it, send an > email to blink-dev+...@chromium.org. |
Re: [blink-dev] Fwd: IDL compiler (V8 bindings generator) switched to Python | Daniel Bratell | 2/28/14 3:14 AM | On Fri, 28 Feb 2014 06:53:58 +0100, Nils Barth <nba...@chromium.org> wrote:
> Blink developers (non-bindings): > * Nothing. The new compiler should just work, run faster, have fewer > bugs, and be improved fasterGeneration times (quad core ~3.1 GHz machines): Linux: 29 s -> 22 s (24% of the time removed) Windows: 182 s -> 69 s (62% of the time removed) So faster. :-) Thanks! /Daniel |
Re: [blink-dev] Fwd: IDL compiler (V8 bindings generator) switched to Python | Mark Pilgrim | 2/28/14 6:15 AM | On Friday, February 28, 2014 1:06:08 AM UTC-5, Adam Barth wrote:Nils, The first (and only) time I had to touch the Perl code generator, I changed 3 lines and introduced 2 bugs while adding 1 feature. Good riddance! -Mark |
Re: [blink-dev] Fwd: IDL compiler (V8 bindings generator) switched to Python | Erik Arvidsson | 2/28/14 8:03 AM | Awesome work Nils (and everyone else who helped out). Thanks for going the distance on this one. --
erik |
Re: [blink-dev] Fwd: IDL compiler (V8 bindings generator) switched to Python | Dirk Pranke | 2/28/14 8:40 AM | Congrats from me as well! As a side note, I don't think I've ever seen so many patches attached to a single bug. I'm gonna kinda miss getting those bug updates :). -- Dirk |
Re: [blink-dev] Fwd: IDL compiler (V8 bindings generator) switched to Python | Nico Weber | 3/3/14 8:50 PM | Thanks for making build performance a priority for your rewrite!
|
Re: [blink-dev] Fwd: IDL compiler (V8 bindings generator) switched to Python | Nils Barth | 3/3/14 9:00 PM | No problem! I've got a few more ideas for improving build performance more ;)
I've also got a GYP question or two to ask you (off-list) so we can minimize full rebuilds (since 1 << 600). |