Fwd: IDL compiler (V8 bindings generator) switched to Python

[Nils Barth]

We rewrote the IDL compiler in Python (from Perl)!

The new Python IDL compiler is much more readable and maintainable.

Comment 418 closed issue 239771: Rewrite the IDL compiler in Python

(opened May 10, 2013, over 9 months ago).

As of r168029, the venerable CodeGeneratorV8.pm is gone, replaced by idl_compiler.py and modules thereof.

What you need to do

Blink developers (non-bindings):

• Nothing. The new compiler should just work, run faster, have fewer bugs, and be improved faster.
• If you want new IDL features, it's much easier now!
• file a bug in Cr-Blink-Bindings (set as blocking 345506: Implement Web IDL spec),
• comment on that bug,
• contact bindings developers (esp. bindings/OWNERS or WATCHLISTS: 'bindings'), or
• ...try implementing it yourself (it's fun now!)

Bindings developers:

• Please have a quick look at the design doc (IDL compiler), particularly Back end (code generator), and the Jinja page for guidance on writing and editing the templates.
• Please add me (nbarth@) as a reviewer for CLs that edit the new compiler (bindings/scripts), until we're all used to hacking on it: it's rather differently structured from the old one.

What happened?

The Tokyo bindings team rewrote the Perl IDL compiler in Python. This was a ground-up rewrite, and included significant cleanups of a hoary code base; it significantly reduces the technical debt in the bindings and should make future velocity much higher.

I (Nils) wrote the code, and haraken tirelessly reviewed it and suggested revisions.

Koji (ex) did initial work on the code generator (and chose the Jinja template processor), while Kouhei's rewrite of the SVG bindings (308818) significantly simplified the rewrite, meaning SVG handling was 1 day of work and a few dozen lines of code, not a month of work and thousands of line.

The rewrite was developed incrementally on-trunk via test-driven development, and switched when the two compilers had identical output (on tests and all actual code). There was virtually no impact on bindings developers, as I synced all Perl changes to Python, with a few developers hacking on the Python themselves even before the switch.

Why?

To improve hackability (ease of modification and understanding of code), primarily.

The Perl compiler was unmaintainable, due to organic growth over the years (since 2005, starting with KDE, then in WebKit). Also, we now use Python, not Perl, and this was the last significant piece of Perl in Blink (347866). Thus it needed to be rewritten in Python, and we took the opportunity to re-engineer it. This was easier once V8 was the only back end, hence why it was done when it was.

Impact

The new code is significantly more hackable, particularly in the code generator (the main component and complexity). This will allow faster and more correct development.

This is due to:

• factoring the code generator into separate modules and templates (e.g., v8_attributes.py/attributes.cpp), rather than one monolithic module;
  
• factoring the compiler into a pipeline (lex/parse/construct/CG logic/CG template);
• the use of templates, rather than building the output by concatenating strings;
  
• moving complexity out of the code generator and into a global context step and the front end.

For example, the Perl code generator called the parser to read IDL files for any interfaces it encountered (e.g., to check the C++ class name)! This was replaced by computing the public information of all interfaces in a preliminary step, and (in rare cases) reading in any interfaces whose private information is needed in the front end.

The new code has a design, and a design document: IDL compiler.

The new code is faster (roughly 25-30% faster in computation time, twice as fast in real time in ninja), and can be optimized further, thanks to its modularity. I expect to speed it up further in the next week (by optimized caching of the lexer-parser).

Followup work

• IDL feature work (345506: Implement Web IDL spec): everyone can pitch in!
• Remove Perl from build (347863): one remaining script (make-css-file-arrays.pl): I'll do this unless anyone else wants to grab it
• Replace embedding resources by ad hoc Python with GRD resources (312586)
• Continue cleanup: (341748, 345137, 345503, 345519): long-term; I'll primarily do this until it's clean enough
• Speed up build (part of 341748):
• caching optimized lexer and parser should improve speed
• precise dependency resolution in GYP to prevent unnecessary full rebuilds

Learn more

• Web IDL interfaces: practical overview
  
• Web IDL in Blink: technical details
  
• IDL compiler: design doc

[Kentaro Hara]

Congratulations! You removed 9000 lines of Perl :)

This was a 9-month work with over 400 CLs. I’d like to emphasize that this is not just a trivial rewrite but a rewrite with a massive amount of clean-ups:

- The new Python compiler removes all the complexity that has existed in the old Perl compiler.

- The new Python compiler removed all special casing for SVG.

- The new Python compiler significantly improves build performance by guaranteeing that only necessary IDL files are rebuilt (Previously we were rebuilding all IDL files when any IDL file is touched).

This significantly improves maintainability of our binding infrastructure. Thanks Nils and Kouhei!

(Nit: But I love Perl.)

--
Kentaro Hara, Tokyo, Japan

[Adam Barth]

Nils,

Thank you for driving this project to completion.  I'm very happy that we've freed ourselves from the old Perl code generator.  That thing was a monster.

Adam

[Yoshifumi Inoue]

Configurations!

I would like to be fast customer for hacking new IDL parser to generate Closure compiler compatible externs.

-yosi

P.S.

I was serious Perl programmer before. I wrote over 200K perl code for Amazon.com ;-)

[Philip Jägenstedt]

I have had the occasional run-in with the Perl IDL compiler, so this
makes me very happy. Thank you Nils for all your work on this!

Philip

> To unsubscribe from this group and stop receiving emails from it, send an
> email to blink-dev+...@chromium.org.

[Daniel Bratell]

On Fri, 28 Feb 2014 06:53:58 +0100, Nils Barth <nba...@chromium.org> wrote:

> Blink developers (non-bindings):
> * Nothing. The new compiler should just work, run faster, have fewer

> bugs, and be improved faster

Generation times (quad core ~3.1 GHz machines):

Linux: 29 s -> 22 s (24% of the time removed)
Windows: 182 s -> 69 s (62% of the time removed)

So faster. :-)

Thanks!

/Daniel

[Mark Pilgrim]

On Friday, February 28, 2014 1:06:08 AM UTC-5, Adam Barth wrote:

Nils,

Thank you for driving this project to completion.  I'm very happy that we've freed ourselves from the old Perl code generator.  That thing was a monster.

The first (and only) time I had to touch the Perl code generator, I changed 3 lines and introduced 2 bugs while adding 1 feature. Good riddance!

-Mark 

[Erik Arvidsson]

Awesome work Nils (and everyone else who helped out). Thanks for going the distance on this one.

--
erik

[Dirk Pranke]

Congrats from me as well!

As a side note, I don't think I've ever seen so many patches attached to a single bug. I'm gonna kinda miss getting those bug updates :).

-- Dirk

[Nico Weber]

Thanks for making build performance a priority for your rewrite!

 

Thanks!

/Daniel

[Nils Barth]

No problem!

I've got a few more ideas for improving build performance more ;)

I've also got a GYP question or two to ask you (off-list) so we can minimize full rebuilds (since 1 << 600).