Retiring build-time execution of Perl

[Adam Barth]

People of blink-dev,

We're frighteningly close to removing all build-time execution of Perl from Blink.  As far as I know, we have only three Perl scripts remaining:

1) code_generator_v8.pm

2) xxd.pl

3) make-css-file-arrays.pl

Nils is working on removing (1) by rewriting the IDL compiler in Python.  I believe (2) and (3) can be replaced by generalizing make-file-arrays.py a bit, although we should study whether we'd be better off using a GRD file.

At this time, I'd like to propose that we not add any more Perl scripts to the build and consider Perl a linguam non grata.

Thoughts?  Volunteers to squash (2) and/or (3)?  :)

Adam

[Nils Barth]

Adam Barth:

At this time, I'd like to propose that we not add any more Perl scripts to the build and consider Perl a linguam non grata.

Thoughts?

I'm understandably in favor of "no new Perl".

(For context, I'm told Perl is one reason for needing cygwin, which makes Windows development more painful than it needs to be, so removing Perl helps with this.)

Volunteers to squash (2) and/or (3)?  :)

...and I can get a hint:

42793005: Port xxd.pl to xxd.py (Perl to Python)

(solves (2), quick win).

(3) is a bit more involved, so anyone should feel free to grab (since I'm focusing on IDL compiler rewrite).

[Kouhei Ueno]

I'd like to take (3) as a chance to improve my python skills :p

2013/10/25 Nils Barth <nba...@chromium.org>

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--

Kouhei Ueno

[Adam Barth]

On Thu, Oct 24, 2013 at 8:24 PM, Nils Barth <nba...@chromium.org> wrote:

Adam Barth:

At this time, I'd like to propose that we not add any more Perl scripts to the build and consider Perl a linguam non grata.

Thoughts?

I'm understandably in favor of "no new Perl".

(For context, I'm told Perl is one reason for needing cygwin, which makes Windows development more painful than it needs to be, so removing Perl helps with this.)

Volunteers to squash (2) and/or (3)?  :)

...and I can get a hint:

42793005: Port xxd.pl to xxd.py (Perl to Python)

(solves (2), quick win).

Thanks for the CL.

On Thu, Oct 24, 2013 at 8:41 PM, Kouhei Ueno <kou...@google.com> wrote:

I'd like to take (3) as a chance to improve my python skills :p

Thanks!  It might be worth comparing make-css-file-arrays.pl with make-file-arrays.py.  The two seem to be solving very similar problems.

Adam

[Kouhei Ueno]

I just noticed that Nills has already started working on (3) on his refactoring make-file-arrays.py in https://codereview.chromium.org/43083002/.

2013/10/25 Adam Barth <aba...@chromium.org>

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--

Kouhei Ueno

[Nils Barth]

Kouhei Ueno:

I just noticed that Nills has already started working on (3) on his refactoring make-file-arrays.py in https://codereview.chromium.org/43083002/.

Just a little refactoring (landing);

please feel free to use it when porting (3)!

(As Adam notes, they seem very similar.)

[Daniel Bratell]

I like this very much. Perl has a high maintenance cost and no real added benefits to some other languages unless you only know Perl. 

Do you know how much in Chromium outside Blink uses Perl? Would it be possible to aim for the even bigger target to remove that dependency from Chromium?

/Daniel

[Eric Seidel]

Outside of third_party, very little:

https://code.google.com/p/chromium/codesearch#search/&q=lang:%5Eperl$%20-third_party&sq=package:chromium&type=cs

[PhistucK]

What about makegrammar.pl (blink/Source/core) and the rest?

https://code.google.com/p/chromium/codesearch#search/&sq=package:chromium&q=lang:perl%20file:webkit%20-file:tools%20-file:webkitperl&type=cs

☆PhistucK

[Adam Barth]

I removed makegrammar.pl in https://chromium.googlesource.com/chromium/blink/+/bb7df1d1da526dd2ae745662f62ea65839653911

Code search takes a while to update.  :)

Adam

[PhistucK]

And the rest? there are 184 (183, minus makegrammar.pl)...

☆PhistucK

[Adam Barth]

On Fri, Oct 25, 2013 at 6:24 AM, PhistucK <phis...@gmail.com> wrote:

And the rest? there are 184 (183, minus makegrammar.pl)...

As I wrote in the first message in this thread, I believe there are only three Perl scripts remaining in the Blink build.  While we've been discussing this topic, nbarth removed xxd.pl, so now there are only two.

Many of the perl files turned up by your search have already been removed.  Others are not used in the build.  The remainder are part of the three scripts listed above.

Adam

[PhistucK]

Coo. :)

How much time does usually it take for CodeSearch to catch up?

☆PhistucK

[Jochen Eisinger]

around 8h

best

-jochen

[Nico Weber]

Many of these scripts are used at test time, not build time. I'm guessing we're keeping those around.

[PhistucK]

But test is part of the build in some cases (it compiles, tests and the build succeeds), unless I am misunderstanding. The goal here is to drop cygwin entirely, right?

☆PhistucK

[Adam Barth]

My goal is to not have to read or write Perl again.  It's not a language that scales well when written by folks like me who don't know Perl very well (and don't really care to learn).  For example, make_names.pl became an all-sing, all-dance script of amazingness.  I'm hopeful the new Python code will scale better.

Adam

On Fri, Oct 25, 2013 at 6:43 AM, PhistucK <phis...@gmail.com> wrote:

But test is part of the build in some cases (it compiles, tests and the build succeeds), unless I am misunderstanding. The goal here is to drop cygwin entirely, right?

On Fri, Oct 25, 2013 at 4:39 PM, Nico Weber <tha...@chromium.org> wrote:

Many of these scripts are used at test time, not build time. I'm guessing we're keeping those around.

On Oct 25, 2013 6:37 AM, "PhistucK" <phis...@gmail.com> wrote:

Coo. :)

How much time does usually it take for CodeSearch to catch up?

[PhistucK]

Right, so if tests fail and a Perl scripts run them (or do further or pre processing for them), they better be dropped as well, or else it will discourage people from fixing them a bit.

Nevermind - thank you for you work!

☆PhistucK

[Dirk Pranke]

I support this goal.

Didn't we used to have at least a few layout tests w/ server-side cgi scripts written in Perl? Have those been converted over?

-- Dirk

[Adam Barth]

On Fri, Oct 25, 2013 at 8:43 AM, Dirk Pranke <dpr...@chromium.org> wrote:

I support this goal.

Didn't we used to have at least a few layout tests w/ server-side cgi scripts written in Perl?

Yes.

Have those been converted over?

I haven't tried those yet.  I was focusing more on the build system.  It's not clear what a better alternative would be for those tests.  We also use PHP in a bunch of tests.  We don't have as much of a scaling problem on the test side because those scripts tend to be very short and only used for a handful of tests each.  By contrast, make_names.pl and code_generator_v8.pm were falling over under their own weight.

[Mike Sherov]

On Fri, Oct 25, 2013 at 11:54 AM, Adam Barth <aba...@chromium.org> wrote:

On Fri, Oct 25, 2013 at 8:43 AM, Dirk Pranke <dpr...@chromium.org> wrote:

I support this goal.

Didn't we used to have at least a few layout tests w/ server-side cgi scripts written in Perl?

Yes.

Have those been converted over?

I haven't tried those yet.  I was focusing more on the build system.  It's not clear what a better alternative would be for those tests.  

Why not use Python? And while you're at it, convert the PHP tests to Python as well. Why declare 1 language Lingua Non Grata, when you can do 2 for twice the price?

--

Mike Sherov
Chief Technologist
SNAP Interactive, Inc. | Ticker: STVI

http://snap-interactive.com | http://ayi.com

[Dirk Pranke]

Getting rid of all of the PHP-based tests may be more of an uphill battle, as I believe it is the only blessed language by the W3C for writing tests that require server-side scripts. The Python contingency is getting stronger there, though, so that might change over time (and if we pushed for it, that would certainly help).

-- Dirk

[ja...@hoppipolla.co.uk]

On Friday, 25 October 2013 17:03:11 UTC+1, Dirk Pranke wrote:

Getting rid of all of the PHP-based tests may be more of an uphill battle, as I believe it is the only blessed language by the W3C for writing tests that require server-side scripts. The Python contingency is getting stronger there, though, so that might change over time (and if we pushed for it, that would certainly help).

Indeed there is review open [1] to remove all the PHP from web-platforn-tests and replace it with Python. As part of this change there is also a Python-based web server intended for testing use cases. At this point Python should be considered the language used for server-side components of tests at the W3C.

It would be a significant improvement to the overall testing picture for the web platform if the Blink community started using the W3C test formats/environment as their preferred way of writing tests, and contributed the results back upstream.

[1] https://critic.hoppipolla.co.uk/r/368

[Scott Graham]

I'd like to get rid of Perl, and I'd love to get rid of cygwin, but most of the Perl in blink runs on native perl on Windows already (idl gen for example). IIRC there was a handful of perl that relied on sub-shelling to bash scripts (or similar) so they had to be kept using the cygwin version. Maybe that's fixed now.

The chromium bug for cygwin is https://code.google.com/p/chromium/issues/detail?id=123026 . (Some third_party likes to use bash scripts from within gyp, so they'd need to be convinced to do otherwise.)

On Fri, Oct 25, 2013 at 6:43 AM, PhistucK <phis...@gmail.com> wrote:

[Thomas Sepez]

Considering I added a new .pl yesterday to spew some more http spew, I'd say "No.".  There are a bunch of these ... Do we want to put a moratorium on adding new ones at the least?

On Fri, Oct 25, 2013 at 8:43 AM, Dirk Pranke <dpr...@chromium.org> wrote:

[Adam Barth]

There are some things that we can only test via Perl today.  PHP massages/normalizes its output too much for some tests.  Until we have a replacement, I think it's ok to continue to using Perl in tests.

In the build, however, we've shown that we can do everything we need to do without Perl by replacing (almost!) all the existing Perl code.  If someone does the same for the tests, then we should consider stopping using Perl for tests.

Adam

[Thomas Sepez]

So do we believe that for tests, .php > .pl? If a test could be moved from .pl to .php, is there value in that refactoring?

[Adam Barth]

Unless we see a path to removing all the Perl, it doesn't seem worth the effort to me.  Removing Perl from tests is likely going to require introducing a new technology (either Python or node.js or something else).  It doesn't seem worth porting from Perl to PHP just to port to another technology later, but other people might have a different opinion.

Adam

[Philip Jägenstedt]

FWIW, every new LayoutTest I've added to Blink has been using
testharness.js, but I have not been good enough to contribute them to
web-platform-tests. The biggest issue holding me back is that I don't
know of any system to keep the tests in sync after I send them. Does
anyone else have a process for this? Maybe first sending it to
web-platform-tests and having it locally as an imported test only?

Somewhat related, does anyone on the Blink project actively look at
and import new tests from web-platform-tests? It seems that simply
having it a subrepo might be a way to go, eventually.

Philip

[Bem Jones-Bey]

Starting a new thread...

On Oct 25, 2013, at 13:00 , Philip Jägenstedt <phi...@opera.com> wrote:

> FWIW, every new LayoutTest I've added to Blink has been using
> testharness.js, but I have not been good enough to contribute them to
> web-platform-tests. The biggest issue holding me back is that I don't
> know of any system to keep the tests in sync after I send them. Does
> anyone else have a process for this? Maybe first sending it to
> web-platform-tests and having it locally as an imported test only?
>
> Somewhat related, does anyone on the Blink project actively look at
> and import new tests from web-platform-tests? It seems that simply
> having it a subrepo might be a way to go, eventually.

I've been writing tests and contributing them to the CSS WG recently. The way I do it is that I write the tests in the CSS WG's repo, and then I import them into Blink using ./Tools/Scripts/import-w3c-tests. I import them into ./LayoutTests/csswg/... instead of the default location that import-w3c-tests wants to use because dpranke@ has a plan to automate the import of *all* of the tests and at that point I shouldn't have to manually import. I'm sure he'll probably chime in soon, too. :-)

- Bem

[Dirk Pranke]

On Fri, Oct 25, 2013 at 12:35 PM, Adam Barth <aba...@chromium.org> wrote:

There are some things that we can only test via Perl today.  PHP massages/normalizes its output too much for some tests.  Until we have a replacement, I think it's ok to continue to using Perl in tests.

This would be somewhere between kinda surprising and really surprising to me. If you can point me to examples, I'd be happy to look at it.

-- Dirk

[Dirk Pranke]

We do not regularly import the w3c tests. I have done a bunch of work on that front and have a plan for the rest. This was discussed -- but unfortunately not recorded -- at BlinkOn as well. Unfortunately, this has been stuck behind Android and Aura work for me, so I haven't been able to get it over the finish line.

Hopefully I will soon. I'd offer to pass this off to someone, but I'm afraid that even attempting to transfer the knowledge of the current state and what's left would take as much time as just getting it going.

-- Dirk

[Adam Barth]

On Fri, Oct 25, 2013 at 1:58 PM, Dirk Pranke <dpr...@chromium.org> wrote:

On Fri, Oct 25, 2013 at 12:35 PM, Adam Barth <aba...@chromium.org> wrote:

There are some things that we can only test via Perl today.  PHP massages/normalizes its output too much for some tests.  Until we have a replacement, I think it's ok to continue to using Perl in tests.

This would be somewhere between kinda surprising and really surprising to me. If you can point me to examples, I'd be happy to look at it.

LayoutTests/http/tests/encoding/meta-switch-mid-parse-with-title.html is a recent example.  It requires sending broken unicode on the wire and controlling exactly when the underlying socket is flushed.

Adam

[ja...@hoppipolla.co.uk]

On Friday, 25 October 2013 22:04:57 UTC+1, Adam Barth wrote:

On Fri, Oct 25, 2013 at 1:58 PM, Dirk Pranke <dpr...@chromium.org> wrote:

On Fri, Oct 25, 2013 at 12:35 PM, Adam Barth <aba...@chromium.org> wrote:

There are some things that we can only test via Perl today.  PHP massages/normalizes its output too much for some tests.  Until we have a replacement, I think it's ok to continue to using Perl in tests.

This would be somewhere between kinda surprising and really surprising to me. If you can point me to examples, I'd be happy to look at it.

LayoutTests/http/tests/encoding/meta-switch-mid-parse-with-title.html is a recent example.  It requires sending broken unicode on the wire and controlling exactly when the underlying socket is flushed.

For what it's worth; wptserve (code review [1], documentation [2]), the python server developed to replace PHP for the shared public testsuite is specifically designed to allow this level of control, with a high level API for simple cases but raw socket-level access for those cases where it's needed.

[1] https://critic.hoppipolla.co.uk/r/364
[2] http://wptserve.readthedocs.org/en/latest/

[Dirk Pranke]

Somehow I missed this thread when it first happened, but y'all are long overdue on an update on this.

I had been working on aura-related work for the past few weeks, but that is all complete and I'm getting back to working on this stuff again. I should be landing CLs to start importing the repos by default and importing at least a few of the test suites soon. Once I've got a couple of directories working I'll send out another note and document the process on dev.chromium.org.

Upstreaming new tests is a bit trickier as we need to integrate ourselves into the W3C's processes. The first step for this will be to set up forks of their repos on GitHub and make sure we have at least some people that can upload new test suites to our forks and then send pull requests to get them merged into their repos. We have some existing committers to the W3C repos already, so this should be reasonably straightforward, but clearly we're going to need to work through a few iterations on this before we get all the kinks de-kinked.

I have a list of a few existing suites we'd like to upstream and a few others we'd like to definitely downstream and be running, but if you have more you're interested in, feel free to follow up with me off-list.

-- Dirk 

[Philip Jägenstedt]

On Wed, Nov 27, 2013 at 10:36 PM, Dirk Pranke <dpr...@chromium.org> wrote:
> Somehow I missed this thread when it first happened, but y'all are long
> overdue on an update on this.
>
> I had been working on aura-related work for the past few weeks, but that is
> all complete and I'm getting back to working on this stuff again. I should
> be landing CLs to start importing the repos by default and importing at
> least a few of the test suites soon. Once I've got a couple of directories
> working I'll send out another note and document the process on
> dev.chromium.org.
>
> Upstreaming new tests is a bit trickier as we need to integrate ourselves
> into the W3C's processes. The first step for this will be to set up forks of
> their repos on GitHub and make sure we have at least some people that can
> upload new test suites to our forks and then send pull requests to get them
> merged into their repos. We have some existing committers to the W3C repos
> already, so this should be reasonably straightforward, but clearly we're
> going to need to work through a few iterations on this before we get all the
> kinks de-kinked.
>
> I have a list of a few existing suites we'd like to upstream and a few
> others we'd like to definitely downstream and be running, but if you have
> more you're interested in, feel free to follow up with me off-list.

Thanks for the update, Dirk. Will contact you off-list.

Philip