Blink PerformanceTests

[Tony Gentilcore]

Hi Blinkers,

Blink's PerformanceTests are now being run and monitored on the chromium.webkit and chromium.perf waterfalls. They run using the same performance testing framework as chrome, Telemetry. This allows them to be run against any browser (chrome, content_shell, test_shell, etc) on any platform (including android and cros).

To add a new performance test, add a .html file to that directory. It will automatically be run on the waterfalls and monitored for regressions.

To skip a misbehaving performance test, add it to the Skipped file.

To run a test locally, use Telemetry. For example:

  $ tools/perf/run_measurement --browser=release blink_perf third_party/WebKit/PerformanceTests/Parser/html-parser.html

To profile a test:

  mac: $ tools/perf/run_measurement --browser=release --profiler-tool=iprofiler --profiler-dir=/tmp/myprofile blink_perf third_party/WebKit/PerformanceTests/Parser/html-parser.html

  linux: $ tools/perf/run_measurement --browser=release --profiler-tool=perf --profiler-dir=/tmp/myprofile blink_perf third_party/WebKit/PerformanceTests/Parser/html-parser.html

Happy benchmarking!

-Tony

[Adam Barth]

That's awesome.  Thanks for doing that!

If you're like me and you enjoy graphs, here's html5-full-render on a variety of platforms:

https://chromeperf.appspot.com/report?masters=ChromiumPerf&tests=blink_perf%2FParser_html5-full-render&bots=android-nexus10%2Candroid-nexus4%2Cchromium-rel-mac6%2Cchromium-rel-win7-dual%2Clinux-release

Does this mean that these tests are now being monitored for regressions by the perf sheriff? 

Thanks!

Adam

[Tony Gentilcore]

On Tue, Jun 18, 2013 at 10:50 AM, Adam Barth <aba...@chromium.org> wrote:

That's awesome.  Thanks for doing that!

If you're like me and you enjoy graphs, here's html5-full-render on a variety of platforms:

https://chromeperf.appspot.com/report?masters=ChromiumPerf&tests=blink_perf%2FParser_html5-full-render&bots=android-nexus10%2Candroid-nexus4%2Cchromium-rel-mac6%2Cchromium-rel-win7-dual%2Clinux-release

Does this mean that these tests are now being monitored for regressions by the perf sheriff? 

Yep!

[Elliott Sprehn]

This is really cool, but there's no way in the graph to tell which rev number is at a specific point. If I hover the graph or click it I don't get more information. The old perf dashboard tells you all that stuff.

Can we get some more info added? :)

[Elliott Sprehn]

Actually this looks like a bug. There's logic to do hit testing for the graphs to decide if you're hovering the line, but it's not scroll aware. As long as you're scrolled to the top of the page you get the tooltips when hovering the graph lines, but if you scroll down they're broken.

There's code somewhere that doesn't adjust the offset positions with the scrollTop. :)

[Tony Gentilcore]

Annie can confirm, but I believe it is a known polymer bug. It works properly on the stable channel, but not in dev/canary.

[Rebecca Crabb]

Hovering over the graph should give you more information:

We are dogfooding Polymer for the dashboard and there is an open bug right now that breaks our hover when you scroll the page.  If you look at one graph at a time you should be able to work around this issue for now.

[Ojan Vafai]

Is there any way to measure something other than page load? I have a flexbox test-case that exhibits slow layout behavior, but I expect that the load time of the page will be dominated by the HTML parser and style resolver. 

On Tue, Jun 18, 2013 at 10:33 AM, Tony Gentilcore <to...@chromium.org> wrote:

[Tony Gentilcore]

Yeah, could you just force layouts after the page loads?

For example:

function test() {

  var e = document.getElementById('foo');

  e.style.width = "640px";

  e.offsetTop;

  e.style.width = "630px";

  e.offsetTop;

}

PertTestRunner.measureRunsPerSecond({ run: test });

[Ojan Vafai]

Ah. I didn't realize I needed to write test harness code. It seems just adding an html file is not sufficient for a page load test? The harness hung when I tried that.

P.S. This is not to detract from how awesome it is that this is running on the bots...just mentioning pitfalls I hit assuming others would likely hit them as well.

[Tony Gentilcore]

Right. Everything in PerformanceTests/ needs to use the PerfTestRunner API. I didn't mean to mislead with my comment about "just add an .html file." Telemetry hung because it was sitting there waiting to get results from the PerfTestRunner.

There are lots of examples of how to use it in that directory.

[PhistucK]

Works in stable but fails in dev/canary? that sounds like a browser issue...

☆PhistucK

[Eric Seidel]

Does this replace the "run-perf-tests" script? If so, can we remove
run-perf-tests? :)

It's nice to only have one tool to do the job. (Although I have to
admit that typing "run-perf-tests Parser/html-parser.html" is much
easier than the line you gave...

On Tue, Jun 18, 2013 at 10:33 AM, Tony Gentilcore <to...@chromium.org> wrote:

[Tony Gentilcore]

On Mon, Jun 24, 2013 at 4:49 PM, Eric Seidel <ese...@chromium.org> wrote:
> Does this replace the "run-perf-tests" script? If so, can we remove
> run-perf-tests? :)

Yes. I'll put together a patch.

>
> It's nice to only have one tool to do the job. (Although I have to
> admit that typing "run-perf-tests Parser/html-parser.html" is much
> easier than the line you gave...
>

We could make --browser=release the default so the command could be just:
tools/perf/run_measurement blink_perf
third_party/WebKit/PerformanceTests/Parser/html-parser.html

I think the extra path noise might be unavoidable. Kind of like how
layout tests are now webkit/tools/layout_tests/run_webkit_tests.sh
instead of run-webkit-tests.

[Dirk Pranke]

On Mon, Jun 24, 2013 at 7:21 PM, Tony Gentilcore <to...@chromium.org> wrote:

On Mon, Jun 24, 2013 at 4:49 PM, Eric Seidel <ese...@chromium.org> wrote:
> Does this replace the "run-perf-tests" script?  If so, can we remove
> run-perf-tests? :)

Yes. I'll put together a patch.

>
> It's nice to only have one tool to do the job.  (Although I have to
> admit that typing "run-perf-tests Parser/html-parser.html" is much
> easier than the line you gave...
>

We could make --browser=release the default so the command could be just:
tools/perf/run_measurement blink_perf
third_party/WebKit/PerformanceTests/Parser/html-parser.html

I think the extra path noise might be unavoidable. Kind of like how
layout tests are now webkit/tools/layout_tests/run_webkit_tests.sh
instead of run-webkit-tests.

run-webkit-tests works fine, I'm not sure what you mean here? 

Seems like you could change third_party/WebKit/Tools/Scripts/run-perf-tests to wrap run_measurement w/ the right args?

-- Dirk

[Eric Seidel]

Yeah, I think just replacing the guts of run-perf-tests with
run_measurement is a happier solution for me.

run_webkit_tests.sh is just a wrapper around run-webkit-tests. It
doesn't have any smarts itself. In the same way run-perf-tests could
just be made a wrapper around run_measurement or telemetry or
whatever.

[Tony Gentilcore]

Oh cool, that sounds perfect. I'll change it to a wrapper then.