Instructions for measuring the FINAL performance

[Kentaro Hara]

Hi

It looks like that we've finished addressing all substantial regressions. Now we're ready to collect full performance numbers :)

Instructions:

- Use Chromium revision r353009.

- Use chrome official builds (i.e., GYP_DEFINES="buildtype=Official branding=Chrome").

- To restart the browser every time, you need to apply this CL (https://codereview.chromium.org/1393283002).

- Assignments:

Linux: haraken, keishi

Mac: keishi, peria

Win: peria, yutak

Nexus7: yutak

- Benchmarks:

BENCHMARKS = (

    'blink_perf.bindings',

    'blink_perf.css',

    'blink_perf.dom',

    'blink_perf.events',

    'blink_perf.layout',

    'blink_perf.mutation',

    'blink_perf.parser',

    'blink_perf.shadow_dom',

    'blink_perf.svg',

    'blink_style.key_mobile_sites',

    'blink_style.polymer',

    'blink_style.top_25',

    'dromaeo.domcoreattr',

    'dromaeo.domcoremodify',

    'dromaeo.domcorequery',

    'dromaeo.domcoretraverse',

    'memory.long_running_idle_gmail',

    'memory.long_running_idle_gmail_background',

    'memory.memory_health_plan',

    'memory.top_7_stress',

    'memory.idle_multi_tab',

    'memory.mobile_memory',

    'page_cycler.intl_ar_fa_he',

    'page_cycler.intl_es_fr_pt-BR',

    'page_cycler.intl_hi_ru',

    'page_cycler.intl_ja_zh',

    'page_cycler.intl_ko_th_vi',

    'page_cycler.top_10_mobile',

    'page_cycler.typical_25',

    'scheduler.tough_scheduling_cases',

    'smoothness.bidirectionally_scrolling_tough_ad_cases',

    'smoothness.key_mobile_sites_smooth',

    'smoothness.key_silk_cases',

    'smoothness.pathological_mobile_sites',

    'smoothness.polymer',

    'smoothness.scrolling_tough_ad_cases',

    'smoothness.simple_mobile_sites',

    'smoothness.sync_scroll.key_mobile_sites_smooth',

    'smoothness.tough_ad_cases',

    'smoothness.tough_animated_image_cases',

    'smoothness.tough_animation_cases',

    'smoothness.tough_canvas_cases',

    'smoothness.tough_filters_cases',

    'smoothness.tough_pinch_zoom_cases',

    'smoothness.tough_scrolling_cases',

    'smoothness.tough_scrolling_while_zoomed_in_cases',

    'smoothness.tough_webgl_ad_cases',

    'smoothness.top_25_smooth',

    'speedometer',

    'thread_times.key_mobile_sites_smooth',

    'thread_times.key_noop_cases',

    'thread_times.key_silk_cases',

    'thread_times.polymer',

    'thread_times.simple_mobile_sites',

    'thread_times.tough_scrolling_cases',

)

- Run each benchmark with --page-repeat=1. Run each benchmark at least 10 times.

--

Kentaro Hara, Tokyo, Japan

[Kentaro Hara]

This is the result taken on my Linux:

https://drive.google.com/open?id=0B1obCOyvTnPKeFk4bTcybmJ6Nm8

At a first glance, there is no substantial regression.

The only regression we have to justify is the peak memory increase in memory.* benchmarks. I'll do an experiment to confirm that the regression is gone if we increase the frequency of conservative GCs enough. Then we can prove that the peak memory increase is just caused by GC timings.

[Keishi Hattori]

Speedometer and blink_style look good. query-selector could use some improvement.

Mac results:

https://www.xpeg.org/keishi-mac-1009.microbench.html

https://www.xpeg.org/keishi-mac-1009.memory.html

https://www.xpeg.org/keishi-mac-1009.smoothness.html

https://www.xpeg.org/keishi-mac-1009.page_cycler.html

https://www.xpeg.org/keishi-mac-1009.thread_times.html

--

- Keishi

[Hitoshi Yoshida]

2015-10-13 11:17 GMT+09:00 Kentaro Hara <har...@chromium.org>:

This is the result taken on my Linux:

https://drive.google.com/open?id=0B1obCOyvTnPKeFk4bTcybmJ6Nm8

At a first glance, there is no substantial regression.

The only regression we have to justify is the peak memory increase in memory.* benchmarks. I'll do an experiment to confirm that the regression is gone if we increase the frequency of conservative GCs enough. Then we can prove that the peak memory increase is just caused by GC timings.

Could you share the HTML file with http://googledrive.com/host/<docID> style or its redirection?

Otherwise we need to download the file to see it as a HTML page. :)

https://support.google.com/drive/answer/2881970?hl=ja

Google Japan Inc.

Software Engineer

Hitoshi Yoshida (吉田 仁)

mailto:pe...@google.com

[Keishi Hattori]

haraken's linux results look slightly better so I think we should use that.

My linux results:

https://www.xpeg.org/keishi-linux-1009.html

--

- Keishi

[Kentaro Hara]

On Tue, Oct 13, 2015 at 12:43 PM, Hitoshi Yoshida <pe...@chromium.org> wrote:

2015-10-13 11:17 GMT+09:00 Kentaro Hara <har...@chromium.org>:

This is the result taken on my Linux:

https://drive.google.com/open?id=0B1obCOyvTnPKeFk4bTcybmJ6Nm8

At a first glance, there is no substantial regression.

The only regression we have to justify is the peak memory increase in memory.* benchmarks. I'll do an experiment to confirm that the regression is gone if we increase the frequency of conservative GCs enough. Then we can prove that the peak memory increase is just caused by GC timings.

Could you share the HTML file with http://googledrive.com/host/<docID> style or its redirection?

Otherwise we need to download the file to see it as a HTML page. :)

https://support.google.com/drive/answer/2881970?hl=ja

Thanks, done.

haraken's Linux result is https://cf550646961e6bd4a4aa17b4d0ba5b63312f10f7.googledrive.com/host/0B1obCOyvTnPKeFk4bTcybmJ6Nm8

[Kentaro Hara]

As far as I see haraken's Linux, keishi's Linux and keishi's Mac, the only benchmarks that are consistently regressing are as follows:

- css.PseudoClassSelectors

- parser.iframe-append-remove (because it is creating too many iframes)

- shadow_dom.DistributionWithMultipleShadowRoots (multiple shadow roots are deprecated)

- dromaeo.domcorequery:dom_query_getElementsByName

- dromaeo.domcoremodify:dom_modify_createTextNode

- blink_perf.parser:xml-parser

- blink_style.update_style (because it is creating too many StyleResolvers)

Either way, I don't think these regressions are something that blocks oilpan's shipping.

[Kentaro Hara]

On Tue, Oct 13, 2015 at 11:52 AM, Keishi Hattori <kei...@google.com> wrote:

Speedometer and blink_style look good. query-selector could use some improvement.

Mac results:

https://www.xpeg.org/keishi-mac-1009.microbench.html

https://www.xpeg.org/keishi-mac-1009.memory.html

^^^ Can we get a result for memory.top_7_stress?

 

https://www.xpeg.org/keishi-mac-1009.smoothness.html

^^^ I cannot open this one.

[Keishi Hattori]

memory.top_7_stress data is missing. I'll see why.

File was too big. Reduced to 10 runs. (still 100MB)

https://www.xpeg.org/keishi-mac-1009.smoothness.html

--

- Keishi

[Hitoshi Yoshida]

It seems that top-7-stress test is disabled on Yosemite, and I also couldn't get its result.

Is your MacOSX 10.10.5 (Yosemite)?

[Keishi Hattori]

Yes 10.10.5

--

- Keishi

[Hitoshi Yoshida]

This is my result, but the result for Windows seems unreliable.

Mac: https://2fd942869b2638332227cf64f025c4116dd6718a.googledrive.com/host/0B4QUVw-AB8wPbG9UZVMwZjRLbjg

Overall result is similar to keishi@'s result.

Some microbench scored worse than keishi@'s, and others are good.

Windows: https://8c2824b6244981ec465811459b008c2fc837e064.googledrive.com/host/0B4QUVw-AB8wPbElKaWFpLWRCU3c

There is an output for this, but I feel this result unreliable.

I got this result on Sat night, it meant to be done in 24h, and some metrics have very few values.

I started tests again from then, and it is still running over Sun. and Mon.

Let me check how many runs it have done, and update the result.

[Kentaro Hara]

On Tue, Oct 13, 2015 at 1:53 PM, Hitoshi Yoshida <pe...@chromium.org> wrote:

This is my result, but the result for Windows seems unreliable.

Mac: https://2fd942869b2638332227cf64f025c4116dd6718a.googledrive.com/host/0B4QUVw-AB8wPbG9UZVMwZjRLbjg

Overall result is similar to keishi@'s result.

Some microbench scored worse than keishi@'s, and others are good.

It seems a couple of benchmarks are missing from the result (e.g., blink_style, speedometer). However, overall the result looks similar to keishi's Mac, so we can just use keishi's Mac.

 

Windows: https://8c2824b6244981ec465811459b008c2fc837e064.googledrive.com/host/0B4QUVw-AB8wPbElKaWFpLWRCU3c

There is an output for this, but I feel this result unreliable.

I got this result on Sat night, it meant to be done in 24h, and some metrics have very few values.

I started tests again from then, and it is still running over Sun. and Mon.

Let me check how many runs it have done, and update the result.

Thanks! The windows result looks much better than Mac and Linux. This matches our experience -- Mac and Linux are more sensitive than Windows.

[Keishi Hattori]

Just ran the disabled memory.top_7_stress on mac. Looks like plus.google.com and Blogger timeout.

--

- Keishi

[Kentaro Hara]

On Tue, Oct 13, 2015 at 2:22 PM, Keishi Hattori <kei...@google.com> wrote:

Just ran the disabled memory.top_7_stress on mac. Looks like plus.google.com and Blogger timeout.

Thanks!

In conclusion:

- Regarding Linux, we got enough results. We'll use haraken's result.

- Regarding Mac, we got enough results. We'll use keishi's result.

- Regarding Win, we're waiting for peria's result and yutak's result.

- Regarding Nexus7, we're waiting for yutak's result. 

[Yuta Kitamura]

Please be patient for mobile results and my Windows results. Nexus 7 has passed 90%, Windows has completed 17/20 loops, so I hope the complete results for these platforms will be up by the end of tomorrow. I will then compile others' results so we can glance the results from all platforms in one page.

(My Nexus 4 cluster doesn't have enough capacity at this time, so expect no results from them.)

[Hitoshi Yoshida]

This is my new result on Windows. It contains the result of 13 runs (my machine continues running to go through 7 more loops)

https://cf9b9d1a3bd5dbbc41bc4e0f863bb2b9d2740e47.googledrive.com/host/0B4QUVw-AB8wPTHlFYkwyalBLcEU

This page is very large, so I recommend you to open on a powerful machine. (I'll split it into some pages)

Looking the result, I noticed following regressions

- [memory] long running idle GMail regresses 7%

- [memory] in top-7-stress, Blogger regresses largely (~14%)

- [perf] dromaeo regresses in dom_query_getElementsByName and dom_modify_createTextNode ~10%

[Keishi Hattori]

FYI, dumped blink_perf results to spread sheet

https://docs.google.com/spreadsheets/d/1KVoORSHCdEOcluayQZ3yJ7LmmCGMrGdec-VX0rn4zFo/edit?usp=sharing

--

- Keishi

[Yuta Kitamura]

Here's the results from Nexus 7 and Windows (combined):

https://www.googledrive.com/host/0B6XhrzUCq5lhUjloNWR3b0xxRXM/2015/perf-353009/mobile+win/

[Kentaro Hara]

Thanks yuta-san for the result!

I scanned the result of Nexus 7 and Windows. I see no issues except:

- Peak memory increase in memory benchmarks (which we're going to justify).

- Results for blink_perf.bindings on Nexus7 is missing.

keishi-san: yutak's Windows results for blink_perf looks very promising. Would you create the graph (https://docs.google.com/spreadsheets/d/1KVoORSHCdEOcluayQZ3yJ7LmmCGMrGdec-VX0rn4zFo/edit?usp=sharing) for yutak's Windows result as well?

I'll write a brief summary document so that we can publish the result to blink-dev@.

[Kentaro Hara]

yuta-san: Would you apply this CL (https://codereview.chromium.org/1397763003/) and measure memory.* on Nexus 7? I want to make sure that the peak memory regression is mostly gone if we increase GC frequency enough.

[Kentaro Hara]

I started writing the final performance report. (I'm not planning to write a long document.)

https://docs.google.com/document/d/1cm0O3XOaHQWxy-mdlht5eNe3JoLzzXtn6o4tYjbIaVw/edit#

I added edit permissions to you; feel free to edit.

[Kentaro Hara]

I mostly finished analyzing all the data. The result looks pretty good!

In summary, I might want to get the following data.

- blink_perf.bindings and blink_style on Nexus7 (yutak@)

- memory.* with https://codereview.chromium.org/1397763003/ on Nexus7 (yutak@)

[Kentaro Hara]

On Thu, Oct 15, 2015 at 1:10 PM, Kentaro Hara <har...@chromium.org> wrote:

I mostly finished analyzing all the data. The result looks pretty good!

In summary, I might want to get the following data.

 

- blink_perf.bindings and blink_style on Nexus7 (yutak@)

blink_perf.parser is missing as well.

[Yuta Kitamura]

On Thu, Oct 15, 2015 at 1:17 PM, Kentaro Hara <har...@chromium.org> wrote:

On Thu, Oct 15, 2015 at 1:10 PM, Kentaro Hara <har...@chromium.org> wrote:

I mostly finished analyzing all the data. The result looks pretty good!

In summary, I might want to get the following data.

 

- blink_perf.bindings and blink_style on Nexus7 (yutak@)

I removed blink_perf.bindings from the tests list when it was disabled, but I forgot to put it back once it was reenabled. I'm running it now.

blink_style was run successfully, but I forgot to collect the raw data when I compiled the summary.

I will get back to you when these issues are corrected.

 

blink_perf.parser is missing as well.

blink_perf.parser is disabled on Win and Android, so WAI.

[Kentaro Hara]

I finished writing the document. Feel free to edit/comment.

https://docs.google.com/document/d/1cm0O3XOaHQWxy-mdlht5eNe3JoLzzXtn6o4tYjbIaVw/edit#

[Yuta Kitamura]

On Thu, Oct 15, 2015 at 1:50 PM, Yuta Kitamura <yu...@chromium.org> wrote:

On Thu, Oct 15, 2015 at 1:17 PM, Kentaro Hara <har...@chromium.org> wrote:

On Thu, Oct 15, 2015 at 1:10 PM, Kentaro Hara <har...@chromium.org> wrote:

I mostly finished analyzing all the data. The result looks pretty good!

In summary, I might want to get the following data.

 

- blink_perf.bindings and blink_style on Nexus7 (yutak@)

I removed blink_perf.bindings from the tests list when it was disabled, but I forgot to put it back once it was reenabled. I'm running it now.

blink_style was run successfully, but I forgot to collect the raw data when I compiled the summary.

I will get back to you when these issues are corrected.

These are done. https://www.googledrive.com/host/0B6XhrzUCq5lhUjloNWR3b0xxRXM/2015/perf-353009/mobile+win/

[Kentaro Hara]

I'm going to send out the document to blink-dev@ tonight. Leave your comments before then.

[Sigbjorn Finne]

Den 10/15/2015 10:11, Kentaro Hara skreiv:
> I'm going to send out the document to blink-dev@ tonight. Leave your
> comments before then.
>

Thanks everyone for gathering the performance data and going through
them carefully; great work (and the final time this had to be done
across all platforms? :-) )

--sigbjorn

[Kentaro Hara]

Yes, we're done!

We'll start shipping oilpan anytime soon. A bunch of stabilization work & code clean-up will be incoming :)

I already started a thread to discuss a concrete step with Chrome release managers. I'll keep you updated.