Remove (or Deprecate) run-perf-tests?

[Yuta Kitamura]

Hi Blink-dev,

I happened to see someone using Tools/Scripts/run-perf-tests to run Blink performance tests, but as far as I know that script is not really maintained, and Telemetry (src/tools/perf/run_benchmark) has become our standard way to run Blink performance tests.

Is anyone using run-perf-tests now? Can we remove (or deprecate, at least) it to avoid confusions?

Thanks,

Yuta

[TAMURA, Kent]

I think it's ok to remove now if we have PerformanceTests/README.md describing how to run perf tests.

--

TAMURA Kent
Software Engineer, Google

[Dirk Pranke]

This would make me so happy :) ...

-- Dirk

[Elliott Sprehn]

Does telemetry still produce the same dashboard with the diff between revisions and multiple runs?

I've often used that script because it makes a really nice dashboard output you can attach to bugs and also save for looking at later.

I'd prefer we didn't remove it unless the telemetry script had the same features and similar output.

[Timothy Dresser]

+Ned Nguyen 

[Yuta Kitamura]

2017/01/28 午前10:50 "Elliott Sprehn" <esp...@chromium.org>:

Does telemetry still produce the same dashboard with the diff between revisions and multiple runs?

Do you mean the table showing the improvement/regression percentages? I think Telemetry already has the same functionality: you just need to run run_benchmark twice or more, and Telemetry automatically updates results.html so you can grance the differences of the test runs.

[Ned]

+Annie Sullivan 

Elliott: do you mean the dashboard in https://chromeperf.appspot.com/report?sid=d7c59c198d81f30b7563cd869d8944a72a83d5772d000d3fb27fb6b0d998b52c ? That one is produced by telemetry's run_benchmark script.

[Yoichi Osato]

fmm,, Tools/Scripts/run-perf-tests doesn't work now like:

$ ./third_party/WebKit/Tools/Scripts/run-perf-tests --build-directory out_GN --debug  third_party/WebKit/PerformanceTests/DOM/div-editable.html 

Running 1 tests

Running DOM/div-editable.html (1 of 1)

ERROR: CONSOLE DEBUG: line 212: blink_perf: 15933.446044921875ms

FAILED

Finished: 102.931229 s

How do you run each test in Webkit/PerformanceTests in console?

It seems we need to update src/tools/perf/benchmarks/blink_perf.py to run

blink benchmark with src/tools/perf/run_benchmark but it is not convenient 

unlike just above command.

2017年1月30日(月) 23:33 'Ned' via blink-dev <blin...@chromium.org>:

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Elliott Sprehn]

On Mon, Jan 30, 2017 at 6:33 AM, Ned <nedn...@google.com> wrote:

+Annie Sullivan 

Elliott: do you mean the dashboard in https://chromeperf.appspot.com/report?sid=d7c59c198d81f30b7563cd869d8944a72a83d5772d000d3fb27fb6b0d998b52c ? That one is produced by telemetry's run_benchmark script.

No I mean the local compare dashboard. If you change the code and run the script, and change the code again, it keeps appending to a JSON file which lets you see all the runs and how they compare which is really important for local experimentation.

[Ned]

Elliot, the telemetry run_benchmark script does allow comparing between runs by also appending subsequent run results to the local HTML file. 

I run blink_perf.canvas twice locally & here is how it looks:

Documentation of this UI:
https://github.com/catapult-project/catapult/blob/master/docs/metrics-results-ui.md 

[Daniel Bratell]

The other presentation seems to be quite a bit more useful. It tells you for every test who is faster, and by much, including "can't tell" if the results seem noisy. It also has plots that make it trivial to see if the results are skewed because of a few stray values.

In the figure below it took me a while to realize that the two graphs are on completely different scales and can't be compared at all.

As I don't use run-perf-tests daily so I won't be affected much either way, but it doesn't seem to be a straight upgrade.

/Daniel

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Ned]

Daniel, the telemetry's UI does have the comparison feature when you select a referenced column. It uses Mann-Whitney hypothesis test to let you know whether the comparison is statistically significant. From https://github.com/catapult-project/catapult/blob/master/docs/metrics-results-ui.md#the-reference-column-selector   

• a green grinning face when the current Histogram is statistically significantly better than the reference column, or
  
• a red frowning face when the current Histogram is statistically significantly worse than the reference column, or
  
• a neutral face when the current Histogram is not statistically significantly different from the reference column.
  

 

On Tue, Mar 7, 2017 at 3:47 AM Daniel Bratell <bra...@opera.com> wrote:

The other presentation seems to be quite a bit more useful. It tells you for every test who is faster, and by much, including "can't tell" if the results seem noisy. It also has plots that make it trivial to see if the results are skewed because of a few stray values.

This can be fixed in how blink_perf benchmark are integrated with Telemetry. We currently don't set the histogram config for blink_perf data, hence Telemetry's results use the boxplot by default. With the histogram view as above, it should be clear when the results are skewed because of a few stray values.

 

In the figure below it took me a while to realize that the two graphs are on completely different scales and can't be compared at all.

Similarly to above, the UI doesn't use the same scale because we haven't configured the histogram's bin boundary for blink_perf test yet. 

 

As I don't use run-perf-tests daily so I won't be affected much either way, but it doesn't seem to be a straight upgrade.

/Daniel

On Mon, 06 Mar 2017 15:21:15 +0100, 'Ned' via blink-dev <blin...@chromium.org> wrote:

Elliot, the telemetry run_benchmark script does allow comparing between runs by also appending subsequent run results to the local HTML file. 

I run blink_perf.canvas twice locally & here is how it looks:

[Ben Hayden]

Hi everybody!
I’m writing the local test results UI produced by telemetry run_benchmark. I tried to carry over the useful features from the old results.html UI while allowing some new methods of analysis, but it’s possible that I missed or misunderstood some features.

Would there be interest in a brown bag to discuss the new UI and ideas and feature requests?

Either way, I’m always available to listen to feedback and follow up on bugs. Feel free to email/IM me directly or file a catapult bug. Known issues for this UI are in the Results2 label.

Daniel Bratell said

> In the figure below it took me a while to realize that the two graphs are on completely different scales and can't be compared at all.

Working on it!

[Elliott Sprehn]

I've been trying to use this today, the script is definitely much harder to use than the old run-perf-tests script.

- You can't just point it at a file, you need to edit some python configs if you have a new test.

- You specify a set of tests in some telemetry format (blink_perf.dom) and then need to use --story-filter to run a specific test, I guess that just takes experience to know each file is a "story".

- It spews errors by default about not having a browser, took me a while to realize I need to specify --browser=content-shell-release, the old script just chooses content_shell and release by default which is nice.

- It spews stuff to the console with every run:

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

[ RUN      ] long-sibling-list.html

2017-03-13 15:07:47.673 defaults[86426:525960] Couldn't find an application named "/Checkouts/chrome1/src/out/Release/Content Shell.app/Contents/MacOS/Content Shell"; defaults unchanged

That file does actually exist, it looks like the script is not escaping some file path correctly?

- There's no way to get a compact view of the dashboard by default. You have to refresh it every time to get updated perf results from the most recent run, but that expands every graph and a huge list of metrics.

- There's no way to get it to show the result number and a percent delta off the reference like with the old dashboard. I can get it to show percentages and the reference in that column, but not both at the same time.

- Selecting something from the reference dropdown makes it go into a compact view, refresh the page, and then the drop down is blank with no text inside it. If you refresh the page again you'll get the expanded view and the drop down isn't blank.

Not being able to get the compact view makes using it really hard in particular. Every time I refresh for a new result I have to click a ton of little X buttons to get a smaller view.

I don't want this:

I want this, but with the real value and the percent (or delta) next to each other like the old dashboard.

For reference I now need to run:

tools/perf/run_benchmark --browser=content-shell-release blink_perf.dom --story-filter=long-sibling-list.html

I used to be able to do:

Tools/Scripts/run-perf-tests PerformanceTests/DOM/long-sibling-list.html

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Ben Hayden]

Thanks, Elliott!

I'll let Ned respond to the telemetry stuff and I'll respond to the results.html stuff below.

On Sun, Mar 12, 2017 at 9:28 PM Elliott Sprehn <esp...@chromium.org> wrote:

I've been trying to use this today, the script is definitely much harder to use than the old run-perf-tests script.

- You can't just point it at a file, you need to edit some python configs if you have a new test.

- You specify a set of tests in some telemetry format (blink_perf.dom) and then need to use --story-filter to run a specific test, I guess that just takes experience to know each file is a "story".

- It spews errors by default about not having a browser, took me a while to realize I need to specify --browser=content-shell-release, the old script just chooses content_shell and release by default which is nice.

- It spews stuff to the console with every run:

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

[ RUN      ] long-sibling-list.html

2017-03-13 15:07:47.673 defaults[86426:525960] Couldn't find an application named "/Checkouts/chrome1/src/out/Release/Content Shell.app/Contents/MacOS/Content Shell"; defaults unchanged

That file does actually exist, it looks like the script is not escaping some file path correctly?

- There's no way to get a compact view of the dashboard by default. You have to refresh it every time to get updated perf results from the most recent run, but that expands every graph and a huge list of metrics.

- There's no way to get it to show the result number and a percent delta off the reference like with the old dashboard. I can get it to show percentages and the reference in that column, but not both at the same time.

I can change the statistic selector to a dropdown with checkboxes (bug) and display multiple statistics in the cells. Will that help?

- Selecting something from the reference dropdown makes it go into a compact view, refresh the page, and then the drop down is blank with no text inside it. If you refresh the page again you'll get the expanded view and the drop down isn't blank.

Sorry, I don't quite follow. Can you share the html file and maybe a screencast?

Not being able to get the compact view makes using it really hard in particular. Every time I refresh for a new result I have to click a ton of little X buttons to get a smaller view.

I don't want this:

I want this, but with the real value and the percent (or delta) next to each other like the old dashboard.

I have a bug to add a button to the name column so that you don't need to click a ton of little X buttons. Will that help?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Ned]

I think the main difference is the blink_perf benchmarks were designed to fit into the telemetry general Chrome benchmarking model whereas run-perf-tests script is tailor made for the cases of locally run ChromePerformance/ test. 

If you prefer to do less typing, we can totally make a wrapper script "tools/perf/run_blink_benchmark  <..test path..> " built on top of Telemetry that provides similar syntax to  Tools/Scripts/run-perf-tests script.

On Mon, Mar 13, 2017 at 12:28 AM Elliott Sprehn <esp...@chromium.org> wrote:

I've been trying to use this today, the script is definitely much harder to use than the old run-perf-tests script.

- You can't just point it at a file, you need to edit some python configs if you have a new test.

- You specify a set of tests in some telemetry format (blink_perf.dom) and then need to use --story-filter to run a specific test, I guess that just takes experience to know each file is a "story".

- It spews errors by default about not having a browser, took me a while to realize I need to specify --browser=content-shell-release, the old script just chooses content_shell and release by default which is nice.

We removed the behavior of having a default browser because there was case which people were running the benchmark on the Android trybot without argument and it was very unclear which browser was used: the system of browser of Linux host, the release build browser of Android or the system browser of Android. IMO, a little more typing to remove possible confusion about which browser the benchmark was using is an ok trade off.

 

- It spews stuff to the console with every run:

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

There are real world benchmarks that would just hang if flash is missing. So I can't figure out a simple way to inform people about that kind of error except warning about missing flash before it evens happen (this is because Telemetry's interaction with the browser is synchronous, and Telemetry doesn't know what content is happening on the browser unless you poke it). Another reason is even if the page doesn't hang, missing flash can also change the page's performance number & make the test less realistic.

The fact that there are 3 warning messages is probably a bug though.

 

[ RUN      ] long-sibling-list.html

2017-03-13 15:07:47.673 defaults[86426:525960] Couldn't find an application named "/Checkouts/chrome1/src/out/Release/Content Shell.app/Contents/MacOS/Content Shell"; defaults unchanged

That file does actually exist, it looks like the script is not escaping some file path correctly?

- There's no way to get a compact view of the dashboard by default. You have to refresh it every time to get updated perf results from the most recent run, but that expands every graph and a huge list of metrics.

- There's no way to get it to show the result number and a percent delta off the reference like with the old dashboard. I can get it to show percentages and the reference in that column, but not both at the same time.

- Selecting something from the reference dropdown makes it go into a compact view, refresh the page, and then the drop down is blank with no text inside it. If you refresh the page again you'll get the expanded view and the drop down isn't blank.

Not being able to get the compact view makes using it really hard in particular. Every time I refresh for a new result I have to click a ton of little X buttons to get a smaller view.

I don't want this:

I want this, but with the real value and the percent (or delta) next to each other like the old dashboard.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Elliott Sprehn]

On Tue, Mar 14, 2017 at 10:55 AM, Ned <nedn...@google.com> wrote:

I think the main difference is the blink_perf benchmarks were designed to fit into the telemetry general Chrome benchmarking model whereas run-perf-tests script is tailor made for the cases of locally run ChromePerformance/ test. 

If you prefer to do less typing, we can totally make a wrapper script "tools/perf/run_blink_benchmark  <..test path..> " built on top of Telemetry that provides similar syntax to  Tools/Scripts/run-perf-tests script.

It's not just less typing, but also how easy it is for someone new to get started running and using the benchmarks. I had to read pages of help output to figure out what to do here. That was quite intimidating. With the old script I just told a new person "run the script and point it at the file", and it had sensible defaults about what browser to use, how to launch it, and how the output was displayed.

 

On Mon, Mar 13, 2017 at 12:28 AM Elliott Sprehn <esp...@chromium.org> wrote:

I've been trying to use this today, the script is definitely much harder to use than the old run-perf-tests script.

- You can't just point it at a file, you need to edit some python configs if you have a new test.

- You specify a set of tests in some telemetry format (blink_perf.dom) and then need to use --story-filter to run a specific test, I guess that just takes experience to know each file is a "story".

- It spews errors by default about not having a browser, took me a while to realize I need to specify --browser=content-shell-release, the old script just chooses content_shell and release by default which is nice.

We removed the behavior of having a default browser because there was case which people were running the benchmark on the Android trybot without argument and it was very unclear which browser was used: the system of browser of Linux host, the release build browser of Android or the system browser of Android. IMO, a little more typing to remove possible confusion about which browser the benchmark was using is an ok trade off.

I don't know how you can run on the wrong browser locally when the browser actually opens in your face as the test runs. You're focused on Android here, but I'm not using Android at all, this is just running on my Macbook.

 

 

- It spews stuff to the console with every run:

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

There are real world benchmarks that would just hang if flash is missing. So I can't figure out a simple way to inform people about that kind of error except warning about missing flash before it evens happen (this is because Telemetry's interaction with the browser is synchronous, and Telemetry doesn't know what content is happening on the browser unless you poke it). Another reason is even if the page doesn't hang, missing flash can also change the page's performance number & make the test less realistic.

The fact that there are 3 warning messages is probably a bug though.

I wasn't talking about this warning, I was talking about the "file doesn't exist" one below it which is about a file that *does* exist.

 

 

[ RUN      ] long-sibling-list.html

2017-03-13 15:07:47.673 defaults[86426:525960] Couldn't find an application named "/Checkouts/chrome1/src/out/Release/Content Shell.app/Contents/MacOS/Content Shell"; defaults unchanged

^ this one. That's the binary that launches with the benchmark, so I know it exists, and the benchmark script runs it following that line, so the script is somehow both convinced it doesn't exist and able to launch it.

 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Elliott Sprehn]

On Tue, Mar 14, 2017 at 9:34 AM, Ben Hayden <benjh...@google.com> wrote:

Thanks, Elliott!

I'll let Ned respond to the telemetry stuff and I'll respond to the results.html stuff below.

On Sun, Mar 12, 2017 at 9:28 PM Elliott Sprehn <esp...@chromium.org> wrote:

I've been trying to use this today, the script is definitely much harder to use than the old run-perf-tests script.

- You can't just point it at a file, you need to edit some python configs if you have a new test.

- You specify a set of tests in some telemetry format (blink_perf.dom) and then need to use --story-filter to run a specific test, I guess that just takes experience to know each file is a "story".

- It spews errors by default about not having a browser, took me a while to realize I need to specify --browser=content-shell-release, the old script just chooses content_shell and release by default which is nice.

- It spews stuff to the console with every run:

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

[ RUN      ] long-sibling-list.html

2017-03-13 15:07:47.673 defaults[86426:525960] Couldn't find an application named "/Checkouts/chrome1/src/out/Release/Content Shell.app/Contents/MacOS/Content Shell"; defaults unchanged

That file does actually exist, it looks like the script is not escaping some file path correctly?

- There's no way to get a compact view of the dashboard by default. You have to refresh it every time to get updated perf results from the most recent run, but that expands every graph and a huge list of metrics.

- There's no way to get it to show the result number and a percent delta off the reference like with the old dashboard. I can get it to show percentages and the reference in that column, but not both at the same time.

I can change the statistic selector to a dropdown with checkboxes (bug) and display multiple statistics in the cells. Will that help?

I want sensible defaults. Today the dashboard opens with a huge information overload. For a new person it feels like drowning in data. I want to see something much more minimal where you need to opt into the more data.

 

- Selecting something from the reference dropdown makes it go into a compact view, refresh the page, and then the drop down is blank with no text inside it. If you refresh the page again you'll get the expanded view and the drop down isn't blank.

Sorry, I don't quite follow. Can you share the html file and maybe a screencast?

Sure I'll upload one.

 

Not being able to get the compact view makes using it really hard in particular. Every time I refresh for a new result I have to click a ton of little X buttons to get a smaller view.

I don't want this:

I want this, but with the real value and the percent (or delta) next to each other like the old dashboard.

I have a bug to add a button to the name column so that you don't need to click a ton of little X buttons. Will that help?

I want fewer buttons though, less information overload, and the default should be to open in the compact view. Every time I run the benchmark I have to refresh the page, which forces it back into information overload mode. I need it to give me the compact view with the right columns and not have to click on anything. I'm going through this workflow:

1) Build with change

2) Run benchmark

3) Refresh output file in browser

4) Repeat from step 1

and you do it dozens of times while you experiment with various code changes. You need the step 3 where you interact with this dashboard to require no clicks or interaction and instead show the right thing from the start.

 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Ned]

On Mon, Mar 13, 2017 at 8:05 PM Elliott Sprehn <esp...@chromium.org> wrote:

On Tue, Mar 14, 2017 at 10:55 AM, Ned <nedn...@google.com> wrote:

I think the main difference is the blink_perf benchmarks were designed to fit into the telemetry general Chrome benchmarking model whereas run-perf-tests script is tailor made for the cases of locally run ChromePerformance/ test. 

If you prefer to do less typing, we can totally make a wrapper script "tools/perf/run_blink_benchmark  <..test path..> " built on top of Telemetry that provides similar syntax to  Tools/Scripts/run-perf-tests script.

It's not just less typing, but also how easy it is for someone new to get started running and using the benchmarks. I had to read pages of help output to figure out what to do here. That was quite intimidating. With the old script I just told a new person "run the script and point it at the file", and it had sensible defaults about what browser to use, how to launch it, and how the output was displayed.

Agree. I think a more opinionated wrapper script for blink_perf  would help.

 

 

 

On Mon, Mar 13, 2017 at 12:28 AM Elliott Sprehn <esp...@chromium.org> wrote:

I've been trying to use this today, the script is definitely much harder to use than the old run-perf-tests script.

- You can't just point it at a file, you need to edit some python configs if you have a new test.

- You specify a set of tests in some telemetry format (blink_perf.dom) and then need to use --story-filter to run a specific test, I guess that just takes experience to know each file is a "story".

- It spews errors by default about not having a browser, took me a while to realize I need to specify --browser=content-shell-release, the old script just chooses content_shell and release by default which is nice.

We removed the behavior of having a default browser because there was case which people were running the benchmark on the Android trybot without argument and it was very unclear which browser was used: the system of browser of Linux host, the release build browser of Android or the system browser of Android. IMO, a little more typing to remove possible confusion about which browser the benchmark was using is an ok trade off.

I don't know how you can run on the wrong browser locally when the browser actually opens in your face as the test runs. You're focused on Android here, but I'm not using Android at all, this is just running on my Macbook.

What about the case which you have both a release build & a debug build on your checkout?

 

 

 

- It spews stuff to the console with every run:

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

WARNING:root:Chrome build location for mac_x86_64 not found. Browser will be run without Flash.

There are real world benchmarks that would just hang if flash is missing. So I can't figure out a simple way to inform people about that kind of error except warning about missing flash before it evens happen (this is because Telemetry's interaction with the browser is synchronous, and Telemetry doesn't know what content is happening on the browser unless you poke it). Another reason is even if the page doesn't hang, missing flash can also change the page's performance number & make the test less realistic.

The fact that there are 3 warning messages is probably a bug though.

I wasn't talking about this warning, I was talking about the "file doesn't exist" one below it which is about a file that *does* exist.

 

 

[ RUN      ] long-sibling-list.html

2017-03-13 15:07:47.673 defaults[86426:525960] Couldn't find an application named "/Checkouts/chrome1/src/out/Release/Content Shell.app/Contents/MacOS/Content Shell"; defaults unchanged

^ this one. That's the binary that launches with the benchmark, so I know it exists, and the benchmark script runs it following that line, so the script is somehow both convinced it doesn't exist and able to launch it.

This sounds like a bug. I will try to reproduce it.

 

 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Elliott Sprehn]

On 14 Mar. 2017 11:23 am, "Ned" <nedn...@google.com> wrote:

On Mon, Mar 13, 2017 at 8:05 PM Elliott Sprehn <esp...@chromium.org> wrote:

On Tue, Mar 14, 2017 at 10:55 AM, Ned <nedn...@google.com> wrote:

I think the main difference is the blink_perf benchmarks were designed to fit into the telemetry general Chrome benchmarking model whereas run-perf-tests script is tailor made for the cases of locally run ChromePerformance/ test. 

If you prefer to do less typing, we can totally make a wrapper script "tools/perf/run_blink_benchmark  <..test path..> " built on top of Telemetry that provides similar syntax to  Tools/Scripts/run-perf-tests script.

It's not just less typing, but also how easy it is for someone new to get started running and using the benchmarks. I had to read pages of help output to figure out what to do here. That was quite intimidating. With the old script I just told a new person "run the script and point it at the file", and it had sensible defaults about what browser to use, how to launch it, and how the output was displayed.

Agree. I think a more opinionated wrapper script for blink_perf  would help.

 

 

 

On Mon, Mar 13, 2017 at 12:28 AM Elliott Sprehn <esp...@chromium.org> wrote:

I've been trying to use this today, the script is definitely much harder to use than the old run-perf-tests script.

- You can't just point it at a file, you need to edit some python configs if you have a new test.

- You specify a set of tests in some telemetry format (blink_perf.dom) and then need to use --story-filter to run a specific test, I guess that just takes experience to know each file is a "story".

- It spews errors by default about not having a browser, took me a while to realize I need to specify --browser=content-shell-release, the old script just chooses content_shell and release by default which is nice.

We removed the behavior of having a default browser because there was case which people were running the benchmark on the Android trybot without argument and it was very unclear which browser was used: the system of browser of Linux host, the release build browser of Android or the system browser of Android. IMO, a little more typing to remove possible confusion about which browser the benchmark was using is an ok trade off.

I don't know how you can run on the wrong browser locally when the browser actually opens in your face as the test runs. You're focused on Android here, but I'm not using Android at all, this is just running on my Macbook.

What about the case which you have both a release build & a debug build on your checkout?

Perf testing with a debug build rarely makes sense, the debug code makes massively slower. The only time you'd ever do that is to debug a crash, and at that point the dashboard output doesn't make sense anymore you just need to run the test so you might as well run the file directly in content shell.

...

[Ned]

@Elliot: I was able to reproduce the bug around the confusing log spew you were seeing. It is the code that meant to suppress crash dialog on Mac, but not sure why that doesn't work for content-shell. Filed https://github.com/catapult-project/catapult/issues/3381 

[Ben Hayden]

Thanks for the feedback, Elliot!

It does seem like there is too much information shown by default for your use case, so I committed a change to always start at the compact view.

In addition to saner defaults, refreshing should keep exactly the same view, so I’m working on a change to persist all of the configuration in the URL hash instead of only some of it to localStorage.