Re: Design doc: enable tracing support for blink perf tests

[nedn...@chromium.org]

+benchmarking-dev@ for real

On Monday, March 6, 2017 at 5:26:33 PM UTC-5, Ned Nguyen wrote:

We would like to enable support for timing breakdown in blink_perf tests using tracing data. This would allow people to get more accurate performance data on specific area of code in Chrome that they want to optimize for.

Here's the design doc: https://docs.google.com/document/d/1feawFQpadFCcXEsVqz9z7ucSBIQBgFc23z873g69rKc/edit#

Comments are appreciated!

Thanks,

Ned

[Kentaro Hara]

The proposal looks good to me.

I want to use this to measure the performance impact of a couple of our optimizations for an iframe creation (V8 context snapshotting, rewriting extension bindings in gin, splitting context script injections into multiple tasks etc).

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.
To post to this group, send email to platform-architecture-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/41637e35-235e-45ce-a78c-f4f11fb63e7e%40chromium.org.

--

Kentaro Hara, Tokyo, Japan

[nedn...@chromium.org]

+primiano, fadi, caseq who pushed back against computing tracing metrics on the Webkit PerformanceTest runner.js itself. 

[nedn...@chromium.org]

Tracing folks are pushing back supporting ways to collect & analyze trace in Webkit PerformanceTest runner.js itself because they worry that it creates yet another a consumer of tracing. They proposed only support these tracing metrics when you run Webkit PerformanceTest through Telemetry.

My thought on this is that it makes the set of data produced by Webkit PerformanceTest less consistent between running them with & without Telemetry, and both workflows have a substantial user base. On another hand, because Webkit PerformanceTest are micro benchmarking harness for web platform APIs, I can't foresee a future where we add any measurement mechanism more sophisticated than just "give me the total sum of cpu time of the trace events of names X, Y, Z", so the complexity of trace analyzing for this harness be mostly fixed.

+haraken, esprehn: what are your thoughts on this?

[Primiano Tucci]

To be honest, one of the issues I had with the discussion in the past meeting and the doc, is that it is not fully clear to me what are the use cases we want to tackle and what we want to improve. Specifically, the following questions still are not fully clear for me:

Q1: Is this only for manual investigations or for running things on the chromeperf waterfall?

Q2: Is the intended audience chrome developers, other browers (i.e. not just chrome) vendors, of web developers

Q3: is this about getting more details from JS, more details from Chrome, or mixing the two?

> because they worry that it creates yet another a consumer of tracing

I don't think the real problem here is "another consumer of tracing". I think the concern here was creating yet another framework which starts/stop tracing and builds parsers / metrics. We have already quite a lot of independent building blocks (e..g tracing can be started/stopped via the devtools protocol. the trace can be parsed via the TBMv2 metrics). Can we reuse them instead of creating a brand new pipeline that redoes the same?

The suggestion points we had in the meeting were:

- if the point is enriching web test results with trace events, can we have something like run_benchmark that wraps the execution of blink perf tests and attaches a trace in output?

- if the point is being able to inject TRACE_EVENTs from within the JS of the tests (still not sure I understood this use case correctly), can we use the window.performance web API instead, which are already interoperable cross-browser.

- if the point is being able to consume the trace from JS, why we cannot reuse (a subset of) the existing TBMv2 metric system, given that it's already written in JS?

[nedn...@chromium.org]

I leave it to haraken & esprehn to talk about general use cases of Webkit PerformanceTest but I have some answer for infrastructure sharing at below.

On Monday, March 20, 2017 at 4:27:01 PM UTC-4, Primiano Tucci wrote:

To be honest, one of the issues I had with the discussion in the past meeting and the doc, is that it is not fully clear to me what are the use cases we want to tackle and what we want to improve. Specifically, the following questions still are not fully clear for me:

Q1: Is this only for manual investigations or for running things on the chromeperf waterfall?

Q2: Is the intended audience chrome developers, other browers (i.e. not just chrome) vendors, of web developers

Q3: is this about getting more details from JS, more details from Chrome, or mixing the two?

> because they worry that it creates yet another a consumer of tracing

I don't think the real problem here is "another consumer of tracing". I think the concern here was creating yet another framework which starts/stop tracing and builds parsers / metrics. We have already quite a lot of independent building blocks (e..g tracing can be started/stopped via the devtools protocol. the trace can be parsed via the TBMv2 metrics). Can we reuse them instead of creating a brand new pipeline that redoes the same?

The suggestion points we had in the meeting were:

- if the point is enriching web test results with trace events, can we have something like run_benchmark that wraps the execution of blink perf tests and attaches a trace in output? 

- if the point is being able to inject TRACE_EVENTs from within the JS of the tests (still not sure I understood this use case correctly), can we use the window.performance web API instead, which are already interoperable cross-browser.

This is not the point.

- if the point is being able to consume the trace from JS, why we cannot reuse (a subset of) the existing TBMv2 metric system, given that it's already written in JS?

This is only necessary if the parsing & computing metrics is complicated. IMO, it's more work to reuse the  existing TBMv2 metric system in this case because you need to be able to vulcanize tracing code base & link it in the JS core of PerformanceTest properly. 

The code to compute the simple trace metrics in PerformanceTest's runner.js should look something like:

trace_events = json.load(raw_trace_data);

for (var event of trace_events) {

   for (var name of test_config.interested_event_names) {

       if (event.name === name) 

          total_cpu_duration[name] += event.tdur;

   }

}

IMO, not using existng TBMv2 metric system also discourage people from doing more complicated metrics which is a feature, not a bug. This harness should focus on simple micro benchmarking cases only.

[Ned]

Some other folks are confused about what's the problem here, so I try my best to summarize the situation here. 

Proposal: we want to support output tracing breakdown metrics in blink_perf harness because they give us lower noise & more detailed data. 

There are two ways of doing that & each will have different implications.

Way 1 (tracing through Telemetry):

Only support tracing metrics when people run blink_perf through telemetry. The picture of which metrics output in what setting will look like this:

	Run on Chrome + Telemetry	Run on Chrome directly	Run on other browsers
Producing total test duration	Yes	Yes	Yes
Producing tracing breakdown metrics	Yes	No	No

Pros: be able to reuse Telemetry infra for collecting & analyzing tracing. Least amount of extra infra require.

Cons: require running Telemetry to get the tracing metrics & Telemetry doesn't play with other local debugging tool nicely (e.g: gdb/sampling profiler/...).

 Way 2 (tracing through Javascript):

Support collecting & analyzing traces in PerformanceTest/runner.js. The picture of which metrics output in what setting will look like this:

	Run on Chrome + Telemetry	Run on Chrome directly	Run on other browsers
Producing total test duration	Yes	Yes	Yes
Producing tracing breakdown metrics	Yes	Yes	No

Pros: Chrome developers who are familiar with BlinkPerf but not Telemetry will be able to get extra breakdown metrics. Be able to get these metrics just by opening the test file (written in HTML & JS) in Chrome directly is nice & make it much easier to iteratively tweak the test.

Cons: requires an entirely different infra for collecting & analyzing tracing in JS land. 

P/s: I am sorry for not communicating this well as first, so please let me know if you still have further questions.

On Monday, March 20, 2017 at 4:27:01 PM UTC-4, Primiano Tucci wrote:

[Kentaro Hara]

Thanks for the great write-up! :)

In terms of the requirement, I think the Way 1 (tracing through Telemetry) would be enough. The Way 2 would be nice to have but I think it's enough if we can measure metrics through Telemetry.

--

You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.
To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/e885a1a7-1ea1-4a8e-b353-7db2b0a68799%40chromium.org.

[Nat Duca]

How sure are we that blink members are going to be ok with having some of their perf tests not runnable in content shell anymore? Thats the implication here... as I understood, a lot of people like to open a single html file and iterate on it just reloading it. That currently works with blink perf. With plan A, we're going to be in a very awkward state of sometimes you can run the html file standalone, and sometimes you need to run it inside telemetry. Tbh that just sounds weird. But if blink-dev peeps are fine with that, then cool I guess?

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsubsc...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/e885a1a7-1ea1-4a8e-b353-7db2b0a68799%40chromium.org.

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "Chrome benchmarking" group.
To unsubscribe from this group and stop receiving emails from it, send an email to benchmarking-dev+unsubscribe@chromium.org.
To post to this group, send email to benchmar...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/benchmarking-dev/CABg10jzCTCb9%3DShWnhTkvY2_XY9eyzrcUo4KMJv-sLeb0-4nPw%40mail.gmail.com.

[Kentaro Hara]

Oh, with Plan A I was assuming:

- the benchmark just runs without tracing on all browsers as a standalone html.

- the benchmark runs with tracing if we use Chrome + telemetry.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAAMsTOvhcCD4wKUwg%2B10Cd4R9h%2B88drS5dG6BLmb%2B5ktKO7pqA%40mail.gmail.com.

[Ned Nguyen]

Kentaro's assumption about the plan in correct. With plan A, the tests will still run without tracing on all browsers as a standalone html. They will just not output tracing based metrics.

Since A & B are not in conflict, I think we can view this as A is phase 1 (support tracing breakdown metrics on Telemetry only), and B as phase 2 (support tracing breakdown metrics on Telemetry & Chrome). 

In both phases, test author will still be able to write tests & get the "total test duration metric" on Chrome & other browsers directly. The hope is that after phase 1 & more people start to write tests that use this features, we will have more input to assess the importance of extending the support to get tracing based metrics in Chrome content-shell directly.

I updated the implementation plan in the doc to reflect the 2 phases approach, please takes a look and let me know if you have any concerns with this plan

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.
To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/e885a1a7-1ea1-4a8e-b353-7db2b0a68799%40chromium.org.

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "Chrome benchmarking" group.
To unsubscribe from this group and stop receiving emails from it, send an email to benchmarking-dev+unsubscribe@chromium.org.
To post to this group, send email to benchmar...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/benchmarking-dev/CABg10jzCTCb9%3DShWnhTkvY2_XY9eyzrcUo4KMJv-sLeb0-4nPw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.
To post to this group, send email to platform-architecture-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAAMsTOvhcCD4wKUwg%2B10Cd4R9h%2B88drS5dG6BLmb%2B5ktKO7pqA%40mail.gmail.com.