Defining a benchmark with A/B diffing built-in

[Kalvin Lee]

Hi infra-dev,

(Rather than emailing the individuals given as benchmark owners, I thought it better to try this list, but please bounce me if I'm wrong.)

My goal is described in this issue. Roughly:

• We'd like to subclass Chromium's competitive benchmarks - Speedometer, MotionMark, and Jetstream.
• We will use these to measure the long-term performance disparity between MiraclePtr (explainer) being enabled vs. disabled in the renderer process.
• Each of the benchmarks will run twice - once vanilla and a second time with --enable-features=... fed in.
• Then some special plumbing kicks in to display the disparity in each story/page when the measurements are tallied.
• At the end, we'd like the score to be a simple delta. In prose, for example, "MotionMark total score regresses 0.1% with Renderer BRP enabled."

Can this be done? A naive read of the code suggests I would have to make some weird icky changes to RunStorySet() to manually propagate changes from these "special" new benchmark classes, just to accept a directive to run the browser twice for each story set.

Feedback much appreciated. Thanks -

Kalvin

[Takuto Ikuta]

https://groups.google.com/a/chromium.org/g/benchmarking-dev or https://chromium.googlesource.com/catapult.git/+/refs/heads/main/telemetry/README.md#contact-us-or-follow-along might be a better place to ask this kind of question.

I'm not sure if benchmark folks subscribe this mailing list.

--
You received this message because you are subscribed to the Google Groups "infra-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/infra-dev/ca7c1c3a-e7b8-4dba-90ad-7de261148211n%40chromium.org.

[Kalvin Lee]

Thanks for the bounce, Ikuta-san - unfortunately, those groups look inactive and overrun by spam, respectively. I'll try pinging listed owners directly.

To unsubscribe from this group and stop receiving emails from it, send an email to infra-dev+unsubscribe@chromium.org.