Speed Benchmark Changes

[Jason Kersey]

Humans of Chromium,

Chrome's Speed Operations team has been working for the last few months to understand, categorize, and assign owners to the benchmarks currently run on our infrastructure. Starting today, we plan to make some substantial changes to what benchmarks we’re running, and how they are run.

First, we’re moving to having a small set of harnesses that Operations will be supporting to build and run new benchmarks on our infrastructure. You can find out more about the harnesses we’ll be supporting, how to use them, and other details.

Second, we’re going to be much more opinionated in general about what benchmarks we accept to run on our main infrastructure. We have posted our current set of guidelines and expect to start enforcing those starting today.

This will be a large change to how we measure Speed factors in Chromium, in an effort to get us more streamlined and able to handle the new benchmarks and frameworks we plan to add. Our end goal is to have a comprehensive set of metrics and benchmarks that cover Chromium’s performance, (including memory, power, responsiveness, etc).

kerz@ and sullivan@

on behalf of Speed Operations