Can we have tests that keep track of running speed?

400 views
Skip to first unread message

Ramin Halavati

unread,
Sep 25, 2024, 9:54:14 AM9/25/24
to Chromium-dev
Hi,

I want to add tests to monitor changes in the processing speed of an external library, for Windows, Linux, and Mac. Is it possible in the chromium commit queue or as an FYI test to run tests on rather unchanging hardware configs?

We are running similar tests for ChromeOS (as TAST tests) on different hardwares and have speed expectations based on each specific hardware, but I am not sure if this is possible for other platforms.

Best,
Ramin

Greg Thompson

unread,
Sep 26, 2024, 10:25:54 AM9/26/24
to rhal...@google.com, Chromium-dev
I think it should be possible to make a perf test for this. Pinpoint runs perf tests on actual hardware and makes pretty pictures/alerts.

https://chromium.googlesource.com/chromium/src/tools/+/HEAD/perf/README.md

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAORodQjVxozA98s8-Z1N_yLDUiwK1aDGMj%3DkyW4D5O3xoM5JfQ%40mail.gmail.com.

Ramin Halavati

unread,
Sep 27, 2024, 1:19:13 AM9/27/24
to Greg Thompson, Chromium-dev
Thank you Greg.

Erik Chen

unread,
Sep 27, 2024, 1:30:13 AM9/27/24
to rhal...@google.com, Dana Fried, Eshwar Stalin, Robert Liao, ochoa...@google.com, Greg Thompson, Chromium-dev
+Dana Fried +Eshwar Stalin +Robert Liao +ochoa...@google.com

The existing perf framework is built around the concept of running tests that measure web-page performance. We want to have something similar for native code, there's a project under way to make this easy to do via kombucha.

Erik Chen

unread,
Sep 27, 2024, 1:30:41 AM9/27/24
to rhal...@google.com, Dana Fried, Eshwar Stalin, Robert Liao, ochoa...@google.com, Greg Thompson, Chromium-dev
Incidentally, what is your use case? IMO it's much better to track a CUJ than a microbenchmark for a library.

Ramin Halavati

unread,
Sep 27, 2024, 1:35:58 AM9/27/24
to Erik Chen, Dana Fried, Eshwar Stalin, Robert Liao, ochoa...@google.com, Greg Thompson, Chromium-dev
Thank you. I want to monitor the performance speed of the OCR module.
We have browser tests such as OpticalCharacterRecognizerTest::PerformOCR_Simple to monitor quality of the service, but they cannot measure speed reliably.
(The test  uses CIPD to have access to the library binary that is downloaded on clients via component updater.)

danakj

unread,
Sep 27, 2024, 9:24:56 AM9/27/24
to rhal...@google.com, Erik Chen, Dana Fried, Eshwar Stalin, Robert Liao, ochoa...@google.com, Greg Thompson, Chromium-dev

Ramin Halavati

unread,
Sep 30, 2024, 12:28:15 AM9/30/24
to danakj, Erik Chen, Dana Fried, Eshwar Stalin, Robert Liao, ochoa...@google.com, Greg Thompson, Chromium-dev
Thank you, there are external histograms, but since the data comes back slowly and gradually and over a wide range of hardware and inputs, it will take a long time until we notice that something is wrong.

Ramin Halavati

unread,
Apr 28, 2025, 12:06:30 PM4/28/25
to danakj, Erik Chen, Dana Fried, Eshwar Stalin, Robert Liao, ochoa...@google.com, Greg Thompson, Chromium-dev
I started a short doc on the options so far, please contribute if you have any other suggestions:
Reply all
Reply to author
Forward
0 new messages