Can we have tests that keep track of running speed?

[Ramin Halavati]

Hi,

I want to add tests to monitor changes in the processing speed of an external library, for Windows, Linux, and Mac. Is it possible in the chromium commit queue or as an FYI test to run tests on rather unchanging hardware configs?

We are running similar tests for ChromeOS (as TAST tests) on different hardwares and have speed expectations based on each specific hardware, but I am not sure if this is possible for other platforms.

Best,

Ramin

[Greg Thompson]

I think it should be possible to make a perf test for this. Pinpoint runs perf tests on actual hardware and makes pretty pictures/alerts.

https://chromium.googlesource.com/chromium/src/tools/+/HEAD/perf/README.md

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAORodQjVxozA98s8-Z1N_yLDUiwK1aDGMj%3DkyW4D5O3xoM5JfQ%40mail.gmail.com.

[Ramin Halavati]

Thank you Greg.

[Erik Chen]

+Dana Fried +Eshwar Stalin +Robert Liao +ochoa...@google.com

The existing perf framework is built around the concept of running tests that measure web-page performance. We want to have something similar for native code, there's a project under way to make this easy to do via kombucha.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAORodQhi%3DsPSWqqO%3DfcAD0aB9bkm_GkJViGy%3DXsAgeLAAUweRA%40mail.gmail.com.

[Erik Chen]

Incidentally, what is your use case? IMO it's much better to track a CUJ than a microbenchmark for a library.

[Ramin Halavati]

Thank you. I want to monitor the performance speed of the OCR module.

We have browser tests such as OpticalCharacterRecognizerTest::PerformOCR_Simple to monitor quality of the service, but they cannot measure speed reliably.

(The test  uses CIPD to have access to the library binary that is downloaded on clients via component updater.)

[danakj]

It might be that you'd prefer to use a histogram which can measure performance in the wild https://chromium.googlesource.com/chromium/src/+/master/tools/metrics/histograms/README.md

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAORodQgut%3DBaMzNd8gfdVCDXemyVzxb-MfBKWs1o4Nzkik24SQ%40mail.gmail.com.

[Ramin Halavati]

Thank you, there are external histograms, but since the data comes back slowly and gradually and over a wide range of hardware and inputs, it will take a long time until we notice that something is wrong.

[Ramin Halavati]

I started a short doc on the options so far, please contribute if you have any other suggestions:

 Adding performance tests for OCR in Chrome