Retrieve benchmark results programmatically

[cybrow...@gmail.com]

Is it possible to write a benchmark that has a pass/fail condition? I would like to give users the option to run benchmarks as a pass/fail test in CI. If some calculated benchmark value exceeds a threshold, I would like the binary to exit with non-zero exit code so that it can be identified by a CI test runner as a failure.

I realize I could run the binary and then parse the json output (or use compare.py), but it would be convenient to do this in a single c++ binary. Is there a way for user code to access the benchmark value or otherwise specify a hard threshold for a benchmark?

[Dominic Hamon]

It's come up before (internally) but what we generally prefer to do (due to potential noise in benchmarks) is to monitor trends and statistically significant anomalies, which are hard to capture in the library.

If you wanted to do something like this i imagine you could define a custom reporter (an instance of BenchmarkReporter* can be passed at runtime using this api). That reporter receives information about each group of benchmark runs. You could couple this with an expectation for the given group, i suppose.

Dominic Hamon | Google

There are no bad ideas; only good ideas that go horribly wrong.

On Mon, May 6, 2019 at 2:25 PM <cybrow...@gmail.com> wrote:

Is it possible to write a benchmark that has a pass/fail condition? I would like to give users the option to run benchmarks as a pass/fail test in CI. If some calculated benchmark value exceeds a threshold, I would like the binary to exit with non-zero exit code so that it can be identified by a CI test runner as a failure.

I realize I could run the binary and then parse the json output (or use compare.py), but it would be convenient to do this in a single c++ binary. Is there a way for user code to access the benchmark value or otherwise specify a hard threshold for a benchmark?

--
You received this message because you are subscribed to the Google Groups "benchmark-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to benchmark-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/benchmark-discuss/6256dac3-96ed-4bfe-aeb7-9ba507e26053%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[cybrow...@gmail.com]

Thanks, this looks promising. What would be the best way to register the expectation though? I was imagining an API inside a BENCHMARK definition to allow expectation declaring, some kind of "ASSERT_BENCHMARK_GT()". It looks like I'd have to have that write to some kind of global map of benchmark names to expectation conditions. What objects are accessible in BENCHMARK scope that are appropriate to write this information to?

On Monday, May 6, 2019 at 2:47:15 PM UTC-7, Dominic Hamon wrote:
> It's come up before (internally) but what we generally prefer to do (due to potential noise in benchmarks) is to monitor trends and statistically significant anomalies, which are hard to capture in the library.
>
>
>
> If you wanted to do something like this i imagine you could define a custom reporter (an instance of BenchmarkReporter* can be passed at runtime using this api). That reporter receives information about each group of benchmark runs. You could couple this with an expectation for the given group, i suppose.
>
>
>
>
>
>
> Dominic Hamon | Google
> There are no bad ideas; only good ideas that go horribly wrong.
>
>
>
> On Mon, May 6, 2019 at 2:25 PM <cybrow...@gmail.com> wrote:
> Is it possible to write a benchmark that has a pass/fail condition? I would like to give users the option to run benchmarks as a pass/fail test in CI. If some calculated benchmark value exceeds a threshold, I would like the binary to exit with non-zero exit code so that it can be identified by a CI test runner as a failure.
>
>
>
> I realize I could run the binary and then parse the json output (or use compare.py), but it would be convenient to do this in a single c++ binary. Is there a way for user code to access the benchmark value or otherwise specify a hard threshold for a benchmark?
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "benchmark-discuss" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to benchmar...@googlegroups.com.

[Dominic Hamon]

I imagined you'd want the expectations to be available at the reporter scope, rather than the benchmark scope. Ie, you would (as you say) have a map from benchmark name (the thing reported per row in the console output) to expectations stored in the custom reporter. Then on a given run report, you can determine if the expectations match reality.

Ie, it's not like googletest where the test itself asserts truth, it's the responsibility of the benchmark to run the thing and time the thing and the responsibility of the reporter to do something with the results (assert, or report or whatever).

To unsubscribe from this group and stop receiving emails from it, send an email to benchmark-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/benchmark-discuss/fcda5da1-e235-4012-bc03-4dfefa5979ec%40googlegroups.com.