asserting benchmark results

[jonathan...@gmail.com]

I searched around but didn't see a solution.

I'd like to have my benchmark times as tests.

I see there is testing.Benchmark() which is great to get the NsPerOp() but if I call testing.Benchmark(BenchmarkFoo) within a TestFoo() my benchmarks are run twice.

Is there a way to get at the benchmark result inside the benchmark so I can just use the normal .Error functions to fail the benchmark if it didn't meet the goal? 

[jonathan...@gmail.com]

I understand I can just name the function different and the normal benchmark runner won't find it, but I'd like to still get the normal output from the benchmarks and be able to use benchcmp.

So perhaps this is just a future request to have within testing.B the NsPerOp() that you can get after the loop?

[jonathan...@gmail.com]

Is it foolhardy to assert benchmark results?

[adon...@google.com]

On Monday, 20 July 2015 13:00:42 UTC-4, jonathan...@gmail.com wrote:

Is it foolhardy to assert benchmark results?

Yes.  It's almost impossible to write tests whose running time is sufficiently deterministic that you would want to use them in an assertion.  (Teams that do this at Google, such as the compiler optimization team, use specially configured isolated machines with identical hardware, that run one job at a time.)

More useful is to plot a graph of a metric of interest over time.  You can easily see the changes that caused performance regressions.  It won't stop you from committing a bad change, but the results are easy to understand and you won't go crazy dealing with flaky tests.

[jonathan...@gmail.com]

Thanks that sounds like very sound advice.