Use google benchmark for binary performance

[Luke]

Hi,

I have some application binaries and I'm interested in comparing their performance. Although google benchmark is meant for microbenchmarking, I found it's handy to do something as follows, and the results look meaningful.

---

void BM_app(benchmark::State& state) {
for (auto _ : state) {
int result = std::system("myapp");
benchmark::DoNotOptimize(result);
}
}
BENCHMARK(BM_app)->Iterations(100);

---

I wonder how reliable is it to use google benchmark this way? Or is it a stupid idea to do so?

Thanks!
Luke

[Dominic Hamon]

It's not stupid, but there are some caveats :)

I'd be surprised if you get reliable numbers due to the number of moving parts required to spawn and run the binary. You could try setting Iterations to ~10 and setting Repetitions to ~10 so you get some statistics across runs. The degenerate case of Iterations(1)->Repetitions(100) might be even better for this use-case, assuming the binary takes a while (~seconds) to run.

It's usually more useful to have the binaries decomposed into libraries so you can like a benchmark main against them and benchmark the underlying functionality directly, but you may not have the flexibility in your system to do that, I suppose.

Dominic Hamon | Google

There are no bad ideas; only good ideas that go horribly wrong.

--
You received this message because you are subscribed to the Google Groups "benchmark-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to benchmark-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Roman Lebedev]

Also, you may want to "->UseRealTime()".