Integration with Google Test?

[irwin...@gmail.com]

Hi all,

I came across this package recently, and it looks great. It seems very easy to use and I'd like to get started with it.

That being said, I'd like to integrate Google Benchmark within my existing test suite done with Google Test. And it is not obvious to me how to do this.

I understand that I'm supposed to define benchmarks as a function that takes a benchmark::State&, and then use the BENCHMARK macro. That is all okay. However, it seems that I'm supposed to run the benchmarks using the benchmark::RunSpecifiedBenchmarks() function. Ideally, I'd like to be able to run selected benchmarks within a Google Test TEST(...) macro. The idea is that benchmarks are also tests themselves.

Looking at the source code, it seems this should be possible by choosing a benchmark, then calling RunInstance for each Instance returned with CreateBenchmarkInstances.

Am I missing something obvious here? Otherwise, if I say above is true, maybe we could make this integration easy?

Cheers,

Irwin

[Dominic Hamon]

Hi Irwin

I did originally make some efforts to integrate this directly with Google Test, but it was decided that separate projects would be better.

You almost certainly don't want to run the benchmark as a TEST as the method used to get good timings and track iterations would need tighter integration.

The best way is to define tests and benchmarks in one executable and then set up a custom main function to run both test and benchmark instances.

- dominic

--
You received this message because you are subscribed to the Google Groups "benchmark-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to benchmark-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Irwin Zaid]

Hi Dominic,

Thanks for the speedy reply!

> You almost certainly don't want to run the benchmark as a TEST as the
> method used to get good timings and track iterations would need tighter
> integration.

What I'm proposing is running some BENCHMARK(...) functions within a
Test, so that only those BENCHMARK(...) functions are benchmarked. I
certainly wouldn't want to benchmark the whole TEST. (Or was this clear
before?)

Is something wrong with my proposal of using RunInstance over the
created instances? (I'd just like to understand.)

> The best way is to define tests and benchmarks in one executable and
> then set up a custom main function to run both test and benchmark instances.

This could work, but it's a bit tricky. For instance, I think I'd have
to create a lot of wrapper functions around what I want to benchmark.
Then I'd have to call those within various TEST(...) and also wrap those
in functions that accept a benchmark::State&.

Irwin

[Dominic Hamon]

On Sat, Nov 29, 2014 at 10:28 AM, Irwin Zaid <irwin...@physics.ox.ac.uk> wrote:

Hi Dominic,

Thanks for the speedy reply!

You almost certainly don't want to run the benchmark as a TEST as the
method used to get good timings and track iterations would need tighter
integration.

What I'm proposing is running some BENCHMARK(...) functions within a Test, so that only those BENCHMARK(...) functions are benchmarked. I certainly wouldn't want to benchmark the whole TEST. (Or was this clear before?)

Ah, I see. The BENCHMARK() macros need to be outside of a test as they have all sorts of magic going on. I agree that you don't want to benchmark the whole test.

 

Is something wrong with my proposal of using RunInstance over the created instances? (I'd just like to understand.)

You can't create instances without a 'family', and that's created as part of creating a new 'Benchmark' object. You could possibly create the Benchmark objects, passing in a name and the function you want to benchmark, and then calling the right methods. I'm just not convinced that all the state will be set up correctly.

 

The best way is to define tests and benchmarks in one executable and
then set up a custom main function to run both test and benchmark instances.

This could work, but it's a bit tricky. For instance, I think I'd have to create a lot of wrapper functions around what I want to benchmark. Then I'd have to call those within various TEST(...) and also wrap those in functions that accept a benchmark::State&.

That's true - you'd end up with a test and a benchmark that calls a function. Having said that, you should be calling the function repeatedly until the state lets you know it's ready to complete and that boilerplate is necessary only for benchmarking, not for testing.

You could always have a benchmark that also checks for correctness using the google test macros. I haven't tried that, but I don't know a reason why it wouldn't work for simple cases. If you have test fixtures, it gets a little more complicated.

 

Irwin

[dma]

I just posted https://github.com/google/benchmark/pull/70 which I think helps your use case a little bit. You can link with this new library 'libbenchmark_test_main' alongside 'libbenchmark' to get an executable that runs both googletest and benchmark tests.

It doesn't help you define the tests and benchmark using the same functions, but it at least saves you writing your own main.