Running benchmark on a list of tests in a vector (or array)

[ccl...@nuodb.com]

I want to benchmark an interpreter on a wide variety on inputs.  Each input is a single string contained in a vector and I would like each one reported as a separate benchmark.  Is there a way to specify that to Google benchmark?

The original code was something like:

const char *inputs = { "1 + 1", "1 + 2" , "1 * 1" ....};

Test()
{
   for (auto input: inputs) {
     preliminary work;
     code I want to benchmark;
     verification code;
   }
}

If I do

TEST(benchmark::State state)
{

   for (auto input: inputs) {
     preliminary work;
     for (auto _ : state) {

       code I want to benchmark;
     }
     verification code;
   }
}

BENCHMARK(TEST)->RANGE etc.

It doesn't give me what I want as results.  In fact, it complains with an assertion "!started && !finished" which I think means it doesn't expect to see the auto _ : state loop run multiple times in one benchmark.

[Dominic Hamon]

Your assessment is correct.

I think the way to do this would be to use runtime registration of benchmarks. Something like this (note, untested code):

```

static void BM_Interpret(benchmark::State& state, const char* input) {

  for (auto _ : state) {

    Interpret(input);

  }

  st.SetLabel(input)

}

int main(int argc, char** argv) {

  std::array<std::pair<const char*, const char*>> cases = {

    {"test1", "1+1"},

    {"test2", "1+2"},

    {"test3", "1*1"}};

  for (auto const& c : cases) {

    benchmark::RegisterBenchmark(c.first, BM_Interpret, c.second);

  }

  benchmark::Initialize(&argc, argv);

  size_t num_ran = benchmark::RunSpecifiedBenchmarks();

  assert(num_ran == cases.size());

  return 0;

}

```

Dominic Hamon | Google

There are no bad ideas; only good ideas that go horribly wrong.

--
You received this message because you are subscribed to the Google Groups "benchmark-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to benchmark-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ccl...@nuodb.com]

I got something closer to working by defining a two argument benchmark and using CustomArgs.  The first argument is the index into the vector of strings that the evaluator should process and the second is the number of repetitions.

However, the complexity stats are summed over the entire set.  I'd really like to treat each variation on the first argument as a separate benchmark.  Even better if I could label the output with the strings once they are separated.

[ccl...@nuodb.com]

Excellent.  Looks worth a try.  Thanks.  It sounds like it will get me the rest of what I want.