Benchmark and timer

[Mathieu Lonjaret]

Hi,

I need to run some specific benchmarks on a fairly large database. I
wanted to create and fill the database on the fly in the benchmark
function with something like this:

func BenchmarkDB(b *testing.B) {
b.StopTimer()
createAndFillDb() // takes more than 600s
b.StartTimer()
for i := 0; i < b.N; i++ {
actualBenchmark()
}
}

but apparently stopping the timer does not affect the 600s limit on
the duration of tests. And the -timeout flag does not seem to work for
benchmarks either, just for tests.

Arguably, createAndFillDb() could be improved to not take so long,
but out of curiosity I'd like to know what would be the recommended
way to achieve that kind of setup+benchmark.

Some solutions have been proposed on #go-nuts, but I'd like to have
other opinions please.

Cheers,
Mathieu

[Rob 'Commander' Pike]

The simplest thing is probably to set the -test.timeout flag.

-rob

[Mathieu Lonjaret]

This does not seem to affect benchmarks. I've tried:

go test -bench'='BenchmarkDB -timeout'='10s benchIndices_test.go

and it does not stop after 10 seconds. Am I using it incorrectly? I
haven't looked at the code myself, but someone on #go-nuts told me the
timeout flag does not have any effect on benchmarks, only tests.

[Rob 'Commander' Pike]

My reading of the code says three things:

1) Benchmarks are not subject to the -test.timeout flag directly. I
was wrong there.
2) The test binary itself does not run any timeout on the benchmarks,
so you could do
go test -c
foo.test -bench ...
3) The 'go test' command sets an overall timeout for the binary's run
to be 1 minute longer than the value of -test.timeout, if set, so you
could also do
go test -timeout 1000m ...
Because the delta is 1 minute, a 10s check run won't trigger it.

Again, this is just my reading of the code; I might have missed something.

-rob

[Mathieu Lonjaret]

Ah yes, -timeout 10s indeed killed it at ~(60 + 10 =) 70s so all is well.

thanks,
Mathieu