go test -test.benchtime not working?

[Boris Solovyov]

Hi all,

I want to benchmark/profile some things for longer time to see how GC happens and find where memory spent.

$ time go test -bench='.*' -memprofile mem.out -test.benchtime 300

PASS

BenchmarkAggregation 2000000000         0.02 ns/op

ok   <package> 0.220s

real 0m0.732s

user 0m0.577s

sys 0m0.070s

It doesn't seem to make any difference what -test.benchtime value I specify, always completes in fraction of a second. I want to run a LONG benchmark. Do I do something wrong? Also, BenchmarkAggregation function has a big loop in it, 10000 iterations over something that is not terribly expensive but I expect to take much more than 0.02ns. Something seems wrong to me.

[Jan Mercl]

`BenchmarkAggregation` source?

-j

[minux]

time go test -bench=. -memprofile mem.out -benchtime 300s

note the argument to benchtime is a Go time.Duration, so you need to append the time unit.

also note that, when providing the options on the "go test" command line, you can omit "test."

prefix.

[Boris Solovyov]

Hi,

On Thu, Feb 7, 2013 at 3:16 PM, minux <minu...@gmail.com> wrote:

time go test -bench=. -memprofile mem.out -benchtime 300s

note the argument to benchtime is a Go time.Duration, so you need to append the time unit.

That does not work for me. 

$ time go test -bench=. -memprofile mem.out -benchtime 30s

invalid value "30s" for flag -test.benchtime: strconv.ParseFloat: parsing "30s": invalid syntax 

It also does not match the docs, which says -test.benchtime N, where N is the number of seconds. Are you using a different version of Go? I am using 1.0.3.

[Russ Cox]

Your BenchmarkAggregation is not running b.N iterations.

The confusion about the benchtime argument is a change in the go tool. In Go 1 it is a number of seconds; in Go 1.1 it will be a true duration.

Russ

[Boris Solovyov]

Hi,

On Thu, Feb 7, 2013 at 11:19 PM, Russ Cox <r...@golang.org> wrote:

Your BenchmarkAggregation is not running b.N iterations.

Aha, I understand now! Thanks. May I suggest that the package documentation, which currently says this,

The benchmark package will vary b.N until the benchmark function lasts long enough to be timed reliably. The output

would be good to say this:

Note that the code iterates from 0 to b.N. The benchmark package will vary b.N until the benchmark function lasts long enough to be timed reliably. The output

 

[Jan Mercl]

On Fri, Feb 8, 2013 at 6:01 PM, Boris Solovyov <boris.s...@gmail.com> wrote:
> Aha, I understand now! Thanks. May I suggest that the package documentation,
> which currently says this,
>
> The benchmark package will vary b.N until the benchmark function lasts long
> enough to be timed reliably. The output
>
> would be good to say this:
>
> Note that the code iterates from 0 to b.N. The benchmark package will vary
> b.N until the benchmark function lasts long enough to be timed reliably. The
> output

But that's not correct. Consider for example:

func BenchmarkMakeByteSlice(b *testing.B) {
b := make([]byte, b.N)
b.Log(len(b))
}

-j

[Jan Mercl]

On Fri, Feb 8, 2013 at 6:10 PM, Jan Mercl <0xj...@gmail.com> wrote:
> func BenchmarkMakeByteSlice(b *testing.B) {
> b := make([]byte, b.N)
> b.Log(len(b))
> }

func BenchmarkMakeByteSlice(b *testing.B) {
s := make([]byte, b.N)
b.Log(len(s))
}

// No, not two time 'b'

-j

[Boris Solovyov]

Then I change my suggestion for the docs to this:

Note the use of b.N. The benchmark package will vary b.N until the benchmark function lasts long enough to be timed reliably. The output

[Russ Cox]

On Fri, Feb 8, 2013 at 12:10 PM, Jan Mercl <0xj...@gmail.com> wrote:

But that's not correct. Consider for example:

func BenchmarkMakeByteSlice(b *testing.B) {
        b := make([]byte, b.N)
        b.Log(len(b))
}

Actually, it is this example that is incorrect, at least in general. In order for the benchmark function's approach of dividing by b.N to produce meaningful numbers, the work being done by the function must scale linearly with b.N. Performance relative to input size typically has significant discontinuities that will be incorrectly smoothed over by using b.N as an input size. Iterating is best.

Russ

[Kevin Gillette]

I was thinking about that a while ago: I think it'd be useful, because timings can be "smoothed over" like that, and because it'd be useful in general, it may be worth calculating a regression line and displaying that as a rate of (potentially negative) exponential growth. If you're expecting linear behavior, but the growth is sublinear, then it would indicate that the benchmark is incorrect. It'd also serve as a sanity check when, e.g. developing n^2 algorithms (in which case any growth larger than 2 may indicate problems).