Fun with benchmarks?

[Greg Ward]

Hi all --

I've just written my first benchmark using the testing package. Seems
to work just fine. However, one could obviously have a lot more fun
with the *testing.B object as a starting point:

* track past runtimes on the current machine and make sure that
particular benchmarks don't get slower (unless you say it's OK)

* assert that an allegedly constant-time algorithm is really
(roughly) constant time (or linear, or lg(N), or whatever)

Has anyone played around with either of those? The first seems doable.
The second one feels ... big. Interesting. Scary. Master's thesis at
least. ;-)

Greg

[Dave Cheney]

I solve the first by renaming the output of go test -c to $PKG.$(hg
id) allowing me to checkpoint various revisions while making changes.
Then

./$PKG.$(hg id) -test.bench=. > old.txt && ./$PKG.test -test.bench=. >
new.txt && $GOROOT/misc/benchcmp {old,new}.txt

Crude, but it works for me.

Cheers

Dave

> --
>
>

[Russ Cox]

On Mon, Nov 19, 2012 at 5:49 PM, Greg Ward <gr...@gerg.ca> wrote:
> I've just written my first benchmark using the testing package. Seems
> to work just fine. However, one could obviously have a lot more fun
> with the *testing.B object as a starting point:
>
> * track past runtimes on the current machine and make sure that
> particular benchmarks don't get slower (unless you say it's OK)

We used to do this automatically for Go benchmarks as part of the
build.golang.org dashboard. We turned it off because due to various
failures of vision our implementation of the back end storage wasn't
scaling well. I still think it's a good idea.

> * assert that an allegedly constant-time algorithm is really
> (roughly) constant time (or linear, or lg(N), or whatever)

That's interesting, and it's not something I've seen automated in the
Go context. I don't think it's that hard, but it would need to be
separate from testing.B's N, which is supposed to control a linear
parameter.

Russ

[minux]

On Mon, Nov 26, 2012 at 4:15 AM, Russ Cox <r...@golang.org> wrote:

>   * assert that an allegedly constant-time algorithm is really

>     (roughly) constant time (or linear, or lg(N), or whatever)

That's interesting, and it's not something I've seen automated in the
Go context. I don't think it's that hard, but it would need to be
separate from testing.B's N, which is supposed to control a linear
parameter.

I like this idea. 

It will be cool if the testing package can tell me the function's rate of

growth. It can also help removing some duplicate benchmark functions

like this:

func f(m int) { /* actual function to be benchmarked */ }

func BenchmarkF1(b *testing.B) { for /* */ { f(1) }

func BenchmarkF2(b *testing.B) { for /* */ { f(2) }

func BenchmarkF10(b *testing.B) { for /* */ { f(10) }

I think what we really want is the rate of growth of f() here.