Built-in benchmark tracking and regression failures

[Will Faught]

I'm wondering if it would be useful to build benchmark regression detection into Go test benchmarks, and policy enforcement in terms of making regressions fail a benchmark "test".

If I understand correctly, as Go benchmarks currently stand, they tell you the numbers, but it's left to you to interpret them, store them, compare them, and enforce policies around them. For Go users who aren't performance and benchmarking experts, it would be useful to build in a simple process for doing this. Perhaps benchmark "bars" could be stored under $GOPATH like fuzz data, and a benchmark bar could be specified or saved in the package itself.

This would be useful for any project that had a policy of no performance regressions, or only permitting a certain amount of regression, such as a browser renderer. The Go Team could use it to track Go performance and enforce its own regression policies for Go changes.

Perhaps something like this:

$ pwd

/proj

$ ls

go.mod go.sum proj.go proj_test.go

$ go test -bench . # result saved under $GOPATH

PASS
BenchmarkThing   5000000               500 ns/op
ok      github.com/my/proj       3.0s

$ vi proj.go # make changes

$ go test -bench . # much slower

PASS, REGRESSION
BenchmarkThing   5000000               900 ns/op
ok      github.com/my/proj       3.1s

$ vi proj.go # revert changes

$ go test -bench . # much faster

PASS, REGRESSION FIXED
BenchmarkThing   5000000               501 ns/op
ok      github.com/my/proj       3.0s

$ ls

go.mod go.sum proj.go proj_test.go

$ go test -bench . -benchsave # save benchmark bar to go.bench

PASS
BenchmarkThing   5000000               502 ns/op
ok      github.com/my/proj       3.0s

$ ls

go.bench go.mod go.sum proj.go proj_test.go

$ vi proj.go # make changes

$ go clean -benchcache # delete results saved under $GOPATH

$ go test -bench . # compares to go.bench

FAIL, REGRESSION
BenchmarkThing   5000000               901 ns/op
ok      github.com/my/proj       3.1s

Will