[ANN] benchls - Benchmark Performance Curve Fitting

[Jonathan Lawlor]

Hello,

I've written a command line tool benchls in the same vein as benchstat that fits a least squares model to parameterized benchmark output.

The basic idea is to take benchmarks like this:

PASS
BenchmarkSort10-4                1000000          1008 ns/op
BenchmarkSort100-4                200000          8224 ns/op
BenchmarkSort1000-4                10000        152945 ns/op
BenchmarkSort10000-4                1000       1950999 ns/op
BenchmarkSort100000-4                 50      25081946 ns/op
BenchmarkSort1000000-4                 5     302228845 ns/op
BenchmarkSort10000000-4                1    3631295293 ns/op
BenchmarkStableSort10-4          1000000          1260 ns/op
BenchmarkStableSort100-4          100000         16730 ns/op
BenchmarkStableSort1000-4           5000        362024 ns/op
BenchmarkStableSort10000-4           300       5731738 ns/op
BenchmarkStableSort100000-4           20      88171712 ns/op
BenchmarkStableSort1000000-4           1    1205361782 ns/op
BenchmarkStableSort10000000-4          1    14349613704 ns/op
ok      github.com/jonlawlor/benchls    138.860s

Where the 10, 100, 1000, ... indicate a number of unsorted elements, and the ns/op indicate the approximate amount of time it took to sort, and then fit a least squares model to describe the relationship between the number of elements and how long it takes:

$ benchls -vars="/?(?P<N>\\d+)-\\d+$" -xtransform="math.Log(N) * N, 1.0" bench.txt
group \ Y ~          math.Log(N) * N    1.0             R^2
BenchmarkSort        2.254e+01±6.4e-02  -2e+06±3.9e+06  0.9999949426719544
BenchmarkStableSort  8.906e+01±1.8e-01  -7e+06±1.1e+07  0.9999973642760738

Where each row indicates the least squares estimate of the performance, parameterized by N * log(N).  Here we can see (for illustration only) that sort.Sort is about 4x faster than sort.Stable for large N.

You can put any expression involving only float64's, +, -, *, /, and functions from the math library into both the explanatory and response, and you can identify input variables using named regexp, so functions of multiple variables are possible.

Currently it only reports the model parameters, 95% confidence intervals, and the R^2, but if anyone wants additional goodness of fit stats I can add them.  The code is partially derived from (and inspired by) benchstat.

Feedback is appreciated!

[Jonathan Lawlor]

Forgot to mention - if the sub-benchmarks proposal is incorporated into go, it would be very easy to construct benchmarks in this form.

[Seb Binet]

thinking out loud...
maybe have a switch (or another program altogether) to also produce a
gonum/plot.Plot with the data points plus the fit overlaid ?

-s

[Damian Gryski]

On Thursday, February 25, 2016 at 7:11:52 PM UTC+1, Sebastien Binet wrote:

thinking out loud...
maybe have a switch (or another program altogether) to also produce a
gonum/plot.Plot with the data points plus the fit overlaid ? 

I've seen a few tools that produce a static html + javascript charting library (for example, flot) as output for this graphing case.

Damian 

[Seb Binet]

yep, I've dabbled into those as well, and also replicated a gcvis[1]
-like thingy from Dave to understand flot, interactive charting,
websockets and other things I didn't know I needed to know :)
and, for monitoring and digging into performance regressions via
(probably) a web interface, what you suggest is probably the best
medium.
but for quick one-off comparisons, I believe a quick PNG/PDF plot
created via gonum/plot would be easier to use.

-s

[1] https://github.com/davecheney/gcvis

[Damian Gryski]

On Friday, February 26, 2016 at 10:34:33 AM UTC+1, Sebastien Binet wrote:

but for quick one-off comparisons, I believe a quick PNG/PDF plot
created via gonum/plot would be easier to use.

The snippet of code I use as a template:

https://github.com/dgryski/trifles/blob/master/grinderplot/main.go#L84

Damian