Understanding benchstat's output

[Tong Sun]

Hi, 

I'm trying to understand the benchstat's output. 

The benchstat that I have is fresh from golang.org/x/perf/cmd/, installed hours ago.

This is understandable:

name            old time/op    new time/op    delta

FormatEmoji-2     1.79µs ± 0%    1.82µs ± 1%   ~     (p=0.100 n=3+3)

FormatEmoji-4     1.99µs ± 1%    1.91µs ± 4%   ~     (p=0.200 n=3+3)

FormatEmoji-8     1.97µs ± 3%    1.99µs ± 1%   ~     (p=0.700 n=3+3)

FormatEmoji-12    2.05µs ± 1%    2.03µs ± 1%   ~     (p=0.400 n=3+3)

name            old alloc/op   new alloc/op   delta

FormatEmoji-2       384B ± 0%      384B ± 0%   ~     (p=1.000 n=3+3)

FormatEmoji-4       386B ± 0%      385B ± 0%   ~     (p=0.700 n=3+3)

FormatEmoji-8       387B ± 0%      386B ± 0%   ~     (p=0.400 n=3+3)

FormatEmoji-12      388B ± 0%      388B ± 0%   ~     (p=1.000 n=3+3)

name            old allocs/op  new allocs/op  delta

FormatEmoji-2       7.00 ± 0%      7.00 ± 0%   ~     (all equal)

FormatEmoji-4       7.00 ± 0%      7.00 ± 0%   ~     (all equal)

FormatEmoji-8       7.00 ± 0%      7.00 ± 0%   ~     (all equal)

FormatEmoji-12      7.00 ± 0%      7.00 ± 0%   ~     (all equal)

However, 

Why benchstat thinks there is no differences between the following two runs?

$ benchstat /tmp/old.log /tmp/new.log

name                  old time/op    new time/op    delta

FormatEmojiString-2     1.79µs ± 0%    6.81µs ± 2%   ~     (p=0.100 n=3+3)

FormatEmojiString-4     1.99µs ± 1%    7.22µs ± 1%   ~     (p=0.100 n=3+3)

FormatEmojiString-8     1.97µs ± 3%    7.55µs ± 2%   ~     (p=0.100 n=3+3)

FormatEmojiString-12    2.05µs ± 1%    7.64µs ± 1%   ~     (p=0.100 n=3+3)

name                  old alloc/op   new alloc/op   delta

FormatEmojiString-2       384B ± 0%     1459B ± 0%   ~     (p=0.100 n=3+3)

FormatEmojiString-4       386B ± 0%     1463B ± 0%   ~     (p=0.100 n=3+3)

FormatEmojiString-8       387B ± 0%     1468B ± 0%   ~     (p=0.100 n=3+3)

FormatEmojiString-12      388B ± 0%     1473B ± 0%   ~     (p=0.100 n=3+3)

name                  old allocs/op  new allocs/op  delta

FormatEmojiString-2       7.00 ± 0%     25.00 ± 0%   ~     (p=0.100 n=3+3)

FormatEmojiString-4       7.00 ± 0%     25.00 ± 0%   ~     (p=0.100 n=3+3)

FormatEmojiString-8       7.00 ± 0%     25.00 ± 0%   ~     (p=0.100 n=3+3)

FormatEmojiString-12      7.00 ± 0%     25.00 ± 0%   ~     (p=0.100 n=3+3)

I'd think that the delta will be quite big. 

Now take a look at this. benchstat thinks there is no differences between the following two runs is beyond imagination:

$ benchstat /tmp/old.log /tmp/new.log

name                 old time/op    new time/op      delta

FormatEmojiBlock-2     1.79µs ± 0%  13543.09µs ± 4%   ~     (p=0.100 n=3+3)

FormatEmojiBlock-4     1.99µs ± 1%  14597.09µs ± 1%   ~     (p=0.100 n=3+3)

FormatEmojiBlock-8     1.97µs ± 3%  16390.62µs ± 2%   ~     (p=0.100 n=3+3)

FormatEmojiBlock-12    2.05µs ± 1%  17409.37µs ± 2%   ~     (p=0.100 n=3+3)

name                 old alloc/op   new alloc/op     delta

FormatEmojiBlock-2       384B ± 0%   15270330B ± 0%   ~     (p=0.100 n=3+3)

FormatEmojiBlock-4       386B ± 0%   15270671B ± 0%   ~     (p=0.100 n=3+3)

FormatEmojiBlock-8       387B ± 0%   15271206B ± 0%   ~     (p=0.100 n=3+3)

FormatEmojiBlock-12      388B ± 0%   15271727B ± 0%   ~     (p=0.100 n=3+3)

name                 old allocs/op  new allocs/op    delta

FormatEmojiBlock-2       7.00 ± 0%     3467.00 ± 0%   ~     (p=0.100 n=3+3)

FormatEmojiBlock-4       7.00 ± 0%     3467.00 ± 0%   ~     (p=0.100 n=3+3)

FormatEmojiBlock-8       7.00 ± 0%     3467.00 ± 0%   ~     (p=0.100 n=3+3)

FormatEmojiBlock-12      7.00 ± 0%     3467.00 ± 0%   ~     (p=0.100 n=3+3)

What's happening? What I've done wrong?

Thx

[Kurtis Rader]

You didn't show us the actual commands you are running to produce the two log files. It is also unclear why you expect benchstat to report a difference. In other words: insufficient information for anyone to make even a guess why benchstat isn't producing the output you expect.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a56a17b1-a8c3-4926-a27a-21cd86074f67n%40googlegroups.com.

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

[Dan Kortschak]

Your sample size is too small. By default the statistical test
performed by benchstat in a Mann-Whitney U test which is a non-
parametric test that works by comparing the relative orderings of the
samples in the compared sets; without looking at the magnitudes of
difference. 3 is too small.

Dan

[Tarmigan]

From the benchstat help text:

benchstat -h
usage: benchstat [options] old.txt [new.txt] [more.txt ...]
options:
-alpha α
consider change significant if p < α (default 0.05)
....


Specifically the "p" in your results is reported to be at least 0.100
which is probably because you have a small sample size with N=3. Try
again with N=10 and see if that helps.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD8R2NBveVArAchBOkrL1qZ6oNRpFqWLqbF7aKG5o3L%2Bxw%40mail.gmail.com.