Thank you David and Caleb, those were helpful suggestions, especially
benchstat! Check out my conclusions though... :-/ I am running these
four:
$ go test -bench=. -count=10 >normal.txt
$ go test -run=XXX -bench=. -count=10 >filter.txt
$ go test -bench=. -count=10 >comment.txt
$ go test -run=XXX -bench=. -count=10 >comment_filter.txt
As you might guess, the last two are when I manually comment out the
body of the test function. Here's the result:
$ benchstat normal.txt filter.txt
name old time/op new time/op delta
PushBack-2 380µs ± 0% 380µs ± 1% ~ (p=0.447 n=10+9)
So filtering the test case out with -run seems to make no difference.
$ benchstat normal.txt comment.txt
name old time/op new time/op delta
PushBack-2 380µs ± 0% 355µs ± 1% -6.45% (p=0.000 n=10+9)
So commenting it out (the body!) leads to what I observed earlier,
just more reliably now thanks to running it 10 times.
$ benchstat comment.txt comment_filter.txt
name old time/op new time/op delta
PushBack-2 355µs ± 1% 357µs ± 1% ~ (p=0.278 n=9+10)
So applying the filter on top of commenting things out (just sanity
checking!) changes nothing again.
Since Caleb said he cannot detect a difference on his machine with his
version of Go, I am wondering if what I am seeing is just a weird
artifact of my ancient machine and old Go release. I'll try upgrading
soon. Also I'll try to see what I get on different machines I have
access too.
Thanks again!
--
Peter H. Froehlich <
http://www.cs.jhu.edu/~phf/>
Senior Lecturer | Director, Johns Hopkins Gaming Lab
The views stated in this communication are my own and do not express
the position or views of The Johns Hopkins University.