Which benchmark case is the reliable one?

[T L]

I write a benchmark, but the results of the following 3 cases are much different.
Which one is reliable?

package main

import (
    "testing"
    "runtime"
)

const N = 200 * 1000
type Element int64
var sum int

func Benchmark_MaybeOptimizedOut(b *testing.B) {
    var a [N]Element
    b.ResetTimer()
    var g int
    for i := 0; i < b.N; i++ {
        for j := 0; j < len(a); j++ {
            g += int(a[j])
        }
    }
}

func Benchmark_AssignToGlobal(b *testing.B) {
    var a [N]Element
    b.ResetTimer()
    var g int
    for i := 0; i < b.N; i++ {
        for j := 0; j < len(a); j++ {
            g += int(a[j])
        }
    }
    sum = g
}

func Benchmark_KeepAlive(b *testing.B) {
    var a [N]Element
    b.ResetTimer()
    var g int
    for i := 0; i < b.N; i++ {
        for j := 0; j < len(a); j++ {
            g += int(a[j])
        }
    }
    runtime.KeepAlive(&g)
}

/* output:
$ go test -bench=. -benchtime=3s
Benchmark_MaybeOptimizedOut-4          50000         87318 ns/op
Benchmark_AssignToGlobal-4             20000        205934 ns/op
Benchmark_KeepAlive-4                  10000        620351 ns/op
*/

[Jan Mercl]

You need to define what do _you_ consider 'reliable' to get a useful answer.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

-j

[T L]

On Sunday, April 23, 2017 at 12:20:52 AM UTC+8, Jan Mercl wrote:

You need to define what do _you_ consider 'reliable' to get a useful answer.

calculate the sum of a number array elements.
 

[Dave Cheney]

The middle one, however you should make sum exported for the most reliable results.

[T L]

On Sunday, April 23, 2017 at 6:08:32 AM UTC+8, Dave Cheney wrote:

The middle one, however you should make sum exported for the most reliable results.

Thanks. Yes I just remembered that the g in the last one will escape to heap, maybe this is the reason why it is slow.
 

[Dave Cheney]

-gcflags=-m will help confirm this l.