Help: confused by the go1 benchmark test result

[Ben Shi]

Dear go developers,

I am doing arm64's ssa compiler optimization with register indexed load/store, but quite confused with the go1 benchmark result.

It seems most test cases get improvements via my patch, but the total performance of time cost go down?

Below is the test result and the original test logs are attached as old.txt and new.txt(with my patch applied)

How should I understand the test result?

Ben Shi

[powerman1st]

usually i treated all difference below 2％ as noise on my rpi3 board.

but i think there is indeed improvement for the ones exceed -2%

发自我的小米手机

在 Ben Shi <power...@163.com>，2018年4月16日 18:01写道：

[David Chase]

I am also confused by your results, though I see the same.

If on the other hand I take your new and old results and past them into a spreadsheet, and take ratios of runtimes (new/old), with all entries I see

average 97.57%
geomean 97.45%
median 98.52%

and with ~ results removed:

average 97.79%
geomean 97.67%
median 99.06%

20 out of 30 improved, excluding ~ results, 17 out of 26 improved.

This looks good to me.

Do you have a CL we can look at/play with?

I also have an arm64 box (ODroid), and have collected a second set of benchmarks from github.

(github.com/dr2chase/bent -- it downloads, compiles, benchmarks, you do need to first configure which systems to test.

It does not stay constant over time -- I'm always looking for new ones, and old ones drop out if they fail to compile and run).

[cherry]

It seems that old.txt contains BenchmarkSubMem-4, whereas new.txt doesn't. becnhstat doesn't work well if old and new are not the exact set of benchmarks. I believe that when computing geomean, it uses all the results in its own set, regardless of whether a benchmark presents in the other set. I.e. old's geomean includes go1+SubMem, whereas new's geomean includes only go1. They are not comparable.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Josh Bleecher Snyder]

On Mon, Apr 16, 2018 at 7:06 AM cherry <luna...@gmail.com> wrote:

It seems that old.txt contains BenchmarkSubMem-4, whereas new.txt doesn't. becnhstat doesn't work well if old and new are not the exact set of benchmarks. I believe that when computing geomean, it uses all the results in its own set, regardless of whether a benchmark presents in the other set. I.e. old's geomean includes go1+SubMem, whereas new's geomean includes only go1. They are not comparable.

https://github.com/golang/go/issues/19565

On Mon, Apr 16, 2018 at 8:35 AM, 'David Chase' via golang-dev <golan...@googlegroups.com> wrote:

I am also confused by your results, though I see the same.

If on the other hand I take your new and old results and past them into a spreadsheet, and take ratios of runtimes (new/old), with all entries I see

average 97.57%
geomean 97.45%
median 98.52%

and with ~ results removed:

average 97.79%
geomean 97.67%
median 99.06%

20 out of 30 improved, excluding ~ results, 17 out of 26 improved.

This looks good to me.

Do you have a CL we can look at/play with?

I also have an arm64 box (ODroid), and have collected a second set of benchmarks from github.

(github.com/dr2chase/bent -- it downloads, compiles, benchmarks, you do need to first configure which systems to test.

It does not stay constant over time -- I'm always looking for new ones, and old ones drop out if they fail to compile and run).

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Ben Shi]

Really sorry for such a low mistake. The SubMem case I forgot to remove.

The real geomean data should be 

[Geo mean]                  717µs          698µs        -2.55%

[Geo mean]               7.57MB/s       7.79MB/s        +2.98%

I thought total 2% improvement is really big for arm64.

I have created a CL for my change https://go-review.googlesource.com/c/go/+/107376

Ben