How to reduce the fluctuation of micro benchmark results

[Xiangdong JI]

Hi,

Significant fluctuation is observed when running micro benchmarks, even with the same version of go, e.g the following numbers from math/big package, could anyone please share practices that may help generate stable result?

FloatAdd/10000-64                     669ns ± 0%     676ns ± 1%    +1.05%  (p=0.008 n=5+5)

FloatAdd/100000-64                   5.29µs ± 1%    5.56µs ± 0%    +5.08%  (p=0.008 n=5+5)

FloatSub/10-64                        125ns ± 0%     125ns ± 0%      ~     (p=0.444 n=5+5)

FloatSub/100-64                       126ns ± 0%     126ns ± 0%      ~     (all equal)

FloatSub/1000-64                      150ns ± 0%     150ns ± 0%      ~     (all equal)

FloatSub/10000-64                     393ns ± 0%     447ns ± 0%   +13.79%  (p=0.008 n=5+5)

FloatSub/100000-64                   3.20µs ± 0%    3.20µs ± 0%      ~     (p=0.056 n=5+5)

ParseFloatSmallExp-64                25.3µs ±15%    23.8µs ± 0%    -6.26%  (p=0.016 n=5+5)

ParseFloatLargeExp-64                84.7µs ± 0%    84.8µs ± 1%      ~     (p=0.690 n=5+5)

GCD10x10/WithoutXY-64                 166ns ± 0%     165ns ± 0%    -0.60%  (p=0.029 n=4+4)

GCD10x10/WithXY-64                    746ns ± 1%     747ns ± 1%      ~     (p=0.810 n=4+5)

GCD10x100/WithoutXY-64                438ns ± 0%     439ns ± 0%      ~     (p=0.730 n=5+5)

GCD10x100/WithXY-64                  2.35µs ±14%    2.23µs ± 2%      ~     (p=0.175 n=5+5)

GCD10x1000/WithoutXY-64              1.16µs ±13%    1.37µs ±43%      ~     (p=0.310 n=5+5)

GCD10x1000/WithXY-64                 6.20µs ±59%    4.71µs ±29%      ~     (p=0.841 n=5+5)

GCD10x10000/WithoutXY-64             16.8µs ±52%    11.7µs ±46%      ~     (p=0.151 n=5+5)

GCD10x10000/WithXY-64                58.6µs ±53%    80.2µs ±53%      ~     (p=0.310 n=5+5)

GCD10x100000/WithoutXY-64            72.0µs ±20%    77.1µs ±16%      ~     (p=0.421 n=5+5)

GCD10x100000/WithXY-64                169µs ±19%     271µs ±28%   +60.47%  (p=0.016 n=5+5)

GCD100x100/WithoutXY-64              2.37µs ±18%    2.31µs ± 8%      ~     (p=0.841 n=5+5)

GCD100x100/WithXY-64                 4.22µs ± 7%    4.05µs ± 1%      ~     (p=0.556 n=5+4)

GCD100x1000/WithoutXY-64             3.73µs ± 7%    4.02µs ±27%      ~     (p=0.841 n=5+5)

GCD100x1000/WithXY-64                7.44µs ± 3%   10.67µs ±22%   +43.33%  (p=0.008 n=5+5)

GCD100x10000/WithoutXY-64            32.4µs ±63%    20.8µs ± 3%   -35.83%  (p=0.016 n=5+4)

GCD100x10000/WithXY-64               71.3µs ±93%   118.5µs ±44%      ~     (p=0.151 n=5+5)

GCD100x100000/WithoutXY-64            199µs ±56%     195µs ±29%      ~     (p=1.000 n=5+5)

GCD100x100000/WithXY-64               459µs ±33%     892µs ±60%   +94.33%  (p=0.016 n=5+5)

GCD1000x1000/WithoutXY-64            22.8µs ±14%    22.5µs ±10%      ~     (p=1.000 n=5+5)

GCD1000x1000/WithXY-64               39.8µs ± 0%    43.9µs ±10%   +10.26%  (p=0.008 n=5+5)

GCD1000x10000/WithoutXY-64           41.6µs ± 0%    56.0µs ±16%   +34.70%  (p=0.008 n=5+5)

GCD1000x10000/WithXY-64               154µs ± 0%     302µs ±29%   +96.15%  (p=0.008 n=5+5)

GCD1000x100000/WithoutXY-64           240µs ± 0%     268µs ±11%   +11.64%  (p=0.016 n=4+5)

GCD1000x100000/WithXY-64             2.51ms ±60%   9.28ms ±197%  +270.21%  (p=0.032 n=5+5)

GCD10000x10000/WithoutXY-64           545µs ± 0%     549µs ± 0%    +0.69%  (p=0.008 n=5+5)

GCD10000x10000/WithXY-64             1.20ms ± 0%    1.20ms ± 1%      ~     (p=0.222 n=5+5)

GCD10000x100000/WithoutXY-64         1.40ms ± 0%    1.67ms ±27%   +19.66%  (p=0.008 n=5+5)

GCD10000x100000/WithXY-64            11.7ms ± 9%    11.4ms ±14%      ~     (p=0.841 n=5+5)

GCD100000x100000/WithoutXY-64        38.8ms ± 0%    38.8ms ± 0%      ~     (p=0.690 n=5+5)

GCD100000x100000/WithXY-64           84.9ms ± 0%    85.0ms ± 0%      ~     (p=0.905 n=5+4)

Hilbert-64                           1.93ms ± 0%    1.93ms ± 1%      ~     (p=1.000 n=4+5)

Binomial-64                          3.42µs ± 2%    3.59µs ± 8%      ~     (p=0.095 n=5+5)

QuoRem-64                            4.55µs ± 0%    4.55µs ± 0%      ~     (p=0.659 n=5+5)

Exp-64                               17.1ms ± 0%    17.1ms ± 0%      ~     (p=0.841 n=5+5)

Exp2-64                              17.1ms ± 0%    17.1ms ± 0%      ~     (p=0.421 n=5+5)

Bitset-64                            20.8ns ± 0%    20.8ns ± 0%      ~     (p=1.000 n=5+5)

BitsetNeg-64                          105ns ± 0%     106ns ± 1%      ~     (p=0.238 n=4+5)

BitsetOrig-64                         231ns ±20%     183ns ± 2%   -20.92%  (p=0.016 n=5+5)

BitsetNegOrig-64                      786ns ±49%     543ns ±32%      ~     (p=0.143 n=5+5)

[Xiangdong JI]

BTW. 'drop cache' is made, while setting cpu governor is not supported by the platform. 

Thanks.

[robert engels]

Take a look at https://github.com/golang/go/issues/24735

You probably want to increase benchtime to a larger value, even 10x.

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0e0e1779-bba6-4636-87c4-5f323933a4ea%40googlegroups.com.

[roger peppe]

I'd recommend watching this talk from Daniel Marti, which explores exactly this issue and might be relevant to you: https://www.youtube.com/watch?v=oE_vm7KeV_E

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/5d57b1f4-7e9e-4aa5-bc74-85fac11c00c8%40googlegroups.com.

[Xiangdong JI]

Thanks for the replying, tried -benchtime=Nx and the results appeared a little stable now.

On Wednesday, January 8, 2020 at 10:19:25 AM UTC+8, Xiangdong JI wrote: