How to actually limit the number of executions?

[phill...@gmail.com]

Hi,

I've read the docs and it seems that these four options will limit the number of executions:

exec.minWarmupRuns
exec.maxWarmupRuns
exec.benchRuns
exec.independentSamples

However, I've set them to the following:

exec.minWarmupRuns -> 5
exec.maxWarmupRuns -> 5
exec.benchRuns -> 15
exec.independentSamples -> 1

I would expect this, based on the documentation, to run the code precisely 20 times - 5 of which are warmup. I am 'using' Gen.single. 

However, it consistently runs the 'in' block 83 times.

Could someone explain what is happening? I would like the code tested only 20 times.

Thanks for any help.

[bertra...@gmail.com]

A test should normally run 5 iterations of warmup, followed by at least 15 benchmark measurements.

Above, the "at least" part means that a ScalaMeter Measurer can decide to run the benchmark snippet more often.

In particular, the OutlierElimination measurer (https://github.com/scalameter/scalameter/blob/master/scalameter-core/src/main/scala/org/scalameter/Measurer.scala#L232) is used in regression detection, and is enabled by default in certain configurations.

This measurer exists to provide more stable results, so that a performance regression gets reported only if there is a real regression, and not just random jitter.

My guess is that, what might be going on, is that the OutlierElimination measurer is repeating some measurement.

If you turn the verbose -> true configuration option on, you might be able to see this.

In this case, the solution is to override the measurer, like here:

https://github.com/scalameter/scalameter-examples/blob/master/offline-regression-report/src/bench/scala/org/scalameter/examples/ListBenchmark.scala#L24

In your case, you would have to return "new Measurer.Default" (https://github.com/scalameter/scalameter/blob/master/scalameter-core/src/main/scala/org/scalameter/Measurer.scala#L114).

Also, if you provide a runnable example, it would be easier to give you an exact answer.