trials should be > 1 by default?

[Kevin Burton]

The number of independent measurements to take per benchmark scenario; a positive integer (default: 1). JIT is often non-deterministic. Performance-related decisions should almost always be based on data from multiple trials to account for that fact.

ok... so if it's "almost always" shouldn' the default be > 1 ...? say 10? 

[Kevin Bourrillion]

I'm not sure.  My benchmarking process involves a ton of iteration before I'm finally ready to make an official run that I'm going to "base performance-related decisions on", and I usually don't want to run multiple trials for all of those.  It's like there's a "development mode" and a "for real mode", which is a distinction that we long ago tried to formalize into two different sets of caliper default behaviors, but we had some concerns with how it worked out, because who are we to choose what are your best "for real" parameters?

On Fri, Jul 26, 2013 at 2:14 PM, Kevin Burton <burto...@gmail.com> wrote:

The number of independent measurements to take per benchmark scenario; a positive integer (default: 1). JIT is often non-deterministic. Performance-related decisions should almost always be based on data from multiple trials to account for that fact.

ok... so if it's "almost always" shouldn' the default be > 1 ...? say 10? 

--
--
guava-...@googlegroups.com
Project site: http://caliper.googlecode.com
This group: http://groups.google.com/group/caliper
 
This list is for general discussion.
To report an issue: http://code.google.com/p/caliper/issues/entry
To get help: http://stackoverflow.com/questions/ask (use the tag "caliper")
 
---
You received this message because you are subscribed to the Google Groups "caliper" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caliper+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Kevin Bourrillion | Java Librarian | Google, Inc. | kev...@google.com

[Kevin Burton]

I guess that's a good point.  I feel the same way.  Of course I think this is another argument to bring back the console output ;)

Kevin