"Small" bits of code

[Brian Harris]

Could someone talk a little more about the notion a "small" bit of code, mentioned in the first sentence here https://code.google.com/p/caliper/wiki/JavaMicrobenchmarks

I'd like to benchmark a message sender with a big message, and this might take a half second. There's a lot of code that will be exercised when sending this message. I wouldn't call it "small". I'll stub the transport of course.

Does something break down when testing a "larger" operation that takes a bit "longer" than usual?

Thanks,

Brian

[Brian Harris]

I also have cases where I'd be benchmarking an operation that involves a "large" amount of code, but it only takes a few millis. Wondering if Caliper is appropriate for that too.

[Kevin Bourrillion]

To be honest, I don't really know how suitable caliper is for testing larger amounts of code.  It's just a different beast from the beast we use Caliper to slay.  I suspect that if I had perfect knowledge of the relevant concerns for benchmarking larger amounts of code it would probably lead to the creation of a separate "MacrobenchmarkInstrument" which would both do more reasonable things and be more convenient to use (i.e., no reason to make you code the for loop yourself in each and every time() method anymore, etc.).

When micro-benchmarking we can reason "by the time the performance of this method matters at all, we can be sure that hotspot will have compiled it and everything it depends on," and so in our benchmark we warm up until that happens.  For a large enough blob of code, that's probably not going to be true; there are operations you may want to make faster but not every part of the operation was compiled. How to deal with that in a benchmark? Things like that...

--
guava-...@googlegroups.com
Project site: http://caliper.googlecode.com
This group: http://groups.google.com/group/caliper
 
This list is for general discussion.
To report an issue: http://code.google.com/p/caliper/issues/entry
To get help: http://stackoverflow.com/questions/ask (use the tag "caliper")

--

Kevin Bourrillion | Java Librarian | Google, Inc. | kev...@google.com

[Jesse Wilson]

My own threshold is 100 milliseconds. If each rep takes longer than 100 milliseconds, it's outside of the scenario that Caliper was designed for. Beyond that, Caliper gets grumpy: it won't aggregate reps and becomes susceptible to noise.

The upcoming new Caliper will have a better story here.

[Gregory Kick]

To get a little more technical about it, the compile threshold is probably the biggest the problem.  Since Caliper can't really know what code you're intending to microbenchmark, the best it can do is figure out whether or not _something_ has compiled.  So, the microbenchmark instrument will warm up for some amount of time, but since your operations take so long, it's unlikely that the top-level method will actually be invoked the requisite (10,000 by default) number of times for it to be JIT'd.  Then, you'll have compilation during the middle of your timing which means that you'll probably just have some ugly, bi-modal data.

So, if you were to increase the warm up time, probably decrease the compile threshold and, unless this operation doesn't allocate, increase the heap by a whole ton, you might might be able to get some useful results.

Hope that helps.

On Thu, Dec 6, 2012 at 10:50 AM, Jesse Wilson <je...@swank.ca> wrote:

My own threshold is 100 milliseconds. If each rep takes longer than 100 milliseconds, it's outside of the scenario that Caliper was designed for. Beyond that, Caliper gets grumpy: it won't aggregate reps and becomes susceptible to noise.

The upcoming new Caliper will have a better story here.

--

guava-...@googlegroups.com
Project site: http://caliper.googlecode.com
This group: http://groups.google.com/group/caliper
 
This list is for general discussion.
To report an issue: http://code.google.com/p/caliper/issues/entry
To get help: http://stackoverflow.com/questions/ask (use the tag "caliper")

--
Greg Kick
Java Core Libraries Team