Integrating Caliper into Maven Structure & Output

[Michael Nitschinger]

Hey folks,

I've been quite happy with caliper and thanks to the new tutorial, its quite easy to get started.

The other thing I'm wondering about is how to best integrate caliper tests into the "daily" workflow. I know that you can run it from the command line with mvn exec plugin and this is fine, but I also want to use it in an automated way. I know that someone built a ci plugin (?), but I'm also more talking about doing the tests and then storing the results away (not uploading them(!)) for later analysis. 

Also, is it possible to integrate it into a "junit runner way", so you can get results directly from your IDE? If not thats not an issue, but It would be great somehow to "assert" certain performance requirements before passing the build, even locally.

The other thing thats more best practice: whats the common way to integrate perf tests with caliper into the codebase? I dont want to ship it as part of src/main, but maybe src/test/perf ? What do you guys use in your project?

Thanks for helping me out :)

Michael

[Gregory Kick]

On Tue, Jul 23, 2013 at 8:02 AM, Michael Nitschinger <michael.n...@gmail.com> wrote:

Hey folks,

I've been quite happy with caliper and thanks to the new tutorial, its quite easy to get started.

Glad to hear it.

 

The other thing I'm wondering about is how to best integrate caliper tests into the "daily" workflow. I know that you can run it from the command line with mvn exec plugin and this is fine, but I also want to use it in an automated way. I know that someone built a ci plugin (?), but I'm also more talking about doing the tests and then storing the results away (not uploading them(!)) for later analysis. 

Data is always being persisted into ~/.caliper, so the storing is happening automatically.  If you want to turn off uploading, just set results.upload.options.url to the empty string.  E.g. -Cresults.upload.options.url=

 

Also, is it possible to integrate it into a "junit runner way", so you can get results directly from your IDE? If not thats not an issue, but It would be great somehow to "assert" certain performance requirements before passing the build, even locally.

It's certainly possible.  At google, we do --dry-run runs as part of our unit testing just to make sure that the benchmarks are still compiling and running without exceptions.  I've thought about making a JUnit runner so that you could do @RunWith(CaliperDryRun.class) to that effect.  That would be a very welcome contribution if you're interested.

Actually doing performance runs is much trickier.  Your average CI setup is probably running a ton of jobs and constantly churning, so measuring runtime is pretty messy.  That said, if you can figure out the infrastructure, we hope to have two-dimensional visualizations for looking at performance over time (or whatever dimension you'd like) in the webapp.  It's still a ways off and the project has been a volunteer effort, so no guarantees on a timeframe.

 

The other thing thats more best practice: whats the common way to integrate perf tests with caliper into the codebase? I dont want to ship it as part of src/main, but maybe src/test/perf ? What do you guys use in your project?

We don't use maven internally, but I've been advocating src/benchmark/java.  It seemed like the logical place to me.

 

Thanks for helping me out :)

Michael

--
--
guava-...@googlegroups.com
Project site: http://caliper.googlecode.com
This group: http://groups.google.com/group/caliper
 
This list is for general discussion.
To report an issue: http://code.google.com/p/caliper/issues/entry
To get help: http://stackoverflow.com/questions/ask (use the tag "caliper")
 
---
You received this message because you are subscribed to the Google Groups "caliper" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caliper+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Greg Kick
Java Core Libraries Team