Benchmark times in version 7 versus version 4 of JUnitBenchmarks

[pich acri]

Hi,

I started experimenting with version 4 of Junit benchmarks about a year ago and found a significant difference in the benchmark results times when I performed the same experiments with version 7. I found out that in version 4 the @Before method is called only once before each benchmark test (not in every round) and the time spent in @Before method is not included in the final benchmark results for the test. When I recently started running version 7, I found that the @Before method is called in every round for a test and the time spent in this method is included in the final benchmark result. I find this version 7 implementation causes all my tests results to be very long as I am doing more work in the @Before method than in the actual test method (@Test). I really would like to only capture the time spent in the test method like in version 4. Can anyone comment on this major difference in behaviour or am I the only one experiencing this? Thank you. 

[Dawid Weiss]

Yeah... this could be a regression resulting from a patch I accepted
that moves the benchmarking code from @MethodRule to a @Rule. So the
change is caused by how JUnit implements these (the order of nested
rule calls). I will have to investigate -- filed this issue to track
this:

http://issues.carrot2.org/browse/JUNITBENCH-54

Dawid

> --
> You received this message because you are subscribed to the Google Groups
> "JUnitBenchmarks: Performance Benchmarking for JUnit4" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to junit-benchmar...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Dawid Weiss]

Unfortunately this isn't something that can be fixed in
JUnitBenchmarks. The rule order (and method order returned from java
reflection, for that matter) is not guaranteed. JUnit's MethodRule and
TestRule nesting is different so indeed there was a change-of-behavior
between 0.4.0 and further versions.

If your test case requires a setup use a trick and set up an explicit
rule chain that calls the benchmarking rule as the *last* rule. Then
you can inject an additional rule that will call your before and after
hooks once. This is shown here, for example:

https://github.com/carrotsearch/junit-benchmarks/blob/master/src/test/java/com/carrotsearch/junitbenchmarks/TestJUnitHooksWorkAround.java#L21

Dawid

[pich acri]

Thank you Dawid for your response. I will try out the trick with the TestJunitHooksWorkAround sample and see how it goes. Thanks a lot.