Benchmarks not running

[Olivier Eymere]

Hello all,  I have been trying to get JUnitBenchmarks running but am not getting the results I expect.  The original test is an existing junit test using the @Test annotation.  Following the instructions added the following to the test:

    @Rule
    public TestRule benchmarkRun = new BenchmarkRule();
   
    @BenchmarkOptions(benchmarkRounds = 20, warmupRounds = 0)

The test compiles fine, no warnings or obvious issues.  However, when i run the test (in eclipse) the test still only executes one time.  There is no JUnitBenchmarks output in the console.  To illustrate I have a modified, very simple, test as follows:

import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TestRule;

import com.carrotsearch.junitbenchmarks.BenchmarkOptions;
import com.carrotsearch.junitbenchmarks.BenchmarkRule;

public class NotificationPerformanceTester {
    @Rule
    public TestRule benchmarkRun = new BenchmarkRule();
   
    @BenchmarkOptions(benchmarkRounds = 20, warmupRounds = 0)
    @Test
    public void sendNotify() throws Exception{
        System.out.println("**** Running ****");
        Thread.sleep(1000);
    }
}

Since I have set the number of rounds to 20 I would expect to see "**** Running ****" printed in the console 20 times and i am expecting to see some performance statistics from JUnitBenchmarks.  When I run the test I only see the expected text printed once, no JUnitBenchmarks output and the test time is around 1.005 s.  I imagine I am missing something simple but I'm not sure what that is, does anyone see something that I am missing?

Regards.

[Dawid Weiss]

I've just tried and your example works for me just fine (it does print
Running 20 times). Which version of JUnit are you using? What IDE?

Dawid

> --
> You received this message because you are subscribed to the Google Groups
> "JUnitBenchmarks: Performance Benchmarking for JUnit4" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to junit-benchmar...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Olivier Eymere]

Thanks for testing it out.  I am using eclipse 4.3 and JUnit 4.9.  I can't think of anything that is notably unique about my setup.  I also tried adding 'extends AbstractBenchmark' instead of using the @Rule annotation with the same results.  The puzzling part is that the classes are imported and the class compiles.  I will also try running this via maven instead of in the IDE just to to validate.

[Dawid Weiss]

Can you put a breakpoint inside your test method, then dump the stack
trace? I want to see what's running the test class.

Dawid

[Olivier Eymere]

Here is the stack trace.  Unless I am missing something I don't see any of the JUnit-Benchmarks classes here.  That makes sense considering my results.  I don't know why that should be.

Thread [main] (Suspended (breakpoint at line 18 in NotificationPerformanceTest))   
    NotificationPerformanceTest.sendNotify() line: 18   
    NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]   
    NativeMethodAccessorImpl.invoke(Object, Object[]) line: 60   
    DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 37   
    Method.invoke(Object, Object...) line: 611   
    TestMethodRunner.executeMethodBody() line: 99   
    TestMethodRunner.runUnprotected() line: 81   
    TestMethodRunner(BeforeAndAfterRunner).runProtected() line: 34   
    TestMethodRunner.runMethod() line: 75   
    TestMethodRunner.run() line: 45   
    TestClassMethodsRunner.invokeTestMethod(Method, RunNotifier) line: 66   
    TestClassMethodsRunner.run(RunNotifier) line: 35   
    TestClassRunner$1.runUnprotected() line: 42   
    TestClassRunner$1(BeforeAndAfterRunner).runProtected() line: 34   
    TestClassRunner.run(RunNotifier) line: 52   
    JUnit4TestClassReference(JUnit4TestReference).run(TestExecution) line: 45   
    TestExecution.run(ITestReference[]) line: 38   
    RemoteTestRunner.runTests(String[], String, TestExecution) line: 460   
    RemoteTestRunner.runTests(TestExecution) line: 673   
    RemoteTestRunner.run() line: 386   
    RemoteTestRunner.main(String[]) line: 196   

[Dawid Weiss]

Olivier,

this seems to be some seriously outdated junit runner you're using. --
the TestClassRunner has been deprecated in junit 4.4. You should be
using junit >= 4.10. In short, all the rules and annotations get
ignored by that runner so there is no way for JUnitBenchmarks to work.

Either you have an ancient Eclipse or you have an old junit in
classpath and Eclipse downgrades to that.

Dawid

[Olivier Eymere]

That was it.  There was an old Junit in the classpath which eclipse was picking up.  Thanks for the help.