Including custom measurements in Macrobenchmark

[martin....@gmail.com]

Hello,

I'm using the following structure for a Macrobenchmark:

public class MyBenchmark extends Benchmark {

  @Param({ "10", "30", "50", "70", "90", "110" })
  private int n;

  @Param
  CalcType type;
  public enum CalcType {
    CPU, GPU
  };

  @Override
  protected void setUp() throws Exception {
    // ...
  }

  @Override
  protected void tearDown() throws Exception {
    // ...
  }

  @Macrobenchmark
  public void timeCalculate() {
    doBenchmark(0);
  }

  public int doBenchmark(int sum) {
    switch (type) {
      case CPU:
        sum = doCPU(sum);
        break;
      case GPU:
        sum = doGPU(sum);
        break;
      default:
        break;
    }
    return sum;
  }

  private int doCPU(int sum) {
    try {
      sum += ToolRunner.run(new KMeans(false), null);
    } catch (Exception e) {
      e.printStackTrace();
    }
    return sum;
  }

  private int doGPU(int sum) {
    try {
      sum += ToolRunner.run(new KMeans(true), null);
    } catch (Exception e) {
      e.printStackTrace();
    }
    return sum;
  }

  private class Example extends Configured implements Tool {
    private boolean useGPU;

    public Example(boolean useGPU) {
      this.useGPU = useGPU;
    }

    @Override
    public int run(String[] arg0) throws Exception {
      if (useGPU) {
        // run GPU Program with parameter n
      } else {
        // run CPU Program with parameter n
      }
      return 0;
    }
  }

  public static void main(String[] args) {
    CaliperMain.main(MyBenchmark.class, args);
  }
}

I'm executing this Marcobenchmark with the following command:

java -Xmx4G -javaagent:../../../lib/allocation-instrumenter-2.1.jar -jar KMeans-Benchmark.jar  --time-limit 600s --instrument macro -Cinstrument.macro.options.measurements=5 -Cinstrument.macro.options.warmup=30s

 Class loading breakage: Will not be able to instrument JDK classes
 Experiment selection:
 Instruments:   [macro]
 User parameters:   {n=[10, 30, 50, 70, 90, 110], type=[CPU, GPU]}
 Virtual machines:  [default]
 Selection type:    Full cartesian product
 This selection yields 12 experiments.

But I'm always observing the following warnings:

Starting experiment 1 of 12: {instrument=macro, method=timeCalculate, vm=default, parameters={k=10, type=CPU}}
WARNING: Hotspot compilation occurred during timing. Depending on the scope of the benchmark, this might significantly impact results. Consider running with a longer warmup.
WARNING: Hotspot compilation occurred after warmup, but outside of timing. Depending on the scope of the benchmark, this might significantly impact results. Consider running with a longer warmup.

1) May I ignore these warnings or should I use some other configuration?
Usually my applications are running 20 seconds and more.

2) How can I add custom measurements in caliper?
My applications do a more fine grained time measurement and I want to add these results to the measurement output.
Maybe I have to use an other caliper method? (not timeCalculate)

e.g., my program is measuring serialization, execution and deserialization time and I want to include these values in measurements? (like magnitude of timeCalculate)

Something like this?
    @Override
    public int run(String[] arg0) throws Exception {
      Results r;
      if (useGPU) {
        r = runGPUProgram(n);
      } else {
        r = runGPUProgram(n);
      }
      CaliperMeasurement.addValue("serializationTime", r.getSerializationTime());
      return 0;
    }

Thanks in advance, Martin

[Gregory Kick]

From the looks of your invocation (specifically the 600s time limit), it seems that you're probably executing a pretty large amount of code.  Depending on the structure, it's could be quite likely that the compilation threshold (10,000 invocations by default) for some corner of the code is being hit long into execution.  If you run with -verbose you can see exactly what is being compiled and make a determination about whether it is likely to impact your measurements.  Otherwise, you'll just have to increase the warmup even further.

 

2) How can I add custom measurements in caliper?
My applications do a more fine grained time measurement and I want to add these results to the measurement output.
Maybe I have to use an other caliper method? (not timeCalculate)

e.g., my program is measuring serialization, execution and deserialization time and I want to include these values in measurements? (like magnitude of timeCalculate)

Something like this?
    @Override
    public int run(String[] arg0) throws Exception {
      Results r;
      if (useGPU) {
        r = runGPUProgram(n);
      } else {
        r = runGPUProgram(n);
      }
      CaliperMeasurement.addValue("serializationTime", r.getSerializationTime());
      return 0;
    }

Thanks in advance, Martin

The current solution for adding a custom measurement is the @ArbitraryMeasurment annotation.  It allows you emit custom data from an independent experiment - it must be different from timeCalculate.  At the moment, each new measurement requires a new @ArbitraryMeasurement method.  If that is particularly prohibitive, please file a feature request.

--
--
guava-...@googlegroups.com
Project site: http://caliper.googlecode.com
This group: http://groups.google.com/group/caliper
 
This list is for general discussion.
To report an issue: http://code.google.com/p/caliper/issues/entry
To get help: http://stackoverflow.com/questions/ask (use the tag "caliper")
 
---
You received this message because you are subscribed to the Google Groups "caliper" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caliper+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hope this helps.

--
Greg Kick
Java Core Libraries Team

[mar...@illecker.at]

Hello,

thanks for your response!

The current solution for adding a custom measurement is the @ArbitraryMeasurment annotation.  It allows you emit custom data from an independent experiment - it must be different from timeCalculate.  At the moment, each new measurement requires a new @ArbitraryMeasurement method.  If that is particularly prohibitive, please file a feature request.

As far as I can see, it's only possible to return one value by ArbitraryMeasurment [1]?
Would it be possible to return multiple measurements by the @ArbitraryMeasurment annotation?
Or how could I implement the following benchmark?
e.g.,
  @ArbitraryMeasurment
  public double[] arbitraryBenchmark() {
    Result res = doBenchmark();
    return new double[] { res.serializationTime(), res.executionTime, res.deserializationTime };
  }

Please can you give me a short example of an ArbitraryMeasurment?

Thanks!

Martin

[1] http://code.google.com/p/caliper/source/browse/caliper/src/main/java/com/google/caliper/model/ArbitraryMeasurement.java

[gak]

On Monday, February 10, 2014 7:12:05 AM UTC-6, mar...@illecker.at wrote:

Hello,

thanks for your response!

The current solution for adding a custom measurement is the @ArbitraryMeasurment annotation.  It allows you emit custom data from an independent experiment - it must be different from timeCalculate.  At the moment, each new measurement requires a new @ArbitraryMeasurement method.  If that is particularly prohibitive, please file a feature request.

As far as I can see, it's only possible to return one value by ArbitraryMeasurment [1]?

That is correct. You can have multiple arbitrary measurements, but not all returned from the same experiment.

 

Would it be possible to return multiple measurements by the @ArbitraryMeasurment annotation?

It would be possible, but would require a new (or modified) API.  If this is important to you, please file a feature request at https://code.google.com/p/caliper/issues/list .

 

Or how could I implement the following benchmark?
e.g.,
  @ArbitraryMeasurment
  public double[] arbitraryBenchmark() {
    Result res = doBenchmark();
    return new double[] { res.serializationTime(), res.executionTime, res.deserializationTime };
  }

Please can you give me a short example of an ArbitraryMeasurment?

https://code.google.com/p/caliper/source/browse/examples/src/main/java/examples/CompressionSizeBenchmark.java