Compare two runs directly (i.e. output % increase/decrease from 2 runs)

[Matthew de Detrich]

I am currently using ScalaMeter to compare different API's and the performance of the API's. The test currently is very simple, since the API's are segregated, however I was wondering if there is a way to report the % difference between runs (instead of having to manually read the output of the runs in the logs and divide 2 numbers to provide a percentage).

Like in terms of code, something like

  lazy val a = performance of "API ONE" in {


    measure method "A" in {
      using(strings) in {
        s => apiOne.someCall
      }
    }
  }




  lazy val b = performance of "API TWO" in {


    measure method "A" in {
      using(strings) in {
        s => apiTwo.someCall
      }
    }
  }

  compareResult = compare(a,b)

Does something like this exist?

[Aleksandar Prokopec]

That's a fair feature request - it could be useful to have.
At the moment, there is nothing that compares this.

I would have to think a little bit about what's the easiest way to include it into ScalaMeter's existing model.
I see several options:
1) Augmenting the HtmlReporter, so that users can customize the report - e.g. define new curves based on existing ones.
2) Adding a new kind of a benchmark snippet, that is not defined in terms of a snippet, but in terms of other snippets.
3) Adding a postprocessor step (invoked after Executor and before Reporter) that allows adding new curves in terms of existing ones.

I'd almost be inclined to go for 3.
We will be doing a major Measurer reorganization, and 3 could fit well into that.

[Trevor Hartman]

+1, I would find a compare feature very useful.

[Aleksandar Prokopec]

See the following commit for an example of how to achieve this:

https://github.com/scalameter/scalameter/commit/3bdcd313dc085e08b5050e0bb713cd3fad37fc77

(this commit does validation on each curve separately, but it is straightforward to modify this to a reporter that compares two curves)