Only one html report for all the scalameter tests

[Sébastien Mouthuy]

Hi all,

thank you for scalameter. It is a very useful framework :-)

I wrote several scalameter tests, however I cannot figure out how to create only one html report for all the scalameter tests.

Say I have two tests such as TimePerformanceTester and MemoryPerformanceTester with extend OnlineRegressionReport.

If I run the SBT test task, then both tests will be performed but only MemoryPerformanceTester will be outputed in the html report.

I know of the include method, but then by running the SBT test task, all the scalameter will be run twice.

Any advice on this ?

It would greatly help us for our continuous integration process.

Hope this message finds you well,

Sébastien

[Aleksandar Prokopec]

Hi!

You're welcome! :)

We do have support for something like this. Here:

http://scala-blitz.github.io/home/documentation/benchmarks//i7mac.html

you should be able to see that memory and running time tests are
produced jointly.

https://github.com/scala-blitz/scala-blitz/blob/master/src/test/scala/org/scala/optimized/test/scalameter/benchmarks.scala

It is true that this setup means that running 'test' runs your tests
twice, and that is ugly.

Possible work-arounds:
a) putting benchmark-related tests in one class, and using the
'testOnly' sbt command to run specific tests
b) putting benchmarks in a separate configuration (recommended), and run
them with 'bench:testOnly' - example here:

https://github.com/scalameter/scalameter-examples/blob/master/basic-with-separate-config/build.sbt

The general problem here is that a report gets generated after the test
completes. We could maybe work around this by introducing a new type of
a test class, e.g. ReportAggregator or smth like that, which uses the
persisted benchmark data to re-generate the html report.
Trouble is, it would be non-straightforward to ensure in sbt that it is
run after all real tests complete.

I'm open to suggestions, of course, if you have some other idea of how
to do this.

--
Aleksandar Prokopec

[Aleksandar Prokopec]

Hey, I had an inspiration the other day.

What if, in addition to the `PerformanceTest` class, we define a `PerformanceTestTemplate`.
This would be a class that can only:
a) be used to declare generators, and test snippets
b) used in the `include` statement

But, from SBT's perspective, it would not actually be a runnable test.
To use a `PerformanceTestTemplate`, one has to include it in a real `PerformanceTest`.
It seems to me that this would solve your problem.

Thoughts?

[Sébastien Mouthuy]

Hi,

that would be great.

Being able to define test snippets that are not run would be very useful.

I see two comments:

1) that would be great to be able to write 

include(myTestTemplate)

where myTestTemplate is an object, and not a class. Objects are great because there is less trouble with serialization when we use separate JVM's.

2) We should be able to create several reports. I personally need it because I have big projects. Now it seems this cannot be done; the report directory can only be indicated when launching the tests in sbt, which means I cannot create two different reports in my CI server.

object MyPerformanceReport extends PerformanceTest.OnlineReport{
  report of "MyTimePerformance" in "target/time-benchmarks"{
    include(MyFistExperiments)
    include(MySecondExperiments)
  }

  report of "MyMemoryPerformance" in "target/memory-benchmarks"{
    include(MyMemoryBenchmarks)
  }
}

Cheers, thank you for this great project,

seb

[Aleksandar Prokopec]

Hi - ScalaMeter 0.7 is now out. See comments below.

On Monday, 10 August 2015 12:31:02 UTC+2, Sébastien Mouthuy wrote:

Hi,

that would be great.

Being able to define test snippets that are not run would be very useful.

I see two comments:

1) that would be great to be able to write 

include(myTestTemplate)

where myTestTemplate is an object, and not a class. Objects are great because there is less trouble with serialization when we use separate JVM's.

PerformanceTest (now called Bench classes) are serializable, so they should not be more problematic than objects.

I found a more compact (and I think, more idiomatic) solution - instead of having a special BenchTemplate class, users now have to declare their 'template' benchmarks as traits.

trait MyTemplateBenchmark extends Bench {
  // ...
}

These templates can then be included in Bench.Group benchmarks:

object MyGroup extends Bench.Group {
  include(new MyTemplateBenchmark {})
}

 

2) We should be able to create several reports. I personally need it because I have big projects. Now it seems this cannot be done; the report directory can only be indicated when launching the tests in sbt, which means I cannot create two different reports in my CI server.

object MyPerformanceReport extends PerformanceTest.OnlineReport{
  report of "MyTimePerformance" in "target/time-benchmarks"{
    include(MyFistExperiments)
    include(MySecondExperiments)
  }

  report of "MyMemoryPerformance" in "target/memory-benchmarks"{
    include(MyMemoryBenchmarks)
  }
}

Yep, this should now work, although the syntax is a bit different and relies on existing 'config' functionality - see example project here:

https://github.com/scalameter/scalameter-examples/tree/master/include-statements
 

Cheers, thank you for this great project,

seb

Glad to hear it helps!

Cheers,
Alex
 

[Joshua Portway]

Hi Alex,

I've followed the example code you've posted for this, but I still can't get it to solve the original poster's problem of combining multiple benchmarks into one report. When I make a Bench.Group as in your example, if I include more than one Benchmark like this :

class Benchmarks extends Bench.Group {

  performance of "time" in {

    include(new ProtocolBufferBenchmarks {})

    include(new CorrelationBenchmark {})

  }

}

then only one of the included benchmarks is in the report - I assume that the other one gets overwritten.

Am I doing something wrong?

There seems to be a lack of documentation on what I would assume is quite a common requirement.

[Aleksandar Prokopec]

Hi Joshua,

You need to specify the reports.resultDir configuration value so that
ScalaMeter knows in which directory to place the report.
An example is in this source file:

https://github.com/scalameter/scalameter-examples/blob/master/include-statements/src/bench/scala/org/scalameter/examples/collection-benchmarks.scala

Unfortunately, there is no way to combine the reports at the moment -
you will end up with two separate (sub)directories and two reports.
The reason is that different Reporters for the included benchmarks could
report entirely different types of values (e.g. running time and a
method profiling table). It is not clear how to combine them.

It could be possible to write a new Reporter implementation that assumes
that all the included benchmarks use the HtmlReporter, and then
reads the data in their directories and somehow builds a new Html report.
This is not implemented in ScalaMeter right now.

Note that the `include` directive is specifically meant for this case -
you have multiple benchmark templates with different reporter types.
If they have the same reporter type (i.e. both ProtocolBufferBenchmarks
and CorrelationBenchmarks measure running time), it might be better to
mix them into the benchmark class directly.

class Benchmarks extends ProtocolBufferBenchmarks with CorrelationBenchmark

(didn't try it, but it should work)

--
Aleksandar Prokopec

[Joshua Portway]

Hi Aleksander,

Thanks so much for the quick reply. I tried this :

class Benchmarks extends ProtocolBufferBenchmarks with CorrelationBenchmark 

but it didn't really work because although it combined the reports into one it also combined all of the axes from the sub-benchmarks, so the X-axis is incorrect on the graphs.

I guess I will just have to generate separate reports.

For a beginner, the configuration of Scalameter seems very confusing and complex once you get beyond the simple case of subclassing one of the presets.

[Aleksandar Prokopec]

On Sunday, 13 September 2015 18:13:50 UTC+2, Joshua Portway wrote:

Hi Aleksander,

Thanks so much for the quick reply. I tried this :

class Benchmarks extends ProtocolBufferBenchmarks with CorrelationBenchmark 

but it didn't really work because although it combined the reports into one it also combined all of the axes from the sub-benchmarks, so the X-axis is incorrect on the graphs.

This might be a bug.
I'm not sure exactly what you mean by combining the axes from the sub-benchmarks - could you elaborate a bit more how the X-axis is incorrect?
 

I guess I will just have to generate separate reports.

For a beginner, the configuration of Scalameter seems very confusing and complex once you get beyond the simple case of subclassing one of the presets.

I understand - there are many configuration options one can change.
I guess we could make a flow diagram that leads the user to specific examples depending on what the user wants to do, and add it to the documentation on the ScalaMeter website.

[Joshua Portway]

Hi Aleksander,

Here's a screenshot of the report generated when I inherit from both test classes: 

The "numParticles" axis is generated by the "correlationForce.update" test, but it's being used on the "ProtocolBuffers" test, which means that the graph generated for "ProtocolBuffers" is meaningless.

Maybe I'm misunderstanding the way that ScalaMeter is supposed to be used - is the normal way to use it to have all of the benchmarks in a single class? Doesn't that get difficult to manage for anything except for small projects? If all the benchmarks aren't in a single class how is it possible to generate reports, since they all seem to overwrite each other?

thanks,

Josh

[Aleksandar Prokopec]

Hi Josh,

On Sunday, 13 September 2015 19:33:39 UTC+2, Joshua Portway wrote:

Hi Aleksander,

Here's a screenshot of the report generated when I inherit from both test classes: 

The "numParticles" axis is generated by the "correlationForce.update" test, but it's being used on the "ProtocolBuffers" test, which means that the graph generated for "ProtocolBuffers" is meaningless.

Did you try dragging the axis "size" (which I assume is for the ProtocolBuffers test) up to the top.
That should place the "size" parameter on the x-axis, and use the other parameter combinations to display distinct curves.
 

Maybe I'm misunderstanding the way that ScalaMeter is supposed to be used - is the normal way to use it to have all of the benchmarks in a single class?

This depends on what you need.
If the benchmarks are somehow related, then I would put them in the same class.
Alternatively, you can put them into traits which you then mixin together into the final benchmark class.

If the benchmarks are not related to each other (i.e. one measures time, the other one measures memory consumption, and the third one produces execution times of methods in your program), then you have to use separate benchmark classes for each of them.
These separate benchmark classes will generate separate reports in separate directories (see below).

If you want to group these separate benchmark classes into one big benchmark group class, then you should extend JBench.Group (as shown in the example below).
This will give you the benefit of running all the benchmarks included into that group class.

When using the JBench.Group class, the reports will still be generated in separate directories.
You will have separate html file with the report in each of these directories.
We could in the future implement special reporters that gather all the results from different directories, and combine them into one super-report.
How exactly would this super-reporter look and work is an open problem.
 

Doesn't that get difficult to manage for anything except for small projects?

In bigger projects, you should either put benchmarks into separate traits, or you should put them into separate benchmark classes.
 

If all the benchmarks aren't in a single class how is it possible to generate reports, since they all seem to overwrite each other?

They will not overwrite each other if you place them in separate subdirectories. Please follow the example in the link (scroll all the way to the bottom to see how to set result dir):

https://github.com/scalameter/scalameter-examples/blob/master/include-statements/src/bench/scala/org/scalameter/examples/collection-benchmarks.scala
 

[Jakub Liska]

I tried this model out, however the Html Report index.html : 

target/benchmarks/time/*

target/benchmarks/memory/*

target/benchmarks/report/index.html

shows just one of time or memory benchmarks, sort of non-deterministicaly.

So that I cannot create something like http://scala-blitz.github.io/home/documentation/benchmarks//i7mac.html

[Jakub Liska]

Sorry, forgot to paste the snippet : 

https://gist.github.com/l15k4/2af77341a4e20148626c890145bef2d7