ScalaMeter issues

[Aleksandar Prokopec]

I'm forwarding a part of the personal conversation to the mailing list
for documentation purposes.

> Essentially I have a few things that don't seem to work for me, though I
> tried to follow the instructions on the website as best as I could.
> I use Scala version 2.11.6 and sbt version 0.13.9.
>

Should work.

> I have tried to make a separate folder for Scalameter (following this
> example:
>
https://github.com/scalameter/scalameter-examples/blob/master/basic-with-separate-config/build.sbt
> ),
> i.e. I have appended the content of that build.sbt to my own build.sbt.
> However, I have found that my project would not compile any more. Several
> cleans/compiles did not help.
>

Should work, I have several setups like the one above.
In particular, changing your build that way should not break your main
code, which is then separate from the benchmark configuration.
The code in the bench configuration should compile with the command
`bench:compile`.

> So for now, I have the benchmarks in the test folder.

That is fine too.

> But I also have unit
> tests there which use ScalaTest. Then, even if I do test-only, the
-silent
> option causes the following error:
> java.lang.IllegalArgumentException: Argument unrecognized by ScalaTest's
> Runner: -silent

Looks like some sbt regression - it passes the arguments to all test
suites, including scalatest.
It sucks, but separate bench configuration should fix it.

> In combination with other tests, the options to ScalaMeter don't seem to
> work. Is there a way to use the silent option programmatically?
>

You might want to try to set the Key.verbose to false in your benchmark
configuration, at the top level.
That should be enough to eliminate standard output (assuming you really
need this, standard output can be useful to know what's going on).

> I have tried to extend LocalTime and ForkedTime. To my surprise,
ForkedTime
> measured longer running times, but even more surprisingly, out of three
> snippets computing the same thing, the fastest on LocalTime was not the
> fastest on ForkedTime (one snippet took about the same time, another's
> runtime differed by two orders of magnitude). So now I am unsure
about the
> validity.

It almost sounds like you should increase the minimum and maximum number
of warmup runs, to allow the separate JVM to warm up properly.

Also, make sure that whatever you measure is at least on the order of
milliseconds.

> (My tool, to which the snippets belong, will most likely be run on
separate
> JVMs.)
> I assume LocalTime uses the LocalExecutor, whereas ForkedTime uses the
> SeparateJvmsExecutor, is this correct?

Should be - quick search in ScalaMeter code tells me:

abstract class ForkedTime extends Forked[Double]

and:

abstract class Forked[U: Pickler: PrettyPrinter] extends Bench[U] {
def warmer: Warmer = new Warmer.Default

def aggregator: Aggregator[U]

def executor: Executor[U] = new execution.SeparateJvmsExecutor(
...

> Aggregator.average would make more sense for me instead of
Average.min, but
> when I copy the following config from the website (now extending Bench,
> just like on the web.):
> lazy val executor = LocalExecutor(
> new Executor.Warmer.Default,
> Aggregator.average,
> new Measurer.Default)
> lazy val reporter = new LoggingReporter
> lazy val persistor = Persistor.None
>
> I get the following compile error:
> diverging implicit expansion for type Ordering[T]
> [error] starting with method Tuple9 in object Ordering
> [error] Aggregator.min,
> [error] ^
> [error] one error found

The API changed somewhat, but the documentation didn't.
The min method now takes an implicit parameter:

object Aggregator {

def min[T: Ordering]: Aggregator[T] = Aggregator("min") {
_.minBy(_.value) }

def average: Aggregator[Double] = Aggregator("average") { xs =>
val units = xs.head.units
Quantity(utils.Statistics.mean(xs.map(_.value)), units)
}

}


You're encouraged to become a contributor by submitting a PR with a docs
fix.

Btw, I wouldn't use the `min` method when measuring time - you should
probably average it.

> Funny enough, when I change Aggregator.min to Aggregator.average the
> compile error becomes:
> No Pickler available for Double.
> [error] Please import org.scalameter.picklers.Implicits for existing
ones,
> [error] or define your own,
> [error] or import org.scalameter.picklers.noPickler._ and use
> SerializationPersistor.
> [error] lazy val executor = LocalExecutor(
> [error] ^
> [error] one error found
>

Error message tells you here what you need to do

import org.scalameter.picklers.Implicits._

Explanation:
This class now takes a pickler object for (de)serialization of historic
results. This changed in the latest version of ScalaMeter - we now have
picklers for more binary-compatible serialization:

class LocalExecutor[V: Pickler](val warmer: Warmer, val aggregator:
Aggregator[V])

package picklers {
object Implicits {
implicit val doublePickler = DoublePickler
...

> Then I tried this from an example from Mano:
> def executor = SeparateJvmsExecutor(
> new Executor.Warmer.Default,
> Aggregator.min,
> new Measurer.Default)
>
> def reporter = new LoggingReporter
> def persistor = Persistor.None
>
> which results in:
> diverging implicit expansion for type Ordering[T]
> [error] starting with method Tuple9 in object Ordering
> [error] Aggregator.min,
> [error] ^
> [error]
/home/eva/repo/daisy/src/test/scala/bench/AffineFormBench.scala:19:
> type mismatch;
> [error] found : org.scalameter.api.Measurer.Default
> [error] required: org.scalameter.Measurer[Nothing]
> [error] Note: Double >: Nothing (and
org.scalameter.api.Measurer.Default <:
> org.scalameter.Measurer[Double]), but trait Measurer is invariant in
type V.
> [error] You may wish to define V as -V instead. (SLS 4.5)
> [error] new Measurer.Default)
> [error] ^
> [error] two errors found
>

Same comment as before - min takes a type parameter - probably you want
Aggregator.min[Double].

But, I recommend using `average`.

> I will stop here. This is not urgent, and you are probably busy. But
if you
> can find a few minutes to tell me what I am doing wrong, that'd be great!
> Also, feel free to forward this mail to whoever you see fit.
>

Hope this helps!


--
Aleksandar Prokopec