Re: GSoC 2015 - progress report

[janosz.k...@gmail.com]

I'm doing now preparation step for a JMH frontend implementation.
As it was suggested, I've started from comparing both JMH and
ScalaMeter. I think that I should finish it and make available for a
review today or tomorrow.

PS:
Weird, looks like "replay all" button in my mail client didn't work properly this time.
To be consistent, I placed my post here and removed the other one (but topic is still on mailing list since I can't remove it)

Regards,
Chris

[Aleksandar Prokopec]

Sounds good.
Make sure that the comparison includes some code examples of doing
things in the two frameworks. Once you have it, let's have a meeting to
decide on the next steps.

Cheers,
Alex

--
Aleksandar Prokopec

[Krzysztof Janosz]

In the passing week I've prepared the doc with JMH and ScalaMeter
comparison.
As it was determined, I've started to work on proposal for the new
frontend (and other features).
Results are incrementally updated on that doc.

Regards,
Chris

[Krzysztof Janosz]

New frontend is already implemented and PR is during iteration phase.
I have also initial implementation of the throughput measurer (gist is
in the proposal doc). I think I will start poking around custom
classloader for the invocation count measurer from previous tasks and
also think of the last milestone.

Cheers,
Chris

[Aleksandar Prokopec]

Nice work!

We should double-think how we want to do the measurers.
If we make them typed, we should consolidate them with reporters as well.
Thinking of possible next steps, a Thyme-style profiler measurer is
something you could explore and play around with.

About reworking the measurers:
If we parametrize Measurer with a type parameter T, then, depending on
the measurer type, we should also consolidate different reporters (for
example, a Reporter[Double] can produce charts like our HtmlReporter
does, but something like Measurer.MethodProfiler would need a
Reporter[MethodProfile], which would show the profiling breakdown in
some sort of a table).
We should start with a doc before we start coding.

--
Aleksandar Prokopec

[Krzysztof Janosz]

From the new Java frontend milestone I've updated the documentation
recently so I guess that we should iterate over it in the next few days.

About the measurer reorganization - I'm still thinking about the best
way to achieve measurer composition.
I will update the proposal doc and ping you if I come up with some new
ideas.

[Aleksandar Prokopec]

Nice work! I reviewed your PR for the docs.

Yes, the measurer reorganization seems tricky.
My thoughts:
1. Measurers and Reporters should become typed - this would probably
mean that their surrounding test class also carries a type parameter
2. Measurers would benefit from a `map` method - i.e. it would be both
useful and straightforward to make them functors
3. I'm still not sure how much we would win if we add the `zip` method
to Measurer. There are some possible gains here, but it seems extremely
tricky to implement a general composition of two measurers. I'm not sure
if even the Runner/Gauge split solve all the problems we've identified.
4. I like that you identified and proposed the Quantity class. I would
still try to simplify it a bit:
- I would try to eliminate the second type parameter
- The quantity is essentially a pair of an arbitrary value and some
string that represents a unit. In your gist, you defined several
categories of quantities, along with constrained conversion within them.
I would instead try to keep it simple - the user should be able to
transform the quantity arbitrarily (there is even a use case for this -
if you measured 10 iterations of some loop, but want to display the
running time for 10000 iterations, then you really want to scale the
value, but keep the unit intact). I would argue for allowing users to do
something (more or less) like this:

val scaledMeasurer = originalMeasurer.map {
case (value, unit) => (value * 1000, unit)
}

Or, if the users want to use the stack profiling measurer, but map it to
the percentage of a particular method, they should be able to do
something like this:

val boxMeasurer = profileMeasurer.map {
case (profile, "") => (profile("BoxesRuntime.boxToInteger"), "%")

[Krzysztof Janosz]

I've made a PR with initial measurer reorganization changes. I've left
you a few comments to list possible caveats or questions.
Unfortunately it's quite big and I didn't find a way to split it - all
are necessary changes to compile the project successfully and pass the
tests.
I will now work on things listed in the TODO section in the proposal.

Btw, couple of days ago I've pushed changelog sketch to the
scalameter/planning. I will be updating it gradually.