GSoC 2015 - progress report

[Krzysztof Janosz]

Hello,

As of today following tasks has been resolved:
- https://github.com/scalameter/scalameter/issues/98
Looks like it was already fixed in master. See:
https://github.com/scalameter/scalameter/issues/98#issuecomment-105992708
- https://github.com/scalameter/scalameter/issues/94
- examples of ScalaMeter usage and travis build for them
Done, although one PR for example with separate configuration is waiting.

There is one last warmup task - automation of release process.
I think I should be able to make a PR tomorrow.
Also, one thing had come to my mind. What about adding some kind of
automation script for scalmeter-examples repo?
I mean version bumping, which currently would be a pain to update n sbt
build files on ScalaMeter update, and git tagging? This would allow to
clearly know when examples were valid for specific ScalaMeter version.

Regards,
Chris

[Aleksandar Prokopec]

On 5/29/2015 8:01 PM, Krzysztof Janosz wrote:
> Hello,
>
> As of today following tasks has been resolved:
> - https://github.com/scalameter/scalameter/issues/98
> Looks like it was already fixed in master. See:
> https://github.com/scalameter/scalameter/issues/98#issuecomment-105992708
> - https://github.com/scalameter/scalameter/issues/94
> - examples of ScalaMeter usage and travis build for them
> Done, although one PR for example with separate configuration is waiting.
>
> There is one last warmup task - automation of release process.
> I think I should be able to make a PR tomorrow.

Nice work, I left a few minor comments in your PR!

> Also, one thing had come to my mind. What about adding some kind of
> automation script for scalmeter-examples repo?
> I mean version bumping, which currently would be a pain to update n sbt
> build files on ScalaMeter update, and git tagging? This would allow to
> clearly know when examples were valid for specific ScalaMeter version.

It sounds good to me, although it would be good if this script ran
directly from the main scalameter repo along with the release.
Perhaps a separate release step could run one script to verify that the
scalameter-examples compile, and then another release step could do the
versioning in -examples after ScalaMeter is published.

> Regards,
> Chris
>

--
Aleksandar Prokopec

[Krzysztof Janosz]

>> Also, one thing had come to my mind. What about adding some kind of
>> automation script for scalmeter-examples repo?
>> I mean version bumping, which currently would be a pain to update n sbt
>> build files on ScalaMeter update, and git tagging? This would allow to
>> clearly know when examples were valid for specific ScalaMeter version.
>
> It sounds good to me, although it would be good if this script ran
> directly from the main scalameter repo along with the release.
> Perhaps a separate release step could run one script to verify that the
> scalameter-examples compile, and then another release step could do the
> versioning in -examples after ScalaMeter is published.
>

Wouldn't this require that scalameter-examples must be a git submodule
for scalameter? Could you elaborate more on this approach, please?


Currently releasing of version should trigger scalameter-examples
compilation in travis.
So, what do you think about such approach:
- modify tools/builds_examples.sh to read version.sbt and send it as
environment variable before requesting example rebuild
- in scalameter-examples instead of running sbt $TASKS for each
$TEST_DIR do version check
- if there is a mismatch do nothing
- if versions match pass $TASKS to sbt
- use something like https://github.com/dmakhno/travis_after_all which
will do actual version bumping, tagging and push if necessary
This solution seems to be pretty straighforward although it's not a
clean one, and I also haven't tested it yet.

[Aleksandar Prokopec]

Not necessarily a submodule. I'd instead just write a bash script that creates a tmpdir, checks out scalameter-examples there, does the versioning and branching. Invoking this script could be a separate custom release step in the scalameter build.

But I don't feel strongly, if you want to suggest some different approach.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Aleksandar Prokopec]

On 5/31/2015 10:23 PM, Aleksandar Prokopec wrote:

Not necessarily a submodule. I'd instead just write a bash script that creates a tmpdir, checks out scalameter-examples there, does the versioning and branching. Invoking this script could be a separate custom release step in the scalameter build.

But I don't feel strongly, if you want to suggest some different approach.

On May 31, 2015 10:07:39 PM GMT+02:00, Krzysztof Janosz <janosz.k...@gmail.com> wrote:

 Also, one thing had come to my mind. What about adding some kind of
 automation script for scalmeter-examples repo?
 I mean version bumping, which currently would be a pain to update n sbt
 build files on ScalaMeter update, and git tagging? This would allow to
 clearly know when examples were valid for specific ScalaMeter version.
 
 It sounds good to me, although it would be good if this script ran
 directly from the main scalameter repo along with the release.
 Perhaps a separate release step could run one script to verify that the
 scalameter-examples compile, and then another release step could do the
 versioning in -examples after ScalaMeter is published.



Wouldn't this require that scalameter-examples must be a git submodule
for scalameter? Could you elaborate more on this approach, please?

I missed this part when reading the e-mail on my phone.

Currently releasing of version should trigger scalameter-examples
compilation in travis.
So, what do you think about such approach:
- modify tools/builds_examples.sh to read version.sbt and send it as
environment variable before requesting example rebuild

Sounds good.

- in scalameter-examples instead of running sbt $TASKS for each
$TEST_DIR do version check
- if there is a mismatch do nothing
- if versions match pass $TASKS to sbt
- use something like https://github.com/dmakhno/travis_after_all which
will do actual version bumping, tagging and push if necessary
This solution seems to be pretty straighforward although it's not a
clean one, and I also haven't tested
it yet.

To reduce complexity, I would actually recommend that you put version.sbt in the top-level directory,
and use sbt-release again for version bumping, just as you did in scalameter.

You could just use vanilla sbt-release functionality - I don't think branches for versions are necessary in scalameter-examples,
so we could just tag versions.
This would mean that you wouldn't have to implement the extras as you did in the scalameter repo.

Thoughts?

[Krzysztof Janosz]

>> - modify tools/builds_examples.sh <http://examples.sh> to read version.sbt and send it as

>> environment variable before requesting example rebuild
>>
> Sounds good.
>
>> - in scalameter-examples instead of running sbt $TASKS for each
>> $TEST_DIR do version check
>> - if there is a mismatch do nothing
>> - if versions match pass $TASKS to sbt
>> - use something like https://github.com/dmakhno/travis_after_all which
>> will do actual version bumping, tagging and push if necessary
>> This solution seems to be pretty straighforward although it's not a
>> clean one, and I also haven't tested
>> it yet.
>>
> To reduce complexity, I would actually recommend that you put
> version.sbt in the top-level directory,
> and use sbt-release again for version bumping, just as you did in
> scalameter.
>
> You could just use vanilla sbt-release functionality - I don't think
> branches for versions are necessary in scalameter-examples,
> so we could just tag versions.
> This would mean that you wouldn't have to implement the extras as you
> did in the scalameter repo.
>
> Thoughts?
>
>

I've started writing solution and it's based on your proposal.
I think it can be actually more simple and safe than using my initial idea.
However, I'm not writing bash script, only just release step using
sbt.IO and sbt.Process API. So it will look like:
- clone examples repo
- loop over directories and swap content of sbt.version to releaseVersion
- tag using sbt-release git api
- loop over directories and swap content of sbt.version to nextVersion
- push changes using sbt-release git api
I cant change it and add global version.sbt in examples although I think
it's not so good idea, because it's making examples less independent and
less readable.

[Aleksandar Prokopec]

That sounds good.
Btw, when you say sbt.version above, you mean some version file that would contain the version of the scalameter artifact?
My understanding is that, in examples, you don't need to version the examples projects themselves (since they will not be published to Maven), but instead version the scalameter library dependency that they use.

[Krzysztof Janosz]

Ouch, silly mistake: sbt.version -> version.sbt.
You're right, currently examples are using version.sbt both for
unnecessary versioning and for setting scalameter artifact version.
I was doing it to avoid definition of custom setting key for ScalaMeter
artifact versioning.
I can make a PR and change version.sbt content from `version in
ThisBuild := "xxx"` to something like `scalaMeterVersion in ThisBuild :=
"xxx"` and define this setting key somewhere - in version.sbt or in
build.sbt.

[Krzysztof Janosz]

Since last week I've finished rest of warmup tasks, which was mainly
automating release process - https://scalameter.github.io/home/releasing/
I've started also implementing binary compatible persistor(s) as
outlined in proposed solution.
I've already coded some solution that works, but it's a little bit messy
so I must refactor it first and then it should be more appropriate for a PR.

Regards,
Chris

[Aleksandar Prokopec]

That sounds great!
Let me know if you need help with anything.

--
Aleksandar Prokopec

[Krzysztof Janosz]

I'm finishing up binary compatible json persistor.
As you advised, I've modified Gen to include Manifest(s) in Parameters
and later I'm serializing them.
This and optional annotating of any class inheritance should give enough
flexibility. One thing to consider is that Jackson does not support
serializing of class hierarchy that include object, given e.g.
> trait T
> case class A extends T
> object B extends T
and serializing:
> Parameters(ListMap("xyz" -> List[T](A() :: B :: Nil)))
Jackson would complain that it does not know subtypes of T. There is
annotation @JsonSubtypes which accepts Array[Class[_]] but typing
B.getClass for B object won't work.

I must basically refactor code now.
As of PR, would you prefer to split it into smaller ones (e.g changes to
Parameters and Key class as separate PR) or keep one big PR with several
commits?

Regards,
Chris

[janosz.k...@gmail.com]

Now I'm fixing bug in custom picklers.
As soon as I finish I will add test for them and reopen the https://github.com/scalameter/scalameter/pull/119
I will continue my work, as it's written in https://github.com/scalameter/scalameter/pull/105#issuecomment-113155300
Since I've finished all classes on my university - I have only master thesis left - I am able to spend more time on GSoC now.

Regards,
Chris

[Aleksandar Prokopec]

That's great, Chris!
Looking forward to the PR.

Given that the time is limited, perhaps it would be more prudent to
merge boxing and method invocation counters into one step, and do the
more general thing, as you suggested in the doc at one point. We can
talk about that after you're done with the Persistor part.

--
Aleksandar Prokopec

[Krzysztof Janosz]

I've benchmarked all persistors with regards to execution time.
I've chosen suggested benchmark:
https://github.com/scala-blitz/scala-blitz/blob/master/src/test/scala/org/scala/optimized/test/scalameter/benchmarks.scala
I've measured only saving and loading time (no execution time of whole
benchmark) using ScalaMeter inline benchmarking with 20 benchmark runs
and default Warmer.

java serialization: ~2.95 ms
java deserialization: ~3.16 ms
json serialization: ~0.85 ms
json deserialization: ~0.88 ms
json gz serialization: ~1.21 ms
json gz deserialization: ~1.09 ms

[Aleksandar Prokopec]

Those numbers look pretty good.
How big were the files that are (de)serialized?

--
Aleksandar Prokopec

[Krzysztof Janosz]

12.9 kB for largest (java serialization), 15.8 kB for json nad 4.4 kB
for gzip json.


I've made also some artificial benchmark now (just zip(listA, listB)
over large enough lists) and for it results are as follows:
java serialization: ~79.41 ms
java deserialization: ~46.83 ms
json serialization: ~9.51 ms
json deserialization: ~9.46 ms
json gzip serialization: ~17.20 ms
json gzip deserialization: ~11.00 ms

file sizes:
java - 76.8 kB
json - 72.6 kB
json.gz - 18.6 kB

[Aleksandar Prokopec]

Ok, this sounds reasonable, and much better than what we had before.

[Krzysztof Janosz]

Since Tuesday I'm working on method invocation count.
I have invesigated suggested solutions a little bit and I think that
I've come up to some conclusions.
I will update proposal gdoc today.

[Aleksandar Prokopec]

That sounds great, Chris. Make sure you spend as much time as necessary
on investigating the assumptions and hacking up the proof-of-concept
solution for the invocation counter - the more time you spend on that,
the less time you will spend backtracking later.

Btw, make sure that you update the Planning document on GitHub.

--
Aleksandar Prokopec

[Krzysztof Janosz]

I've finished design phase of method invocation counting.
Since mid of a week I've started to make PRs with an actual
implementation that is mostly prepared - I just need to finish
documentation and add the ClassPath class.

Regards,
Chris

[Aleksandar Prokopec]

That sounds great!
I will take a look at your latest PR tomorrow evening!

[Krzysztof Janosz]

I got a little delay with an actual measurer implementation, but now I
think that everything is on a good track - last PR from the core
implementation is on Github.
After it I have 2/3 tasks left that I have mainly prepared and they are
almost ready for the review process.
Making an example and documentation should be a quick task, and I think
I will be able to do it tomorrow evening or on Sunday.

Regards,
Chris

[Aleksandar Prokopec]

That is awesome - I am very happy with the way things are progressing!

I didn't have a chance to review your latest PR today, but I should be
able to do so tomorrow.

As a side note: just remember that the most important thing is to
implement a robust solution. Even if you end up falling behind the
schedule, and aren't able to finish some work items, it's more important
to do a solid implementation on the ones you are tackling.
For example, if we need to unexpectedly invest some time into
refactoring things like the context for the setup zipper (as you did),
it's worth doing that. This ensures smooth continued development in the
future.

Cheers,
Alex

--
Aleksandar Prokopec