Clojure: How you can help! (Hint: Alioth Benchmarks)

90 views
Skip to first unread message

Chris Redinger

unread,
Jun 23, 2011, 2:34:23 PM6/23/11
to cloju...@googlegroups.com
Now that we've shipped Clojure 1.3 Beta 1, we need to start moving other parts of the ecosystem towards a final release. One of the areas we'd really like to see completed is the Alioth Benchmarks [1].

Stuart Halloway and I have added a benchmark for the binary trees that should demonstrate how to contribute going forward.

The official repo can be found here: https://github.com/clojure/test.benchmark

Our approach was to start with the Java solution and do a direct port. Then, examine where we might have bottle-necks and improve. Repeat until we are on par with Java performance.

It would be fantastic if we could get all of the benchmarks to have compatible performance to the Java solutions when we ship 1.3. Having these benchmarks would be a great way to test (and demonstrate) our current performance with 1.3. And also provide a regression test against our performance in the future. 

Are you up for the challenge?


--
Christopher Redinger
Clojure/core

pmbauer

unread,
Sep 13, 2011, 7:09:35 AM9/13/11
to cloju...@googlegroups.com, ch...@clojure.com
I had a free day so took a crack at one of the alioth benchmarks.
Any feedback/criticism appreciated. http://dev.clojure.org/jira/browse/TBENCH-8

Stuart Halloway

unread,
Sep 13, 2011, 8:05:56 AM9/13/11
to cloju...@googlegroups.com
I had a free day so took a crack at one of the alioth benchmarks.
Any feedback/criticism appreciated. http://dev.clojure.org/jira/browse/TBENCH-8

Hi Paul,

Looks great, and I have verified your performance numbers locally. Nice.

We don't want to create any confusion by committing Java code that isn't under the CA, but the need to have it for comparison is obvious. One thing we could do is include bash scripts that would download the Java code for comparison. 

If you will recreate the patch without the Java code I will apply it. Or if you plan to keep at this you can just have commit access to the repos. Let me know.

Stu

Stuart Halloway
Clojure/core
http://clojure.com

pmbauer

unread,
Sep 13, 2011, 1:10:39 PM9/13/11
to cloju...@googlegroups.com
Thanks. I will remove the java code and resubmit after work.
The commit access would be sweet.  I probably won't get to another benchmark for another week though.

Since test.benchmark could be used as a performance regression test, I'd like to add a set of java/clojure comparison tests.
Provided there's an automated way in place to download java bits, do you think that's worth pursuing?
If so, what are your thoughts on downloading the java code from someplace other than alioth (e.g. my own github account)?  The alioth benchmark code needed tweaking for this goal.

Stuart Halloway

unread,
Sep 13, 2011, 2:34:31 PM9/13/11
to cloju...@googlegroups.com
Thanks. I will remove the java code and resubmit after work.

Cool.

The commit access would be sweet.  I probably won't get to another benchmark for another week though.

A week here or there certainly doesn't matter! But commit access represents an ongoing commitment to participate. What say you?

Since test.benchmark could be used as a performance regression test, I'd like to add a set of java/clojure comparison tests.

I think a lot of the alioth benchmarks have expected output files. It would be great to start with correctness tests (as simple as a file compare) and then build perf tests above that.

Provided there's an automated way in place to download java bits, do you think that's worth pursuing?

Sure.

If so, what are your thoughts on downloading the java code from someplace other than alioth (e.g. my own github account)?  The alioth benchmark code needed tweaking for this goal.

Sounds fine. As along as the code isn't in our repos polluting our CA, all is well.

pmbauer

unread,
Sep 13, 2011, 2:43:21 PM9/13/11
to cloju...@googlegroups.com
A week here or there certainly doesn't matter! But commit access represents an ongoing commitment to participate. What say you?

Yes.
 
It would be great to start with correctness tests (as simple as a file compare) and then build perf tests above that.
  
I have correctness tests in the patch for mandelbrot, so we're covered there.   I was planning to add tests for the others later.
 
Sounds fine. As along as the code isn't in our repos polluting our CA, all is well.

It's a plan then.

Stuart Halloway

unread,
Sep 13, 2011, 3:14:29 PM9/13/11
to cloju...@googlegroups.com
A week here or there certainly doesn't matter! But commit access represents an ongoing commitment to participate. What say you?

Yes.

Great, it's done.

Stu
Reply all
Reply to author
Forward
0 new messages