CLJS: Are we fast yet?

[Brian Taylor]

CLJS devs:

In the spirit of "Are we Fast Yet?" (http://arewefastyet.com/) I've put together a quick benchmarking / reporting tool that shows how the performance of certain Clojurescript micro-benchmarks has progressed (or regressed) with each update in the repository.

You can see the the results here: http://50ply.com/cljs-bench

You can get the source for the tool here: http://github.com/netguy204/cljs-bench

I'll do my best to keep the site up-to-date with new versions of Clojurescript and new micro-benchmarks as we identify them.

I love feedback of any sort.

Best Regards,

Brian

[David Nolen]

Wow this is great! Which JS engine is this benchmarking? Would be great if we could get information for V8, SpiderMonkey, and JavaScriptCore in there as separate graphs.

Clearly we still have lots of work to do :)

David

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To post to this group, send email to cloju...@googlegroups.com.
To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.

[David Nolen]

Would also be awesome if we could get Clojure on the JVM in there as a baseline.

On Tue, May 29, 2012 at 8:54 PM, Brian Taylor <el....@gmail.com> wrote:

[David Nolen]

One other suggestion, it would be perhaps nice to to have a minimum height on the graphs, 300ms? This way the benchmarks that aren't actually changing much don't look misleading.

David

On Tue, May 29, 2012 at 8:54 PM, Brian Taylor <el....@gmail.com> wrote:

[Brian Taylor]

Right now I'm only running V8. I'll get SpiderMonkey and JavaScriptCore in there as soon as I can get them running on my system.

JVM baseline and minimum height: Definitely! I'll get on it.

[David Nolen]

On Tue, May 29, 2012 at 9:17 PM, Brian Taylor <el....@gmail.com> wrote:

Right now I'm only running V8. I'll get SpiderMonkey and JavaScriptCore in there as soon as I can get them running on my system.

JVM baseline and minimum height: Definitely! I'll get on it.

0 to minimum height would be the most useful. Then comparisons can be easily made between operations.

David 

[Michael Fogus]

Wonderful! Thank you for doing this. I would love to add some reader
tests also. I have some that I've been using, but I'd love to see them
incorporated.

[David Nolen]

On Tue, May 29, 2012 at 9:17 PM, Brian Taylor <el....@gmail.com> wrote:

Right now I'm only running V8. I'll get SpiderMonkey and JavaScriptCore in there as soon as I can get them running on my system.

JVM baseline and minimum height: Definitely! I'll get on it.

You can get pre-built binaries of the latest FireFox JS shell here - http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/

For JavaScriptCore the simplest thing to do is to download a WebKit nightly. Then in your .bash_profile if you're on OS X:

DYLD_FRAMEWORK_PATH=/Applications/WebKit.app/Contents/Frameworks/10.7/JavaScriptCore.framework

export DYLD_FRAMEWORK_PATH

JSC_HOME=/Applications/WebKit.app/Contents/Frameworks/10.7/JavaScriptCore.framework/Resources

Should work for you. For SpiderMonkey you should make sure to set the -m -n -a flags. The existing benchmark script in the CLJS repo should guide you.

David

[David Nolen]

On Tue, May 29, 2012 at 9:28 PM, David Nolen <dnolen...@gmail.com> wrote:

JSC_HOME=/Applications/WebKit.app/Contents/Frameworks/10.7/JavaScriptCore.framework/Resources

You'll probably want to set your PATH to that too.

David 

[David Nolen]

One more time as far as JSC:

DYLD_FRAMEWORK_PATH=/Applications/WebKit.app/Contents/Frameworks/10.7/

export DYLD_FRAMEWORK_PATH

PATH=$PATH:/Applications/WebKit.app/Contents/Frameworks/10.7/JavaScriptCore.framework/Resources

JSC_HOME=/Applications/WebKit.app/Contents/Frameworks/10.7/JavaScriptCore.framework/Resources

export JSC_HOME

[Brian Taylor]

I've updated the plots. In most cases I could clamp the range of the plot from 0 to 300 ms. A few of the tests don't have any points in those bounds so I let them extend a bit higher.

Thank you, David, for the install tips. Homebrew had failed me so I gave up. I'll grab the binaries and re-run the benchmarks.

[David Nolen]

Oops I didn't mean to suggest that a maximum height be set, just a minimum height.

[Brian Taylor]

@fogus Great! Email or pull request them over and I'll include them.

@david Oops, got it. Fixed.

[David Nolen]

Excellent! Your graphs have given me hunch as to why we saw a regression on V8 for PersistentVector ops after adding the IReversible protocol:

http://dev.clojure.org/jira/browse/CLJS-284

I think we're going to get a lot mileage out of this :)

[Brian Taylor]

Wonderful!

Your tips got me up and running with spidermonkey and jsc. The benchmarks are running again and I'll add plots for the new runtimes once I get the data. Thank you.

[Brian Taylor]

The results for the javascriptcore and spidermonkey are online now too.

[David Nolen]

Fantastic stuff. Seems to confirm my suspicion about hitting deoptimization under V8 - the other engines seem to be cruising along as far as PersistentVector ops.

Thank you very, very much.

David

[David Nolen]

It would be nice if the # of iterations for a particular benchmark were shown.

On Wed, May 30, 2012 at 7:56 AM, Brian Taylor <el....@gmail.com> wrote:

[David Nolen]

PersistentVector performance regression addressed in CLJS master thanks to Michal Marczyk.

[Michael Fogus]

> PersistentVector performance regression addressed in CLJS master thanks to
> Michal Marczyk.

Could there be anyone else? :-) This perf reporting is gold.
:F

[Brian Taylor]

I've uploaded the latest plots. Thank you Michal and David! One regression vanquished.

[Brian Taylor]

http://www.50ply.com/cljs-bench/

Cool new plot: Advanced mode gzipped output size. (gzipped because that's likely to be the size that ultimately goes down the wire and the size that closure developers optimize towards.)

I've also added history for a handful of new benchmarks that were added to the clojurescript repo recently.

[David Nolen]

Great stuff! Can we can the Y range on the size graph? 20-40k perhaps?

[Brian Taylor]

Done.

[Brandon Bloom]

http://www.50ply.com/cljs-bench/

Brian: Awesome work on these plots! Thanks for doing this.

 

Cool new plot: Advanced mode gzipped output size. (gzipped because that's likely to be the size that ultimately goes down the wire and the size that closure developers optimize towards.)

I'm looking at: https://github.com/netguy204/cljs-bench/blob/master/test-scaffold/script/benchmark_data#L39 

That seems like an odd choice to be measured. In this case, as you add benchmarks, the size increases. This means that if you add an optimization to the compiler and add a benchmark for that optimization, then you've changed two variables for that graph! Ideally, we should be measuring the compiled output size of a fixed piece of code, like the TwitterBuzz example project.

[Brian Taylor]

On Thu, May 31, 2012 at 7:06 PM, Brandon Bloom <snpr...@gmail.com> wrote:

http://www.50ply.com/cljs-bench/

Brian: Awesome work on these plots! Thanks for doing this.

 

You're welcome!

 

Cool new plot: Advanced mode gzipped output size. (gzipped because that's likely to be the size that ultimately goes down the wire and the size that closure developers optimize towards.)

I'm looking at: https://github.com/netguy204/cljs-bench/blob/master/test-scaffold/script/benchmark_data#L39 

That seems like an odd choice to be measured. In this case, as you add benchmarks, the size increases. This means that if you add an optimization to the compiler and add a benchmark for that optimization, then you've changed two variables for that graph! Ideally, we should be measuring the compiled output size of a fixed piece of code, like the TwitterBuzz example project.

My intent was to re-run the reported swath of git revisions whenever I add new benchmarks so that I can report how previous changes would have affected the new benchmarks. That would ensure that each point on the plot is showing the output size from the same input.

But, I like your idea better because it gives me the freedom to choose not to backfill history when new benchmarks are added and it also makes the absolute size I report more meaningful.

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

To view this discussion on the web visit https://groups.google.com/d/msg/clojure-dev/-/5x2--hfp1WcJ.

[David Nolen]

I just landed a 10X improvement to PersistentVector construction making it JVM competitive under V8.

Would love to get Clojure JVM as the baseline in here where we can :)

David

[Brian Taylor]

Coming soon.

Awesome! Plots are updated. I've also switched to a recent copy of the TwitterBuzz code as the basis for the compile time / compile size benchmark.

[Brian Taylor]

Clojure JVM baseline results are posted! I'm proud to say that there are several cases where CLJS on V8 is faster than the CLJ 1.4 on the JVM.

http://www.50ply.com/cljs-bench/

[David Nolen]

Fantastic stuff!

[David Nolen]

Would be nice to get some of the PersistentHashMap benchmarks in there.

David

[Brian Taylor]

Done. I pulled in the new benchmarks you had added and updated my runs through the latest from GitHub.

[David Nolen]

Thanks again!

[David Nolen]

It would be nice to view yet another update. I imagine compilation slowed down some and perhaps some small performance enhancements here and there.

Thanks again.

[Brian Taylor]

Posted. :-) Happy to do it!

Awesome work on the core.logic / cljs performance improvements! Thank you!