Julia Mathematical JIT Benchmark

[Tim K]

Julia is a new JIT compiled language for mathematical programming, that is intended to be faster than Matlab etc, while offering more expressiveness and convenience.

As such, the developers have developed a small set of micro-benchmarks for numerical performance.

https://github.com/JuliaLang/julia/blob/master/test/perf/perf.js

V8 v3.6.6.11 performs quite well on this, beating C++ on one benchmark and Julia on another.

http://julialang.org

I thought that the Spidermonkey developers might be interested in using this benchmark.

Their dev-list is here:

https://groups.google.com/forum/?fromgroups#!forum/julia-dev

Cheers,

Tim.

[Boris Zbarsky]

On 3/24/12 9:21 PM, Tim K wrote:
> Julia is a new JIT compiled language for mathematical programming, that is intended to be faster than Matlab etc, while offering more expressiveness and convenience.
>
> As such, the developers have developed a small set of micro-benchmarks for numerical performance.
>
> https://github.com/JuliaLang/julia/blob/master/test/perf/perf.js
>
> V8 v3.6.6.11 performs quite well on this, beating C++ on one benchmark and Julia on another.

Huh. When I run this test in V8 v3.7.3 I get an exception when calling
set() in the rand_mat_stat test.

In any case, a comparison of a reasonably recent Spidermonkey with -m -n
with V8 on my hardware (excluding the rand_mat_stat test they throw on)
shows:

Spidermonkey:

javascript,fib,0.185
javascript,parse_int,0.366
javascript,mandel,0.962
javascript,quicksort,7.4
javascript,pi_sum,33.8
javascript,rand_mat_mul,11061

V8:

javascript,fib,0.113
javascript,parse_int,0.498
javascript,mandel,0.627
javascript,quicksort,2.2
javascript,pi_sum,34.5
javascript,rand_mat_mul,9940

Note that pi_sum (which we beat V8 on) is the one that V8 and Julia beat
C++ on.

On a side note, the "quicksort" test in JS is not actually using
quicksort; it's using Array.prototype.sort with the callback
function(a,b) { return a-b; }. That's covered by bug 715181 and the
bugs linked from the comments there.

I looked at the C++ and Julia code for the "quicksort" test, and the C++
and Julia and Python versions of this are both hand-coded naive
quicksort. This is covered by https://github.com/JuliaLang/julia/issues/562

Fwiw, recoding the quicksort in exactly the same way as the Julia or
Python or C++ version makes the times look like this:

Spidermonkey:
javascript,quicksort,1.08

V8:
javascript,quicksort,1.84

There are various other issues in the JS tests (missing var all over
leading to global variable accesses when I think they meant to use
function local variables, for example)....

-Boris

[Tim K]

Thanks for the reply.

Is it worth filing a bug on bugzilla for the perf differences and following the evolution of the benchmark?

[Boris Zbarsky]

On 3/24/12 10:51 PM, Tim K wrote:
> Thanks for the reply.
>
> Is it worth filing a bug on bugzilla for the perf differences and following the evolution of the benchmark?

Probably, yes.

-Boris

[Tim K]

The bug has been filed here:

https://bugzilla.mozilla.org/show_bug.cgi?id=739016
(Performance on Julia language's math operations JIT benchmark)

[patrick...@gmail.com]

On Saturday, March 24, 2012 9:41:44 PM UTC-5, Boris Zbarsky wrote:
> Fwiw, recoding the quicksort in exactly the same way as the Julia or
> Python or C++ version makes the times look like this:
>
> Spidermonkey:
> javascript,quicksort,1.08
>
> V8:
> javascript,quicksort,1.84
>
> There are various other issues in the JS tests (missing var all over
> leading to global variable accesses when I think they meant to use
> function local variables, for example)....

Boris, could you provide your recoded quicksort? I'm sure the Julia folks would appreciate it.

[Boris Zbarsky]

It's really just a 1 for 1 copy of the others, with "var" inserted as
needed, and the single Math.floor() call in the place where floor is needed:

function qsort_kernel(a, lo, hi) {
var i = lo;
var j = hi;
while (i < hi) {
pivot = a[Math.floor((lo+hi)/2)];
while (i <= j) {
while (a[i] < pivot) {
i = i + 1;
}
while (a[j] > pivot) {
j = j - 1;
}
if (i <= j) {
var t = a[i];
a[i] = a[j];
a[j] = t;
i = i + 1;
j = j - 1;
}
}
if (lo < j) {
qsort_kernel(a, lo, j);
}
lo = i;
j = hi;
}
}

function sortperf(n) {
var v = rand(n);
qsort_kernel(v, 0, n);
return v;
}

[patrick...@gmail.com]

On Sunday, March 25, 2012 8:38:17 AM UTC-5, Boris Zbarsky wrote:

> On 3/25/12 9:09 AM Patrick O'Leary wrote:
> > On Saturday, March 24, 2012 9:41:44 PM UTC-5, Boris Zbarsky wrote:
> >> Fwiw, recoding the quicksort in exactly the same way as the Julia or
> >> Python or C++ version makes the times look like this:
> >>
> >> Spidermonkey:
> >> javascript,quicksort,1.08
> >>
> >> V8:
> >> javascript,quicksort,1.84
> >>
> >> There are various other issues in the JS tests (missing var all over
> >> leading to global variable accesses when I think they meant to use
> >> function local variables, for example)....
> >
> > Boris, could you provide your recoded quicksort? I'm sure the Julia folks would appreciate it.
>
> It's really just a 1 for 1 copy of the others, with "var" inserted as
> needed, and the single Math.floor() call in the place where floor is needed:

Thanks, Boris. This helps save us from doing it wrong. :)
https://github.com/JuliaLang/julia/pull/639