Julia and LuaJIT performance

[GrrrWaaa]

I saw this post on the LuaJIT list, and I thought some people here might find it interesting.

Begin forwarded message:

> From: Francesco Abbate <france...@gmail.com>
> Date: September 2, 2012 1:57:59 AM PDT
> To: lua...@freelists.org
> Subject: performance benchmark results on Julia's benchmark
> Reply-To: lua...@freelists.org
>
> Hi all,
>
> I've made some tests using the Julia's benchmark to see the results of
> GSL Shell. The performance is largely determined by LuaJIT so this may
> be interesting for the people here.
>
> Here the document:
>
> https://docs.google.com/presentation/d/1Gwx8rYEV1FqaqALUvnE0VHMhm87CW121cJ0YAoQwfys/edit
>
> and here the results:
>
> Test Julia GSL Shell Python
> ------------------------------------------
> fib 0.429 0.120 3.641
> parse_int 0.367 0.856 3.878
> mandel 0.911 0.171 5.727
> quicksort 0.733 1.114 14.062
> pi_sum 38.59 38.60 764.24
> rand_mat_stat 42.17 57.09 254.92
> rand_mat_mul 175.45 178.70 200.25
> printfd 66.83 74.18
>
> The file used for the test is in attachment. It is standard Lua but it
> does use some GSL Shell modules for some of the tests.
>
> The results are quite good but in some cases Julia is performing
> slightly better than GSL Shell. In particular I was surprised to see
> that it is faster for the quicksort tests where I was thinking that
> LuaJIT was going to rock. Something funny to note is that the integer
> division is needed in this test and both Lua and GSL Shell needs to
> use a trick to do it! :-)
>
> Otherwise LuaJIT is doing a great job in the Mandel test where we have
> a tight loop with complex numbers computations. I guess that in this
> case the work of Mike for objects sinking make the difference for this
> test.
>
> For other tests like rand_mat_stat probably the GSL Shell stuff is
> playing a role by using some slightly suboptimal code. What is
> embarassing here is that the GSL Shell code is not only slower but
> also more ugly. In this case Julia is showing its advantage being
> specialized for matrix computations.
>
> The code for the Julia test can be easily find in the
> http://julialang.org page, there is a link to the perf code in the git
> repository.
>
> Francesco

[Stefan Karpinski]

Very interesting. The LuaJIT numbers are really impressive, although I'm not really surprised — Mike Pall has done amazing work and clearly other people are capitalizing on it and extending it to new areas. Julia definitely isn't the only dynamic language implementation that has outstanding performance — both LuaJIT and V8 prove this. (There have, of course, also been very fast Lisp and Smalltalk implementations.)

The mandelbrot benchmark strikes me as very suspicious because they're clocking in at 5.3x faster than Julia, which on my machine is only 1.9x slower than Fortran and 1.45x slower than C++. That implies, assuming a similar relative performance to C++ and Fortran, that LuaJIT is 2.8x faster than Fortran and 3.7x faster than C++. That just doesn't seem plausible unless the Lua version is is doing less work than it ought to be somehow.

The code for the matrix benchmarks is interesting because the LuaJIT code is so much closer to C++ than it is to Julia, Matlab or NumPy. The difference is most prominent in the random matrix statistics code, which does a lot of fairly complicated but not atypical stuff with small matrices. The fact that Julia's simple, concise randmatstat code does so well, even compared to the Lua version which does C-style explicit matrix allocation, reuses temporaries, and does explicit calls to dgemm, is a testament to Jeff's spectacular work on our compiler.

We should include LuaJIT in our benchmarks now that the code for these exists, modulo figuring out what's up with the suspicious mandelbrot benchmark. Of course, I need to figure out how to the LuaJIT code on my machine first since it's the "official" benchmarking machine. (For no particular reason — just that's where we've been running the benchmarks and installing all the programs required to run the full benchmark suite is kind of a pain.)

--

[Tim Holy]

On Wednesday, September 05, 2012 10:03:30 PM Stefan Karpinski wrote:
> The mandelbrot benchmark strikes me as very suspicious because they're
> clocking in at 5.3x faster than Julia, which on my machine is only 1.9x
> slower than Fortran and 1.45x slower than C++. That implies, assuming a
> similar relative performance to C++ and Fortran, that LuaJIT is 2.8x faster
> than Fortran and 3.7x faster than C++. That just doesn't seem plausible
> unless the Lua version is is doing less work than it ought to be somehow.

Another oddity is that the test results are supposedly in milliseconds. To me
it seems a little unreliable to make serious comparisons among tests that take
less than a millisecond to run--just the overhead in "getting going" might
dominate the test.

--Tim

[Stefan Karpinski]

On Wed, Sep 5, 2012 at 10:21 PM, Tim Holy <tim....@gmail.com> wrote:

Another oddity is that the test results are supposedly in milliseconds. To me
it seems a little unreliable to make serious comparisons among tests that take
less than a millisecond to run--just the overhead in "getting going" might
dominate the test.

Perhaps we ought to bump up those counts a bit. Some of them are kind of short. The trouble is that I'll have to wait all day for the Octave benchmarks to run.

[Stefan Karpinski]

On Wed, Sep 5, 2012 at 10:03 PM, Stefan Karpinski <ste...@karpinski.org> wrote:

The code for the matrix benchmarks is interesting because the LuaJIT code is so much closer to C++ than it is to Julia, Matlab or NumPy.

I should clarify that I'm talking about coding style here, not performance. (Julia is around 2x C++ and LuaJIT is a bit slower.) The LuaJIT code looks a lot like the C++ code, which is not terribly concise and has a lot of manual allocation, etc., whereas the Julia, Matlab, and NumPy codes are quite concise and simple.

[Viral Shah]

As Stefan already said, we need to dig deeper into this - but it is great to see the julia micro-benchmark set becoming popular and driving performance and innovation in other languages as well.

-viral