DeltaBlue in Java

[Nikolay Botev]

Hi,

I do not know if this has been done before (a quick google search revealed nothing), but I ported the benchmark_harness project and the DeltaBlue benchmark from Dart to Java and ran on recent Dart and Java builds on my laptop. The results are pretty impressive (for Dart). I posted them here: http://bonovox.be/blog/?p=128

Keep up the good work.

Cheers

[Alex Tatumizer]

Dart libraries are somewhat falling behind, especially strings and maps. In these areas, Java is faster by factor of 1.5, in some cases 2.

[Srdjan Mitrovic]

Hi Nikolay,

Thank you for your post. I am glad to see Dart VM performance being close to the performance of a statically typed language, even though Dart VM is a much younger system than the Hostpot Java. As regular contributors know, we still have a lot of performance work to do :-). 

FYI: (except on Mac OS X) Hotspot Java has two modes:-server  and -client (java -version). Java's Client VM has much tighter resource constraints than the Server VM, and thus produces slower code. Similarly Dart VM and V8 have also tighter constraints than Java Server VM: e.g., we cannot spend long time optimizing and inlining, otherwise compilation pauses will become too big. I assume the 32-bit Java is the Client VM. It seems that it would be fair to compare only Java's Client VM with Dart and V8. But as they say: all is fair in benchmarking and wars ;-).

Cheers,

- Srdjan

--
Consider asking HOWTO questions at Stack Overflow: http://stackoverflow.com/tags/dart
 
 

[Gilad Bracha]

Hi Nikolay,

Deltablue has been ported to Java  from the original Smalltalk sometime in the past.

On Fri, May 10, 2013 at 2:05 AM, Nikolay Botev <bono...@gmail.com> wrote:

--

Consider asking HOWTO questions at Stack Overflow: http://stackoverflow.com/tags/dart
 
 

--
Cheers, Gilad

[Ladislav Thon]

I do not know if this has been done before (a quick google search revealed nothing)

My quick google search for "deltablue java" immediately found this: https://github.com/xxgreg/deltablue, which interestingly also contains a comparison with Dart (and V8 and C).

LT

[Alex Tatumizer]

In these "macro"-benchmarks, the other side always complains your version in their language is not idiomatic enough, or doesn't take into account *something* very important, or you run it under wrong settings, or something.

That's why I prefer nano-banchmarks. E.g. I can compare substring to substring, and it's hard to question the result.

 

--

[Alex Tatumizer]

If we want to evaluate the cost of polymorphism, won't it be better to test just that?
E.g.
1. create 3 different classes (A, B, C), each implementing interface Foo consisting of a single method foo(int x) { return x; }
 2. create array of 1000 instances of randomly selected (A or B or C)
3. call foo() for each element of array

Would that be a fair test or polymorphism?

[Alex Tatumizer]

Went ahead and tested it.
Interestingly, in java there's a difference between "extends" and "implements".
"extends" : 12.9 ns/iteration
"implements": 15.4 ns/iteration

Dart:
same timing for both "extends" and "implements": 13.9 ns/iteration

"extends" is still marginally faster in java; "implements" is marginally faster in dart.

If the goal of DeltaBlue was to test just that... it could be done with much lesser effort  :-)

[Greg Lowe]

A while back I did some benchmarking with deltablue java, dart and c using the original deltablue impl from Sun.

Code is here:

https://github.com/xxgreg/deltablue

I'll re-publish the results when I get a chance.

[Greg Lowe]

I had a quick look at this. It appears the DartVM is now beating the JVM even at 100,000 iterations (Previously Java edged out Dart at higher iterations).

100,000 iterations takes ~45 seconds to run on my computer.

At a lower number of iterations the DartVM beats the JVM by a significant margin.

The benchmark is easy to run (On linux, maybe macos too). Just check it out, make sure dart, java and gcc are in your path, and execute run.sh.

Note, this code doesn't use the benchmark harness because I wanted to include the warm up time in the measurements. At a high number of iterations this doesn't make much difference.

[Nikolay Botev]

Hi,

Thanks everyone for the great feedback.

I am stoked this garnered so much interest.

* I reran the benchmark with the 32-bit server JVM and got the same results as with the 64-bit JVM.

* I ran Greg's benchmarks using his run.sh script. I do not find the results to be different enough from the ones with the Dart harness to warrant the spending the extra long running time (30-45 seconds vs ~2.5 seconds).

* I ran Greg's version with the Dart harness. The got slightly lower performance than with my DeltaBlue translated from Dart, but not by much. I also changed the harness to warm up for 15 seconds (up from 0.2) and run for 30 seconds (up from 2). The performance improved a bit, but not enough to reach Dart's performance with the default warmup and execution times.

After all this, I think I have given Java a fair chance and the full benefit of the doubt. Dart still comes on top in DeltaBlue, and none of these additional results change the simple conclusion that Dart runs DeltaBlue faster. I also believe that translating Dart directly to Java, as opposed to rewriting the benchmark code in Java from scratch or from another source, make the comparison more fair.

As time permits, I will continue porting the rest of the Dart benchmarks from benchmark_harness to Java and post the results. I am particularly curious about what the Tracer results end up looking like.

[mockturtl]

> https://github.com/xxgreg/deltablue

FWIW:

DeltaBlue Java 100000x 0.37481 ms
DeltaBlue C gcc 100000x 0.7962ms
# Command exited with non-zero status 33
DeltaBlue C gcc -O3 100000x 0.336ms
DeltaBlue d8 100000x 0.50139ms
DeltaBlue Dart 100000x 0.4183ms


java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.7/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.7.2-5'
--with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs
--enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr
--program-suffix=-4.7 --enable-shared --enable-linker-build-id
--with-system-zlib --libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7
--libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu
--enable-libstdcxx-debug --enable-libstdcxx-time=yes
--enable-gnu-unique-object --enable-plugin --enable-objc-gc
--with-arch-32=i586 --with-tune=generic --enable-checking=release
--build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.7.2 (Debian 4.7.2-5)
V8 version 3.19.0 [console: readline]
Dart VM version: 0.5.5.0_r22416 (Mon May 6 10:31:00 2013)

[Entaro Adun]

Are there any benchmarks between python and latest Dart? Comperision?

[Greg Lowe]

Thanks for posting. Are you running this on intel x64, and are you using 64 bit dart?

[TM]

Yes on both counts.

> <http://stackoverflow.com/tags/dart>
> >
> >
>
> --
> For other discussions, see https://groups.google.com/a/dartlang.org/
>
> For HOWTO questions, visit http://stackoverflow.com/tags/dart
>
> To file a bug report or feature request, go to http://www.dartbug.com/new
>
>

[Greg Lowe]

@Tom M

Interesting. The JVM is performing a lot better in your set up (1.12x of optimised C code). I've never seen it run better than 1.2x. I wonder what the difference is? Would be interesting to get a few more people to run the benchmark and post results.

[mockturtl]

On 05/12/2013 06:00 PM, Greg Lowe wrote:> @Tom M

>
> I wonder what the difference is?

I guess it's Java 7 vs. Java 6. I see the same differences in OpenJDK.

----

java version "1.7.0_21"
OpenJDK Runtime Environment (IcedTea 2.3.9) (7u21-2.3.9-4)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

DeltaBlue Java 100000x 0.38389 ms
DeltaBlue C gcc -O3 100000x 0.3418ms
DeltaBlue d8 100000x 0.51826ms
DeltaBlue Dart 100000x 0.42797ms

3839/3418 = 1.12

----

java version "1.6.0_27"
OpenJDK Runtime Environment (IcedTea6 1.12.5) (6b27-1.12.5-1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

DeltaBlue Java 100000x 0.41386 ms
DeltaBlue C gcc -O3 100000x 0.3423ms
DeltaBlue d8 100000x 0.51919ms
DeltaBlue Dart 100000x 0.42862ms

41386/34230 = 1.21

----

[My previous post used Oracle's JDK (7u21).]

[Greg Lowe]

That must be it. I wonder what the difference with Nickolay's benchmark is. He's tested a number of different JVM versions and still sees Dart in-front. Perhaps it's to do with the benchmark harness implementation?

[Nikolay Botev]

Hi,

Only the nightly builds of dart are faster than Java in my tests. That is any dart vm build after the latest jump in dart's delta blue score as seen on dartlang.org/performance. 

Sent from my iPhone

[Greg Lowe]

Ah right - that makes sense.

[Lasse R.H. Nielsen]

On Fri, May 10, 2013 at 9:39 PM, Alex Tatumizer <tatu...@gmail.com> wrote:

In these "macro"-benchmarks, the other side always complains your version in their language is not idiomatic enough, or doesn't take into account *something* very important, or you run it under wrong settings, or something.

That's why I prefer nano-banchmarks. E.g. I can compare substring to substring, and it's hard to question the result.

It's very, very easy to question the result of micro/nano-benchmarks.

Usually, a perfect dead-code eliminator would remove the entire benchmark (what, no "print" or other I/O operations in your program, and only bounded loops - it can all go away!)

Also, if you just repeat the same operation over and over again, it's very susceptible to small changes like code and data alignment. One cross-boundary load instruction in an inner loop of the generated code can easily double the execution time. That would not happen for a larger benchmark, since it won't be using 90% of its time in the same code.

And usually it all runs in a small loop, so you really need on-stack-replacement to optimize the generated code while it's still running, or a suitable warm-up phase. A larger program would not depend so heavily on that for its performance.

(And then there is measuring methodology, but that applies to all benchmarks.)

 

So what I'm saying is: You can get a number, but it's not obvious what that number means. Changing the benchmark even slightly can give a very different number.

/L

 

 

On Fri, May 10, 2013 at 3:00 PM, Ladislav Thon <lad...@gmail.com> wrote:

I do not know if this has been done before (a quick google search revealed nothing)

My quick google search for "deltablue java" immediately found this: https://github.com/xxgreg/deltablue, which interestingly also contains a comparison with Dart (and V8 and C).

LT

--
Consider asking HOWTO questions at Stack Overflow: http://stackoverflow.com/tags/dart
 
 

--
Consider asking HOWTO questions at Stack Overflow: http://stackoverflow.com/tags/dart
 
 

--
Lasse R.H. Nielsen - l...@google.com  

'Faith without judgement merely degrades the spirit divine'
Google Denmark ApS - Frederiksborggade 20B, 1 sal - 1360 København K - Denmark - CVR nr. 28 86 69 84

[Alex Tatumizer]

> Usually, a perfect dead-code eliminator would remove the entire benchmark (what, no "print" or other I/O operations in your program, and only bounded loops - it can all go away!)

I tried to prevent this. Function under test always returns some aggregate of whatever it's doing (e.g. OR'ed elements, or something).

In caller, I do this:

int x=myTest();

if (x<0) print(x);

So there's no dead code, though x is never printed - it's always greater than 0 - but compiler doesn't know this (not obvious), so the thing never gets optimized away.

Actually, I agree with what you are saying, except that if result is bad, it certainly indicates a problem. I'm trying to compare dart vs java on very simple operations.

I've seen only one case when the whole is "smaller" than the sum of parts: in json parsing, dart is faster by factor of 2 vs.json.org parser, but I couldn't explain why. Maybe my bug.

(JSON Serialization is the same, only parsing is different).

In almost every case I tried, java is slightly better: sometimes by 10%, sometimes by 30%, and in a couple of cases by 2  times faster. Though the quality of compiler is on par I think

(except 32-bit words). Dart is slower only because of libraries, which were not yet optimized. If you manage to utilize SIMD in code generation and in libs, you will *easily* outperform java IMO.

Anyway, these benchmarks are not completely useless. In any Intel processor manual, you can find clocks per op. This data is also subject to million factors (alignment of code, alignment of data, cache miss,  states of pipelines, etc, etc etc) - but still, this gives you some idea. E.g. you see that div is slow, so you try to avoid div.

So nanobenchmarks certainly tell just part of the story, but it's better than nothing IMO. 

--

[Nikolay Botev]

Hi,

I posted numbers for the Richards and Tracer benchmarks as well (both ported by hand from benchmark_harness to Java by me). I do understand that DeltaBlue and Richards have existing ports to Java. I have to admit a big part of the purpose of my exercise was to get a feel for the Dart language itself. I do share my experience of stumbling upon unfamiliar Dart features in both posts. FWIW:

http://bonovox.be/blog/?p=127

I was happy to see someone else had already done a comparison (xxgreg) Dart and Java running DeltaBlue. I reran the benchmark under xxgreg's harness and the results were consistent with what I saw and posted on my blog.

Cheers,

Nikolay

[kc]

An interesting comparison would be on Android - DartVM vs Dalvik. 

If Dart - with snapshots - offers comparable performance to Dalvik byte code - interesting.

K.

[Filipe Morgado]

Sexta-feira, 17 de Maio de 2013 11:18:43 UTC+1, kc escreveu:

An interesting comparison would be on Android - DartVM vs Dalvik.

I believe Dalvik is around 2x slower than Hotspot (link), which is understandable since Dalvik is optimized for mobile so it probably uses a lot less memory. Hotspot (and thus Dart) is in another league in terms of raw speed.

Dalvik might be an interesting addition to memory consumption benchmarks. I have yet to see one of those.

[kc]

On Friday, May 17, 2013 1:39:34 PM UTC+1, Filipe Morgado wrote:

I believe Dalvik is around 2x slower than Hotspot (link), which is understandable since Dalvik is optimized for mobile so it probably uses a lot less memory. Hotspot (and thus Dart) is in another league in terms of raw speed.

Dalvik might be an interesting addition to memory consumption benchmarks. I have yet to see one of those.

Yes a comparison that includes memory consumption would be interesting. Also, recent versions of Dalvik in jit-ing so maybe it's getting faster.

This comparison is interesting because the eventual goal should be code once - run on Chrome(OS)/Packages Apps and Android. 

K.

[Greg Lowe]

The resident set size for the JVM when running ~100k iterations of deltablue was ~250MB, the DartVM was ~170MB, C was 2.5MB.

K.

--

[kc]

On Friday, May 17, 2013 9:40:06 PM UTC+1, Greg Lowe wrote:

The resident set size for the JVM when running ~100k iterations of deltablue was ~250MB, the DartVM was ~170MB, C was 2.5MB.

Thanks.

A dalvik comparison would be interesting. JVM's are effectively dead on the client. The only reason to install Java is for Android development!

K.

[Peter Jakobs]

jvm on the client is not dead... there is still minecraft!

Sorry could not resist ;)

[Filipe Morgado]

Sexta-feira, 17 de Maio de 2013 22:05:19 UTC+1, kc escreveu:

JVM's are effectively dead on the client.

They're announcing great stuff for the Java language, making it less verbose and adding "modern" features.
... ok, always with backward compatibility ... will never be a personal choice.

But they're announcing good features as well for the runtime, which affects every language that targets the JVM (I love Scala, too bad it takes forever to compile and only effectively targets Java).

The best of these features (IMO) are modules, breaking up the standard library into small pieces (maybe with deferred loading), greatly reducing startup times and memory footprints.

If done right, this might make the JVM a great option even for mobile.

We can never take anything for granted in computing sciences.