Benchmarks and other lies

[Greg Lowe]

Robert P. has re-run the cross-language deltablue benchmark which I set up 2 years ago. (Thanks Rob!)

Times for 100,000 iterations are:

Dart VM    13.76s

V8                   19.09s

C (gcc -O3)     22.50s

Java                25.36s

C (gcc)            49.08s

Both Dart and V8 have seen significant improvements relative to GCC and Java during this time. 2 years ago Java was just ahead of the Dart VM, and GCC was well ahead.

For this benchmark at least, the Dart VM and V8 are generating better machine code than GCC -O3. Very impressive!

Now putting fingers in ears, and hiding under a rock, and waiting for the usual flames to begin, like last time I published these.

The source code can be found here:

 https://github.com/xxgreg/deltablue

Some flames:
  http://www.reddit.com/r/programming/comments/1e2jhr/dart_vs_java_the_deltablue_benchmark/c9wloq3

And resultant ashes:
  http://blog.headius.com/2013/05/on-languages-vms-optimization-and-way.html

And the raw output, for those obsessed with the details:

java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.8.2-19ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-3
 2=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) 
V8 version 3.30.11 [console: dumb]
d8> 
Dart VM version: 1.7.2 (Tue Oct 14 12:12:42 2014) on "linux_x64"
DeltaBlue   Java        10x 2.6 ms
java DeltaBlue 10 secs 0.09, mem 26088, cpu 173%
DeltaBlue   Java        100x    0.75 ms
java DeltaBlue 100 secs 0.15, mem 41044, cpu 246%
DeltaBlue   Java        1000x   0.34 ms
java DeltaBlue 1000 secs 0.52, mem 79088, cpu 257%
DeltaBlue   Java        10000x  0.258 ms
java DeltaBlue 10000 secs 2.64, mem 210396, cpu 158%
DeltaBlue   Java        100000x 0.2529 ms
java DeltaBlue 100000 secs 25.36, mem 219164, cpu 108%
DeltaBlue   C   gcc 10x 0.9ms
Command exited with non-zero status 26
./deltablue 10 gcc secs 0.00, mem 652, cpu 100%
DeltaBlue   C   gcc 100x    0.51ms
Command exited with non-zero status 28
./deltablue 100 gcc secs 0.05, mem 652, cpu 100%
DeltaBlue   C   gcc 1000x   0.494ms
Command exited with non-zero status 30
./deltablue 1000 gcc secs 0.49, mem 652, cpu 100%
DeltaBlue   C   gcc 10000x  0.4947ms
Command exited with non-zero status 32
./deltablue 10000 gcc secs 4.94, mem 652, cpu 99%
DeltaBlue   C   gcc 100000x 0.49071ms
Command exited with non-zero status 34
./deltablue 100000 gcc secs 49.08, mem 656, cpu 99%
DeltaBlue   C   gcc -O3 10x 0.2ms
Command exited with non-zero status 30
./deltablue-o3 10 gcc -O3 secs 0.00, mem 656, cpu 100%
DeltaBlue   C   gcc -O3 100x    0.26ms
Command exited with non-zero status 32
./deltablue-o3 100 gcc -O3 secs 0.02, mem 656, cpu 100%
DeltaBlue   C   gcc -O3 1000x   0.231ms
Command exited with non-zero status 34
./deltablue-o3 1000 gcc -O3 secs 0.23, mem 660, cpu 100%
DeltaBlue   C   gcc -O3 10000x  0.2283ms
Command exited with non-zero status 36
./deltablue-o3 10000 gcc -O3 secs 2.28, mem 660, cpu 99%
DeltaBlue   C   gcc -O3 100000x 0.22498ms
Command exited with non-zero status 38
./deltablue-o3 100000 gcc -O3 secs 22.50, mem 660, cpu 99%
DeltaBlue   d8      10x 4ms
./d8 deltablue.js -- 10 secs 0.05, mem 15416, cpu 156%
DeltaBlue   d8      100x    0.71ms
./d8 deltablue.js -- 100 secs 0.08, mem 15580, cpu 160%
DeltaBlue   d8      1000x   0.251ms
./d8 deltablue.js -- 1000 secs 0.26, mem 19884, cpu 120%
DeltaBlue   d8      10000x  0.2002ms
./d8 deltablue.js -- 10000 secs 2.01, mem 29160, cpu 103%
DeltaBlue   d8      100000x 0.19085ms
./d8 deltablue.js -- 100000 secs 19.09, mem 46432, cpu 100%
DeltaBlue   Dart        10x 4.8ms
dart DeltaBlue.dart 10 secs 0.11, mem 14896, cpu 93%
DeltaBlue   Dart        100x    0.7ms
dart DeltaBlue.dart 100 secs 0.13, mem 16264, cpu 100%
DeltaBlue   Dart        1000x   0.202ms
dart DeltaBlue.dart 1000 secs 0.26, mem 16268, cpu 101%
DeltaBlue   Dart        10000x  0.1434ms
dart DeltaBlue.dart 10000 secs 1.49, mem 16228, cpu 100%
DeltaBlue   Dart        100000x 0.13698ms
dart DeltaBlue.dart 100000 secs 13.76, mem 18216, cpu 101%

[Peter StJ]

So this is basically a "tweaked" version of a benchmark in which strangely enough Java is running as fast as C but Dart is running twice as fast? Am I missing something? "Is this the real world"?

[Greg Lowe]

My understanding is that Deltablue was originally written in Smalltalk, and used to benchmark Smalltalk VM implementations. The benchmark was ported to C and Java at Sun Microsystems for benchmarking the JVM. The V8 and Dart teams also ported this to benchmark for their languages/VMs.

I believe the benchmark is useful for benchmarking polymorphic calls and array access.

VM's have better information about the program structure at runtime than a static ahead of time compiler has a compile time, so in theory it is possible for them to generate better machine code. It appears that for this benchmark that this is the case.

But I'm sure someone here on the list could give you a more detailed answer. I'd be curious to know more too.