Dart Code Snippet Benchmarks

[Hans Van den Keybus]

Hi All,

Just wanted to let you know that I started a library to benchmark several coding styles / code snippets.

The project can be found here:
https://github.com/dotdotcommadot/dotdotcommadot_dart_benchmarks

The goal is to find out which writing style is faster than the other.

A few remarkable conclusions I can already make are

- Foreach() loops are about 2x slower than using a for(var i...) loop

- When compiled to JavaScript, Operator Overloading works 8x (!!) faster than calling a regular method

- In the Dart VM, String Interpolation ('$hello $world') is 3x faster than the + operator (hello + ' ' + world)
However compiled to JavaScript, this makes no difference.

More tests will be coming up.
Feel free to contribute or make suggestions!

[Joel Trottier-Hébert]

That's interesting! Thanks for the share :)

--
For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new

To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Andrew Skalkin]

I think this is a great idea. In particular, for number crunching, it would be great to see the performance impact for cases when Dart compiler identifies a field as a non-nullable one, and therefore gets rid of null checking when accessing the fields/variables.

[Bob Nystrom]

On Mon, Jun 29, 2015 at 6:37 AM, Hans Van den Keybus <hvande...@gmail.com> wrote:

- Foreach() loops are about 2x slower than using a for(var i...) loop

Your forEach() benchmark is doing something fundamentally different than the for loop one. The former is iterating over a list and the latter is just incrementing a number. You should walk a list with for as well to get an equal comparison. (Because real world code sure as hell should not be creating a list of integers and iterating over them just to do a numeric loop!)

Also, the bodies of the loops are empty. If you aren't careful, the optimizer will notice that and completely eliminate your loop. Benchmarking modern language implementations with complex JITs and optimizers is hard.

For what it's worth, this script:

const trials = 10;

const iterations = 10000000;

void time(String name, benchmark()) {

  var start = new DateTime.now();

  for (var i = 0; i < trials; i++) {

    benchmark();

  }

  var elapsed = new DateTime.now().difference(start).inMilliseconds / trials;

  print("$name: $elapsed ms/trial");

}

void main() {

  var list = [];

  for (var i = 0; i< iterations; i++)

  {

    list.add(i);

  }

  time("forEach()", () {

    var sum = 0;

    list.forEach((i) {

      sum += i;

    });

    if (sum == 1000) print("prevent dead code elimination");

  });

  time("for()", () {

    var sum = 0;

    for (var i = 0; i < list.length; i++) {

      sum += i;

    }

    if (sum == 1000) print("prevent dead code elimination");

  });

  time("while()", () {

    var sum = 0;

    var iterator = list.iterator;

    while (iterator.moveNext()) {

      sum += iterator.current;

    }

    if (sum == 1000) print("prevent dead code elimination");

  });

}

On my Mac laptop outputs this when run on the VM:

forEach(): 46.5 ms/trial

for(): 6.6 ms/trial

while(): 15.7 ms/trial

I don't promise that my script isn't flawed either. :)

- bob

[Yissachar Radcliffe]

Shouldn't your for loop do:

sum += list[i];

For a fair comparison? Currently foreach and while perform list access, while for does not.

On my machine this only adds 3ms to the for loop time, so the overall conclusion remains the same. Though for some reason my while speeds are much closer to the foreach speeds than yours are:

foreach(): 52.6 ms/trial
for(): 9.6 ms/trial
while(): 43.1 ms/trial

[Don Olmstead]

What about a for in loop? I had thought those would be the same as the for though not positive.

[Bob Nystrom]

On Mon, Jun 29, 2015 at 2:24 PM, Yissachar Radcliffe <ydrad...@gmail.com> wrote:

sum += list[i];

Oops, yes! :)

- bob

[Hans Van den Keybus]

Thanks all for the very valid reply!

It is true that some for loops resulted in an empty body, when compiled to JS.
Also, the loops now perform similar functionality.

This has been fixed.

@Andrew Skalkin: Can you give me an example of what you mean?

Grts,
Hans

[Andreas Kirsch]

Are you sure that the operator overloading version works correctly? Maybe there is a bug and dart2js is optimizing away your expression?

--

[Hans Van den Keybus]

@Andreas:
I was just looking into it myself.
I see something very strange going on here.
The generated JS code, is:

NoOperatorOverloading: {

    "^": "BenchmarkBase;myClazzes,myClazz,name,emitter",

    setup$0: function() {

      var t1, i;

      for (t1 = this.myClazzes, i = 0; i < 100000; ++i)

        t1.push(new N.Clazz(i));

    },

    run$0: function() {

      var t1, t2, i, t3;

      for (t1 = this.myClazzes, t2 = this.myClazz, i = 0; i < t1.length; ++i) {

        t3 = t1[i];

        t2.myInteger = t2.myInteger + t3.myInteger;

      }

    },

    teardown$0: function() {

    }

  },

  OperatorOverloading: {

    "^": "BenchmarkBase;myClazzes,myClazz,name,emitter",

    setup$0: function() {

      var t1, i;

      for (t1 = this.myClazzes, i = 0; i < 100000; ++i)

        t1.push(new N.Clazz(i));

    },

    run$0: function() {

      var t1, t2, i, t3;

      for (t1 = this.myClazzes, t2 = this.myClazz, i = 0; i < t1.length; ++i) {

        t3 = t1[i];

        t2.myInteger = t2.myInteger + t3.myInteger;

      }

    },

    teardown$0: function() {

    }

  },

So both are exactly the same!
Only the results are wide apart: 20551ms against 3036.

Weird stuff going on...

[Andreas Kirsch]

Are you running both from the same browser/tab/main? Swap the two around and look at the results. Maybe there is some cross-method optimization happening because both JavaScript objects are the same...

In general, you want to start with a cold environment for every benchmark probably.

[Ahmet A. Akın]

Honestly, I would not trust micro benchmarks. Yes they are fun, but there are too many pitfalls to actually measure platform speed correctly.