Sunspider expects "run" function in shell...

[Joe Millenbach]

Looking at https://trac.webkit.org/wiki/JSC webkit's command line shell supports a "run" function that returns how many milliseconds it took to run a file.  And in the Sunspider framework source code they indicate this is a more accurate timer, but that they can't use it for tests with data sets.

If I try to use the d8 binary with their trunk svn Sunspider code I get errors about the "run" function not existing.  To work around it I use the same "load" function that they use for tests with data sets.

Has there been talk of adding the "run" function to d8 before?  Is there likely any accuracy benefit as the Sunspider comments indicate?

[Jakob Kummerow]

You mean, some milliseconds are more accurate than others? As in, they are precisely 1.0ms long each, rather than just roughly 1ms?

Now, assuming there is a high-accuracy kind of milliseconds, do you really care whether a so-called benchmark finishes in 2ms or 2.0ms? When there's easily a 25% chance that running the same test again will take either 1ms or 3ms?

SunSpider is beyond saving. Trying to make its results look more trustworthy is futile.

That said, here's a "run" function that returns the number of milliseconds it took to run a file:

function run(file) {

  var start = Date.now();

  load(file);

  return Date.now() - start;

}

I don't see any reason why doing this with C++ in the shell would have different accuracy (in either direction). You could argue for a way to get timestamps that are precise to the microsecond, but then again, any benchmark that deserves the name runs long enough that millisecond precision is entirely sufficient, so what would be the point?

--
--
v8-users mailing list
v8-u...@googlegroups.com
http://groups.google.com/group/v8-users
---
You received this message because you are subscribed to the Google Groups "v8-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to v8-users+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Joe Millenbach]

By "more accurate" I meant you might get a smaller spread between results (which I think should be precision) and maybe minus some interpreted code overhead (which I think is accuracy).  I went along with the idea in the comment based on the assumption that native code might be different.  And yes I was imagining the two versions might use different timestamp implementations which could give better or worse results.

But I think I have my answer.  No, this is not something that is desired by the V8 devs.

Your logic for the issue with Sunspider's run time makes sense, but it isn't my choice whether we run Sunspider or not.  It is just another data point, and I'm doing my best to get a good one.

Thanks for your time,

Joe

[Jakob Kummerow]

On Mon, Mar 18, 2013 at 9:19 PM, Joe Millenbach <jmill...@gmail.com> wrote:

By "more accurate" I meant you might get a smaller spread between results (which I think should be precision) and maybe minus some interpreted code overhead (which I think is accuracy).  I went along with the idea in the comment based on the assumption that native code might be different.  And yes I was imagining the two versions might use different timestamp implementations which could give better or worse results.

Well, in JSC you apparently have both methods at your disposal (the original version, and you can just as well run the modifications you'd have to make for d8), so you can compare them to your heart's content.

But I think I have my answer.  No, this is not something that is desired by the V8 devs.

Your logic for the issue with Sunspider's run time makes sense, but it isn't my choice whether we run Sunspider or not.  It is just another data point, and I'm doing my best to get a good one.

Sunspider results are meaningless for modern JS engines. Seriously. Tell that to whoever made the choice for you. 

[Yang Guo]

Jakob is right.

In theory, there is a small difference between high resolution and low resolution timers, if they both round to an even lower resolution. Example, timer A resolves to 1/100 ms, timer B resolves to 1/10 ms. An interval of exactly 1.46 ms would be clocked by A as 1.46, by B as 1.5. A would round the final result to 1ms, and B to 2ms. But given that in both cases the error (0.46ms and 0.54ms) is already very significant compared to the exact result of 1.46, it makes virtually no difference: a benchmark is worthless regardless of whether the relative error is 31% or 37%. This could be solved if the interval was by orders of magnitude larger than the resolution which we round to (1ms), lowering the relative error. That's one of the reasons why Sunspider is a bad benchmark.

Yang

[Joe Millenbach]

I'll pass this all along.  Hopefully we can get people to care about it less internally, but I doubt it'll ever go away as long as anyone else outside (reviewers, competitors) quote results from it.  Like Dhrystone.  We know it isn't a true measure of performance, but we have to still run it.

--

--
v8-users mailing list
v8-u...@googlegroups.com
http://groups.google.com/group/v8-users
---

You received this message because you are subscribed to a topic in the Google Groups "v8-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/v8-users/EPoS0Idbf5I/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to v8-users+u...@googlegroups.com.

[Yang Guo]

If you need to insist on Sunspider, one thing you could do to improve it is to extend the time the benchmark runs. For example for each of the benchmarks you can repeat it until the run time exceeded one second, then divide final run time by the number of runs.

Yang