Update on graphics benchmark performance vs IE9
Robert O'Callahan
ran some commonly-cited tests on my (fast) laptop.
The ** numbers are Firefox with dom.min_timeout_value changed from 10 to 1.
Microsoft FishIE Tank (local copy modified to use 2000 fish): IE 37fps, FF
38fps
Microsoft Speed Reading: IE 7s, FF 6s
GUIMark2 HTML5 Chart: FF 18.86fps, IE 18.2fps
GUIMark2 HTML5 Game: FF 58.87fps, IE 60fps
GUIMark2 HTML5 Text Columns: FF 34.03fps, IE 44.97fps
http://www.kevs3d.co.uk/dev/asteroidsbench: IE 2849, FF 611
Microsoft Mr Potato Gun: IE 42008, FF 15347 (** 28480)
Microsoft Preschool: IE 5.5s, FF 14.05s (** 6.12)
Paul Rouget's HWACCEL: IE 277, FF 85 (** 254)
So:
* We pull ahead on FishIE Tank and Speed Reading (just).
* It's mostly a dead heat on GUIMark 2 except for Text Columns, which is
likely layout not graphics.
* http://www.kevs3d.co.uk/dev/asteroidsbench is definitely us being bad.
I'll look into it.
* In Mr Potato Gun, Preschool and HWACCEL, our score is being dramatically
lowered by the 10ms minimum we impose on setTimeout. Changing that clamp to
1ms makes our score much better; probably it would get better still if we
eliminated the clamp, but setting the pref to 0 seems to break things. Of
course, the clamp is mostly a non-issue for real users since you can't get
more than 60fps onto the screen normally. But the numbers are what they are
and people are drawing conclusions from them:
http://www.conceivablytech.com/6020/products/firefox-4-rc-vs-ie9-rc-the-first-duel/2
Quote on HWACCEL: "IE9 virtually destroys Firefox on the desktop". I expect
we'll see a lot more such comparisons soon.
So what can we do about this?
Note that Paul made HWACCEL cap the reported frame rate at 60, which is the
right thing to do, but conceivablytech somehow reported numbers greater than
60 anyway. I had to modify a local copy of the test to do that.
So here are things we should be checking when people report graphics
benchmark results:
* Was D3D10 GPU acceleration enabled on Firefox? I've seen comparisons where
it wasn't, probably due to blacklisting, and of course we get crushed. It's
not necessarily an unfair comparison since that's what the user sees, but
ideally people would report the results as "Firefox with GPU acceleration
disabled" since it really is almost a different browser.
* Does the benchmark report better results in Firefox if you set
dom.min_timeout_value to 1? If so, it's probably bogus.
Tryserver builds are available at
http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/rocal...@mozilla.com-16644bf2dbd2for
your enjoyment.
Rob
--
"Now the Bereans were of more noble character than the Thessalonians, for
they received the message with great eagerness and examined the Scriptures
every day to see if what Paul said was true." [Acts 17:11]
Boris Zbarsky
> probably it would get better still if we eliminated the clamp
We plan to drop it to 4ms, for what it's worth.
> but setting the pref to 0 seems to break things
We should probably file a bug on that, for what it's worth. It
shouldn't break... In particular, we use 0 intervals if you do an
un-nested |setTimeout(..., 0)|, so they work in at least some cases!
> Of course, the clamp is mostly a non-issue for real users since you can't get
> more than 60fps onto the screen normally. But the numbers are what they are
> and people are drawing conclusions from them:
Does IE9 not have a setTimeout clamp? I was pretty sure it did.
> So what can we do about this?
What are those numbers actually measuring? I can sort of imagine
measurement setups for potato gun and HWACCEL that would be affected by
the timeout clamp in the way you observe (e.g. adding up how many times
X is done, where each callback does X for 2ms and you try to have 500
callbacks per second), but the Preschool result is just weird.
Though I wonder whether IE9 just uses the 4ms clamp and what you're
seeing is that our timers on Windows don't actually go down to below
10-15ms all that well... What does a graph of score vs timeout clamp
look like for you?
> Note that Paul made HWACCEL cap the reported frame rate at 60, which is the
> right thing to do, but conceivablytech somehow reported numbers greater than
> 60 anyway.
We should really ask them about that.
> * Does the benchmark report better results in Firefox if you set
> dom.min_timeout_value to 1? If so, it's probably bogus.
Amen.
-Boris
Robert O'Callahan
(vs 38fps with patches), and Speed Reading is at 32s (vs 6s with patches).
Everything else is about the same, which probably indicates that even the
1ms DOM timeout clamp is still the limiting factor on scores for Mr Potato
Gun, Preschool and HWACCEL.
BTW^2, on GUIMark2 HTML5 Text Columns, I had FF at 34fps in my local build.
But with a PGO build I get 47fps, faster than IE9. Impressive!
Boris Zbarsky
> Everything else is about the same, which probably indicates that even the
> 1ms DOM timeout clamp is still the limiting factor on scores for Mr Potato
> Gun, Preschool and HWACCEL.
We really need to figure out what those tests are measuring. If HWACCEL
is claiming to measure fps, then the timeout clamp should only matter if
it's clamping the fps (to a first approximation). So as long as the fps
it measures is under 100 the current clamp should be fine, and as long
as it's under 1000 the 1ms clamp should be fine, assuming our timer impl
copes.
-Boris
Robert O'Callahan
> Does IE9 not have a setTimeout clamp? I was pretty sure it did.
>
Actually it seems to have a 4ms clamp.
Boris Zbarsky
> Actually it seems to have a 4ms clamp.
OK. So I took a look at exactly what HWACCEL does.
Long story short, it runs this code:
var fps = 0;
setInterval(function() {
// remember value of |fps| in running average
fps = 0;
}, 1000);
function draw() {
// do some stuff
fps++;
setTimeout(draw, 0);
}
If "some stuff" takes time N ms and the timeout clamp is C ms, then this
will measure an fps of 1000/(N + C) (because the call to reset the
draw() timeout is at the end of draw(), not at the beginning). That's
_if_ all the timers are perfect. I think this is a bug in the benchmark.
The second bug is the assumption that the 1s interval will always fire
1s apart. If it's slightly laggy, the benchmark will measure a too-high
fps.
If we posit that our timers are perfect, a score of 85 with a 10ms clamp
means that "some stuff" takes 1.76ms or so. The 254 with a 1ms clamp
then indicates mostly that our timers are imperfect. ;)
IE's 277 indicates that either their clamp sometimes drifts down or that
their 1s interval is laggy. We could tell which by fixing the
fps-measuring function to divide the value of |fps| it sees by the
actual number of seconds elapsed since it last fired.
-Boris
Robert O'Callahan
here:
http://people.mozilla.org/~roc/setInterval-period-canvas.html
If you click on the left button, we do a setInterval(0) with 1000
iterations, drawing to a canvas in each iteration. I consistently get around
3.2ms per iteration. If you click on the right button, we do the same but
without drawing to a canvas. I consistently get around 4ms per iteration.
If you replace the "draw" code with any canvas operation that actually
modifies the canvas --- e.g. stroke/fill/drawImage/drawText --- you get the
same results. If you replace the draw code with a save/restore pair on the
context, the clamp goes back up to 4ms.
Awfully strange bug. If you set our dom.min_timeout_value to 4 to match
IE9's nominal value, this bug seems to explain the remaining performance
advantage IE9 has over my FF build in Mr Potato Gun, Preschool and HWACCEL.
Omega X
> Tryserver builds are available at
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/rocal...@mozilla.com-16644bf2dbd2for
> your enjoyment.
>
> Rob
404 Not found.
--
==================================
~Omega X
MozillaZine Nightly Tester
Omega X
> On 3/8/2011 8:23 PM, Robert O'Callahan wrote:
>
>> Tryserver builds are available at
>> http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/rocal...@mozilla.com-16644bf2dbd2for
>>
>> your enjoyment.
>>
>> Rob
>
>
> 404 Not found.
>
I'm sorry. The "for" got caught in the link.
http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/rocal...@mozilla.com-16644bf2dbd2/
Gary
> On 3/9/2011 3:45 AM, Omega X wrote:
>
> > On 3/8/2011 8:23 PM, Robert O'Callahan wrote:
>
> >> Tryserver builds are available at
>
ROC your Tryserver build is unbelievably fast as far as the IE9 tests
go. For those folks that use these demos to gauge how much faster IE9
is than Fx4 this will be a shock to them when this is released.
Hope you get this out asap.
Jonathan Watt
> On 3/8/11 10:52 PM, Robert O'Callahan wrote:
>> Actually it seems to have a 4ms clamp.
>
> OK. So I took a look at exactly what HWACCEL does.
>
> Long story short, it runs this code:
>
> var fps = 0;
> setInterval(function() {
> // remember value of |fps| in running average
> fps = 0;
> }, 1000);
>
> function draw() {
> // do some stuff
> fps++;
> setTimeout(draw, 0);
> }
>
> If "some stuff" takes time N ms and the timeout clamp is C ms, then this
> will measure an fps of 1000/(N + C) (because the call to reset the
> draw() timeout is at the end of draw(), not at the beginning). That's
> _if_ all the timers are perfect. I think this is a bug in the benchmark.
>
> The second bug is the assumption that the 1s interval will always fire
> 1s apart. If it's slightly laggy, the benchmark will measure a too-high
> fps.
In current browsers (including IE9) the time between each setInterval callback
normally averages out over time to the interval specified. The only time this
seems to not be the case is if the callback /can't/ be called frequently enough
- particularly, if the callback itself takes longer than the interval specified,
then setTimeout will fire a lot less frequently than requested. I'm assuming
that "remember value of |fps| in running average" doesn't take a full second though.
That said, I agree the test really should be checking the elapsed time to avoid
the transient fps values being displayed from being knocked off by slow painting!
Boris Zbarsky
> In current browsers (including IE9) the time between each setInterval callback
> normally averages out over time to the interval specified.
Sort of. Sometimes. Nothing really guarantees that, even just in our
implementation.
> The only time this seems to not be the case is if the callback /can't/ be called frequently enough
No, that's really not the only time you can get drift in the
setInterval... There are all sorts of other ways it could happen.
-Boris