Rough proposal for an API to improve on requestAnimationFrame

[Kevin Gadd]

Hey all,

After a long discussion with vlad and bz a couple days ago, I've put
together a rough proposal for an API that would improve on
requestAnimationFrame:
https://docs.google.com/document/d/1Z9pqFkumi8UTOmKhvHQoXNCzvtAe1Pn0B2LQgFNrSBY/edit

I attempt to provide the reasons behind the proposal, and the
necessary context to understand them, in the document. But it may also
be helpful to look at this bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=731974
And in particular, the IRC chat log attached to it.

To try and summarize here:
The current requestAnimationFrame API is a good starting point and
works well if your goal is to render smooth animations with no
computational component. Unfortunately, it's a poor fit as a
scheduling tool for games and game-like applications, and it's also
not a great fit for vertically-synced rendering. It also requires
fairly precise timer scheduling in order to produce good results.

My proposed API decouples computation and game logic from rendering,
allows rendering to be dropped in response to CPU load without
dropping logic, allows an app developer to clearly communicate their
scheduling requirements to the browser, and allows the browser to
communicate about the constraints affecting its scheduling so that
each side can adapt. Plus, it would allow an app developer to build a
game that runs smoothly and correctly even in the presence of flaky
timer scheduling (like Firefox has right now) - something I personally
suspect is valuable when talking about running on mobile.

I've attempted to keep the API as simple as possible (with some marked
'nice to have' potential routes for useful expansion) and make it
something that can be implemented as a polyfill on top of
requestAnimationFrame/setTimeout (though a polyfill would not provide
many benefits without access to the control and knowledge available to
a browser author).

If anyone has thoughts, questions, or concerns, please feel free to
share them here on the list, or by contacting me directly. I'd also be
glad to have a discussion on moz irc or google talk.

Thanks,
-kg

[Jeff Gilbert]

I filed bug 763122 to back this discussion, as opposed to bug 731974 which has a narrower scope.

-Jeff

_______________________________________________
dev-platform mailing list
dev-pl...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

[Vladimir Vukicevic]

I like it. It's a pretty simple upgrade from rAF, and closer to what
most games (or anything that wants a constant framerate) wants to use.
It'll have the same underlying timer resolution issues that we already
have with rAF, so those need to be fixed regardless -- then we should
probably implement this API in JS on top of rAF as a shim first, and
then do a native impl.

- Vlad

[kevin...@gmail.com]

I managed to get in contact with a couple devs from the Safari and Chrome projects and discuss this with them to try and understand how their respective browsers currently implement requestAnimationFrame and understand the relevant issues and challenges.

Talked w/ Dean Jackson from the Safari team:

Their requestAnimationFrame is in flux right now - it's got some current issues and they're planning to overhaul it significantly in the near future.

They've had difficulty getting every aspect of rAF quite right in their current implementation; in particular whether or not the callbacks should be issued for hidden/obscured windows and tabs.

The design of rAF is an issue for them in that it locks users to a 60hz timer; they have encountered applications that really would have preferred a 10hz setTimeout but used requestAnimationFrame instead because they were told it was the right thing to do. An optimal replacement/refinement of requestAnimationFrame, for them, would allow a content author to specify the intended update rate of their content.

requestAnimationFrame in Safari is hooked up to the display refresh/composition engine instead of going through the normal dom paint pipeline. The compositor tries to give them a callback that aligns with vsync so that they can issue requestAnimationFrame callbacks on timing that lines up well with the retrace. I believe this also means that if you repaint a canvas from within a requestAnimationFrame callback, all they have to do is recomposite instead of invalidate the page containing the canvas and do a full content repaint (I might be wrong on this).

From their perspective the basic model proposed in this API is sound: Separate update and render callbacks; the browser is free to drop render callbacks based on CPU load or based on content being obscured. With the addition of a configurable target rate for updates, their issues with updates being locked to 60hz would be addressed as well.

Dean also expressed interest in an API that allows or even encourages the browser vendor to buffer up 1-3 rendered frames to present at the next vertical sync interval so that those buffered frames can be handed to the compositor and presented at the next realistic opportunity, since this aligns well with how they already render and would dovetail nicely with video playback.

If we move forward with an API like this, Dean suggested the proposal go onto a W3C list. I don't really know anything about that. :D


Talked w/ James Robinson from the Chrome team:

Chromium has an extremely robust implementation of requestAnimationFrame. The core principle is that they attempt to call rAF callbacks at the "correct" rate for a given application. They determine the optimal callback rate based on various inputs: vertical sync timer, computational complexity, etc. Their goal is to present a steady stream of frames at a solid framerate, and optimally align that stream with vertical sync. This means that, for example, if the application is unable to run at 60hz on a 60hz monitor, they would drop to 30hz - not say, 50hz or 40hz.

Chromium's model for this is based on a pipeline of producers and consumers - at one end you have the frame consumer that pulls completely rendered frames out of the pipeline and presents them at vertical sync; on the other end, the requestAnimationFrame callback runs update logic and pushes rendering calls for a frame into the pipeline. They attempt to align the timing on one end of the pipeline (the producers) with the timing on the other end (the consumers), so that eventually the requestAnimationFrame callbacks will drift into near-perfect alignment with vertical sync. This also implies that they can (and will, if necessary) buffer up entire frames, adding a slight amount of latency.

They also measure time spent blocked in various stages of the pipeline: Time blocked on Present/SwapBuffers, time blocked on draw calls, delays on calls to the window manager/compositor. They take that time into account when scheduling frames, so that if the GPU is heavily loaded or the window manager is having trouble keeping up, they will avoid overloading those parts of the system with lots of buffered frames/work. (On my machine, I can actually reproduce this - if I play fullscreen video on my secondary monitor, Chrome will sometimes drop game content down from 60hz to 30hz in response to GPU load and the Aero compositor taking longer to respond.)

At present, Chromium actually hands a 'presentation time' to requestAnimationFrame callbacks that allows them to adjust to callback scheduling variability. They've discussed actually passing a second time to the callback, which is the time at which they anticipate the frame actually being presented to the screen; it's hard to accurately determine this, though, so it would be more informative ('this is roughly how long you have to do your work before the next vsync') than precise ('this is when your next frame is').
This particular detail is inspired by CVDisplayLink, a system they hook into on OS X for presentation: http://developer.apple.com/library/mac/#documentation/QuartzCore/Reference/CVDisplayLinkRef/Reference/reference.html

Since their WebAudio implementation provides very low latency and precise scheduling of audio events, they would optimally like to be in a situation where timing can be synchronized with audio timing, so that game logic, rendering, and audio are all running off the same precise clock. On OS X, CoreVideo and CoreAudio use the same clock so this is possibly easier there.

Right now Chromium's scheduling behavior has a few different modes depending on what kind of content is in a page: WebGL content, for example, will put you into a GPU-accelerated compositing mode, and there are a couple experimental modes that can be turned on in browser configuration.

James described the model they are attempting to move to for scheduling. It's complicated so I'll just paste it here:

jamesr: we basically take a target framerate and a phase factor and then user a timer to produce a set of ticks at the desired framerate (with some careful logic to take care of poor timer precision without drifting)
jamesr: and then produce frames off of that
jamesr: the target framerate is based off of the screen's parameters and the phase factor is used to make sure we don't get aliasing if we are too close to the vsync line
jamesr: then we monitor for backpressure - which may come from the SwapBuffers, from some GL call taking a long time, or from deeper in the pipeline (like the window manager)

They're only part-way there, though, so it's a work in progress for them. It sounds like it does a lot to help them reduce judder and aliasing while still presenting vertically-synced content.

We also discussed the idea of the application being responsible for dropping rendering workloads instead of the browser - i.e. a game can somehow determine that it is overloading the system and as a result not hitting 60fps, and in response, only perform draw calls on every other frame.

Another example he gave was that a physics based game out there runs at 60hz, but only performs physics updates at 30hz - the alternate frames are simply an interpolation of previous game state, to reduce CPU load. This is definitely not a case I gave much thought to when preparing my initial proposal.

One thing James expressed interest in is the idea of providing more data to consumers of requestAnimationFrame so they can understand the constraints the browser is under and how well things are performing, so that the application can take steps to reduce the amount of work it does and get closer to 60hz.

One interesting difference from the gecko model for rAF (as I understand it) is that in Chromium, requestAnimationFrame callbacks occur as sort of a 'last step' in the rendering pipeline - after invalidation has occurred from DOM changes or a composited layer has moved, and they're getting ready to composite, they kick off requestAnimationFrame callbacks so that canvases can get updated. This means that in the case where a game doesn't interact with the DOM at all, the main loop is essentially 'requestAnimationFrame -> draw calls -> composite -> present' without involving any DOM or invalidation machinery. (I might have some of the details wrong here)

A common theme in this discussion is the importance of trying to improve the consistency between browsers - getting Gecko, Safari, Chrome and IE all using the same basic model for scheduling requestAnimationFrame so that developers will be able to build working apps that deliver smooth framerates without having to spend tons of time understanding each browser.


In the next week or so I'll probably spend some time describing concrete use cases and how they work under the existing requestAnimationFrame model, so that we can consider how they would work under the new API proposal. I figure that will help us understand differences between each browser's version of rAF, as well.

-kg

[Robert O'Callahan]

Thanks for looking into this!

On Wed, Jun 20, 2012 at 10:47 AM, <kevin...@gmail.com> wrote:

> I believe this also means that if you repaint a canvas from within a
> requestAnimationFrame callback, all they have to do is recomposite instead
> of invalidate the page containing the canvas and do a full content repaint
> (I might be wrong on this).
>

We do something like this in Gecko. We call it "empty layer transactions".
Whenever, between two frames, the only thing that's changed are the
contents of canvases or video, we can skip the regular content paint path
and just recomposite the layer tree.

One question that came up last time we discussed this on #gfx is whether,
when we can only produce 50fps, it's better to draw 50fps or drop down to
30fps. (Maybe we could give apps control over that, but we'd still need to
choose a sensible default.) Do you have any insight into that?

Rob
--
“You have heard that it was said, ‘Love your neighbor and hate your enemy.’
But I tell you, love your enemies and pray for those who persecute you,
that you may be children of your Father in heaven. ... If you love those
who love you, what reward will you get? Are not even the tax collectors
doing that? And if you greet only your own people, what are you doing more
than others?" [Matthew 5:43-47]

[Kevin Gadd]

I don't have conclusive information on the subject, but from what I
know and from my testing:

If your game is able to run at a fairly consistent 50fps, 50fps
vsynced may in fact be better than 30fps vsynced. You won't be running
at 60hz, but it is close enough that it will probably look smooth in
most cases, and the perceived framerate will be higher than 30.
On the other hand, if your framerate is extremely variable, players
are able to perceive those fluxuations - it is visible that the game
is 'speeding up' and 'slowing down' even if the game logic is running
at exactly the right rate. I think this is why many games cap their
framerate at 30hz instead of drifting up into the 40-50 range when
possible, to eliminate that effect. On the other hand, many of those
30hz games will dip down into the 15-20 range, and they don't limit
their framerate to 15 - there's some sort of a sweet spot you want to
hit, here.

Personally I wouldn't go to the trouble of trying to figure out
precisely when to drop to 30hz - if you get it wrong, the consequence
is that a game now suddenly runs slow for no obvious reason. Chromium
actually has this problem right now - if I play fullscreen video on my
secondary monitor, it brings my GPU load up to near 100%, so chromium
backs off its framerate to 30hz; but if Chromium tried to run at 60hz,
my GPU drivers would clock the card up to its normal speed (instead of
the 2D speed) and both the video and game would be running at full
framerate. So in that particular case, their attempt to be smart
results in a worse experience for me as a user.

Ideally with the proposed API, developers would have the choice of
either method - the default, as specified, would be to get as close as
possible to 60hz (while still maintaining vsync), but a developer
could examine the performance data being handed in via callbacks, and
if the game is consistently running below 60, they could adjust the
target framerate to 30.

-kg

On Wed, Jun 20, 2012 at 6:36 AM, Robert O'Callahan <rob...@ocallahan.org> wrote:
> Thanks for looking into this!
>
>
> On Wed, Jun 20, 2012 at 10:47 AM, <kevin...@gmail.com> wrote:
>>

>> I believe this also means that if you repaint a canvas from within a
>> requestAnimationFrame callback, all they have to do is recomposite instead
>> of invalidate the page containing the canvas and do a full content repaint
>> (I might be wrong on this).
>
>

> We do something like this in Gecko. We call it "empty layer transactions".
> Whenever, between two frames, the only thing that's changed are the contents
> of canvases or video, we can skip the regular content paint path and just
> recomposite the layer tree.
>
> One question that came up last time we discussed this on #gfx is whether,
> when we can only produce 50fps, it's better to draw 50fps or drop down to
> 30fps. (Maybe we could give apps control over that, but we'd still need to
> choose a sensible default.) Do you have any insight into that?
>
> Rob
> --
> “You have heard that it was said, ‘Love your neighbor and hate your enemy.’
> But I tell you, love your enemies and pray for those who persecute you, that
> you may be children of your Father in heaven. ... If you love those who love
> you, what reward will you get? Are not even the tax collectors doing that?
> And if you greet only your own people, what are you doing more than others?"
> [Matthew 5:43-47]
>

--
-kg