Re: RAF Aligned Input

[Rick Byers]

[+input-dev - this thread is awesome, thank you so much Alexander!]

On Wed, Sep 14, 2016 at 9:29 AM, Dave Tapuska <dtap...@google.com> wrote:

#3) This is really valuable input. Tim and I were debating about the current solution. And it isn't much work to do an alternate solution where we grab the events that are in the queue at the rAF time and process those instead of grabbing events one at a time during the rAF callback. I'll see if I can get a patch together on this and you can re-run it to see if it solves the issue. Will let you know when it is in a build.

Guaranteeing exactly one event per frame sounds like a good idea to me.  In some applications it's not unreasonable to do a bunch of work in your touchmove/mousemove handler, ideally developers wouldn't need to reason about how to debounce that work so they're not wasting effort.

dave.

On Wed, Sep 14, 2016 at 8:30 AM, Alexander Suvorov <alexs...@unity3d.com> wrote:

Hello Dave.

#1) Historical points API would be a great feature, as Unity already has a mechanism to provide the event history to the user scripts. Nevertheless, at the current moment this functionality is not widely used in WebGL builds (i.e. the games I know seem to only rely on 1 mouse move event per frame, even though they can technically obtain additional events). Historical points API should be therefore considered not critical for Unity WebGL at the current moment, and should not block the shipping of the rAF-aligned input.

#2) As I can see on the charts, the mouse input event handler is executed during VBlank (in some cases it misses VBlank and is executed a bit later). It seems to me that this timing only gives you an advantage if you are lucky enough to receive it during VBlank and adjust your rendered content appropriately. In case of Unity however, the updated mouse position will go all the way though the main rendering loop and will only affect the next rendering frame, so it does not make much difference whether the event handler has been executed during VBlank or right before the rAF callback, as all the input events are just put into the internal queue in order to be processed in the main loop in rAF callback. For the same reason there is not much of performance difference for Unity WebGL, because the only operation that actually happens in the event handler is putting the event into the internal input queue. However, getting higher resolution of events (through the future historical points API) sounds really great!

Makes sense.  We just want this behavior to be the default, rather than something the app needs to be smart enough to opt-in to.   This should improve performance and smoothness of a bunch of existing web pages, and have no impact on sophisticated apps like Unity.  This basically extends the Android "project butter" lessons to Chrome on all platforms.

#3) This point seems to be a little bit risky to me. As charts have shown, the handler can be executed up to 5 times or more depending on the mouse frequency. So if the developer has a normal frequency mouse, he would expect the handler to be executed only once (or maximum twice), and can put some heavy code into the handler, while all the testing will show decent performance. Still some end user with a high-frequency mouse (or another high-frequency device that controls the mouse position), might experience performance degradation, which would be absolutely unexpected. Of course, due to the design tradeoff this risk might be acceptable, still it would be really great to have a guarantee that the handler is executed only once per frame, so there will be no unexpected issues with future devices.

Best Regards,

Alexander

On Mon, Sep 12, 2016 at 3:53 PM, Dave Tapuska <dtap...@google.com> wrote:

#1) We are looking at providing a historical points API which would give you all the coalesced points thus far. So in fact it would give you more resolution than you have today; see issue 22. What we can do is consider mouse and touch separately. Today touch is essentially at most 60 Hz because of the way android delivers touch events.

Nit: this isn't quite true.  Android also has a historical points API (see Batching), and so when we expose such an API to the web we'll be sure to ask the OS for any historical points it has as well.  Many devices (eg. Nexus 5) sample touch at 120hz.

 

Mouse is based on the hardware of the mouse so it really depends on how many applications depend on >60 Hz mice. Do you have any opinions in this space? Should we block shipping rAF aligned input for mouse on the historical points API but go ahead shipping it for touch? These are all open questions we have.

#2) Performance is the benefit. The hover state, hit testing, etc can all can get updated multiple times per frame. If we just don't process the mouse events until we know it makes sense to produce a frame there is no visual difference to the user. But yes drawing apps can be affected (as in #1) by lower resolution mouse moves. And the real question is what impact does this have. 

#3) There was a design tradeoff to be made processing between exactly one mousemove event per rAF call or to always process the freshest input. I took the later approach in the design. But we can easily change it if there is an explicit problem. So yes there is potential that when as a first mouse move is being processed, a second event is queued before the first is done processing the second event will also get handled before the rAF callback.

Thanks very much for your feedback. I do really appreciate it.

dave.

On Mon, Sep 12, 2016 at 8:58 AM, Alexander Suvorov <alexs...@unity3d.com> wrote:

Hello.

I think this all makes sense. Though after testing on 55.0.2858.0 canary 32-bit --enable-features=RafAlignedInput I got a few questions:

1) Since coalesced mouse move events are not sent to the mousemove, then how it is possible to access them? Consider drawing a circle with a mouse at low fps (let's say 5-10 fps), even though you get only one mousemove call per frame, in order to draw a smooth curve you need access to the event history (preferably with accurate timestamps).

2) The rAF callback is executed after the actual vsync (as displayed on the attached graphs). So what exactly is the practical difference between executing the event handler a moment before the rAF callback versus executing the event handler right before the rAF callback? (the latter scheme can be implemented by the application itself, as it can use it's own buffer for the incoming events and process this buffer in the rAF callback)

3) It seems that during the period between the event processing signal and the rAF callback input events are processed in the usual manner (just with a small delay), which means that the mousemove handler can be executed multiple times during this short period (see the attached images, where frame callback takes about 1/120 sec, and the requestAnimationFrame takes the rest 1/120 sec accordingly, the description of the images can be found here http://files.unity3d.com/alexsuvorov/webgl/raf/). If those calls are intentional, then what exactly is the practical purpose of this?

Hope this helps, and thank you very much for providing information about this scheme in advance.

Best Regards,

Alexander

On Fri, Sep 9, 2016 at 4:50 PM, Dave Tapuska <dtap...@google.com> wrote:

Tim pointed out an obvious thing that was missed. This went into M55; so it is only available in Canary right now.

dave.

On Fri, Sep 9, 2016 at 10:40 AM, Dave Tapuska <dtap...@chromium.org> wrote:

Alex,

I wanted to reach out and ask you for some feedback; specifically as you were quite helpful on issue 599152. A few weeks ago I landed a concept of RAF aligned input.

Where touchmove, mousemove and mousewheel events are dispatched to javascript just before the RAF event is fired.

It is enabled via passing an arg to the command line "--enable-features=RafAlignedInput"

Rick wrote an example a while back that demonstrates the timing between the event and the raf callback. 

We are planning on trying the feature out on some canary and dev builds but I wanted to reach out to you if you had anything that you could explicitly try with it and provide feedback if you had any.

thanks,

dave.

[Dave Tapuska]

As I indicated my fix just landed https://chromium.googlesource.com/chromium/src.git/+/ed255334e459137808c04ede96e8a3253ca3dde5 It should be included in a Canary build tomorrow morning. Depending on roll out you might see it the same day or a day after.

dave.