Frame Timing API synchronization with JS

[Ignacio Solla Paula]

hello,

we're currently working on adding visual state callbacks to the Android WebView and wonder if anyone working on the Frame Timing API has already considered synchronization with Javascript. In other words being able to retrieve the sourceFrameNumber of the frame that shows the result of executing some JS. For example:

document.documentElement.appendChild(aChild);

var now = new Date().getTime();

var sourceFrameNumber =    

    getSourceFrameNumberUsingTimingApiForFrameBefore(now);

Given that the DOM mutations (blink applying styles/layout/etc...) resulting from running some block of JS could be applied asynchronously, then the call to appendChild(aChild) could return before the DOM is updated, so the frame with sourceFrameNumber might not contain aChild. Are we planning to support this use case and if so what is the plan here?

thanks,

Ignacio

[Rick Byers]

+michaelblain@ and nduca@ for frame timing API

[Nat Duca]

Hi Ignacio -

We've so far avoided exposing the sourceFrameNumber directly since as you note, you can write down window.performance.now() (note, high resolution time!) and then look yourself for MainFrameEntries with a start time >= now.

We've wrapped some of this up into a helper requestFirstFrameCallback(cb) api in https://github.com/GoogleChrome/frame-timing-polyfill --- you do your dom mutation and then the callback fires when everything to that point has made it onscreen.

One of the things I dont love about our implementation in the polyfill is that it does dirty things to the performace timeline event buffer to avoid polling the performance timeline for newly-available frames. Its blah.

I believe we're on track for a better solution though, which you can follow along with on webperf hereish: http://lists.w3.org/Archives/Public/public-web-perf/2014Dec/0016.html

tldr, PerformanceObserver. Its kind of like mutation observers: you create an observer and say you're intrested in frame records. Then, until you disconnect() the newly created observer, you will get periodic callbacks about new performance events. Off of this, you can construct the similar requestFirstFrameCallback() but with much less mess, and still without exposing the actual source frame number as an actual property.

Hope this context helps!

[Martin Kosiba]

Nat,

  Thanks for the explanation. If I understand correctly the flow you described assumes that a commit will occur at some point in the future. This makes sense for the case Ignacio described (mutate some dom, wait for the commit) but doesn't help us with the flush API.

  First, some assumptions we're making (which may or may not be correct):

• the 'flush' API consists of 2 parts: flush request and flush response (callback), both of which take place on the UI thread. The response should only be delivered after the effects of all operations initiated on the UI thread (such as evaluateJavaScript) and all callbacks received on the UI thread (such as didComitProvisionalLoad) are reflected in the state of the active tree,
• blink may deconstruct certain operations into a set of smaller tasks, where only the final task results in changes to the LayerTree (== a call to LayerTreeHost::setNeedsCommit),
• the 'flush' API needs to guarantee that side-effects of operations _initiated_ before the call are taken into account.
  In other words, if running some JS results in tasks A, B, anc C, and if only C actually causes the LayerTree to change then if the 'flush request' was issued after the JS was evaluated the 'flush response' can't be delivered before the activation of the LayerTree corresponding to task C,
• it may be possible now (or in the future given the Blink main thread scheduler work that's currently ongoing) that the following sequence of operations takes place on the main thread:
  
  [JS eval] [Task A] [setNeedsCommit] [Task B] [commit -> early out] [Task C] [setNeedsCommit] [commit for realz]
  
  What I'm trying to say is that we're _not_ assuming we can just setNeedsCommit on the blink main thread to guarantee that all of the 'in flight' changes will be included in the next commit.
• there may not be a 'next commit' at all. If the WebView's contents are '<div> haha </div>' then a 'flush request' should still result in a 'flush response',
• introducing unnecessary latency is not desirable the 'flush response' should come back as soon as possible.

I hope this explains our reasoning. Feel free to poke at holes in the above, maybe we're trying to solve something that doesn't needs fixing.

--

Thanks!

    Martin

[Ignacio Solla Paula]

We would really appreciate your help understanding how blink works. Please find some additional questions inline.

On Fri, Dec 12, 2014 at 2:42 PM, Martin Kosiba <mko...@google.com> wrote:

Nat,

  Thanks for the explanation. If I understand correctly the flow you described assumes that a commit will occur at some point in the future. This makes sense for the case Ignacio described (mutate some dom, wait for the commit) but doesn't help us with the flush API.

  First, some assumptions we're making (which may or may not be correct):

• the 'flush' API consists of 2 parts: flush request and flush response (callback), both of which take place on the UI thread. The response should only be delivered after the effects of all operations initiated on the UI thread (such as evaluateJavaScript) and all callbacks received on the UI thread (such as didComitProvisionalLoad) are reflected in the state of the active tree,
• blink may deconstruct certain operations into a set of smaller tasks, where only the final task results in changes to the LayerTree (== a call to LayerTreeHost::setNeedsCommit),

Are operations decompose inside ThreadProxy::BeginMainFrame or somewhere else? I can see that sometimes we early-out after having requested a new commit, for example when 1) any animation requests generated by the apply or animate trigger another frame or when 2) objects that only layout when painted trigger another commit

• the 'flush' API needs to guarantee that side-effects of operations _initiated_ before the call are taken into account.
  In other words, if running some JS results in tasks A, B, anc C, and if only C actually causes the LayerTree to change then if the 'flush request' was issued after the JS was evaluated the 'flush response' can't be delivered before the activation of the LayerTree corresponding to task C,

would it be feasible to request a commit when we get the flush request and then wait for task C before delivering the flush response? 

 

• it may be possible now (or in the future given the Blink main thread scheduler work that's currently ongoing) that the following sequence of operations takes place on the main thread:
  
  [JS eval] [Task A] [setNeedsCommit] [Task B] [commit -> early out] [Task C] [setNeedsCommit] [commit for realz]
  
  What I'm trying to say is that we're _not_ assuming we can just setNeedsCommit on the blink main thread to guarantee that all of the 'in flight' changes will be included in the next commit.

Is the sequence above possible? If decomposing the tasks happens inside ThreadProxy::BeginMainFrame then I think we would see the following:

[JS eval] [Task A] [setNeedsCommit] [Task B] [BeginMainFrame start] [setNeedsCommit] [commit -> early out] [BeginMainFrame end] [Task C]  [commit for realz]

In that case we could check whether a new commit was requested before we early out, and if so wait for that commit before we deliver the flush response.

 

• there may not be a 'next commit' at all. If the WebView's contents are '<div> haha </div>' then a 'flush request' should still result in a 'flush response',

if we always request a commit when we get the flush request then we could make that commit a forced commit. A forced commit will be deliver even if there are no changes from the previous frame. This will probably keep things simple. Would this be feasible?

thanks,

Ignacio

 

[Martin Kosiba]

On Wed, Dec 17, 2014 at 7:14 PM, Ignacio Solla Paula <igs...@google.com> wrote:

We would really appreciate your help understanding how blink works. Please find some additional questions inline.

On Fri, Dec 12, 2014 at 2:42 PM, Martin Kosiba <mko...@google.com> wrote:

Nat,

  Thanks for the explanation. If I understand correctly the flow you described assumes that a commit will occur at some point in the future. This makes sense for the case Ignacio described (mutate some dom, wait for the commit) but doesn't help us with the flush API.

  First, some assumptions we're making (which may or may not be correct):

• the 'flush' API consists of 2 parts: flush request and flush response (callback), both of which take place on the UI thread. The response should only be delivered after the effects of all operations initiated on the UI thread (such as evaluateJavaScript) and all callbacks received on the UI thread (such as didComitProvisionalLoad) are reflected in the state of the active tree,
• blink may deconstruct certain operations into a set of smaller tasks, where only the final task results in changes to the LayerTree (== a call to LayerTreeHost::setNeedsCommit),

Are operations decompose inside ThreadProxy::BeginMainFrame or

somewhere else? I can see that sometimes we early-out after having requested a new commit, for example when 1) any animation requests generated by the apply or animate trigger another frame or when 2) objects that only layout when painted trigger another commit

The early out in BeginMainFrame has nothing to do with what I described. What I wanted to say is that blink sometimes does layout/parsing/style recalc/etc.. asynchronously. It's not that they're "broken down" somewhere, they're just implemented as a series of smaller tasks.

 

• the 'flush' API needs to guarantee that side-effects of operations _initiated_ before the call are taken into account.
  In other words, if running some JS results in tasks A, B, anc C, and if only C actually causes the LayerTree to change then if the 'flush request' was issued after the JS was evaluated the 'flush response' can't be delivered before the activation of the LayerTree corresponding to task C,

would it be feasible to request a commit when we get the flush request and then wait for task C before delivering the flush response? 

how do you wait for task C? How do you even know you need to wait for it or what it is?

 

 

• it may be possible now (or in the future given the Blink main thread scheduler work that's currently ongoing) that the following sequence of operations takes place on the main thread:
  
  [JS eval] [Task A] [setNeedsCommit] [Task B] [commit -> early out] [Task C] [setNeedsCommit] [commit for realz]
  
  What I'm trying to say is that we're _not_ assuming we can just setNeedsCommit on the blink main thread to guarantee that all of the 'in flight' changes will be included in the next commit.

Is the sequence above possible? If decomposing the tasks happens inside ThreadProxy::BeginMainFrame then I think we would see the following:]

There is no decomposing in BeginMainFrame, that's not what I was talking about.

 

[JS eval] [Task A] [setNeedsCommit] [Task B] [BeginMainFrame start] [setNeedsCommit] [commit -> early out] [BeginMainFrame end] [Task C]  [commit for realz]

In that case we could check whether a new commit was requested before we early out, and if so wait for that commit before we deliver the flush response.

 

• there may not be a 'next commit' at all. If the WebView's contents are '<div> haha </div>' then a 'flush request' should still result in a 'flush response',

if we always request a commit when we get the flush request then we could make that commit a forced commit. A forced commit will be deliver even if there are no changes from the previous frame. This will probably keep things simple. Would this be feasible?

Would it be sufficient?

 

thanks,

Ignacio

 

--

Thanks!

    Martin

[Elliott Sprehn]

On Thu, Dec 18, 2014 at 1:41 AM, 'Martin Kosiba' via blink-dev <blin...@chromium.org> wrote:

On Wed, Dec 17, 2014 at 7:14 PM, Ignacio Solla Paula <igs...@google.com> wrote:

We would really appreciate your help understanding how blink works. Please find some additional questions inline.

On Fri, Dec 12, 2014 at 2:42 PM, Martin Kosiba <mko...@google.com> wrote:

Nat,

  Thanks for the explanation. If I understand correctly the flow you described assumes that a commit will occur at some point in the future. This makes sense for the case Ignacio described (mutate some dom, wait for the commit) but doesn't help us with the flush API.

  First, some assumptions we're making (which may or may not be correct):

• the 'flush' API consists of 2 parts: flush request and flush response (callback), both of which take place on the UI thread. The response should only be delivered after the effects of all operations initiated on the UI thread (such as evaluateJavaScript) and all callbacks received on the UI thread (such as didComitProvisionalLoad) are reflected in the state of the active tree,
• blink may deconstruct certain operations into a set of smaller tasks, where only the final task results in changes to the LayerTree (== a call to LayerTreeHost::setNeedsCommit),

Are operations decompose inside ThreadProxy::BeginMainFrame or

somewhere else? I can see that sometimes we early-out after having requested a new commit, for example when 1) any animation requests generated by the apply or animate trigger another frame or when 2) objects that only layout when painted trigger another commit

The early out in BeginMainFrame has nothing to do with what I described. What I wanted to say is that blink sometimes does layout/parsing/style recalc/etc.. asynchronously. It's not that they're "broken down" somewhere, they're just implemented as a series of smaller tasks.

 

Layout and style recalc are never scheduled independently of BeginMainFrame, but JS or input events may cause them to happen at other times inside other tasks (ex. because you touch offsetTop in script, or because an input event needs to do a hit test).

- E

[Ignacio Solla Paula]

Sorry I wasn’t very clear, let me try to clarify.

I have been looking at this code and observed some properties that might help us solve this problem, but we need help from the blink team to understand whether what I have in mind makes any sense at all. This is what I have noticed:

1. The document cycles (in an unordely manner jumping back and forth) through the states in DocumentLifecycle.h (ex. InStyleRecalc, InPerformLayout, InCompositingUpdate, etc.)
   
2. The document goes through these states regardless of whether a commit has been requested or not
   
3. My guess is that transitions between these states happen when the asynchronous tasks that you mention complete
   
4. At some point we send a commit request to the impl thread. When the impl thread decides that this is a good moment then we get the BeingMainFrame call on the main thread. My assumption is that the impl thread always send us the BeginMainFrame after we have requested a commit.
   
5. The actual commit and the DocumentLifecycle are synchronized, ie. the  layer_tree_host()->CommitComplete() call in BeginMainFrame always happens when the document is in the CompositingClean state.
   
6. BeginMainFrame will request new commits if those are needed. In other words, once BeginMainFrame ends there may or may not be an active commit request in the impl thread.
   

Now given all of those, and I do understand that there are a lot of assumptions in there that may be incorrect (blink team are 3, 4, 5 correct?), then I can glimpse a possible solution. The basic idea is to request a forced commit whenever we get a flush request. A forced commit is a commit in which any early outs in BeginMainFrame will be ignored, and the commit will complete even if there are no updates (this could result in new frame with the same contents as the previous frame).

Let me go through the possible scenarios and explain how this would work.

A) Flush request received when there is nothing pending to do (no new frames)

We request the forced commit which will complete and deliver the same frame again. The flush response will be delivered with that commit with the mechanism that we already have.

B) Flush request received when there is pending work, but no active commit request

In this case we are just requesting the commit ahead of time, for example say when the document is in state 2 instead of state 4. My assumption is that given that commits only complete when the document is in state CompositingClean then requesting the commit ahead of time has no practical effects.

C) Flush request received when there is pending work and an active commit request

The forced commit that we request when we get the flush request will just convert the active commit into a forced commit without actually needing to notify the impl thread (see SendCommitRequestToImplThreadIfNeeded). Only the main frame needs to distinguish between normal and forced commits.

D) Flush request received but updating the visual state requires more than 1 commit.

This is the tricky case. Essentially we need a mechanism to decide whether we need to wait for a new frame or not. My solution is based on this assumption:

Once BeginMainFrame completes then a new commit will have been requested if there is still pending work. 

So at the appropriate point in BeingMainFrame we need to check whether there is an active commit request. If there is none then we know that we need to deliver the flush response with the current commit. If there is one then we know that we need to wait.

This raises the question of termination. We could continue waiting forever if there is always some extra work to do. In that case we can perhaps introduce some heuristic and wait for a maximum of N commits before delivering the response.