Intent to Ship: video.requestAnimationFrame()

[Thomas Guilbert]

Contact emails

tgui...@chromium.org

Explainer

https://github.com/WICG/video-raf/blob/gh-pages/explainer.md

Spec

Link to the spec (the explainer has all the information for now). WICG discourse.

Github repo.

TAG review.

Summary

HTMLVideoElement.requestAnimationFrame() registers a one-shot callback, called after a video frame has been presented for composition. It also provides useful metadata about that frame. This is useful for WebGL and <canvas> applications, automated quality/metric analysis, and writing reliable web tests.

Link to “Intent to Prototype” blink-dev discussion

Link

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Risks

Interoperability and Compatibility

Video stacks and compositor implementations can be very different from browser to browser. There will have to be some balance between defining a spec that offers useful guarantees, while still being implementable by all browsers.

Edge: Public support

Firefox: No signals

Web / Framework developers: Positive comments on the WICG discussion thread. Safari: No "official" signals. One response expressing issues with the naming, and when this is fired within the event loop processing model.

The naming issue has come up during TAG review. I have opened a spec issue to address the second point.

Ergonomics

Are there any other platform APIs this feature will frequently be used in tandem with? 

This API might be used in tandem with window.rAF, and so their interactions should be well defined. Could the default usage of this API make it hard for Chrome to maintain good performance

This is unlikely.

Activation Will it be challenging for developers to take advantage of this feature immediately, as-is?

The usage of the API should be relatively straightforward and intuitive, and provide immediate benefits.

Is this feature fully tested by web-platform-tests?

This bug to migrate the current tests (file 1, file 2, file 3)

Entry on the feature dashboard

[Philip Jägenstedt]

Hi Thomas,

I've taken a look through the explainer and the spec and filed a number of issues and a few simple PRs. Some of the issues are trivial, but some would make it hard for another engine to implement this API. https://github.com/WICG/video-raf/issues/1 is the most important IMHO, and my https://github.com/WICG/video-raf/issues/10 is probably a dupe of this. Can you look over that and the other open issues and see which you think are important to fix before we ship?

The naming issue is one we shouldn't block on once the alternatives have been explored, but I'll ping Simon on webkit-dev to see if there were concrete options.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABrVPoY5tzDUMDtC8Jo-aWcALnw2rd5kZTLkadG5Ac1%3DQye7Rg%40mail.gmail.com.

[Daniel Bratell]

We talked about this at the Blink API Owner's meeting today and since the issues you mentioned are still unresolved (naming, exactly when to fire, and maybe more), we decided that we'll wait for those to reach some kind of conclusion. Meanwhile, you could also file an issue with https://github.com/mozilla/standards-positions/issues to get an official response from Mozilla about their position here.

Can you please ping the thread when you think it's ready to proceed?

/Daniel

To unsubscribe from this group and stop receiving emails from it, send an email to blin...@chromium.org.

[Thomas Guilbert]

Hello,


I believe I've addressed all outstanding issues, and that the spec is worth another look. Please let me know if there is anything else that is blocking this feature's approval.


Here is the issue I filed with Mozilla in January, but forgot to include in my original email. There is some additional support from developers.

I've updated the spec to include the timing about exactly when the callbacks fire, relative to the existing HTML spec. I issued a pull request to add a reference from the HTML spec, and was told that I should be merging my spec into the HTML spec instead, when it graduates incubation.

I addressed Philip's feedback on the various issues raised.

Here are the WPT results.

As for the naming, there is as of yet no concrete alternative name proposed, while there is support for keeping the already familiar name (still in the TAG review). I personally still think that requestAnimationFrame() is the best name, especially after incorporating the feedback from the webkit-dev thread which makes it behave closely to window.requestAnimationFrame().

Thank you!
Thomas

[Philip Jägenstedt]

LGTM1

Longer version:

Thomas and I spent some time going over the feedback from https://github.com/mozilla/standards-positions/issues/250 and open issues. We filed issues for outstanding changes at https://github.com/WICG/video-raf/issues, a notable one being for more examples. I also filed a few issues relating to testing this in more depth.

Overall, my judgement is that now that this integrates with the rendering loop in HTML, that settles most timing issues and what guarantees do or don't exist. The naming also seems pretty good to me, but I recognize it's different from window.rAF in some important ways. A few things need to be spelled out with examples or notes, but there are few normative changes to make. https://github.com/WICG/video-raf/issues/25 is perhaps the most observable one outstanding.

I believe that this is in good shape to ship with the understanding that the outstanding issues will be closed well before this reaches stable.

And as always, if we do get more feedback after this intent has 3xLGTM, then changes are always on the table even post shipping.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABrVPoYMsshLVWQs2cnoym35cmN%3DJC-04ijKjfwFGX7iVqN7CQ%40mail.gmail.com.

[Mounir Lamouri]

non-owner LGTM

The proposal has been shared widely and feedback has been taken into account to change the design of the specification which is a great sign for compatibility with future implementations. Even if more improvements can always be done, the specification looks good and hopefully this work will be merged into the HTML spec as soon as it has incubated.

As mentioned above, Safari provided feedback and it was listened to, leading to a change in the specification and Chrome's implementation. Mozilla also left some feedback. I unfortunately didn't have time to read it all but I trust Philip if he thinks this API is shippable as-is after working on Mozilla's feedback. I reached out personally to Paul from Mozilla today to check whether shipping the API as-is is okay. As Philip pointed out, non-breaking changes are always on the table.

-- Mounir

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYdK9a6aqBjUqDVn%2BEKxh%3Dy1nvane6PXYGUHT29-oVaFDA%40mail.gmail.com.

[Daniel Bratell]

LGTM2

/Daniel

[sligh...@chromium.org]

LGTM3

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABrVPoYMsshLVWQs2cnoym35cmN%3DJC-04ijKjfwFGX7iVqN7CQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Thomas Guilbert]

Thank you very much!

CL to enable this by default:
https://chromium-review.googlesource.com/c/chromium/src/+/2101242

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABrVPoYMsshLVWQs2cnoym35cmN%3DJC-04ijKjfwFGX7iVqN7CQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Boris Zbarsky]

On 3/10/20 7:45 PM, Philip Jägenstedt wrote:
> Overall, my judgement is that now that this integrates with the
> rendering loop in HTML

> <https://html.spec.whatwg.org/multipage/webappapis.html#update-the-rendering>,

> that settles most timing issues and what guarantees do or don't exist.

Just so we're all on the same page:

1) This does not match Mozilla's view of the current state of things.

2) Paul (who has been engaging from our side on this) feels like his
feedback is more or less being ignored, and is _definitely_ not being
understood. He is trying to follow up on this some more, but feels like
it's mostly wasted effort at this point because you all are just not
listening at all.

-Boris

[Chris Harrelson]

Hi Boris,

Could you point me to Paul's feedback or add me to threads with him that may not be on public repos? I'm having a hard time finding it so far.

 

-Boris

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0084894f-77c6-4b0f-0a68-edfe09fda621%40mit.edu.

[Boris Zbarsky]

On 3/13/20 1:18 PM, Chris Harrelson wrote:
> Could you point me to Paul's feedback or add me to threads with him that
> may not be on public repos? I'm having a hard time finding it so far.

Chris,

I assume that you've read
<https://github.com/mozilla/standards-positions/issues/250>, right?

Past that, I believe a large fraction of the feedback has been in
private mails, but I'll let Paul speak for himself; adding him to cc.

-Boris

[Paul Adenot]

All,

Yes there has been a single private email on this, but it's mostly paraphrases of the standard-position link above with some re-explanation maybe, but nothing drastically different from my initial feedback.

I'll put in in the public thread on monday (it's getting late here) because there are some answers to the new questions asked in the answer to my initial points, but yes, it feels like most of my questions and concerns haven't been answered, and I do believe they are legitimate.

Thanks,

Paul.

[Daniel Bratell]

Thomas, I know you already have the 3 LGTMs, one of them being mine, but it seems like the discussion was not as complete as you and I and the rest of us thought. There was one thing in particular that I would like to be investigated and that is if this feature, as it's designed, can introduce performance problems through the sync between the threads.

Until we know that there are no more misunderstandings, I'm putting my previous lgtm "on hold".

/Daniel

[Dale Curtis]

I think it's unfair to say we're not listening. As Philip notes, we've been soliciting and addressing feedback on the discourse and elsewhere since July 2019. I pinged Jean-Yves @ Mozilla privately (as well as Microsoft and Safari) about it at that time, but did not hear back, so we filed the official request for feedback in January.

Just so the timeline is clear: The discourse has been up since July 2019. Thomas requested feedback on Jan 21st from Mozilla; almost 2 months ago. When he did receive feedback on Mar 9th, he replied to that feedback the same day, but has since gotten no reply. All of his conversations have been in public as well, so we're unsure who those private e-mails are with.

As is, much of Paul's feedback seems like a misunderstanding of the API -- hence all the issues filed for examples and clarity that Philip notes above. We'd love for Paul to elaborate if that's not the case. If it's easier to meet via VC we can set that up. We're definitely interested in understanding his feedback and will absolutely pull the API if there's a show stopper issue.

- dale

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CANWt0Wrt1mJYf26cMQ2QhJiVG6PbqdtuXbcn36eDv7-_HwUPjg%40mail.gmail.com.

[Dale Curtis]

On Fri, Mar 13, 2020 at 12:57 PM Daniel Bratell <brat...@gmail.com> wrote:

Thomas, I know you already have the 3 LGTMs, one of them being mine, but it seems like the discussion was not as complete as you and I and the rest of us thought. There was one thing in particular that I would like to be investigated and that is if this feature, as it's designed, can introduce performance problems through the sync between the threads.

Definitely let's continue discussion on Paul's feedback, but at least on this point - this is not possible. The current API just posts trivial tasks from one thread to another. There's no more cross thread synchronization (at least in Chrome) than there is for the existing paint() and texImage2D mechanisms. If this doesn't answer your Q, please feel free to write it up as an issue on the GitHub so we can address it more specifically.

Until we know that there are no more misunderstandings, I'm putting my previous lgtm "on hold".

/Daniel

On 2020-03-13 18:33, Paul Adenot wrote:

All,

Yes there has been a single private email on this, but it's mostly paraphrases of the standard-position link above with some re-explanation maybe, but nothing drastically different from my initial feedback.

I'll put in in the public thread on monday (it's getting late here) because there are some answers to the new questions asked in the answer to my initial points, but yes, it feels like most of my questions and concerns haven't been answered, and I do believe they are legitimate.

Thanks,

Paul.

On Fri, Mar 13, 2020 at 6:25 PM Boris Zbarsky <bzba...@mit.edu> wrote:

On 3/13/20 1:18 PM, Chris Harrelson wrote:
> Could you point me to Paul's feedback or add me to threads with him that
> may not be on public repos? I'm having a hard time finding it so far.

Chris,

I assume that you've read
<https://github.com/mozilla/standards-positions/issues/250>, right?

Past that, I believe a large fraction of the feedback has been in
private mails, but I'll let Paul speak for himself; adding him to cc.

-Boris

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/01ca02b9-88aa-9e94-b1bd-a0c9abc79637%40gmail.com.

[Thomas Guilbert]

Since LGTMs are on hold, here is the revert of the API shipping:
https://chromium.googlesource.com/chromium/src/+/d369ad7ba3f463c052f4fca25da54a942d9a90cf

[Philip Jägenstedt]

Thanks Boris and Paul for raising flags and sorry for misjudging how much there could still be to discuss.

I had made two assumptions here:

• Synchronization guarantees are actually impossible, because video keeps decoding and rendering even if the main thread is blocked. (Similar to scrolling.) Because of that, if the main thread takes too long, the information delivered could be stale.
• Hooking this into the rendering loop in HTML answers basically all questions. Because the conditions for when animation frame callbacks are invoked aren't changed, this is just providing information about what the video frame is likely to be, but it cannot be guaranteed.

However, it sounds from Paul in https://github.com/mozilla/standards-positions/issues/250 like guarantees aren't impossible, at least not in all engines, which is interesting. This bit is also potentially critical feedback:

Adaptive sync is the most important thing that has happened to video playback since battery-powered video playing devices exist. We can't simply ignore it here. We might find that it just works however, but we need to look into it and to allow for it to kick-in.

https://github.com/WICG/video-raf/issues/29 will hopefully make it quite clear what use cases can and can't be addressed by this proposal.

Hoping to see this make progress. Thomas, are there additional issues that should be filed in https://github.com/WICG/video-raf/issues for any of the feedback? Can you update this thread when the issues have been addressed?

Thanks all!

[Dale Curtis]

On Mon, Mar 16, 2020 at 3:28 PM Philip Jägenstedt <foo...@chromium.org> wrote:

Thanks Boris and Paul for raising flags and sorry for misjudging how much there could still be to discuss.

I had made two assumptions here:

• Synchronization guarantees are actually impossible, because video keeps decoding and rendering even if the main thread is blocked. (Similar to scrolling.) Because of that, if the main thread takes too long, the information delivered could be stale.
• Hooking this into the rendering loop in HTML answers basically all questions. Because the conditions for when animation frame callbacks are invoked aren't changed, this is just providing information about what the video frame is likely to be, but it cannot be guaranteed.

These assumptions are correct.

 

However, it sounds from Paul in https://github.com/mozilla/standards-positions/issues/250 like guarantees aren't impossible, at least not in all engines, which is interesting. This bit is also potentially critical feedback:

I don't think it's impossible in Chrome either, but it would mean blocking the start of page composition until after video composition. Which for reasons you mention above and more isn't something we really want to do. Other 

 

Adaptive sync is the most important thing that has happened to video playback since battery-powered video playing devices exist. We can't simply ignore it here. We might find that it just works however, but we need to look into it and to allow for it to kick-in.

There's no special consideration necessary for VRR modes with this API since it ties to the existing rAF, but only triggers at the video rate. So whatever ends up decided for HTML5/rAF in general is fine: https://github.com/whatwg/meta/issues/158

 

https://github.com/WICG/video-raf/issues/29 will hopefully make it quite clear what use cases can and can't be addressed by this proposal.

Hoping to see this make progress. Thomas, are there additional issues that should be filed in https://github.com/WICG/video-raf/issues for any of the feedback? Can you update this thread when the issues have been addressed?

 

We're going to VC with Paul tomorrow to discuss. Thanks for your help!

[Philip Jägenstedt]

Thanks Dale!

On Mon, Mar 16, 2020 at 11:47 PM Dale Curtis <dalec...@chromium.org> wrote:

On Mon, Mar 16, 2020 at 3:28 PM Philip Jägenstedt <foo...@chromium.org> wrote:

Thanks Boris and Paul for raising flags and sorry for misjudging how much there could still be to discuss.

I had made two assumptions here:

• Synchronization guarantees are actually impossible, because video keeps decoding and rendering even if the main thread is blocked. (Similar to scrolling.) Because of that, if the main thread takes too long, the information delivered could be stale.
• Hooking this into the rendering loop in HTML answers basically all questions. Because the conditions for when animation frame callbacks are invoked aren't changed, this is just providing information about what the video frame is likely to be, but it cannot be guaranteed.

These assumptions are correct.

 

However, it sounds from Paul in https://github.com/mozilla/standards-positions/issues/250 like guarantees aren't impossible, at least not in all engines, which is interesting. This bit is also potentially critical feedback:

I don't think it's impossible in Chrome either, but it would mean blocking the start of page composition until after video composition. Which for reasons you mention above and more isn't something we really want to do. Other

Thanks for confirming, sounds like my assumptions weren't wrong for Chromium at least.

Adaptive sync is the most important thing that has happened to video playback since battery-powered video playing devices exist. We can't simply ignore it here. We might find that it just works however, but we need to look into it and to allow for it to kick-in.

There's no special consideration necessary for VRR modes with this API since it ties to the existing rAF, but only triggers at the video rate. So whatever ends up decided for HTML5/rAF in general is fine: https://github.com/whatwg/meta/issues/158

Thanks, following that issue.

 

https://github.com/WICG/video-raf/issues/29 will hopefully make it quite clear what use cases can and can't be addressed by this proposal.

Hoping to see this make progress. Thomas, are there additional issues that should be filed in https://github.com/WICG/video-raf/issues for any of the feedback? Can you update this thread when the issues have been addressed?

 

We're going to VC with Paul tomorrow to discuss. Thanks for your help!

Sounds good, hope we arrive at something that solves the right use cases and everyone can implement!

[Thomas Guilbert]

Here's a quick update on the status of this API.

Dale, Paul, Philip, Mounir and I met on VC on 3/17. Paul posted a nice summary of what we discussed on the Mozilla positions thread.

Overall, my understanding is that there are no outstanding technical concerns with the API. Some changes were made to the wording of the spec, and to the names of the attributes. There were no noteworthy changes made to the implementation.

The only remaining blocker is the API name, which is being discussed here.

Thanks,
Thomas

[Thomas Guilbert]

The API name discussion is resolved, and we have settled on video.requestVideoFrameCallback().

The spec has been updated, the repo has been renamed and the Chrome implementation has been updated.

The web-platform-tests are temporarily disabled. I am waiting for the IDL updates to land upstream before updating them.

Mozilla's latest position is "worth-prototyping".

I do not think that there are any other blocking issues regarding this API. Can API OWNERS evaluate this API's ship-readiness in its latest state?

Thank you!
Thomas

[Philip Jägenstedt]

LGTM1, great to see that a name was found that everyone can live with in the end.

I've merged https://github.com/web-platform-tests/wpt/pull/22875 which should conflict with / unblock https://chromium-review.googlesource.com/c/chromium/src/+/2145965/.

The README says the tests are still basic, do tests tests added in the mentioned CL address that or are there still more tests to be added?

[Yoav Weiss]

LGTM2

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYdbEnHCK4%2BVYpefOiMXo%2B5GG8tOjnTN3RE_MhMX0K65Dw%40mail.gmail.com.

[Thomas Guilbert]

I am adding more tests, after the initial mechanical renames, and I will update the README. For example, Johannes (kron@) has provided me with a test that will allow us to check all of the WebRTC optionals, and close this testing issue.

[Philip Jägenstedt]

That's great, thanks Thomas!

API OWNERS, still one LGTM missing here :)

[j.j.]

I do not think that there are any other blocking issues regarding this API.

https://www.chromestatus.com/feature/6335927192387584

has the old name.

[Thomas Guilbert]

FYI, all WPTs have been updated and re-activated, and the new tests I mentioned have landed as well.

[Mike West]

LGTM3. Thank you for following up on the tests.

-mike

LGTM2

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Thomas Guilbert]

Thank you j.j., I updated the chromestatus page.

The API is now on-by-default:
https://chromium.googlesource.com/chromium/src/+/da4aa5d1898d941250a4fd3f5b5575903ba06ca9

Thank you so much everyone!

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[siupra...@o2.pl]