Intent to Implement and Experiment: Single buffered canvas

[Justin Novosad]

Contact emails

ju...@chromium.org

Spec

https://github.com/whatwg/html/issues/2659

Summary

This feature provides a low latency rendering mode for canvases. This is useful, for example, for stylus input. Single buffered mode is enabled with canvas.getContext('2d', {singleBuffered: true}); Caveats: Opting in to this mode breaks synchronization between canvas commands and DOM edits. Motivation: NaCl/PPAPI provides native OpenGL rendering surfaces which are currently used to implement low latency interfaces. The deprecation of NaCl creates a need for this API.

Goals for experimentation

Gather developer feedback in order to resolve design issues. In particular we want to determine whether the browser should internally use non-synchronized double buffering to mitigate flickering, or let developers decide whether or not to mitigate flickering in JavaScript, or whether the API could provide a special mode for that (single buffered vs. low latency double-buffered).

Another goal for the experiment is to validate the impact of this feature on user experience with production web sites in order to justify moving forward with standardization.

For the experiment, the intent is to provide only the raw single buffered mode to determine whether developers would prefer a more fool-proof solution (possibly at the expense of performance).

Experimental timeline

Start: M64

End: M66

Any risks when the experiment finishes?

If the experiment ends without the feature being shipped, web apps that were using it will not suffer any loss of functionality, they will just lose the advantage of single buffered mode. Request for for the feature will be silently ignored (it is a parameter in a dictionary).

Ongoing technical constraints

To offer truly single buffered operation, hardware overlay buffers must be available. Otherwise the implementation will make a best effort to reduce latency, but rendering will not be truly single buffered due to the need to go through a compositing pass upstream of device scanout.

Debuggability

No DevTools support needed.

Will this feature be supported on all five Blink platforms supported by Origin Trials (Windows, Mac, Linux, Chrome OS, and Android)?

Yes.

Link to entry on the feature dashboard

https://www.chromestatus.com/feature/5648950090530816

[Antoine Labour]

On Fri, Oct 27, 2017 at 12:45 PM, Justin Novosad <ju...@chromium.org> wrote:

Contact emails

ju...@chromium.org

Spec

https://github.com/whatwg/html/issues/2659

Summary

This feature provides a low latency rendering mode for canvases. This is useful, for example, for stylus input. Single buffered mode is enabled with canvas.getContext('2d', {singleBuffered: true}); Caveats: Opting in to this mode breaks synchronization between canvas commands and DOM edits. Motivation: NaCl/PPAPI provides native OpenGL rendering surfaces which are currently used to implement low latency interfaces. The deprecation of NaCl creates a need for this API.

I would like to point out that this API only really works on Chrome OS, and only if we can guarantee that the Pepper plugin's surface is always promoted to an overlay (i.e. never composited by the compositor), because it completely bypasses the compositor signaling mechanism (to achieve that low latency). This was put in place for a very specific use case and is not general-purpose at all.

Reintroducing the signaling mechanism would I believe make latency no better than the Offscreen Canvas case.

 

Goals for experimentation

Gather developer feedback in order to resolve design issues. In particular we want to determine whether the browser should internally use non-synchronized double buffering to mitigate flickering, or let developers decide whether or not to mitigate flickering in JavaScript, or whether the API could provide a special mode for that (single buffered vs. low latency double-buffered).

Another goal for the experiment is to validate the impact of this feature on user experience with production web sites in order to justify moving forward with standardization.

For the experiment, the intent is to provide only the raw single buffered mode to determine whether developers would prefer a more fool-proof solution (possibly at the expense of performance).

Experimental timeline

Start: M64

End: M66

Any risks when the experiment finishes?

If the experiment ends without the feature being shipped, web apps that were using it will not suffer any loss of functionality, they will just lose the advantage of single buffered mode. Request for for the feature will be silently ignored (it is a parameter in a dictionary).

Ongoing technical constraints

To offer truly single buffered operation, hardware overlay buffers must be available. Otherwise the implementation will make a best effort to reduce latency, but rendering will not be truly single buffered due to the need to go through a compositing pass upstream of device scanout.

Debuggability

No DevTools support needed.

Will this feature be supported on all five Blink platforms supported by Origin Trials (Windows, Mac, Linux, Chrome OS, and Android)?

Yes.

Given the above, I'm not clear that this can be possible on all platforms.

Antoine

 

Link to entry on the feature dashboard

https://www.chromestatus.com/feature/5648950090530816

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABpaAqSyEnStX-h7ZfZ9p42LkPJz%2BfC5bP%3DSWe59RJpk5RZQ1A%40mail.gmail.com.

[Justin Novosad]

It won't be possible to be completely single buffered on all platforms, but it will be possible to reduce latency by one or two vsync periods on all platforms (at the expense of breaking display update synchronization).  This can be achieved by allowing canvas rendering contexts to draw to shared bitmaps or textures after they've been committed to the compositor, thus catching up with the previous frames before they get scanned out.

[Justin Novosad]

Correction: (...), thus catching up with the previous frames before they are consumed by the surface renderer.

[Antoine Labour]

So this is like the update model for OffscreenCanvas, which doesn't cause tearing? Shouldn't we separate the 2 concepts?

1- canvas update decoupled from the page (possible on all platforms)

2- single-buffered, requires #1, causes tearing, and is only possible on a subset of the platforms (and has undocumented/changing constraints on the canvas transform/opacity and other indirect properties such as whether it's overlapped by other elements)?

[Justin Novosad]

Yep, that is a possibility. It's not clear whether there is demand for a mode that explicitly guarantees no tearing, which is exactly the sort of feedback I hope to get from the experiment.

 

 

Antoine

 

Link to entry on the feature dashboard

https://www.chromestatus.com/feature/5648950090530816

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABpaAqSyEnStX-h7ZfZ9p42LkPJz%2BfC5bP%3DSWe59RJpk5RZQ1A%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAMeTaZdmcxWvBe3cQc1bqfAnhT-Ruk1h1qtZt1TFQe4UbuyOjA%40mail.gmail.com.

[Antoine Labour]

I guess my question is: what is the expected behavior for #2 when you don't fall into the path that works - i.e. you're on the right platform (effectively Chrome OS right now, maybe Mac, I'm not sure about the effective restrictions on that one), your canvas is screen-aligned on pixel boundaries, not clipped and on top of everything else, no filter, and there is no transparency at any level of compositing (e.g. UI, things that are out of control from the point-of-view of the site or even the renderer), and I'm probably missing more. If you just try to do what the current PPAPI hack does, basically it would mean the canvas changes don't show up any more, unless something else triggers compositing, which can also cause out-of-order frames.

Antoine

[Justin Novosad]

We'd need to keep pushing frames to make sure the compositor is triggered. It's just that we'd push frames with mailboxes that reference the same pixel buffer/texture as the previous frame.

[Antoine Labour]

On Tue, Oct 31, 2017 at 8:41 AM Justin Novosad <ju...@chromium.org> wrote:

We'd need to keep pushing frames to make sure the compositor is triggered. It's just that we'd push frames with mailboxes that reference the same pixel buffer/texture as the previous frame.

How do we prevent out-of-order frames when transitioning from scanned-out canvas native buffer to composited?

Antoine

[Justin Novosad]

On Tue, Oct 31, 2017 at 12:11 PM, Antoine Labour <pi...@google.com> wrote:

On Tue, Oct 31, 2017 at 8:41 AM Justin Novosad <ju...@chromium.org> wrote:

We'd need to keep pushing frames to make sure the compositor is triggered. It's just that we'd push frames with mailboxes that reference the same pixel buffer/texture as the previous frame.

How do we prevent out-of-order frames when transitioning from scanned-out canvas native buffer to composited?

Interesting... Had not though of that case. May need to use frame sequence numbers to time the transition correctly, possibly drop frames (which is better than out of order).  

[Rick Byers]

These are all great design questions that we should figure out, but I don't think they should block doing an origin-trial for SOME design.  There sadly hasn't been much engagement on the GitHub issue, Antoine can you and Justin work out these design questions there?

Justin, do you have any "production websites" lined up who actually want to use this functionality ASAP?

FWIW I believe it's important we ship something for this scenario soon. My personal Pixelbook arrived last night and I find the stylus writing experience to be quite awesome (<10ms latency), but we just have no way to create that experience on the web yet :-(. If (as seems to be the case by the lack of engagement on the github issue) no other engine is interested in engaging with us on the design, we should just do our best to get whatever design consensus we can and ship something for real ASAP based on a spec PR (we don't want to let lack of engagement hold back the web).

Rick

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAMeTaZd5DBSKvOs1VZaqF_WC3c_vmKs8tz%3DhOyT6zmVKva0FJw%40mail.gmail.com.

[Justin Novosad]

On Tue, Oct 31, 2017 at 2:02 PM, Rick Byers <rby...@chromium.org> wrote:

These are all great design questions that we should figure out, but I don't think they should block doing an origin-trial for SOME design.  There sadly hasn't been much engagement on the GitHub issue, Antoine can you and Justin work out these design questions there?

Justin, do you have any "production websites" lined up who actually want to use this functionality ASAP?

At the moment, Google Keep intends to sign-up for the experiment.

FWIW I believe it's important we ship something for this scenario soon. My personal Pixelbook arrived last night and I find the stylus writing experience to be quite awesome (<10ms latency), but we just have no way to create that experience on the web yet :-(. If (as seems to be the case by the lack of engagement on the github issue) no other engine is interested in engaging with us on the design, we should just do our best to get whatever design consensus we can and ship something for real ASAP based on a spec PR (we don't want to let lack of engagement hold back the web).

Right, I think this origin trial could help make a compelling case that could help get things moving.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFUtAY_k3wkgZpTN8HUa%2BK0zkQVZkVZf8Ynvmk5Nnu8aA%3D0dQw%40mail.gmail.com.

[Rick Byers]

On Wed, Nov 1, 2017 at 11:59 AM, Justin Novosad <ju...@chromium.org> wrote:

On Tue, Oct 31, 2017 at 2:02 PM, Rick Byers <rby...@chromium.org> wrote:

These are all great design questions that we should figure out, but I don't think they should block doing an origin-trial for SOME design.  There sadly hasn't been much engagement on the GitHub issue, Antoine can you and Justin work out these design questions there?

Justin, do you have any "production websites" lined up who actually want to use this functionality ASAP?

At the moment, Google Keep intends to sign-up for the experiment.

Great.  Keep low-latency stylus is a great experience on ChromeOS today, but it's really sad that it relies on NaCL.  I'm anxious to support them (and anyone else who is interested) get this working on the web ASAP.

LGTM to experiment.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABpaAqQrqMzkr%2BGJpxxnJiOM%3DXZ8OpT9tOPFcKHpnepKJh5qpQ%40mail.gmail.com.