Intent to Ship: OffscreenCanvas and DedicatedWorker.requestAnimationFrame

[Fernando Serboncini]

Contact emails

fs...@chromium.org

Spec

https://html.spec.whatwg.org/#the-offscreencanvas-interface

https://github.com/whatwg/html/pull/3677

https://github.com/whatwg/html/pull/3708

https://github.com/w3ctag/design-reviews/issues/141

https://github.com/w3ctag/design-reviews/issues/288

Summary

This would launch OffscreenCanvas and requestAnimationFrame on DedicatedWorkers.

OffscreenCanvas allows developers to use canvas APIs to render to screen or to textures from window or workers. It does so with a slightly less overhead.

requestAnimationFrame on workers allow developers to have a vsync and rendering bound event on workers.

Link to “Intent to Implement” blink-dev discussion

https://groups.google.com/a/chromium.org/forum/?fromgroups#!searchin/blink-dev/offscreencanvas|sort:date/blink-dev/ckHvzFXn5zQ/fRwhkiE8BAAJ

https://groups.google.com/a/chromium.org/forum/?fromgroups#!searchin/blink-dev/offscreencanvas|sort:date/blink-dev/3dAVa13THa8/NK4KgZVgAgAJ

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Debuggability

There may be issues regarding the rendering specific part of DevTools (FPS meter, for example). But apart from that, no extra attention needed.

Risks

Edge: No signal.

Firefox: Developed under a flag. Latest changes LGTMed.

Safari: No signal.

The final spec details are still to be submitted, but have been LGTMed and issues brought up on W3C Tag were all addressed.

Is this feature fully tested by web-platform-tests? Link to test suite results from wpt.fyi.

Yes. https://wpt.fyi/results/offscreen-canvas

Entry on the feature dashboard

https://www.chromestatus.com/feature/5681560598609920

[Daniel Bratell]

This seems like a useful thing. 

You wrote "No signal" for MS and Safari. Have you tried to solicit feedback from them, for instance by reporting an issue in their issue trackers?

There is also nothing about feedback from Web Developers? Are there any web developers that have tried it and confirmed they need/want it and that it works for them?

I'm also missing some subsections from the risk description. Could you please add those too. I'm not familiar enough with this feature to know all the potential problems and it's good to know what the potential bumps will be.

(Note to self: is it too hard to find/use the right template?)

/Daniel

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADp2-T8HPDAN7PM7q6w5pc8SFoWJ4szzMdw3R7T%2B%2BBt0Ad_ihQ%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

[Fernando Serboncini]

We do have issues on their trackers for implementation that have been ack'ed and assigned. I didn't know that qualified as "public signal".

On Web Developers feedback, sorry for not adding those:

There's been some feedback from people experimenting with it on beta: https://twitter.com/ebidel/status/975788240463585281?lang=en

related reddit thread: https://www.reddit.com/r/javascript/comments/85reh7/excited_that_offscreencanvas_is_finally_happening/

Microsoft/TypeScript folks are also on it: https://twitter.com/drosenwasser/status/1008941418608381952

Autodesk: https://twitter.com/camwest/status/941163404252901377

ReactJS: https://twitter.com/sebmarkbage/status/788567698565324801

I removed some sections on risk description because I thought I didn't have much to say on them. But I was wrong. Let me try again.

Ergonomics

OffscreenCanvas is used together with Canvas and other graphics-related APIs (texImage2D on WebGL, for example). There's a minimal increase in complexity that comes from that.

OffscreenCanvas also has a .commit() function that only executes on workers and is a synchronous function. The commit() function is made to be used by busy-loop type of applications, and was added to support new WebAssembly use cases.

On the Worker.requestAnimationFrame side, it does add a new complexity for DedicatedWorkers. That added complexity comes from the fact that now Worker may have a logical connection to the display/Window they were created in. This is not exposed directly to developers in any way.

Activation

OffscreenCanvas itself is simple to use. Polyfill-wise, it's almost a drop in replacement for Canvas. That said, it does come with some extra functions that allow people to, for example, use it on Workers. Developers will need to learn about that.

It could benefit a lot from more external documentation and outreach, not so much of libraries.

We already have custom modification to support, for example, Three.js on Workers.

Thanks.

[Daniel Bratell]

LGTM1

/Daniel

[PhistucK]

> On the Worker.requestAnimationFrame side, it does add a new complexity for DedicatedWorkers. That added complexity comes from the fact that now Worker may have a logical connection to the display/Window they were created in.

I was/am worried about this one. If workers are ever moved to different processes at some point (note - I have not heard of such plans, but it does not sound so infeasible at the moment and as RAM capacity and CPU speed grows, I can imagine it happening. Kind of like Microsoft splitting SVCHost services to their own separate processes per-service for higher RAM devices on Windows 10), will this make such a move much more complex?

Workers used to be completely separate, disconnected and uncoupled from the main-frame rendering and this is a beginning of a coupling which may close some optimization/performance/security doors...

☆PhistucK

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADp2-T_vrkkji-9NzHybZ0HbmpKosZr%2BJ10E2DNNPXexxLEONA%40mail.gmail.com.

[Fernando Serboncini]

Ahn, I can clarify that. :)

I don't think such a move would be made more complex by this. Workers are still independent from main-frame rendering. I really meant a "logical connection". :) What we do now is, at Worker creation time, we pass some info to the worker that it then uses to communicate with the GPU process directly. There's no extra coupling of any sort with the main thread. Does that clarify it?

[Ben Kelly]

On Thu, Jun 21, 2018 at 12:36 PM, 'Fernando Serboncini' via blink-dev <blin...@chromium.org> wrote:

Ahn, I can clarify that. :)

I don't think such a move would be made more complex by this. Workers are still independent from main-frame rendering. I really meant a "logical connection". :) What we do now is, at Worker creation time, we pass some info to the worker that it then uses to communicate with the GPU process directly. There's no extra coupling of any sort with the main thread. Does that clarify it?

Does this require having a connection to a particular main thread window?  Are there issues with dedicated workers nested in SharedWorkers?  See:

https://groups.google.com/a/chromium.org/d/msg/blink-dev/KZx0i3e5nZM/I7txb1mmAgAJ

Nesting within a SharedWorker means you cannot draw a connection from the dedicated worker to any particular window.  It will always be connected through the shared worker to at least one window (or worker if nested SharedWorkers are ever implemented), but that set of connected windows may change over time.

Also, there is a desire to expose nested workers in ServiceWorker in the future where there will be no associated window.

Just curious how this all interacts in your implementation.  Thanks.

Ben

[Fernando Serboncini]

Those are valid points. The current spec states that only DedicatedWorkers that come (directly or indirectly) from a scope with a Window have valid requestAnimationFrame. If there's no associated Window down the line, then we won't provide requestAnimationFrame (for now, we are throwing an exception, until there's some IDL mechanism to describe this).

[PhistucK]

Yes, I think so. However, this might open the door for more timing attack vectors (I am not a security guy, so I cannot even explain how, but it is very much imaginable that this will be used for timing attacks when it directly communicates with the main-and-single GPU process of the browser. Even if you phrased it too loosely, it still might be another invisible vector to the user).

I bet the security team will think of how (or be pretty certain how not), just raising a flag.

☆PhistucK

[Ken Russell]

On Wed, Jun 20, 2018 at 8:21 AM 'Fernando Serboncini' via blink-dev <blin...@chromium.org> wrote:

We do have issues on their trackers for implementation that have been ack'ed and assigned. I didn't know that qualified as "public signal".

On Web Developers feedback, sorry for not adding those:

There's been some feedback from people experimenting with it on beta: https://twitter.com/ebidel/status/975788240463585281?lang=en

related reddit thread: https://www.reddit.com/r/javascript/comments/85reh7/excited_that_offscreencanvas_is_finally_happening/

Microsoft/TypeScript folks are also on it: https://twitter.com/drosenwasser/status/1008941418608381952

Autodesk: https://twitter.com/camwest/status/941163404252901377

ReactJS: https://twitter.com/sebmarkbage/status/788567698565324801

OffscreenCanvas is also critically needed for WebAssembly's threading support. Querying a WebGLRenderingContext from an OffscreenCanvas on a background thread is required to bring existing multithreaded rendering engines, and game engines, to the web.

-Ken

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADp2-T_vrkkji-9NzHybZ0HbmpKosZr%2BJ10E2DNNPXexxLEONA%40mail.gmail.com.

[Ben Kelly]

On Thu, Jun 21, 2018 at 12:57 PM, Fernando Serboncini <fs...@google.com> wrote:

Those are valid points. The current spec states that only DedicatedWorkers that come (directly or indirectly) from a scope with a Window have valid requestAnimationFrame. If there's no associated Window down the line, then we won't provide requestAnimationFrame (for now, we are throwing an exception, until there's some IDL mechanism to describe this).

I think throwing an exception is actually the right thing here.  With a dedicated worker nested under a SharedWorker the contents of the owner set can change over time.  So if we add SharedWorker nested under ServiceWorker (which is seems we want), then you could end up with a situation where requestAnimationFrame() is supported for a while and then becomes unsupported.  This might be a bit weird, but at least it can be represented with the exception approach.  We can't really make interfaces appear and disappear from the global dynamically via webidl.

Thanks!

Ben

[Anne van Kesteren]

On Fri, Jun 22, 2018 at 3:53 PM, Ben Kelly <bke...@mozilla.com> wrote:
> I think throwing an exception is actually the right thing here. With a
> dedicated worker nested under a SharedWorker the contents of the owner set
> can change over time. So if we add SharedWorker nested under ServiceWorker
> (which is seems we want), then you could end up with a situation where
> requestAnimationFrame() is supported for a while and then becomes
> unsupported. This might be a bit weird, but at least it can be represented
> with the exception approach. We can't really make interfaces appear and
> disappear from the global dynamically via webidl.

The idea was to scope it to agent clusters, and excluding any agent
cluster that does not contain a similar-origin window agent. That
should be static and deterministic, but we only have low-level
infrastructure for this and even that is not great.


--
https://annevankesteren.nl/

[Ben Kelly]

I don't understand how that can work if both a window and a ServiceWorker can attach to the same SharedWorker.  They can't right now, but we have uses cases that want this.  Maybe this agent cluster concept is a blocker for exposing SharedWorker in ServiceWorker.

[Anne van Kesteren]

On Fri, Jun 22, 2018 at 8:06 PM, Ben Kelly <bke...@mozilla.com> wrote:
> I don't understand how that can work if both a window and a ServiceWorker
> can attach to the same SharedWorker. They can't right now, but we have uses
> cases that want this. Maybe this agent cluster concept is a blocker for
> exposing SharedWorker in ServiceWorker.

It's not. Agent clusters are not completely isolated from each other.
They define the memory boundary and what you might implement as a
process boundary. In the case where a window and a service worker
reach a shared worker, you'd have 3 agent clusters. As defined only
dedicated workers that directly chain up to the window (can only have
intermediate dedicated workers) have the API exposed (and are in the
window's agent cluster).


--
https://annevankesteren.nl/

[Yoav Weiss]

Agreed. Would also be great to have a feature detection mechanism that let's the developer know RAF is not available (either not supported, or not supported in current worker).

That way, developers can tackle both cases at the same time (and not be forced to handle legitimate failures using exceptions).

  

Thanks!

Ben

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2B1UsbR3cdndP389NQA%3D%2B7PP%3DJEd2LrhLmfRWdQdwCTyuriAxw%40mail.gmail.com.

[Chris Harrelson]

I think this feature is ready to ship. The Canvas team has put in a lot of effort to spec and test this feature thoroughly. Great work!

However, I'm recusing myself from LGTM since I was directly involved in the project.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACj%3DBEga%2BV%2BaGQyoVr9K%3DR5p06_t3-7NapVxaOGHB1NHD13%2BSQ%40mail.gmail.com.

[Rick Byers]

I'm excited to see this shipping! Being able to draw in a way that's performance isolated from other script is a very important and powerful primitive. LGTM2

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw8OP2bK1923%2BCrRN26tL2QfvJaZTVJkxw%3DYvTu_A-XTpw%40mail.gmail.com.

[Yoav Weiss]

LGTM3

[Gregg Tavares]

> OffscreenCanvas also has a .commit() function that only executes on workers and is a synchronous function.

Where is this in the spec? How does a worker which is in spin loop on commit receive other events? I'm probably looking at the wrong document but there doesn't seem to be anything in here about commit being blocking https://html.spec.whatwg.org/#the-offscreencanvas-interface ?

Do other events get processed while one event is blocked waiting for a commit()? Can another even call commit() while some other event is already waiting on a commit()

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADp2-T8HPDAN7PM7q6w5pc8SFoWJ4szzMdw3R7T%2B%2BBt0Ad_ihQ%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADp2-T_vrkkji-9NzHybZ0HbmpKosZr%2BJ10E2DNNPXexxLEONA%40mail.gmail.com.

[Fernando Serboncini]

This change doesn't address event receiving on spin loop, which is something that will need to be done in the future to support the full spin loop use case. This is orthogonal to OffscreenCanvas, the ability to commit() is a stepping stone in supporting spin loop. The commit interface is already partially defined on the spec, but there are a few wording issues that need to be cleared up there, which we will address very soon. That said, events do not get processed while a thread is waiting for commit.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADp2-T8HPDAN7PM7q6w5pc8SFoWJ4szzMdw3R7T%2B%2BBt0Ad_ihQ%40mail.gmail.com.

--

/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

[gm...@greggman.com]

I'm far from an expert but if the goal is for commit to block and not process events then I guess I really don't see how this spec is ready to ship.

That sounds like a huge change from everything previously. Is there some other worker api that already blocks that shows how all the issues are solved?

If you're spinning on commit()

      while(true) {

         render();

         commit();

      }

And commit blocks then you're never processing other events. If commit doesn't block then you have all kinds of nesting issues 

      // in worker

      onmessage = function() {

         renderIntoContext();   // the spin loop is blocked when this arrives.

      };

      while(true) {

         renderStuff();

         postMessage({});  

         commit();

      }

      // in main thread

      onmessage = function() {

         worker.postMessage({});

      }

Sorry I'm not familiar enough with the the web to date but my understanding at the moment is this goes pretty much against everything previous and seems like it would have lots of issues so it seems premature to ship with the intent of such a large change if that change hasn't been specced and tested behind a flag for a while. I get that OffscreenCanvas has been tested but blocking commit has not has it?

2018年6月29日金曜日 21時03分50秒 UTC+9 Fernando Serboncini:

[PhistucK]

While non-standard by now, the file system API has a synchronous variant that can (only) be used by workers, I believe it behaves the same way (blocking until it returns).

Also, any other tight loop would do that same, right? I am not sure this is a new concept...

☆PhistucK

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fbcd9636-df4b-45eb-8132-7d9f7baea70d%40chromium.org.

[Marijn Kruisselbrink]

On Fri, Jun 29, 2018 at 3:37 PM PhistucK <phis...@gmail.com> wrote:

While non-standard by now, the file system API has a synchronous variant that can (only) be used by workers, I believe it behaves the same way (blocking until it returns).

Also, any other tight loop would do that same, right? I am not sure this is a new concept...

Haven't read much of the rest of this thread, but yes, while the File API (i.e Blob/File/FileReader/FileReaderSync) does have a sync variant, and the (chrome only) FileSystem API also has a sync variant, we're (well, I am) sort of regretting having that sync variant for workers. I don't think if we were designing either of those APIs today we'd include the sync variants, as with promises and async/await there really isn't much benefit to them imho.

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_%2BrAgiHXnmvyOFYJu0r%2B6fU_h_XXyxKjMjNAD5SZnNfzQ%40mail.gmail.com.

[Ken Russell]

In order for WebAssembly threads to use the OpenGL ES API to render via WebGL, it's necessary to have a way to present frames on demand, and to throttle the application if it's swamping the graphics processor. That's why OffscreenCanvas.commit() is required, and why it has to be able to block execution until it completes.

It's understood that WebAssembly's thread execution model – which is C's threading model, where the main loop of the thread never exits – is incongruous with web workers' event-driven model. APIs using promises don't work well, and since the main loop never exits, messages posted to the wasm thread will never be received. This area will require more work. APIs  WebAssembly threads are finally on the verge of shipping, so let's get them turned on and start figuring out whether more primitives are needed, for example manually running a single posted message or event.

-Ken

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BOSsVaxROTaSuM5HTQukHquArZ0WM-3pG-iyPfJvv%3Dqcyp3bw%40mail.gmail.com.

[Gregg Tavares]

How does commit work if the page is not the front tab?

With rAF you just don't deliver an event so the worker/page can still process other events (a XHR that arrives, a websocket message, etc...)

With commit though what happens?

     commit blocks forever (in which case the worker is stuck)

     commit is no-op (in which case it's associated canvas never has the correct content)

     commit is throttled (1 fps?)

     ???

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fbcd9636-df4b-45eb-8132-7d9f7baea70d%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_%2BrAgiHXnmvyOFYJu0r%2B6fU_h_XXyxKjMjNAD5SZnNfzQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

[Ken Russell]

Not sure yet. Maybe fserb@ will have an opinion, but let's get it working for basic uses first.

☆PhistucK

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fbcd9636-df4b-45eb-8132-7d9f7baea70d%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_%2BrAgiHXnmvyOFYJu0r%2B6fU_h_XXyxKjMjNAD5SZnNfzQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

[Gregg Tavares]

Maybe I'm mis-understanding

Intent to Ship = move to stable not behind a flag correct?

If so the issues of a blocking commit need to be solve before shipping to stable not after.

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fbcd9636-df4b-45eb-8132-7d9f7baea70d%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_%2BrAgiHXnmvyOFYJu0r%2B6fU_h_XXyxKjMjNAD5SZnNfzQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

[Gregg Tavares]

On Sat, Jun 30, 2018 at 1:40 PM Ken Russell <k...@chromium.org> wrote:

Not sure yet. Maybe fserb@ will have an opinion, but let's get it working for basic uses first.

Even the basics don't work yet AFAICT.  Filed bugs with examples that it doesn't work. Maybe I'm mis-understanding how it's supposed to be used but given the bug repos this doesn't seem ready for shipping and seems like the shipping commit should be reverted until these issues are addressed?  

[Fernando Serboncini]

Hi.

Regarding the background page throttle. Right now we do no throttle, as in, the renderer keeps consuming images (that are not shown, because the tab is in the background) and we keep blocking at Vsync. It's not a problem in the spec to throttle this, as we state that commit() returns when it's ready to consume another frame and no promise is made in regard to what the UA does to the produced frame. Properly throttle is a important feature that we will work on next, but we didn't consider a top priority for launching.

As Ken tried to explain, this is not a "DedicatedWorker now has full support for non-event based applications". This is a stepping stone to support that. Once SharedArrayBuffer is back, applications will be able to use it to send events back to Workers (as they have been doing in WebAssembly), but won't have to send drawing commands anymore, since commit() will solve this issue.

That said, OffscreenCanvas works without 'commit()', as canvas normally does (by rendering as soon as the JS context yields AND we get a new begin-frame signal). The case of calling 'commit()' inside a requestAnimationFrame() is unsupported and unnecessary.

Regarding the event bug, we currently don't enforce that commit() is not called inside RAF, but it would be trivial to either make it a no-op or throw an exception. We haven't focused on this case much, since there's no real use for it, but it's a nice quality-of-life change and it def has a important learning aspect to OffscreenCanvas usage that we will address in the very near future. 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOct_BFL2%3DxtbZe9dTjnE9K8qcc26ehdxPQe7G1nyp4LFLJjCg%40mail.gmail.com.

[Gregg Tavares]

I want the point out that AFAIK what Chrome is shipping is NOT the spec linked above.

The spec linked above says all over it that it's required to call commit for updates to appear in the placeholder and for the placeholder's intrinsic size to change but that is not what Chrome is shipping.

Is there a spec that matches what Chrome is shipping?

As for throttling it is far more complicated than you're letting on. As has been made clear in other discussions commit is now pretty much a WebAssembly only API. Using it prevents all messaging from working. So, you're required to use SharedArrayBuffers to do any communication with a thread using a spin loop on commit. If you block commit then all code using Atomics to talk to SharedArrayBuffers suddenly becomes vastly more complicated having to deal with structuring their code to deal with lock timeouts.

It seems to me Chrome should ship rAF and the new non-commit needed workflow (once the spec has been updated to make it clear commit is not needed for updates) and that commit itself should still be behind a flag until SharedArrayBuffers are shipping and real tests for how how hard it is to deal with a forever blocking commit are (a commit on a non-front tab)

[Fernando Serboncini]

Just to clarify, we are still missing one spec change, Since there was no IDL change, and we only had to clean up some language around commit() being mandatory, we were waiting for the other changes to land before sending this one. The spec that matches it is:

https://github.com/junov/OffscreenCanvasAnimation/blob/master/OffscreenCanvasAnimation.md

I'm still not sure what you mean by commit making SharedArrayBuffers/Atomics more complicated. The browser may throttle anything at any time. Any implementation will have to deal with this even without commit().

I have no problem in keeping commit behind a flag.

I'll check with more people, and if everyone is fine with it, I'll keep it under a flag until we have multithreaded webassembly cases working again.

[Gregg Tavares]

On Sat, Jul 7, 2018 at 6:11 AM Fernando Serboncini <fs...@google.com> wrote:

Just to clarify, we are still missing one spec change, Since there was no IDL change, and we only had to clean up some language around commit() being mandatory, we were waiting for the other changes to land before sending this one. The spec that matches it is:

https://github.com/junov/OffscreenCanvasAnimation/blob/master/OffscreenCanvasAnimation.md

I'm still not sure what you mean by commit making SharedArrayBuffers/Atomics more complicated. The browser may throttle anything at any time. Any implementation will have to deal with this even without commit().

Stopping rAF works great. I view a shadertoy page that's super heavy and makes my entire machine run slow. I put that tab in the background and it's all good. If rAF was throttled instead, delivering once a second or so the experience would be awful, my entire machine stuttering every 1 second until I closed the tab. That suggests commit needs to block indefinitely when not the front tab.

Throttling is not the same as blocking. If the browser stops delivering rAF events other events are still processed. If the browser freezes on commit forever then nothing is processed period so other threads trying to synchronize via Atomics will suddenly never see the other thread (the one blocked on commit forever) responding. That situation does not exist today. There are no other APIs that block forever the way commit will.

It seems like it would be a good idea to test the implications of commit blocking before shipping it. Make an app (like say Google Maps) that uses commit for rendering and needs constant new data streamed into it. Or maybe Mozilla Hubs since that also has data that needs to constantly be passed to the rendering commit using worker and see what issues there are when freezing on commit. Of the top of my head it might be important for commit return some status as in "you'll be blocked next time you call me" so make preparations, or it might be important to have some other API to call to give you chance to respond. As it is now you'd get an blur event but by the time you get that event it's too late to stop the worker using commit from doing anything. It's already going to be blocked.

In any case, that's why I believe commit should probably stay behind a flag a little longer.

[esp...@chromium.org]

On Friday, June 29, 2018 at 4:26:40 PM UTC-7, Kenneth Russell wrote:

In order for WebAssembly threads to use the OpenGL ES API to render via WebGL, it's necessary to have a way to present frames on demand, and to throttle the application if it's swamping the graphics processor. That's why OffscreenCanvas.commit() is required, and why it has to be able to block execution until it completes.

It's understood that WebAssembly's thread execution model – which is C's threading model, where the main loop of the thread never exits – is incongruous with web workers' event-driven model. APIs using promises don't work well, and since the main loop never exits, messages posted to the wasm thread will never be received. This area will require more work. APIs  WebAssembly threads are finally on the verge of shipping, so let's get them turned on and start figuring out whether more primitives are needed, for example manually running a single posted message or event.

-Ken

I want to raise a concern, as an observer here and the original proposer of the promise based commit(), that blocking the thread inside commit() and having authors use a while (true) loop interacts very poorly with the rest of the platform (which you allude to, though it's not clear to me that shipping even more things to work around it is a good idea...). For example:

- Microtask queue is infinitely growing. Any API that's resolving or creating Promises is enqueueing microtasks, and that queue will grow forever resulting in an OOM.

- Requires adding new sync APIs for all other APIs to use them inside the loop.

- Makes sharing the thread for multiple canvases impossible. The Promise version of commit() was designed to handle drawing to multiple surfaces and waiting independently.

- If a tab is in the background it stops drawing, which means the commit() call would block indefinitely and you then need to introduce new features to abort the commit() if ex. you want to break out of the loop since the user's gaming session timed out.

A promise based version of commit() works fine in JS as while (await context.commit()) { handleInput(); draw(); }, so it seems the blocking is primarily an issue for porting existing content in wasm that wants to write while (true) inside the engine instead of having a JS wrapper around it using the task queue and calling into wasm for the functions inside the loop. Has that been attempted? It only means crossing the boundary twice per frame which should be fine.

Another alternative would be to give workers the explicit ability to spin a nested event loop waiting on a promise, or to block indefinitely on a Promise resolving (though many APIs say "queue a task to resolve the promise" so without an event loop those promises never resolve unless the blocking is also spinning the event loop.). If either of those new things was added then a blocking commit() is not needed though.

I'm not a decider here, and I totally understand both the urgency and the use cases folks want, but in my experience saying "let's get them turned on and start figuring out whether more primitives are needed" has resulted in a lot of the quirks we live with today, and a lot of me explaining to other companies why X API makes the app they're building jank up.

It feels like there should be better alternatives than shipping a brand new API that leaks memory in the microtask and task queues even when used correctly. I'd be happy to file a bug somewhere about this API, but I'm not sure the right forum or people to have such a discussion about event loop semantics. Is junov's OffscreenCanvas bug tracker the right place to discuss an API like that? :)

(fwiw I've also done multiple integrations recently using OffscreenCanvas with the promise based commit() API and folks have been very happy with it. So the existing API does work well for production use cases, though perhaps not all.)

- E

 

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fbcd9636-df4b-45eb-8132-7d9f7baea70d%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_%2BrAgiHXnmvyOFYJu0r%2B6fU_h_XXyxKjMjNAD5SZnNfzQ%40mail.gmail.com.

[Fernando Serboncini]

Since this seems to be a point of confusion, we've decided to put commit() back under a flag until other APIs catch up to it.

The rest of OffscreenCanvas and RAF are still shipping as before.

☆PhistucK

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fbcd9636-df4b-45eb-8132-7d9f7baea70d%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_%2BrAgiHXnmvyOFYJu0r%2B6fU_h_XXyxKjMjNAD5SZnNfzQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BOSsVaxROTaSuM5HTQukHquArZ0WM-3pG-iyPfJvv%3Dqcyp3bw%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/8948de37-7fc4-4df5-b340-804feff9f8c8%40chromium.org.

[Elliott Sprehn]

On Tue, Jul 10, 2018, 10:10 AM Fernando Serboncini <fs...@google.com> wrote:

Since this seems to be a point of confusion, we've decided to put commit() back under a flag until other APIs catch up to it.

The rest of OffscreenCanvas and RAF are still shipping as before.

Great! Very excited to see this ship.