Intent to Experiment: New Canvas 2D API

[Aaron Krajeski]

Contact emails

aar...@chromium.org, fs...@chromium.org

Explainer/Chrome Dev Summit Video

https://github.com/fserb/canvas2d

https://youtu.be/dfOKFSDG7IM

API spec

In progress

  WHATWG:

Text Modifiers

Filters

Perspective Transforms

reset()

CSSColorValue

willReadFrequently

RoundRect

ConicGradient

Context Loss

  MDN 

Conic gradient

Text Modifiers (via ChromeLabs/puppy-content)

Summary

Updated functionality for the Canvas2D API. Adds nine new features/functions to CanvasRenderingContext2D:

  - "ContextLost" and "ContextRestored" events

  - "willReadFrequently" option for canvases where lots of readback is expected

  - CSSColorValues as style inputs

  - More CSS text modifier support

  - A reset function

  - A roundRect draw primitive

  - Full 4x4 transformation matrices

  - Conic gradients

  - Better support for SVG filters

Blink component

Blink>Canvas

Timeline: M92-95

TAG review

None

TAG review status

Not applicable

Risks
Interoperability and Compatibility
Edge: Shipped (partially) (context loss email chain) Very interested in context loss. Helped with implementation.

Gecko: In development (https://github.com/whatwg/html/issues/5431) Already implemented conic gradient. Okay with willReadFrequently, transforms and reset. Push back on filters.

WebKit: Positive (https://github.com/whatwg/html/issues/5619) Some push back on perspective transforms. Positive signal on text modifiers, round rect and color input.

Web developers: Positive (https://www.youtube.com/watch?v=dfOKFSDG7IM) CDN talk in December was received very positively.

Goals for experimentation

Allow third party developers to use the API and provide feedback.

Reason this experiment is being extended

None

Ongoing technical constraints

None

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes (in progress)

RoundRect

reset()

Perspective Transforms

Context Loss (this is not testable on WPT)

Filters

CSSColorValue Input

WillReadFrequently (testable in WPT, work in progress to add tests there)

Conic Gradient

Text Modifiers

Flag name

NewCanvas2DAPI

Tracking bugs

Transforms: https://bugs.chromium.org/p/chromium/issues/detail?id=1140535

Color: https://bugs.chromium.org/p/chromium/issues/detail?id=1187575

Filters: https://bugs.chromium.org/p/chromium/issues/detail?id=1169216

RoundRect: crbug.com/1123971

Reset: crbug.com/1131137

Context Loss: crbug.com/1164059
Will read frequently: https://bugs.chromium.org/p/chromium/issues/detail?id=1090180

Conic gradient: crbug.com/1097034

Text Modifiers: https://bugs.chromium.org/p/chromium/issues/detail?id=1137055

Launch bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1201359

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/6051647656558592

[Aaron Krajeski]

[Aaron Krajeski]

Looks like posting on blink-dev just merged the threads 🤦‍♀️. Double apologies for the noise. This is our real-bonafide intent to experiment thread. 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/ec01b3df-dc82-4cbc-94c4-3efaf1186728n%40chromium.org.

[Mike West]

Hey Aaron,

I have a few questions that I'll add here rather than inline, as the thread's formatting is already a little strange:

1. Can you help me understand why no TAG review was requested? This is a somewhat substantial addition to an existing API, and their input seems like it might be valuable.
   
   
2. Would you mind filing requests for signals with our friends at other vendors via the channels specified in https://bit.ly/blink-signals? It's certainly positive to see participation in GitHub threads, but they've asked us to use their official channels for these intents.
   
   
3. You suggest that Edge is already shipping these APIs, but the link provided isn't public. Have they upstreamed an implementation to Chromium that's already enabled?

-mike

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPhTwY0wkjp57QPLBOT76xCKXP1oktezxTP9J_toe3cVh-chNg%40mail.gmail.com.

[Chris Harrelson]

Hi, one comment.

This link looks broken..

 

[Alex Russell]

Per today's API OWNERs meeting, very conditional LGTM to experiment with all the caveats of Mike's note. Would appreciate you reporting back about both what we learn from the trial from developers as well as any TAG feedback.

[Aaron Krajeski]

Thanks all for the feedback and thanks all for no one pointing out how gracelessly I started this thread!

@Chris Harrelson Sorry about that, here's a better link: https://github.com/whatwg/html/issues/4809. 

1. Can you help me understand why no TAG review was requested? This is a somewhat substantial addition to an existing API, and their input seems like it might be valuable.

I was using Canvas color management I2E as a template. I'll ask for a TAG review and see their feedback.

1. Would you mind filing requests for signals with our friends at other vendors via the channels specified in https://bit.ly/blink-signals? It's certainly positive to see participation in GitHub threads, but they've asked us to use their official channels for these intents.

Yes! will do.

1. You suggest that Edge is already shipping these APIs, but the link provided isn't public. Have they upstreamed an implementation to Chromium that's already enabled?

Yes, they worked on the contextloss api, though ultimately our team ended up implementing it. Nothing is enabled yet, it's still all behind the `NewCanvas2DAPI` flag (which gets enabled with experimental web platform features.

I'll get back to you on 1, 2 (TAG and official signals, inlining comments messed up the list numbering) and respond to this thread.

  Cheers,
    Aaron

[Aaron Krajeski]

TAG review issue: https://github.com/w3ctag/design-reviews/issues/627

[Yoav Weiss]

Friendly ping on bit.ly/blink-signals :)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPhTwY3WmgqAJ8QL9NmmOe4O_1pj-ajF0XUuNWz59SDA42F%3DRQ%40mail.gmail.com.

[Aaron Krajeski]

Thanks for the reminder Yoav!

Mozilla standards positions thread: https://github.com/mozilla/standards-positions/issues/519

I've sent an email describing the proposal to webki...@lists.webkit.org, I assume it will appear here soon https://lists.webkit.org/pipermail/webkit-dev/

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfV14%3DoWr34r%2BiG%2BLWezEU0xtPpmqogO3_p1ohXZeFBtAw%40mail.gmail.com.

[Theodore Olsauskas-Warren]

Hi All,

Some quick feedback from the S&P review, we're have some concerns about the ContextLoss event being delivered to multiple tabs simultaneously, and thus allowing a join across 1P contexts.  Would it be feasible to restrict this to only the active tab? Perhaps delivering the event once the tab has regained focus if it occured whilst the tab didn't have focus? (Apologies if I've missed something that already mitigates this in the design).

[PhistucK]

> allowing a join across 1P contexts

Can you explain the concern given the existing storage event and the BroadcastChannel API (and others, I suppose)?

☆PhistucK

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/04fbf598-e342-4551-ae18-86880b9b7b1cn%40chromium.org.

[Theodore Olsauskas-Warren]

I should clarify, we're concerned this allows a join across different 1P contexts. Both the examples you've given are scoped to the same origin. As an example of our concern, if the user has Site A open in one tab, and Site B open in another tab, and both Site A & Site B receive a context loss event at the same time, then they have a precise timestamp with which they can later use to join the user's separate identities on Site A & Site B. This is type of information is especially concerning as it is independent of other joinable information (such as an IP address, which may be shared or proxied).

[Aaron Krajeski]

sauski@: Talking around with the rest of the team and with the WebGL folks, there is a legitimate privacy concern with the contextloss event. The amount of data would be relatively small, as a tracker would have to be active on two domains simultaneously during a GPU or driver crash. Nevertheless, the data would be unique enough that it's worth designing around this vulnerability. WebGL and WebGPU will have the same issue.

  We're now actively looking into how to delay the event until focus.

[Aaron Krajeski]

Hey everyone! Sorry for the delay in response, blink-dev got removed from the "reply all" chain and I was shouting into the ether. We are code and test complete, just have a few more whatwg specs to merge.

@Theodore Olsauskas-Warren, re context loss: If we delayed context loss until focus events, would that be acceptable from an S&P perspective? 
  The remaining issue is for contexts within iframes. These will not receive a "focus" event unless the user focuses the iframe itself, but in the interest of getting this out we could live with that until we have a better idea.

Some updates from the TAG review (https://github.com/w3ctag/design-reviews/issues/627):
  1. For the CSSColorValue input we are going to rely on the fact that CSSColorValues can implicitly convert to string, and CanvasRenderingContext2D styles all accept strings as input. This makes for a much smaller spec change. We can still use the optimization of taking the object directly, as the functionality is the same.

  2. WebKit says Perspective Transforms are a no-go. With a heavy, affine heart we're putting this feature on the backburner for now.

The mozilla and webkit signals more or less rehash what's already debated on the TAG thread.

Thanks all for the feedback! Think we'll be ready to launch this origin trial soon?

Aaron

[Jeremy Roman]

On Wed, Jun 30, 2021 at 1:33 PM 'Aaron Krajeski' via blink-dev <blin...@chromium.org> wrote:

Hey everyone! Sorry for the delay in response, blink-dev got removed from the "reply all" chain and I was shouting into the ether. We are code and test complete, just have a few more whatwg specs to merge.

@Theodore Olsauskas-Warren, re context loss: If we delayed context loss until focus events, would that be acceptable from an S&P perspective? 
  The remaining issue is for contexts within iframes. These will not receive a "focus" event unless the user focuses the iframe itself, but in the interest of getting this out we could live with that until we have a better idea.

Some updates from the TAG review (https://github.com/w3ctag/design-reviews/issues/627):
  1. For the CSSColorValue input we are going to rely on the fact that CSSColorValues can implicitly convert to string, and CanvasRenderingContext2D styles all accept strings as input. This makes for a much smaller spec change. We can still use the optimization of taking the object directly, as the functionality is the same.

Make sure you check that toString hasn't been replaced on the prototype chain, and otherwise that this skipped step will actually be equivalent to converting to string and then to CSSColorValue. It's a common pitfall to assume this sort of optimization is valid but in general it isn't, at least without accounting for all the weird edges.

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPhTwY2TMCj8y%2BggP62fNsFRPiS%2Bk%3DgwpdbXFN0HOahwXq0Bqw%40mail.gmail.com.

[Theodore Olsauskas-Warren]

@Theodore Olsauskas-Warren, re context loss: If we delayed context loss until focus events, would that be acceptable from an S&P perspective? 

Yep, that sounds good. When this came through the review process that was considered by the S&P reviewers as a reasonable mitigation.

Theo.

[Sushanth Rajasankar]

Hello, 

Just catching up on this discussion. We are super interested in context lost and canvas 2d improvements.

Some questions :

Is it true that we are now limiting this API to fire on Iframes only when they gain focus ? 

This would be problematic for us as some of our properties that want to use this are served from within Iframes. 

- Even without Iframes, can we use page visibility as a trigger to fire these context lost rather than focus. 

This way if a tab is already visible it gets context lost right away. Iframes would use the main frame's page visibility and so

the user should not see broken content.

Thanks,

Sushanth

[Aaron Krajeski]

Is it true that we are now limiting this API to fire on Iframes only when they gain focus ? 

  No, we're just delaying contextloss events randomly now. All events are being delayed by a tunable amount of time.

Also, due to delays in getting this out we need to move the milestones from M92-95 to M95-98.