Intent to Ship: New Canvas 2D API

[Aaron Krajeski]

Contact emails
aar...@chromium.org, fs...@chromium.org

Explainer
https://github.com/fserb/canvas2d
https://youtu.be/dfOKFSDG7IM

Specification
Context Lost Event
Context Restored Event
Will Read Frequently
New Text Modifiers
Reset
RoundRect
Conic Gradient
Filters (still in progress)

Summary
Updated functionality for the Canvas2D API. Adds seven new features/functions to CanvasRenderingContext2D:

- "ContextLost" and "ContextRestored" events
- "willReadFrequently" option for canvases where lots of readback is expected
- More CSS text modifier support
- A reset function
- A roundRect draw primitive
- Conic gradients
- Better support for SVG filters

https://github.com/fserb/canvas2d

Blink component
Blink>Canvas

TAG review status
Resolution: satisfied

Risks
Security and privacy team expressed concerns with ContextLost and ContextRestored events during the intent to implement phase. These concerns were addressed by re-designing the event to not launch simultaneously across different contexts.

Interoperability and Compatibility
Gecko: In development (https://github.com/whatwg/html/issues/5431) Already implemented conic gradient. Okay with willReadFrequently, transforms and reset.

WebKit: Positive (https://github.com/whatwg/html/issues/5619). Positive signal on text modifiers, round rect and color input.

Web developers: Positive (https://www.youtube.com/watch?v=dfOKFSDG7IM) CDN talk in December was received very positively.

Signals
Gecko: https://github.com/mozilla/standards-positions/issues/519
WebKit: https://lists.webkit.org/pipermail/webkit-dev/2021-May/031833.html

Is this feature fully tested by web-platform-tests?
Yes

Flag name
#new-canvas-2d-api

Requires code in //chrome?
False

Tracking bug
https://bugs.chromium.org/p/chromium/issues/detail?id=1201359

Estimated milestones
OriginTrial desktop first
95

OriginTrial desktop last
98

OriginTrial android first
95

OriginTrial android last
98

Link to entry on the Chrome Platform Status
https://www.chromestatus.com/feature/6051647656558592

This intent message was generated by Chrome Platform Status.

[Manuel Rego Casasnovas]

On 14/10/2021 23:06, Aaron Krajeski wrote:
> *Specification*
> Context Lost Event
> <https://html.spec.whatwg.org/multipage/indices.html#event-contextlost>
> Context Restored Event
> <https://html.spec.whatwg.org/multipage/indices.html#event-contextrestored>
> Will Read Frequently
> <https://html.spec.whatwg.org/multipage/canvas.html#concept-canvas-will-read-frequently>
> New Text Modifiers
> <https://html.spec.whatwg.org/multipage/canvas.html#canvastextdrawingstyles>
> Reset
> <https://html.spec.whatwg.org/multipage/canvas.html#dom-context-2d-reset>
> RoundRect
> <https://html.spec.whatwg.org/multipage/canvas.html#dom-context-2d-roundrect>Conic
> Gradient
> <https://html.spec.whatwg.org/multipage/canvas.html#dom-context-2d-createconicgradient>Filters
> <https://github.com/whatwg/html/pull/6763> (still in progress)

These are a bunch of features, so it'd be nice understand if other
parties are on board with all of them.

> *TAG review status
> *Resolution: satisfied
> <https://github.com/w3ctag/design-reviews/issues/627#issuecomment-838726027>

Is the TAG on board with all the features we're shipping?

> *Interoperability and Compatibility*
> Gecko: In development (https://github.com/whatwg/html/issues/5431
> <https://www.chromestatus.com/admin/features/launch/6051647656558592/5?intent=1>)

> Already implemented conic gradient. Okay with willReadFrequently,
> transforms and reset.

Any link to bugzilla?

> *Signals*
> Gecko: https://github.com/mozilla/standards-positions/issues/519

> <https://github.com/mozilla/standards-positions/issues/519>
> WebKit:
> https://lists.webkit.org/pipermail/webkit-dev/2021-May/031833.html

> <https://lists.webkit.org/pipermail/webkit-dev/2021-May/031833.html>

Both Gecko and WebKit have concerns about some of the features we asked
signals for.

Could you summarize if they agree on all the things shipped here, or
just a few. Or maybe they disagree in some of them.

Thanks,
Rego

[Aaron Krajeski]

These are a bunch of features, so it'd be nice understand if other
parties are on board with all of them.

Yes, the original list had extra features that are not included here. Namely Conic Curves, CSS Color Input and Perspective Transforms. These features were removed due to concerns from other parties. Apologies, I should have made this clearer in the original email.

Is the TAG on board with all the features we're shipping?

Yes, they are. They asked for some changes to SVG Filters and Text modifiers, these changes were taken into account on the spec changes there. They also nixed Perspective Transforms and CSS Color Input.

Any link to bugzilla?

No

Both Gecko and WebKit have concerns about some of the features we asked
signals for.

Gecko pushed back on Perspective Transforms, which was removed from the final feature list. They also had pushback on roundRect on that thread, but we reached a consensus on the WHATWG pull request.

WebKit pushed back strongly on Perspective Transforms, which was why it was canceled. They had reservations for reset and SVG Filters that were addressed the spec changes.

In conclusion, the proposal has changed quite a bit from the intent to experiment thread due to addressing concerns from Gecko, Webkit and TAG. At this point everything we are shipping has been codified into WHATWG except for SVG Filters, which is close, but still in progress.

Cheers!
  Aaron

[Arthur Hemery]

Hi Aaron!
Security and Privacy review envoy here. There was some confusion as to what the actual mitigation ended up being. We found in the code the 1-100ms delay implemented here but some people remembered discussion about firing the event when regaining focus. Can you confirm that it was dropped?
We also discussed including the mitigation mean in the spec and we think it should be in there. Would it be possible to update the HTML spec?
Thanks,
Arthur 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPhTwY1OXaX8W_D2nP9XbJopca4zC0RsSHtXkPzhO5HOKhCm2Q%40mail.gmail.com.

[Fernando Serboncini]

Arthur,

that CL is only for OffscreenCanvas, where we don't have a notion of focus at all.

For the regular canvas, you can see here that we only trigger on focus, as discussed.

Regarding adding the mitigation to the spec, we didn't do it as it does look more like an implementation detail (even in our case, we discussed multiple potential solutions that were satisfactory). I think sticking this to spec

would probably be a mistake.

That said, I can bring up with the WhatWG and ask if we can maybe add an implementation note that UAs should be aware of not triggering this immediately on all contexts in different origins. Would that address this?

[]s

F.

[Anne van Kesteren]

On Tue, Oct 19, 2021 at 5:57 PM Fernando Serboncini <fs...@chromium.org> wrote:
> Regarding adding the mitigation to the spec, we didn't do it as it does look more like an implementation detail (even in our case, we discussed multiple potential solutions that were satisfactory). I think sticking this to spec
> would probably be a mistake.
> That said, I can bring up with the WhatWG and ask if we can maybe add an implementation note that UAs should be aware of not triggering this immediately on all contexts in different origins. Would that address this?

I think this should be in the specification. And definitely not just a
note. Privacy and security considerations need to be handled by the
processing model. Any reason this is not tied to visibility? I think
that's what the plan is for equivalent vectors.

[Dominik Röttsches]

Aaron, Fernando,

On Fri, Oct 15, 2021 at 12:06 AM Aaron Krajeski <aar...@chromium.org> wrote:

[...]
Specification
[...]
New Text Modifiers

 

For reference here, after a f2f discussion yesterday with Fernando:

Please do consider adding font-variation-settings and font-feature-settings. In fact, I would find those as important or more important than the new text properties that are added in the New Text Modifiers proposal, as they provide fine-grained low-level control over OpenType features and control over font variations. I think these are important for drawing expressivity on Canvas.

I filed this request back in 2018, and I am happy to see that the general approach of adding the text and font properties to Canvas is getting traction. The technical approach of adding those as properties to Canvas sounds good to me. 

Dominik

[Fernando Serboncini]

It's a bit more nuanced than that, I believe. The visibility part is not the problem, and it probably would be ok to add this to the spec.

The problem is that triggering on visibility allows for cros domain coordination of events (an iframe would trigger the same time as the main document). 

I've moved this conversation to https://github.com/whatwg/html/issues/7130 where we can discuss which parts of this behavior make sense to spec.

 

[Fernando Serboncini]

As discussed offline, those are important and we will be working on them on a separate launch following this.

 

Dominik

[Yoav Weiss]

Thanks for modernizing Canvas! :)

On Thu, Oct 14, 2021 at 11:06 PM Aaron Krajeski <aar...@chromium.org> wrote:

Contact emails
aar...@chromium.org, fs...@chromium.org

Explainer
https://github.com/fserb/canvas2d
https://youtu.be/dfOKFSDG7IM

Great video, thanks for that!!

 

Specification
Context Lost Event
Context Restored Event
Will Read Frequently
New Text Modifiers
Reset

Were the concerns WebKit raised on the issue addressed? Can you expand on that front?

RoundRect
Conic Gradient
Filters (still in progress)

Is the hold-up on the Filters PR editorial or something more fundamental?

I noticed Mozilla's opposition on their position. Is that something that has changed since?

Are these features individually detectable? Do we have reasonable developer advice on how we want folks to use those features with backwards compat/fallbacks in mind?

 

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/eb702e7e-88d6-4b6f-b419-6051511689f1n%40chromium.org.

[Alex Russell]

I'm excited to see these move forward. I probably missed it from the github explainers (which seem pretty IDL heavy, but not particularly use-case driven?), but which of these will be available on Paint Worklets as part of this intent?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Aaron Krajeski]

They are not yet available on paint worklets (they're not in the idls). It doesn't look like it would be too much work to add them. Looking at the WHATWG spec, it doesn't seem like there would need to be any spec change. Am I correct in assuming that?

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Alex Russell]

It would be worth discussing this with the folks responsible for paint worklets; there has been a lot of work to ensure that the worklets don't include APIs that would allow for readback.

Adding Xida and Ian for that.

Best regards,

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Aaron Krajeski]

yoavweiss@:

Great video, thanks for that!!

  Thanks!

Were the concerns WebKit raised on the issue addressed? Can you expand on that front?

  Yes, othermaciej@'s comments lead to us changing the name of that function to `reset` instead of `clear`.

Is the hold-up on the Filters PR editorial or something more fundamental?

  The hold-up is mostly over the question of whether `new CanvasFilter({filter: "someNonExistantFilterType"})` should throw an error. You can read the (very long) discussion in the pull request. Unfortunately it's been extremely hard to reach consensus on that specific issue for the past 3 months. Any comments from this group that would help drive us towards consensus would be much appreciated!

Are these features individually detectable? Do we have reasonable developer advice on how we want folks to use those features with backwards compat/fallbacks in mind?

  We're working with the DevRel team on some polyfills that will be included in a web.dev article that is launching. All the features are trivially detectable except for perhaps contextlost/restored events. I can look further into that if you'd like?
  For polyfills, there are different degrees of effectiveness/difficulty.
  Easy: roundRect (small perf hit), reset
  Hard, maybe we should just advertise a polyfill that avoids errors: filters, conic gradient, text modifiers
  Doesn't make sense to polyfill: contextloss, willReadFrequently

[Domenic Denicola]

On Thu, Oct 28, 2021 at 3:48 PM 'Aaron Krajeski' via blink-dev <blin...@chromium.org> wrote:

yoavweiss@:

Great video, thanks for that!!

  Thanks!

Were the concerns WebKit raised on the issue addressed? Can you expand on that front?

  Yes, othermaciej@'s comments lead to us changing the name of that function to `reset` instead of `clear`.

Is the hold-up on the Filters PR editorial or something more fundamental?

  The hold-up is mostly over the question of whether `new CanvasFilter({filter: "someNonExistantFilterType"})` should throw an error. You can read the (very long) discussion in the pull request. Unfortunately it's been extremely hard to reach consensus on that specific issue for the past 3 months. Any comments from this group that would help drive us towards consensus would be much appreciated!

FWIW the two HTML editors on the thread (myself and Anne, with our HTML editor hats on), as well as Mozilla (via Anne with his Mozilla hat on), prefer the throwing behavior. I think in most cases to overcome that position we would need some really strong reasons why the Chromium project believes the non-throwing behavior is better. It's not clear to me how strong Chromium's position is on this issue, and whether it's worth delaying the feature over. (Or indeed, delaying all the features, since the plan seems to be to bundle them together?)

 

Are these features individually detectable? Do we have reasonable developer advice on how we want folks to use those features with backwards compat/fallbacks in mind?

  We're working with the DevRel team on some polyfills that will be included in a web.dev article that is launching. All the features are trivially detectable except for perhaps contextlost/restored events. I can look further into that if you'd like?

Those two are feature detectable using 'oncontextlost' in Document.prototype, happily enough.

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/ad052b76-cd02-475a-82a2-f010ecb92075n%40chromium.org.

[Yoav Weiss]

Hey Domenic! :)

On Thu, Oct 28, 2021 at 11:00 PM Domenic Denicola <dom...@chromium.org> wrote:

On Thu, Oct 28, 2021 at 3:48 PM 'Aaron Krajeski' via blink-dev <blin...@chromium.org> wrote:

yoavweiss@:

Great video, thanks for that!!

  Thanks!

Were the concerns WebKit raised on the issue addressed? Can you expand on that front?

  Yes, othermaciej@'s comments lead to us changing the name of that function to `reset` instead of `clear`.

Is the hold-up on the Filters PR editorial or something more fundamental?

  The hold-up is mostly over the question of whether `new CanvasFilter({filter: "someNonExistantFilterType"})` should throw an error. You can read the (very long) discussion in the pull request. Unfortunately it's been extremely hard to reach consensus on that specific issue for the past 3 months. Any comments from this group that would help drive us towards consensus would be much appreciated!

FWIW the two HTML editors on the thread (myself and Anne, with our HTML editor hats on), as well as Mozilla (via Anne with his Mozilla hat on), prefer the throwing behavior. I think in most cases to overcome that position we would need some really strong reasons why the Chromium project believes the non-throwing behavior is better. It's not clear to me how strong Chromium's position is on this issue, and whether it's worth delaying the feature over. (Or indeed, delaying all the features, since the plan seems to be to bundle them together?)

My concerns with the throwing behavior are similar to the ones we have discussed in the context of MediaSession actions.

If we go with the throwing behavior, every future addition of filters would have a significant interop risk, in case adopting developers won't use try/catch properly. If they do that and they are not testing in not-yet-supporting browsers, their apps are likely to break entirely in those browsers. 

If we go with a silent failure + feature detection approach, developers using the feature without properly detecting it may not have the desired visual effects they are going for, but won't have unrelated parts of their app break.

From my perspective (with my API owner hat on), less risk is better, and the second approach seems less risky to me.

[Thomas Steiner]

As a developer, I have run into a stupid typo in a filter's name making me pull my hair out for an hour until I realized it. Had there been a console warning or an error about the non-existent filter name, I would not be bald 🧑‍🦲 now.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfVe%3DWB7vT%2BmM3oS%3DJg_nJj7Xu8MGt9eqHqQjkno9Xv1Aw%40mail.gmail.com.

--

Thomas Steiner, PhD—Developer Advocate (https://blog.tomayac.com, https://twitter.com/tomayac)

Google Germany GmbH, ABC-Str. 19, 20354 Hamburg, Germany
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.3.2 (GNU/Linux)

iFy0uwAntT0bE3xtRa5AfeCheCkthAtTh3reSabiGbl0ck0fjumBl3DCharaCTersAttH3b0ttom.hTtPs://xKcd.cOm/1181/
-----END PGP SIGNATURE-----

[Yoav Weiss]

Silent failures + console warnings/errors seem to address this, right?

[Thomas Steiner]

Silent failures + console warnings/errors seem to address this, right?

Yes, as I wrote, as long as it gets surfaced somehow. 

[Mike Taylor]

On 10/29/21 1:23 AM, Yoav Weiss wrote:

Hey Domenic! :)

On Thu, Oct 28, 2021 at 11:00 PM Domenic Denicola <dom...@chromium.org> wrote:

FWIW the two HTML editors on the thread (myself and Anne, with our HTML editor hats on), as well as Mozilla (via Anne with his Mozilla hat on), prefer the throwing behavior. I think in most cases to overcome that position we would need some really strong reasons why the Chromium project believes the non-throwing behavior is better. It's not clear to me how strong Chromium's position is on this issue, and whether it's worth delaying the feature over. (Or indeed, delaying all the features, since the plan seems to be to bundle them together?)

My concerns with the throwing behavior are similar to the ones we have discussed in the context of MediaSession actions.

If we go with the throwing behavior, every future addition of filters would have a significant interop risk, in case adopting developers won't use try/catch properly. If they do that and they are not testing in not-yet-supporting browsers, their apps are likely to break entirely in those browsers. 

If we go with a silent failure + feature detection approach, developers using the feature without properly detecting it may not have the desired visual effects they are going for, but won't have unrelated parts of their app break.

From my perspective (with my API owner hat on), less risk is better, and the second approach seems less risky to me.

I agree with Yoav here (sorry, I don't own any hats). Not throwing will likely result in fewer broken pages in less-well-tested browsers that haven't implemented the APIs yet. And +1 for devtools warnings to help developers figure out "silent" failures.

(I also wonder if requiring try/catch won't trip up new developers trying to use it inside Promises, who don't yet know about `then()/catch()` patterns).

[Domenic Denicola]

Yep. Reasonable people can disagree on the tradeoffs here. The question is whether this is something we want to deadlock over with other implementers.

[Fernando Serboncini]

Since the current implementation is not throwing, we could reasonably launch as-is and the add throwing if it is agreed to do so after.

The added risk is the exact same risk that we are arguing to begin with (i.e., code that uses not-implemented filters without try/catch would break). But since there are no "not-implemented filters" yet, it may not be a problem.

[Domenic Denicola]

I don't think that's true. If someone passes "asdf" to the CanvasFilter constructor, changing from not-throwing to throwing would be a breaking change. We'd need to add use counters, etc. to find out if anyone is passing such invalid values.

Whereas, if we start off with throwing, we can always remove the throwing behavior later, with no use counters required.

[Fernando Serboncini]

yes, of course if you pass "asdf" one would throw and the other wouldn't. There will be a potential behavior difference.

My point is nobody would write "asdf" as a filter, because "asdf" is not a spec'ed or implemented filter anywhere. There are no possible mismatches at this point. If CanvasFilter exists, the set of filters is well defined everywhere.

[Domenic Denicola]

The usual definition of "breaking change" that we use when shipping features is not "if web developers always do valid things, then the change we are making avoids breakages". We can't rely on web developers to write valid code all the time... see e.g. stats about what percent of the web is valid HTML.

For example, while people might not write "asdf", they might write "colourMatrix" (spot the typo), or similar.

[Aaron Krajeski]

During a WHATWG spec meeting this morning we agreed to not throw for malformed canvas filters. There are still a few editorial changes to go through on that PR, but other than that everything is settled for this launch.

[Chris Harrelson]

LGTM1 to ship in parallel with landing the PR.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0612b019-71ee-41e9-aa78-aba0712d0a0dn%40chromium.org.

[Daniel Bratell]

LGTM2

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw__g2FKLgxFsHJ6cRAjS31ONbatHAv26rJ%3Di4dDWprLRA%40mail.gmail.com.

[Manuel Rego Casasnovas]

LGTM3

>>> <https://github.com/w3c/mediasession/issues/228#issuecomment-886455386>

>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0612b019-71ee-41e9-aa78-aba0712d0a0dn%40chromium.org?utm_medium=email&utm_source=footer>.

>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "blink-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to blink-dev+...@chromium.org.
>> To view this discussion on the web visit

>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw__g2FKLgxFsHJ6cRAjS31ONbatHAv26rJ%3Di4dDWprLRA%40mail.gmail.com
>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw__g2FKLgxFsHJ6cRAjS31ONbatHAv26rJ%3Di4dDWprLRA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "blink-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to blink-dev+...@chromium.org

> <mailto:blink-dev+...@chromium.org>.

> To view this discussion on the web visit

> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/8c761ee1-90cc-87b6-a05f-b592871031d4%40gmail.com
> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/8c761ee1-90cc-87b6-a05f-b592871031d4%40gmail.com?utm_medium=email&utm_source=footer>.

[Thomas Steiner]

On Thu 4. Nov 2021 at 20:01 'Aaron Krajeski' via blink-dev <blin...@chromium.org> wrote:

During a WHATWG spec meeting this morning we agreed to not throw for malformed canvas filters. There are still a few editorial changes to go through on that PR, but other than that everything is settled for this launch.

What about the console warnings? Or is this not a spec-level decision, but an implementation detail?

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0612b019-71ee-41e9-aa78-aba0712d0a0dn%40chromium.org.

[Fernando Serboncini]

We are launching with a console warning. Will check on the PR if it makes sense to add a note about it on the spec.

[Thomas Steiner]

On Thu, Nov 4, 2021 at 10:12 PM Fernando Serboncini <fs...@chromium.org> wrote:

We are launching with a console warning. Will check on the PR if it makes sense to add a note about it on the spec.

Perfect, thank you very much!

[Aaron Krajeski]

Filters spec chang was approved and merged. Going to launch now. Thank you all!
https://github.com/whatwg/html/pull/6763