Intent to Ship: Canvas Hit Regions

[Justin Novosad]

Contact emails jinho...@samsung.com,ju...@chromium.org Spec http://www.w3.org/TR/2015/REC-2dcontext-20151119/#hit-regions Summary Allows the creation of pointer hit targets in <canvas> elements, along with accessibility features such as text meta-data and associations with elements (e.g focusable controls) from the fallback DOM content. Motivation To simplify hit testing and solve accessibility issues inherent to the fallback DOM concept. Interoperability and Compatibility Risk Firefox: Shipped Edge: Public support Safari: No public signals (bug with no recent activity: https://bugs.webkit.org/show_bug.cgi?id=82796) Web developers: No signals

Describe the degree of compatibility risk you believe this change poses The feature does not add constraints to existing API and has already survived in the open in Firefox without incident since version 30 (june 2014)

Ongoing technical constraints Interactions with the propsed OffscreenCanvas feature are undefined and will need to be worked out as a part of the WIP specification for OffscreenCanvas. Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? 

Yes OWP launch tracking bug https://crbug.com/328961 Link to entry on the Chrome Platform Status https://www.chromestatus.com/features/5904994859483136 Requesting approval to ship? Yes

[Dominic Cooney]

Non-owner here, but I have a few quick questions about this feature, maybe you can help me out:

The "sample 1" link on the Chrome Status entry seems to behave roughly the same in Chrome 47 and Firefox 43 for me--the canvas checkbox doesn't respond to the mouse, but apparently works with the keyboard. Could you walk me through the example?

Does this interact with Shadow DOM, slots, and event retargeting?

Is there any special interaction with PointerEvent? Or does that come "for free" because PointerEvent : MouseEvent?

[Anne van Kesteren]

On Thu, Jan 7, 2016 at 10:55 PM, Justin Novosad <ju...@chromium.org> wrote:
> http://www.w3.org/TR/2015/REC-2dcontext-20151119/#hit-regions

You mean https://html.spec.whatwg.org/multipage/scripting.html#hit-regions,
right?


--
https://annevankesteren.nl/

[Fredrik Söderquist]

On Fri, Jan 8, 2016 at 9:35 AM, Anne van Kesteren <ann...@annevk.nl> wrote:

On Thu, Jan 7, 2016 at 10:55 PM, Justin Novosad <ju...@chromium.org> wrote:
> http://www.w3.org/TR/2015/REC-2dcontext-20151119/#hit-regions

You mean https://html.spec.whatwg.org/multipage/scripting.html#hit-regions,
right?

I think he actually does mean the former - the implementation as-is is closer to the W3 spec than the WHATWG one. IIRC it was implemented against an older draft of the W3 spec, so for instance it does allow 'path' in the |options| dictionary to addHitRegion(), while it does not handle parentID/label/role/cursor/fillRule or apply the more strict requirements on 'control'.

/fs

[Anne van Kesteren]

On Fri, Jan 8, 2016 at 11:02 AM, Fredrik Söderquist <f...@opera.com> wrote:
> I think he actually does mean the former - the implementation as-is is
> closer to the W3 spec than the WHATWG one. IIRC it was implemented against
> an older draft of the W3 spec, so for instance it does allow 'path' in the
> |options| dictionary to addHitRegion(), while it does not handle
> parentID/label/role/cursor/fillRule or apply the more strict requirements on
> 'control'.

Are the other implementations similarly implementing a forked version?
I haven't seen any issues reported against the HTML Standard, yet we
have been adding new <canvas> features there (with help from Justin,
even). Would be good to get that sorted.


--
https://annevankesteren.nl/

[Fredrik Söderquist]

Just glancing at the Gecko implementation (of addHitRegion) it seems to roughly match what Blink implements.

/fs

[Justin Novosad]

On Fri, Jan 8, 2016 at 3:35 AM, Anne van Kesteren <ann...@annevk.nl> wrote:

On Thu, Jan 7, 2016 at 10:55 PM, Justin Novosad <ju...@chromium.org> wrote:
> http://www.w3.org/TR/2015/REC-2dcontext-20151119/#hit-regions

You mean https://html.spec.whatwg.org/multipage/scripting.html#hit-regions,
right?

Right, we intend to implement and ship https://html.spec.whatwg.org/multipage/scripting.html#hit-regions in the future,

but the first shippable milestone can be a subset of that, i.e. what is currently in the W3C spec. Additional functionality that is spec'ed on the whatwg side can be shipped incrementally.

One little detail that we have in the Blink implementation that is not in the W3C spec is the path dictionary parameter (the W3C spec couldn't have this because it does not even have the Path2D interface.)

Right now, the intent that is being expressed is to ship the first installment of the hit regions feature: the subset that is described in the W3C spec.

 

--
https://annevankesteren.nl/

[Anne van Kesteren]

On Fri, Jan 8, 2016 at 4:28 PM, Justin Novosad <ju...@chromium.org> wrote:
> Right, we intend to implement and ship
> https://html.spec.whatwg.org/multipage/scripting.html#hit-regions in the
> future,
> but the first shippable milestone can be a subset of that, i.e. what is
> currently in the W3C spec. Additional functionality that is spec'ed on the
> whatwg side can be shipped incrementally.
> One little detail that we have in the Blink implementation that is not in
> the W3C spec is the path dictionary parameter (the W3C spec couldn't have
> this because it does not even have the Path2D interface.)
>
> Right now, the intent that is being expressed is to ship the first
> installment of the hit regions feature: the subset that is described in the
> W3C spec.

I see. My concern was that they might not be compatible, but if they
mostly are then this all seems fine.


--
https://annevankesteren.nl/

[Justin Novosad]

Agreed. I will double check for inconsistencies. Luckily, most of the copy-pasted text does not seem to have diverged.

--
https://annevankesteren.nl/

[Justin Novosad]

Hmm... the Blink implementation also adds the 'region' attribute to TouchEvent which is not in the W3C spec, and it also has the fillRule option.  One inconsistency I found is that the W3C version says a NotSupportedError should be thrown if neither an ID or a control are specified

After review, I want to amend the definition of the scope of this intent to ship:

What we intend to ship is a partial implementation of https://html.spec.whatwg.org/multipage/scripting.html#hit-region

We intend to initially ship with support for only the following HitRegion options:  path, fillRule, id, and control.

All other options (parentID, cursor, label, role) will be ignored and the implementation will behave as if those options were set to their default values.

A separate intent will be sent at a later date when those additional options are to be implemented and shipped.

I will modify the feature dashboard entry to reflect this.

[Rick Byers]

On Fri, Jan 8, 2016 at 11:15 AM, Justin Novosad <ju...@chromium.org> wrote:

Hmm... the Blink implementation also adds the 'region' attribute to TouchEvent which is not in the W3C spec, and it also has the fillRule option.

 Can you point me to more details about the TouchEvent change?  I quickly searched / skimmed the spec and current implementation in ToT, but I haven't yet found what you're referring to here.

  One inconsistency I found is that the W3C version says a NotSupportedError should be thrown if neither an ID or a control are specified

After review, I want to amend the definition of the scope of this intent to ship:

What we intend to ship is a partial implementation of https://html.spec.whatwg.org/multipage/scripting.html#hit-region

We intend to initially ship with support for only the following HitRegion options:  path, fillRule, id, and control.

All other options (parentID, cursor, label, role) will be ignored and the implementation will behave as if those options were set to their default values

This is exactly what I was about to suggest you do, thank you!  In particular, indicating (eg. in comments we add to the IDL file) that we're intending to match the WhatWG spec is a reminder that if the behavior changes in the WhatWG  spec then it's either a bug in blink or a bug in the spec that we don't match (i.e. we implement a living standard, not a specific snapshot).

This is especially important when the only other shipping browser is Firefox.  Presumably their implementation completely ignores whatever happens to be written in the W3C spec ;-).  So we maximize interoperability by focusing on the WhatWG spec as well.

[Justin Novosad]

On Fri, Jan 8, 2016 at 5:09 PM, Rick Byers <rby...@chromium.org> wrote:

On Fri, Jan 8, 2016 at 11:15 AM, Justin Novosad <ju...@chromium.org> wrote:

Hmm... the Blink implementation also adds the 'region' attribute to TouchEvent which is not in the W3C spec, and it also has the fillRule option.

 Can you point me to more details about the TouchEvent change?  I quickly searched / skimmed the spec and current implementation in ToT, but I haven't yet found what you're referring to here.

Yeah... you kinda need to know what to look for. The code is implemented as a partial interface in the canvas2D module:

https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Source/modules/canvas2d/TouchHitRegion.cpp&sq=package:chromium&type=cs

See also the EventHitRegion and MouseEventHitRegion classes

 

  One inconsistency I found is that the W3C version says a NotSupportedError should be thrown if neither an ID or a control are specified

After review, I want to amend the definition of the scope of this intent to ship:

What we intend to ship is a partial implementation of https://html.spec.whatwg.org/multipage/scripting.html#hit-region

We intend to initially ship with support for only the following HitRegion options:  path, fillRule, id, and control.

All other options (parentID, cursor, label, role) will be ignored and the implementation will behave as if those options were set to their default values

This is exactly what I was about to suggest you do, thank you!  In particular, indicating (eg. in comments we add to the IDL file) that we're intending to match the WhatWG spec is a reminder that if the behavior changes in the WhatWG  spec then it's either a bug in blink or a bug in the spec that we don't match (i.e. we implement a living standard, not a specific snapshot).

This is especially important when the only other shipping browser is Firefox.  Presumably their implementation completely ignores whatever happens to be written in the W3C spec ;-).  So we maximize interoperability by focusing on the WhatWG spec as well.

Well, people from Microsoft and IBM were particularly active in ironing out the the spec and writing conformance tests on the W3C side. If you look at the mailing list archives (https://lists.w3.org/Archives/Public/public-canvas-api/) you'll see that almost all communication over the past couple years has been about this API (and a few other smaller canvas accessibility features).  So I expect to see this in Edge at some point.  I don't know if any one ever swept through the edits they made unilaterally, but I suspect there might be some details or clarifications that might be worth upstreaming.

[Jinho Bang]

2016년 1월 9일 토요일 오전 7시 9분 33초 UTC+9, Rick Byers 님의 말:

This is exactly what I was about to suggest you do, thank you!  In particular, indicating (eg. in comments we add to the IDL file) that we're intending to match the WhatWG spec is a reminder that if the behavior changes in the WhatWG  spec then it's either a bug in blink or a bug in the spec that we don't match (i.e. we implement a living standard, not a specific snapshot).

This is especially important when the only other shipping browser is Firefox.  Presumably their implementation completely ignores whatever happens to be written in the W3C spec ;-).  So we maximize interoperability by focusing on the WhatWG spec as well.

AFAIK, Firefox has implemented the feature as per W3C 2dcontext level 1 spec[1]. (by Rik Cabanier who is w3c spec writer.)

Basically, Chrome implementation also follows the same spec as with Firefox.

Before the spec[1], there was a another W3C spec[2] but it has separated into level1 and level2 for CR release. (FYI, level2 spec[2] is almost the same with WHATWG[3])

Current canvas implementation(Chrome and Firefox) follows level1 spec as well as part of level2(=WHATWG) spec(e.g. Path2D).

The hit region feature consists of hit detection part and canvas accessibility part.

The hit detection is not only the most useful basic function, but also Chrome and Firefox (and Safari) already enabled Path2D object and related apis(defined in level2).

So, I think it's resonable to add useful options(fillRule and path) for hit detection although they are in level2.

I think we can ship the hit detection part first as follows.

 - We can enable the hit detection part first and then enable the other parts additionally later. (low relationship between two parts.) 

 - Firefox already shipped the hit detection part (level1 + path)

 - The hit detection part is the most basic function in hit region feature. 

[1] https://www.w3.org/TR/2dcontext/#hitregionoptions

[2] https://www.w3.org/TR/2dcontext2/#hitregionoptions

[3] https://html.spec.whatwg.org/multipage/scripting.html#hitregionoptions 

[Philip Jägenstedt]

On Fri, Jan 8, 2016 at 11:28 PM, Justin Novosad <ju...@chromium.org> wrote:

On Fri, Jan 8, 2016 at 5:09 PM, Rick Byers <rby...@chromium.org> wrote:

On Fri, Jan 8, 2016 at 11:15 AM, Justin Novosad <ju...@chromium.org> wrote:

Hmm... the Blink implementation also adds the 'region' attribute to TouchEvent which is not in the W3C spec, and it also has the fillRule option.

 Can you point me to more details about the TouchEvent change?  I quickly searched / skimmed the spec and current implementation in ToT, but I haven't yet found what you're referring to here.

Yeah... you kinda need to know what to look for. The code is implemented as a partial interface in the canvas2D module:

https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Source/modules/canvas2d/TouchHitRegion.cpp&sq=package:chromium&type=cs

See also the EventHitRegion and MouseEventHitRegion classes

Thanks, I found the IDL files and from there the relevant spec sections:

https://html.spec.whatwg.org/multipage/scripting.html#MouseEvent-partial

https://html.spec.whatwg.org/multipage/scripting.html#Touch-partial

I see that the MouseEventInit extensions aren't in Blink, can you add that before shipping?

(The reason that TouchInit isn't extended is because Touch.prototype.region is defined (and implemented) to return a value based on the touch location, and not the value of an internal slot like MouseEvent.)

LGTM1 to ship!

[Justin Novosad]

On Tue, Jan 19, 2016 at 8:36 AM, Philip Jägenstedt <phi...@opera.com> wrote:

On Fri, Jan 8, 2016 at 11:28 PM, Justin Novosad <ju...@chromium.org> wrote:

On Fri, Jan 8, 2016 at 5:09 PM, Rick Byers <rby...@chromium.org> wrote:

On Fri, Jan 8, 2016 at 11:15 AM, Justin Novosad <ju...@chromium.org> wrote:

Hmm... the Blink implementation also adds the 'region' attribute to TouchEvent which is not in the W3C spec, and it also has the fillRule option.

 Can you point me to more details about the TouchEvent change?  I quickly searched / skimmed the spec and current implementation in ToT, but I haven't yet found what you're referring to here.

Yeah... you kinda need to know what to look for. The code is implemented as a partial interface in the canvas2D module:

https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Source/modules/canvas2d/TouchHitRegion.cpp&sq=package:chromium&type=cs

See also the EventHitRegion and MouseEventHitRegion classes

Thanks, I found the IDL files and from there the relevant spec sections:

https://html.spec.whatwg.org/multipage/scripting.html#MouseEvent-partial

https://html.spec.whatwg.org/multipage/scripting.html#Touch-partial

I see that the MouseEventInit extensions aren't in Blink, can you add that before shipping?

Good catch! Filed a bug that is blocking the launch bug: crbug.com/579076

[Rick Byers]

Sorry for the delay, I wanted to take some time to understand this feature better - given my team's involvement in input and hit-testing.  /cc input-dev as FYI there.

That looks like a spec bug to me - probably because we only recently standardized TouchInit.  I filed https://github.com/whatwg/html/issues/547 for this.  Ideally this can be added quickly to the spec and blink implementation, but if not I wouldn't block shipping on that.

But there seem to be other problems here.  Our current implementation for "region" on both MouseEvent and Touch appear to use the pointer's current location.  For touch this doesn't appear to match the spec, which says: "The region attribute on a Touch object representing a touch point T must return the value obtained by running the following algorithm when T was first placed on the surface".  This spec text makes sense because touch events are defined to have "implicit capture" meaning they do a hit-test only at touchstart time, and then following touchmove/touchend events are targetted at the same element that received the touchstart (no new hit-test).  It sounds like Touch.region is defined to match this implicit capture semantics of the events, but our implementation doesn't appear to be doing that.  I filed https://crbug.com/579614 to track this.  I think this probably should block shipping since the behavior difference could cause a bunch of problems / confusion for web developers.

There's also a bunch of strange inconsistencies in the spec between the Touch and MouseEvent case (which don't appear to be reflected in our implementation).  I filed https://github.com/whatwg/html/issues/548 for those.  This doesn't need to block shipping itself IMHO, but it suggests that there may be interop issues here.  Does Firefox really match the spec wrt. the differences between Mouse and Touch behavior?  In which cases do we match the spec vs. match mouse behavior (the implementation looks like Mouse and Touch behave identically - which seems like the right model to me).

LGTM1 to ship!