Intent to Ship: highlightsFromPoint API

[Fernando Fiori]

Contact emails

stephan...@microsoft.com, sa...@microsoft.com, ffi...@microsoft.com

Explainer

https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/highlight/HighlightsFromPointsExplainer.md

Specification

https://drafts.csswg.org/css-highlight-api-1/#interactions

Summary

The highlightsFromPoint API enables developers to interact with custom highlights by detecting which highlights exist at a specific point within a document. This interactivity is valuable for complex web features where multiple highlights may overlap or exist within shadow DOM. By providing precise point-based highlight detection, the API empowers developers to manage dynamic interactions with custom highlights more effectively, such as responding to user clicks or hover events on highlighted regions to trigger custom tooltips, context menus, or other interactive features.

Blink component

Blink>CSS

TAG review

https://github.com/w3ctag/design-reviews/issues/1043

TAG review status

Issues addressed

Risks

Interoperability and Compatibility

The 'highlightsFromPoint' API is a new feature so there isn’t a compatibility risk. The interoperability risk is limited to the usual risk of limited adoption and/or inconsistent browser support for a new feature.

Gecko: No signal (https://github.com/mozilla/standards-positions/issues/1068)

WebKit: No signal (https://github.com/WebKit/standards-positions/issues/394)

Web developers: Positive (https://github.com/w3c/csswg-drafts/issues/7513#issuecomment-1211033472, https://github.com/w3c/csswg-drafts/issues/7447#issuecomment-2386160133, https://github.com/w3c/csswg-drafts/issues/7447#issuecomment-1183422904)

Other signals:

Ergonomics

None

Activation

None

Security

None

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

None

Debuggability

None

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes

https://wpt.fyi/results/css/css-highlight-api/HighlightRegistry-highlightsFromPoint.html?label=experimental&label=master&aligned

https://wpt.fyi/results/shadow-dom/HighlightRegistry-highlightsFromPoint.html?label=master&label=experimental&aligned

Note that these tests are currently failing in wpt.fyi because the feature still has the status test in code, so it’s not activated for these tests that run under experimental flag, but worth mentioning they’re passing in chromium CI. They’re expected to pass in wpt.fyi as well once the feature status is upgraded.

Flag name on about://flags

None

Finch feature name

HighlightsFromPoint

Requires code in //chrome?

False

Tracking bug

https://issues.chromium.org/issues/365046212

Estimated milestones

Shipping on desktop

134

Anticipated spec changes

Open questions about a feature may be a source of future web compat or interop issues. Please list open issues (e.g. links to known github issues in the project for the feature specification) whose resolution may introduce web compat/interop risk (e.g., changing to naming or structure of the API in a non-backward-compatible way).

None

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/4552801607483392?gate=4762112543686656

Links to previous Intent discussions

Intent to Prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/SJ0PR00MB130193097BB3B418C676D88CEC642%40SJ0PR00MB1301.namprd00.prod.outlook.com

This intent message was generated by Chrome Platform Status.

 

[Stephen Chenney]

I support this as a strongly desired feature for CSS Custom Highlights.

I believe the target milestone should be M-135 given M-134 is in Beta already for a week.

It's also worth pointing out that this method only returns custom highlights, not selection or spelling or any other highlight pseudos.  Furthermore, the highlights returned are all from the highlight registry specific to a given realm. Hence it does not expose any information not already known to the script (because script created the highlights in the first place).

Stephen.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/BL4PR00MB26636D7174ADA73FC348333DDCF92%40BL4PR00MB2663.namprd00.prod.outlook.com.

[Robert Flack]

When you add a click handler to get the highlights from a given point and interact with the highlight, how can you tell whether the returned highlight is topmost and should be handled? E.g. how can you tell that there's not a dialog element or some other absolute positioned element painted overtop of the highlight (and highlighted content)?

On Saturday, February 15, 2025 at 9:17:17 AM UTC-5 Stephen Chenney wrote:

I support this as a strongly desired feature for CSS Custom Highlights.

I believe the target milestone should be M-135 given M-134 is in Beta already for a week.

It's also worth pointing out that this method only returns custom highlights, not selection or spelling or any other highlight pseudos.  Furthermore, the highlights returned are all from the highlight registry specific to a given realm. Hence it does not expose any information not already known to the script (because script created the highlights in the first place).

Stephen.

On Fri, Feb 14, 2025 at 7:59 PM 'Fernando Fiori' via blink-dev <blin...@chromium.org> wrote:

Contact emails

stephan...@microsoft.com, sa...@microsoft.com, ffiori@microsoft.com

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Robert Flack]

Ah, I put together a test page with some examples and I see that in fact we only return the highlights if they were not obscured. This wasn't clear to me from the spec, especially as the analogous API elementsFromPoint returns all elements, even those obscured.

It might make sense to have a unified API for hit test results that could return a combination of highlights and elements. E.g. if we had options for elementsFromPoint to also return highlights, to include elements with pointer-events: none, to stop at the first hit testing opaque element, etc.

Also, as was discussed in the linked issue, using event handlers would be the ideal way I think for developers to handle this.

On Wed, Feb 19, 2025 at 8:56 AM Robert Flack <fla...@chromium.org> wrote:

When you add a click handler to get the highlights from a given point and interact with the highlight, how can you tell whether the returned highlight is topmost and should be handled? E.g. how can you tell that there's not a dialog element or some other absolute positioned element painted overtop of the highlight (and highlighted content)?

On Saturday, February 15, 2025 at 9:17:17 AM UTC-5 Stephen Chenney wrote:

I support this as a strongly desired feature for CSS Custom Highlights.

I believe the target milestone should be M-135 given M-134 is in Beta already for a week.

It's also worth pointing out that this method only returns custom highlights, not selection or spelling or any other highlight pseudos.  Furthermore, the highlights returned are all from the highlight registry specific to a given realm. Hence it does not expose any information not already known to the script (because script created the highlights in the first place).

Stephen.

On Fri, Feb 14, 2025 at 7:59 PM 'Fernando Fiori' via blink-dev <blin...@chromium.org> wrote:

Contact emails

stephan...@microsoft.com, sa...@microsoft.com, ffi...@microsoft.com

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/BL4PR00MB26636D7174ADA73FC348333DDCF92%40BL4PR00MB2663.namprd00.prod.outlook.com.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/6ab7c016-61a9-4b09-add0-30d27a86785dn%40chromium.org.

[Fernando Fiori]

Hi all, thanks for the feedback.

@Stephen, thanks for the target milestone suggestion, I just changed it.

@Robert, I'll respond inline:

"Ah, I put together a test page with some examples and I see that in fact we only return the highlights if they were not obscured. This wasn't clear to me from the spec, especially as the analogous API elementsFromPoint returns all elements, even those obscured.

It might make sense to have a unified API for hit test results that could return a combination of highlights and elements. E.g. if we had options for elementsFromPoint to also return highlights, to include elements with pointer-events: none, to stop at the first hit testing opaque element, etc."

Custom highlights are supposed to be annotations on top of content, the use cases are about highlights that are visible, so the way I see it, it should return the ones that are not obscured. For example, for spellcheck annotations you wouldn't expect the highlights to react to you hovering over them if there's another element on top (like in your test page for instance). I'm filing an issue in CSSWG about this to define if it should include highlights that are obscured or not and will be updating the spec.

Regards this API also returning elements, I think it doesn't fit the use cases either, we could think about the same example I gave above.

"Also, as was discussed in the linked issue, using event handlers would be the ideal way I think for developers to handle this."

For this kind of solution there's been some discussion previously with the CSSWG as well and I wrote some explanation in the explainer for highlightsFromPoint.

Summarizing, all the event-based options analyzed carried significant complexity. This API was selected since it is much simpler to implement and still satisfies the use cases proposed adequately.

Setting an event listener on some user action to get them (as you did in your sample page) is a good way for developers to use it (see also the example in the spec).

Please let me know if you've got any further questions or concerns.

Thanks,

Fernando

[Stephen Chenney]

Reading Rob's reply and the response, I am reminded of the related issue of supporting ::highlight::hover etc, aka "pseudo-classes on highlight pseudo elements" (another requested CSS feature)..

But to do that, it would be very useful to get the Range in addition to the Highlight. Say you have a syntax highlighted code example. The same element, say a line or block of code, might have multiple instances of a variable or keyword highlighted with the same highlight. Maybe the app is processing pointer move events and looking for highlights under the pointer. Having found a highlight, to get the highlighted word you would need to use caretPositionFromPoint() or similar then process the ranges to determine which range covered that caret position.

That could be made simpler if the highlightsFromPoint() API also returned the particular Range that was highlighted under the hit point

Stephen.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/23c57487-54df-47d2-8c4b-94be46b4df2cn%40chromium.org.

[Fernando Fiori]

@Stephen, thanks for bringing this up, that's definitely a possible scenario. There's also been some discussion about it here [css-highlight-api] Is a HighlightPointerEvent type needed? · Issue #7512 · w3c/csswg-drafts

I think it'd be good to add a function like intersects(x,y) to Highlight, which returns a list of ranges in the highlight that intersect the given coordinates. Or even something like highlightRangesFromPoint to CSS.highlights. I would think of these as future iterations or extensions to existing types and keep highlightsFromPoint as simple as it was proposed because this can already be achieved as you mentioned -- by using caretPositionFromPoint() or similar to determine which range was targeted. Also there are some use cases that the current highlightsFromPoint cover, for example custom selection where there's usually just one range, or in a scenario of online collaboration where there's one Highlight registered per user and we want to show let's say some user information when the highlighted content is interacted with.

I think something like you suggested would be a great addition to the API surface to simplify code for developers. I also think there's still some discussion needed to define it properly and that could be done as future work. Let me know your thoughts.

Thanks,

Fernando

[Stephen Chenney]

Thanks for the reply Fernando. I think it's fine to proceed with the current API and continue iterating on further improvements. I agree that there are various issues to resolve when it comes to custom highlights and user interactions and fortunately there does seem to be enough interest around the spec issues to make progress.

Also, once this is shipping I'll do a blog post and include an example of how to identify a sub-range.

In case it's not obvious, I also support the decision to go with an API like this rather than an event.

Cheers,

Stephen.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c30b8c24-8907-4719-967d-8eebc63ec7acn%40chromium.org.

[Robert Flack]

On Wed, Feb 26, 2025 at 9:43 AM Stephen Chenney <sche...@chromium.org> wrote:

Thanks for the reply Fernando. I think it's fine to proceed with the current API and continue iterating on further improvements. I agree that there are various issues to resolve when it comes to custom highlights and user interactions and fortunately there does seem to be enough interest around the spec issues to make progress.

Also, once this is shipping I'll do a blog post and include an example of how to identify a sub-range.

In case it's not obvious, I also support the decision to go with an API like this rather than an event.

I think we should aim to eventually have both. Just as you can query elementsFromPoint or add event listeners to elements, you should be able to query highlights at a point or add an event listener to a highlight.

Cheers,

Stephen.

On Tue, Feb 25, 2025 at 10:16 PM 'Fernando Fiori' via blink-dev <blin...@chromium.org> wrote:

@Stephen, thanks for bringing this up, that's definitely a possible scenario. There's also been some discussion about it here [css-highlight-api] Is a HighlightPointerEvent type needed? · Issue #7512 · w3c/csswg-drafts

I think it'd be good to add a function like intersects(x,y) to Highlight, which returns a list of ranges in the highlight that intersect the given coordinates. Or even something like highlightRangesFromPoint to CSS.highlights. I would think of these as future iterations or extensions to existing types and keep highlightsFromPoint as simple as it was proposed because this can already be achieved as you mentioned -- by using caretPositionFromPoint() or similar to determine which range was targeted. Also there are some use cases that the current highlightsFromPoint cover, for example custom selection where there's usually just one range, or in a scenario of online collaboration where there's one Highlight registered per user and we want to show let's say some user information when the highlighted content is interacted with.

I think something like you suggested would be a great addition to the API surface to simplify code for developers. I also think there's still some discussion needed to define it properly and that could be done as future work. Let me know your thoughts.

Thanks,

Fernando

El lunes, 24 de febrero de 2025 a la(s) 5:00:15 p.m. UTC-8, Stephen Chenney escribió:

Reading Rob's reply and the response, I am reminded of the related issue of supporting ::highlight::hover etc, aka "pseudo-classes on highlight pseudo elements" (another requested CSS feature)..

But to do that, it would be very useful to get the Range in addition to the Highlight. Say you have a syntax highlighted code example. The same element, say a line or block of code, might have multiple instances of a variable or keyword highlighted with the same highlight. Maybe the app is processing pointer move events and looking for highlights under the pointer. Having found a highlight, to get the highlighted word you would need to use caretPositionFromPoint() or similar then process the ranges to determine which range covered that caret position.

That could be made simpler if the highlightsFromPoint() API also returned the particular Range that was highlighted under the hit point

Stephen.

On Mon, Feb 24, 2025 at 3:56 PM 'Fernando Fiori' via blink-dev <blin...@chromium.org> wrote:

Hi all, thanks for the feedback.

@Stephen, thanks for the target milestone suggestion, I just changed it.

@Robert, I'll respond inline:

"Ah, I put together a test page with some examples and I see that in fact we only return the highlights if they were not obscured. This wasn't clear to me from the spec, especially as the analogous API elementsFromPoint returns all elements, even those obscured.

It might make sense to have a unified API for hit test results that could return a combination of highlights and elements. E.g. if we had options for elementsFromPoint to also return highlights, to include elements with pointer-events: none, to stop at the first hit testing opaque element, etc."

Custom highlights are supposed to be annotations on top of content, the use cases are about highlights that are visible, so the way I see it, it should return the ones that are not obscured. For example, for spellcheck annotations you wouldn't expect the highlights to react to you hovering over them if there's another element on top (like in your test page for instance). I'm filing an issue in CSSWG about this to define if it should include highlights that are obscured or not and will be updating the spec.

Regards this API also returning elements, I think it doesn't fit the use cases either, we could think about the same example I gave above.

My thinking here was something like possibly adding a dictionary to elementsFromPoint or designing an API for a unified result.

E.g.

document.hitTest(x, y, options)

Where options could specify:

• elements: true | false (include elements? possibly add a tree-abiding pseudo-elements flag in the future when the CSSPseudoElement API in css-pseudo-4 is ready)
• highlights: true | false (include highlights? Or perhaps this is just pseudoElements)
• opaqueToHitTest: true | false (i.e. should we include pointer-events: none targets - this is highly desirable for developer tooling)
• until: 'first', 'opaqueToHitTest', 'all'

Can you explain what you mean by this doesn't fit the use cases? I don't understand why it wouldn't be useful to know for example which element the intersected highlight was on.

[Alex Russell]

Excited to see a solution to this, but have some concerns:

• The Explainer includes IDL (an anti-pattern) but doesn't outline the problem developers have, or how this design will solve them, with example code
• I understand that this a close analogue with `document.elementsFromPoint()` in that it is both synchronous and returns a list, but it would be good to address the lingering concern about this being synchronous. Will this cause style recalc and layout if they were otherwise deferred? Did you consider API designs that avoid that hazard?

Best,

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.