Intent to Implement: CSS Spatial Navigation (only spatialNavigationSearch API)

[hyojin...@lge.com]

Contact emails

hyojin...@lge.com, junho0...@lge.com

Explainer

https://github.com/WICG/spatial-navigation/blob/master/docs/spatialNavigationSearch_explainer.md

Design doc/Spec

Spec: https://drafts.csswg.org/css-nav-1/#dom-element-spatialnavigationsearch

TAG review: https://github.com/w3ctag/design-reviews/issues/287

CSS WG discussion: https://github.com/w3c/csswg-drafts/issues/3743

Summary

CSS Spatial Navigation provides a directional focus navigation with arrow keys, mainly used in smart TV with remote control and touchless smartphone. It could be used for one of accessibility features in several devices as well. In CSS Spatial Navigation specification, there are a dozen of APIs such as DOM methods, CSS properties and UI Events. This is an intent to implement only for spatialNavigationSearch API as a first phase. spatialNavigationSearch() is a proposed API to get a next target element to be focused via directional inputs. It returns a focusable element as a best candidate element in a given direction.

Motivation

The basic processing model of spatial navigation has been supported with a bit unstable quality in WebKit and Blink for more than a decade years behind a flag (--enable-spatial-navigation), but it has recently been enhanced in terms of stability by Vewd, Google and LG. spatialNavigationSearch() provides an overriding option to authors based on the internal processing model in Blink. Several spatial navigation JS frameworks such as js-spatial-navigation(Mozilla) and Enact spotlight(LG), needs to use the option to customize the original behavior of spatial navigation.

Debuggability

No need any support from DevTools for now. This feature has already been verified by polyfill and chrome extension tool. Developers can use them for debugging this feature as well.

Risks

Interoperability and Compatibility

Edge: No signals

Firefox: No signals

Safari: No signals

Web / Framework developers: Positive (positive feedback from TV web developers)

Ergonomics

The proposed DOM API is just a get method, so there isn't any performance risk.

Activation

This features works only if the following runtime flag enables. (--enable-spatial-navigation) It means that this features would be basically supported in some specific user agents such as TV with remote control and touchless smartphone. Several JS libraries provides almost same functionalities without consistency, but the libraries with this web engine feature will provide the functionalities with a high estimation accuracy in a consistent way.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes. This feature will be platform agnostic. In Android, directional inputs (i.e. arrow keys) could be triggered by a bluetooth keyboard.

Link to entry on the feature dashboard

TBD

Requesting approval to ship?

Not yet, but we would implement this feature in a month.

[Yoav Weiss]

On Wed, Nov 13, 2019 at 8:23 AM <hyojin...@lge.com> wrote:

Contact emails

hyojin...@lge.com, junho0...@lge.com

Explainer

https://github.com/WICG/spatial-navigation/blob/master/docs/spatialNavigationSearch_explainer.md

Design doc/Spec

Spec: https://drafts.csswg.org/css-nav-1/#dom-element-spatialnavigationsearch

TAG review: https://github.com/w3ctag/design-reviews/issues/287

CSS WG discussion: https://github.com/w3c/csswg-drafts/issues/3743

Summary

CSS Spatial Navigation provides a directional focus navigation with arrow keys, mainly used in smart TV with remote control and touchless smartphone. It could be used for one of accessibility features in several devices as well. In CSS Spatial Navigation specification, there are a dozen of APIs such as DOM methods, CSS properties and UI Events. This is an intent to implement only for spatialNavigationSearch API as a first phase. spatialNavigationSearch() is a proposed API to get a next target element to be focused via directional inputs. It returns a focusable element as a best candidate element in a given direction.

Motivation

The basic processing model of spatial navigation has been supported with a bit unstable quality in WebKit and Blink for more than a decade years behind a flag (--enable-spatial-navigation), but it has recently been enhanced in terms of stability by Vewd, Google and LG. spatialNavigationSearch() provides an overriding option to authors based on the internal processing model in Blink. Several spatial navigation JS frameworks such as js-spatial-navigation(Mozilla) and Enact spotlight(LG), needs to use the option to customize the original behavior of spatial navigation.

Debuggability

No need any support from DevTools for now. This feature has already been verified by polyfill and chrome extension tool. Developers can use them for debugging this feature as well.

Risks

Interoperability and Compatibility

Edge: No signals

Firefox: No signals

Safari: No signals

Could be a good idea to reach out now.

 

Web / Framework developers: Positive (positive feedback from TV web developers)

Ergonomics

The proposed DOM API is just a get method, so there isn't any performance risk.

Activation

This features works only if the following runtime flag enables. (--enable-spatial-navigation) It means that this features would be basically supported in some specific user agents such as TV with remote control and touchless smartphone. Several JS libraries provides almost same functionalities without consistency, but the libraries with this web engine feature will provide the functionalities with a high estimation accuracy in a consistent way.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes. This feature will be platform agnostic. In Android, directional inputs (i.e. arrow keys) could be triggered by a bluetooth keyboard.

Link to entry on the feature dashboard

TBD

Requesting approval to ship?

Not yet, but we would implement this feature in a month.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/3c8f4f9d-4aee-48e4-bf18-8aceb3fd9780%40chromium.org.

[Joe Medley]

Can you please create a Chrome Status entry and provide a tracking bug?

Thanks,

Joe

Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195

If an API's not documented it doesn't exist.

On Wed, Nov 13, 2019 at 8:23 AM <hyojin...@lge.com> wrote:

--

[mi...@ubo.ro]

Hi guys,

Will the Spatial Navigation API be implemented with or without Shadow DOM (both open and closed) support? We are currently using the polyfill developed by mozilla and this forced us to give up on using shadow dom because the polyfills are unable to get focus inside shadow dom. Our app is a desktop/electron app designed to be controlled remotely thus spatial navigation is quite important part of the app.

>> This features works only if the following runtime flag enables. (--enable-spatial-navigation) It means that this features would be basically supported in some specific user agents such as TV with remote control and touchless smartphone.

It would be a shame not to have this API work on PC/ desktop devices. People using their PC as media device would very much appreciate great keyboard support/spatial navigation.

All the best,

Mihai.

To unsubscribe from this group and stop receiving emails from it, send an email to blin...@chromium.org.

[Jervis Muindi]

> spatialNavigationSearch() provides an overriding option to authors based on the internal processing model in Blink. 

To clarify scope, the initial goal is to expose the existing internal spatial navigation processing in Blink via this API ? Or is tweaking and updating the blink spatial navigation processing model also in scope ? 

- Jervis

Sent on the go.
   

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/3f838395-648a-4fca-9784-23830dbf40c0%40chromium.org.

[Hyojin Song]

I've updated the explainer link from Github MD to Google Docs for putting comments.

https://docs.google.com/document/d/1zU_O5DFiBwW4ClXyaM5FuIx5wODLt3PfWQVFCU_SZJg/

[Hyojin Song]

Thank you all for the interest. I put the inline comments as follows:

>> Edge: No signals, Firefox: No signals, Safari: No signals

> (Yoav) could be a good idea to reach out now.

We've already checked those vendors about the spatial navigation in CSS WG meeting and BlinkOn event. MS had an interest in terms of XBox console so that internally made progress on spatial navigation PoC based on Blink AFAIK. Mozilla and Apple had also a bit interest in terms of technical approach, but had not considered in terms of their browsers yet.

> (Joe) Can you please create a Chrome Status entry and provide a tracking bug?

Will do this week.

> (Mihai) Will the Spatial Navigation API be implemented with or without Shadow DOM (both open and closed) support?

Not yet, but will be supported soon. I believe that it's one of the important use cases for Blink spatial navigation. (Not supported by JS) Could you elaborate the situation you mentioned? You can briefly see the current status on Blink SpatNav at the following sample. (need to enable the runtime flag "chrome.exe --enable-spatial-navigation")

https://wicg.github.io/spatial-navigation/demo/sample/sample_shadowDOM_SpatNav.html 

>> This features works only if the following runtime flag enables. (--enable-spatial-navigation) It means that this features would be basically supported in some specific user agents such as TV with remote control and touchless smartphone.

> (Mihai) It would be a shame not to have this API work on PC/ desktop devices. People using their PC as media device would very much appreciate great keyboard support/spatial navigation.

It could be discussed with how to expose the spatial navigation basic capability and its APIs. The previous approach was Ctrl + arrow keys on the keyboard in Windows. It depends on the voice on the use cases from developers.

>> spatialNavigationSearch() provides an overriding option to authors based on the internal processing model in Blink.

> (Jervis) To clarify scope, the initial goal is to expose the existing internal spatial navigation processing in Blink via this API ? Or is tweaking and updating the blink spatial navigation processing model also in scope?

Both. Without spatialNavigationSearch API, there is no way to get any information on the internal spatial navigation processing. It helps JS SpatNav works well via spatialNavigationSearch API. For the second one, I proposed the sample code and result at the main text above. (a looping example)

On Thursday, November 14, 2019 at 1:23:12 AM UTC+9, Hyojin Song wrote:

[David Bokan]

This is a long reply since I've provided parts of this feedback in various forums and we had a discussion around this during BlinkOn so I'm going to try to summarize and consolidate it all below. To be clear, I don't think the broader API is bad or spatialNavigationSearch in particular is harmful but I do think that the general approach is likely to lead to significant limitations and challenges gaining broad usage so I'd argue for a change in direction.

My main concern is that the proposal is heavily imperative, rather than declarative. It provides deep programmatic hooks into the engine. In addition to added complexity to the rest of the engine and platform, I think this limits its use in a number of important ways:

For example, this forces spatial-navigation to block input on the main thread and makes it difficult to move it off-thread. We've spent years fighting performance issues that resulted from similarly designed touch and wheel input APIs. If we had to redesign those today, I don't think we'd make it so easy to block, for example scrolling, on the main thread. The problem is even more pronounced on devices where spat-nav is prevalent: on low-powered devices like KaiOS, the main thread can be busy for _seconds_ at a time.

It also means spatial navigation is localized to a single renderer; how would something like spatialNavigationSearch work in the presence of out-of-process iframes? I have a hard time seeing how the proposed API and a spatial navigation implementation conforming to it could be made to work intuitively on pages with out of process frames. This is a major limitation of the spatial navigation implementation in Blink today; fixing it would mean moving it to the browser process. If these APIs were implemented this would make it difficult or impossible to do.

Lastly, this ties spatial-navigation intimately to page content which can interfere with the user's navigation: the focus ring can be clipped, overlayed, and hidden by other content on the page; script can move or lock focus; key events can be prevented; etc. In short, page script can interfere with the user's primary mode of interaction. I think these can mostly be addressed (e.g. the "interest" concept that attempts to solve focus trapping) but in bolted-on ways that seem inconsistent with the rest of the web platform.

I think a major complicating factor has been that today's spatial navigation implementation tries to work with general web content that doesn't expect to be spatially navigated (i.e. built for mouse/touch). Because of that it relies on many heuristics and complicated algorithms to handle all the corner cases of the web. These make it unpredictable and irrational and means building apps that *are* meant for spatial navigation is difficult for authors. My impression is the proposed APIs provide authors a way to hook into these algorithms and replace them with something more rational. I think we could significantly reduce the problem with a simple switch (e.g. <meta name="simple-spatial-navigation">) that tells the UA that a page has been built for and tested with spat-nav and to "turn off the hacks". That is, I think authors would be satisfied with a much simpler API if we start from the premise that it doesn't have to work on the general-purpose web and the browser provides a clean, sane (and specified) spatial navigation algorithm.

Spat-Nav as a Browser Service

A proposed approach (one we've discussed at BlinkOn and before) would be to have the renderer build a tree of spatially-accessible elements and their geometry information and pass it to the browser process (my understanding is Microsoft had some success prototyping this recently). The browser would collect these "SpatNav Trees" and stitch them together into a single page, thus working with OOPIF. A spat-nav service in the browser would then be able to use this information to navigate around pages without blocking on script and without issues of interacting poorly with page content.

Where available, this information should be used to augment the accessibility tree and have the spat-nav service use that. This means spat nav could leverage all the existing functionality of accessibility (e.g. form control interaction) and apps built for spat nav would automatically be accessible. However, this approach is largely incompatible with the proposed API. I think parts of it could be made to work but not naturally and I think it goes against the spirit of the approach (which is that spat nav input should be semantically hinted) vs the API (which exposes deep programmatic hooks).

To give an example, I've heard in the past that infinite scrollers present a challenge for spat-nav. At BlinkON we saw the newly proposed aria-virtualcontent. Having the spat-nav service consume this kind of information and act on it appropriately seems much better than allowing script hooks and letting the author build the functionality specifically for spatial-navigation. It encourages apps to be built by describing semantics (as ARIA attributes, where appropriate, perhaps with some specific spatial attributes otherwise) which will be helpful not just to spat nav but all input modalities. I don't think spat-nav is unique in facing these challenges so we shouldn't build solutions that only work with spatial navigation.

I believe one of the major use cases here is building apps for devices like TVs and consoles. These aren't web exposed, target a specific device, and often have specific needs. It makes sense to me that, where existing APIs like above aren't sufficient, vendors could provide proprietary attributes and augment the "SpatNav tree" with extra data. The spat-nav service could then use the proprietary extension to provide vendor-specific customization. One advantage of building spat-nav as a browser service rather than inside Blink is that vendors are free to implement this service as they wish without gating changes on slow-moving web APIs.

SpatNav with Web APIs

For cases that truly want to build their own, highly-customized, spat-nav interface, I think that should be possible with existing web-APIs. The explainer lists some of the reasons why this isn't practical to build with existing web functionality. I believe this comes from experience with existing JS polyfills. These attempt to also solve the problem of general-purpose web browsing of content designed for mice/touch. To be clear, I don’t believe that problem can be solved with JS for many of these reasons. However, IMHO if the scope is narrowed to content built ground-up for spatial navigation, I’m unconvinced a reasonable library can’t be built out of existing tools. (e.g. if you have content built to be spatially navigated it should avoid corner cases like overlapping boxes and line-broken links, I don’t see why you can’t build something like spatialNavigationSearch out of Element.getBoundingClientRect)

 

I've added some questions and replies to each of the points listed in the explainer below:

• Limitation from JS capability  (Form elements (slider, text input, etc.), Shadow DOM, OOIPFs, e.g. no way to get the current caret position or the current slider bar for now)

Shadow DOM is only limited for closed trees which are rare and presumably closed for this reason. It's true you can't do much with an OOPIF from JS but I'd point out this is also the case for much of the proposed API.

Could you elaborate on what the limitation in form elements is? 

• Uncertainty on animated pages (With animated object on a web page, Blink SpatNav is better than JS SpatNav in terms of responsiveness.)

Do you have more details or an example here? It's not obvious to me how SpatNav should be interacting with an animation or why implementing this in Blink would perform significantly better since you can run JS during each frame with requestAnimationFrame (and if the animation is composited a Blink implementation won't fare well here either). I think I don't understand the use case here.

• Uncertainty on keydown handlers (Author-defined keydown handlers could affect to the existing SpatNav operation provided by JS SpatNav, e.g. preventDefault in the author-defined keydown handeler could cancel the JS SpatNav logic as well.)

 

This wouldn't be an issue on content built for spatial-navigation. I agree we need a built-in spat-nav for existing content for these kinds of cases.

• Inconsistency between several JS frameworks (Several JS SpatNav provides a bit different user experience. (e.g. rules for next target selection))

 

This seems more like a feature than a bug to me. This would allow apps/libraries to customize the behavior to specific situations when needed and iterate quickly on improvements and functionality. This would only be an issue if a spatial app is targeting multiple spat nav libraries which seems unlikely to me?

One challenge we have with Blink's existing implementation is that every time we fix some corner case seen on one page, it breaks a different case on another page. Versioned JS libraries provide a solution to this kind of churn ("if it works today it'll work tomorrow unless I change it"). 

• Difficult to align native scroll behaviors with SpatNav when moving the focus to offscreen element (With JS, a bit complex codes are needed to support the offscreen element management in a scroll container.)

Could you provide more details about this? Off hand, I'm not sure what about scrolling and offscreen elements isn't exposed in web APIs already (e.g element.scrollIntoView). I'm curious if there's something we're missing here.

• Difficult to align the virtual focus management with a11y supports (The a11y functionality works based on the focus movement, but some JS SpatNav works with virtual focus.)

I'm not an a11y expert so I'll leave the finer points to those who are but I believe the approach I outlined above using a browser-side spat-nav service would have a much better outcome for accessibility.

• Performance degradation due to a bit expensive cost of DOM Access whenever pressing arrow keys (In case of the huge number of focusable elements in a web page, it works slowly when pressing arrow keys)

Do you have any data here? I'd be a bit surprised if DOM access is the bottleneck here rather than the fact we're walking the entire DOM on each key press which I'd expect to be slow in Blink as well. As well, there are new tools in the platform like IntersectionObserver that would allow you to limit the size of your search space.

That said, I can see the benefits of integrating with a browser's built-in service so perhaps we should aim for a world where building ground-up in JS is truly niche.

Thanks,
David

On Wednesday, November 13, 2019 at 11:23:12 AM UTC-5, hyojin...@lge.com wrote:

Contact emails

hyojin...@lge.com, junho0...@lge.com

j

[Hyojin Song]

Hello Bokan,

Thank you for the elaborate and experienced description. It discloses the overall spatial navigation status with several problematic issues so that it couldn't secure its place with a stable quality on the web until now. (so long history with spatial navigation for general web contents since early of 2000 on the web..)

First of all, I deeply respect your experience on performance issues on the main thread(esp. scrolling) and efforts for low-powered devices. Honestly, we also started the spatial navigation(aka. SpatNav) for smart TV as one of the low-powered devices with low performance and issues occurred rather frequently in SpatNav area. In a long-term perspective, I would totally agree with the approach you described.

Before replying your questions (I will respond it next week, though most of the comments make sense.), I would like to share some background and desired directions in phases.

<< SpatNav as a browser service >>

As you mentioned, most of the web contents have only considered mouse and touch as input modalities. It would be one of the biggest challenges on the spread of web-based TV platform ecosystem. The lean-back UX with remote control would be less competitive with desktop and smartphone, even though TV has the awesome big screen. In this background, we would like to embrace the existing web contents on the smart TV browser (which is included in top 5 apps) via the enhanced input modality. SpatNav was one of the key improvement area to be achieved on TV browser and TV installed apps, so we have tried to apply the SpatNav requirements into W3C spec and Blink until now.

I think TV browser almost doesn't need to support spatialNavigationSearch() and kind of the new meta syntax with simple-spatial-navigation. It would be difficult to manage the directional focus path on each page. If needed for some cases, the web contents offset could be adjusted in a proper way or handled by focusout event handler (see exmaple), so we hasn't taken account of the browser service for SpatNav.

<< SpatNav for TV installed apps >>

TV installed apps runs on a web runtime with a specific environments, so it's independent with TV browser. To make the install apps to be easily developed, we provides TV-friendly UI framework named Enact(https://enactjs.com/), which supports spatial navigation capability via the Spotlight JS library(https://enactjs.com/docs/modules/spotlight/) inside Enact. It works quite well, but sometimes it works abnormally or tricky with unexpected behavior. That's the reason we made the decision to migrate part of the JS logics on it into Blink.

At the current moment, I'm not convinced that the decision would be reasonable, because we've put much efforts more than expected. However, we've tried to define an appropriate processing model and standards API in a right direction so that I believe that the activities affects the competitiveness of our product and framework. TV installed apps are one of the important customers for spatialNavigationSerch API. If we secure that API, we can meaningfully use it on Enact framework and other frameworks as well, though it could be eliminated after migrating SpatNav to the browser process.

<< Suggestion >>

I believe that spatialNavigationSearch API affects no performance degradation without any risk, if authors don't use it. We plan to present the performance experiment when it's executed on the focusout handler on a current focused element as a synced event followed by focusin and focus on a next target. We can use it for the time being before all set of future SpatNav with a bit higher quality (e.g. interest, out of Blink, OOPIF, closed Shadow DOM). I guess it takes some time. In addition, the release period of TV product release is about a year unlike desktop and mobile, so we would hope that the API could be applied to M80. We've already started to implement the API, but need to get its review from several valuable people as usual. If I missed something or another opinion from different perspective, I'm willingly open to it.

Thanks,

Hyojin

---------- Original Message ----------

From : David Bokan <bo...@chromium.org>
To : blink-dev <blin...@chromium.org>
Date : 2019/11/21 01:55:26 [GMT+09:00]
Subject : [blink-dev] Re: Intent to Implement: CSS Spatial Navigation (only spatialNavigationSearch API)

--
You received this message because you are subscribed to a topic in the Google Groups "blink-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/blink-dev/Mjm4FzfxekY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/e65e56ae-2a5c-48e1-9cc6-4e8d6e3cc7b6%40chromium.org.

[Hyojin Song]

Hello Bokan,

Sorry for the delay. I put some feedback on your questions as follows:

===

> I don't see why you can't build something like spatialNavigationSearch out of Element.getBoundingClientRect)

I understand your suspicion on the motivation of spatialNavigationSearch API which could be supported by JS with getBoundingClientRect(). The polyfill has already supported spatialNavigationSearch in that way. I understand you think that SpatNav-considered app can be built totally by JS libraries(e.g. polyfill) so that it doesn't need to use Blink SpatNav at all.

However, we would like to use Blink SpatNav on JS SpatNav framework for securing quite light-weight JS SpatNav. It has several advantages that I mentioned above, and at the moment, it needs only one API which supports how to get a mid-result of Blink SpatNav processing model. If someone think it doesn't make sense, all APIs described in css-nav spec seems for JS library. (not browser engine..)

We have bunch of standards API which has implemented in the browser engines and most of the standard APIs could be implemented by JS as well. The important criteria would be its generality and widely usage, but I hope Chromium open source can embrace powerless devices with a minimum coverage. I believe spatialNavigationSearch has few risk and generous API as the most fundamental one for SpatNav.

===

>> Limitation from JS capability (Form elements (slider, text input, etc.), Shadow DOM, OOIPFs, e.g. no way to get the current caret position or the current slider bar for now)

> Shadow DOM is only limited for closed trees which are rare and presumably closed for this reason. It's true you can't do much with an OOPIF from JS but I'd point out this is also the case for much of the proposed API. Could you elaborate on what the limitation in form elements is?

Inside Blink SpatNav, OOPIF could be properly handled and spatialNavigationSearch can handle OOIPF as well. The JS SpatNav can't handle OOPIF without spatialNavigationSearch.

The limitation of form elements means no way to get kind of a slider thumb or cursor position so that JS SpatNav can't handle them for now. The focusless mode or new primitives API can solve the problem.

===

>> Uncertainty on animated pages (With animated object on a web page, Blink SpatNav is better than JS SpatNav in terms of responsiveness.)

> Do you have more details or an example here? It's not obvious to me how SpatNav should be interacting with an animation or why implementing this in Blink would perform significantly better since you can run JS during each frame with requestAnimationFrame (and if the animation is composited a Blink implementation won't fare well here either). I think I don't understand the use case here.

The following example is what I wanted to show: https://codepen.io/anawhj/pen/rNaNbOJ

I thought that it somethime doen't work well when moving the focus via arrow keys, but it seems a tricky case.

===

>> Uncertainty on keydown handlers (Author-defined keydown handlers could affect to the existing SpatNav operation provided by JS SpatNav, e.g. preventDefault in the author-defined keydown handeler could cancel the JS SpatNav logic as well.)

> This wouldn't be an issue on content built for spatial-navigation. I agree we need a built-in spat-nav for existing content for these kinds of cases.

The polyfill provides some restriction that authors can't use window.addEventListerer(keydown or keyup) due to a reserved usage from JS SpatNav. (document or specific DOM elements is no problem, though) In the background, authors need to use those event handlers based on JS SpatNav. If Blink SpatNav is supported, no need to consider it at all.

===

>> Inconsistency between several JS frameworks (Several JS SpatNav provides a bit different user experience. (e.g. rules for next target selection))

> This seems more like a feature than a bug to me. This would allow apps/libraries to customize the behavior to specific situations when needed and iterate quickly on improvements and functionality. This would only be an issue if a spatial app is targeting multiple spat nav libraries which seems unlikely to me?

> One challenge we have with Blink's existing implementation is that every time we fix some corner case seen on one page, it breaks a different case on another page. Versioned JS libraries provide a solution to this kind of churn ("if it works today it'll work tomorrow unless I change it").

We seem to have a different perspective between Blink developer and JS developer. I totally agree that we need to avoid so many band-aids in Blink SpatNav. That's one of the points that SpatNav module would be very difficult to properly handle.

===

>> Difficult to align native scroll behaviors with SpatNav when moving the focus to offscreen element (With JS, a bit complex codes are needed to support the offscreen element management in a scroll container.)

> Could you provide more details about this? Off hand, I'm not sure what about scrolling and offscreen elements isn't exposed in web APIs already (e.g element.scrollIntoView). I'm curious if there's something we're missing here.

With JS code, it seems a bit complex code so that it sometimes raises some field issues. However, it's not impossible to support the cases.

BTW, I just want to get your opinion about the scroll alignment API. You can see the scroll alignment JS API at the following link. https://scroll-into-view-if-needed.netlify.com/ Is it enough as a JS API or could it be considered as a standard API to be supported by browser engines?

===

>>Difficult to align the virtual focus management with a11y supports (The a11y functionality works based on the focus movement, but some JS SpatNav works with virtual focus.)

> I'm not an a11y expert so I'll leave the finer points to those who are but I believe the approach I outlined above using a browser-side spat-nav service would have a much better outcome for accessibility.

I definitely agree on the architecture of a browser-side spat-nav service.

===

>> Performance degradation due to a bit expensive cost of DOM Access whenever pressing arrow keys (In case of the huge number of focusable elements in a web page, it works slowly when pressing arrow keys)

> Do you have any data here? I'd be a bit surprised if DOM access is the bottleneck here rather than the fact we're walking the entire DOM on each key press which I'd expect to be slow in Blink as well. As well, there are new tools in the platform like IntersectionObserver that would allow you to limit the size of your search space.

I've checked it but I couldn'd get any meaningful data. In a general case, it has no performance degradation so no objection to your opinion.

===

Once again, I indeed respect your opinion and suggestion on SpatNav and agree most of your concerns. You're the person who has been known Blink SpatNav in Google so please give a summarized opinion in terms of the review of spatialNavigationSearch() into Blink. Thank you.

To unsubscribe from this group and all its topics, send an email to blink-dev+unsubscribe@chromium.org.

[David Bokan]

Sorry for the delay, replies inline.

 

We have bunch of standards API which has implemented in the browser engines and most of the standard APIs could be implemented by JS as well. The important criteria would be its generality and widely usage, but I hope Chromium open source can embrace powerless devices with a minimum coverage. I believe spatialNavigationSearch has few risk and generous API as the most fundamental one for SpatNav.

I suspect it might be difficult to determine clipping and overlap of elements from JS but otherwise I'm still not totally clear on the advantages of building this as a browser API. However, I agree the risk here is low. My summary opinion is that this isn't harmful to add but I also don't know what problem this is solving.

Inside Blink SpatNav, OOPIF could be properly handled and spatialNavigationSearch can handle OOIPF as well. The JS SpatNav can't handle OOPIF without spatialNavigationSearch.

How would this work? The API returns a Node. If you've got an OOPIF on your page, the DOM tree of the OOPIF isn't accessible to the current frame so you can't return a node to something inside it. Even if you did return some kind of proxy, you'd still need to do a cross-process call - the API shape appears to be synchronous which doesn't match well with an cross-process call.

The following example is what I wanted to show: https://codepen.io/anawhj/pen/rNaNbOJ

I thought that it somethime doen't work well when moving the focus via arrow keys, but it seems a tricky case.

These animations are ticked on the compositor thread while this API works on the main thread, I don't think a blink-side API here will help much. That aside though, is this really a realistic use case for spatial navigation to solve?

The polyfill provides some restriction that authors can't use window.addEventListerer(keydown or keyup) due to a reserved usage from JS SpatNav. (document or specific DOM elements is no problem, though) In the background, authors need to use those event handlers based on JS SpatNav. If Blink SpatNav is supported, no need to consider it at all.

Why is this a restriction?  If authors want to do something with key events other than spat nav they can still register event listeners and preventDefault the key events before they reach spat nav.

Or is the intent that you don't want authors to be able to override spat nav? (this would seem like an odd design for apps designed specifically for spat-nav)

In addition, a JS library can provide authors with an API to allow overriding/listening key events if this is really an issue. 

Perhaps I don't fully understand the key event listener use cases you're worried about.

BTW, I just want to get your opinion about the scroll alignment API. You can see the scroll alignment JS API at the following link. https://scroll-into-view-if-needed.netlify.com/ Is it enough as a JS API or could it be considered as a standard API to be supported by browser engines?

This link appears to be a polyfill of functionality already shipped in Blink in scrollIntoView and scrollIntoViewIfNeeded so this is already available.

Cheers,

David

To unsubscribe from this group and all its topics, send an email to blink-dev+...@chromium.org.

[Hyojin Song]

Thank you for the opinion. Most of your comments make sense so it seems difficult to make you clear for now.

I'll bring some specific evidence on our platform later for showing what problem the API can solve if there are some.

I reply inline on your comments below.

On Tuesday, January 14, 2020 at 11:49:16 PM UTC+9, David Bokan wrote:

Sorry for the delay, replies inline.

 

We have bunch of standards API which has implemented in the browser engines and most of the standard APIs could be implemented by JS as well. The important criteria would be its generality and widely usage, but I hope Chromium open source can embrace powerless devices with a minimum coverage. I believe spatialNavigationSearch has few risk and generous API as the most fundamental one for SpatNav.

I suspect it might be difficult to determine clipping and overlap of elements from JS but otherwise I'm still not totally clear on the advantages of building this as a browser API. However, I agree the risk here is low. My summary opinion is that this isn't harmful to add but I also don't know what problem this is solving.

We've just wanted to use the Blink SpatNav so that we can delegate the SpatNav features from several JS SpatNav libraries owned by several vendors to Blink as a general feature provider. It doesn't have any dramatic advantage, but I imagined that it could be a trigger for extending TV-specific features in Blink. Several small vendors have their own downstream features and it used to require many resources. That's one of the biggest challenges for using Chromium open source, though your comment is absolutely reasonable and acceptable to me as well.

 

Inside Blink SpatNav, OOPIF could be properly handled and spatialNavigationSearch can handle OOIPF as well. The JS SpatNav can't handle OOPIF without spatialNavigationSearch.

How would this work? The API returns a Node. If you've got an OOPIF on your page, the DOM tree of the OOPIF isn't accessible to the current frame so you can't return a node to something inside it. Even if you did return some kind of proxy, you'd still need to do a cross-process call - the API shape appears to be synchronous which doesn't match well with an cross-process call.

 

I've expected that Blink SpatNav can solve the issue even by changing API design in an asynchronous way, but JS SpatNav can't. We can discuss it in further detail after resolving other critical blockers in the future.

The following example is what I wanted to show: https://codepen.io/anawhj/pen/rNaNbOJ

I thought that it somethime doen't work well when moving the focus via arrow keys, but it seems a tricky case.

These animations are ticked on the compositor thread while this API works on the main thread, I don't think a blink-side API here will help much. That aside though, is this really a realistic use case for spatial navigation to solve?

 

Honestly, the example isn't a realistic use case, but just an assumption for showing that Blink processing might be slightly better than JS processing in terms of latency and memory usage. Agree on your comment.

The polyfill provides some restriction that authors can't use window.addEventListerer(keydown or keyup) due to a reserved usage from JS SpatNav. (document or specific DOM elements is no problem, though) In the background, authors need to use those event handlers based on JS SpatNav. If Blink SpatNav is supported, no need to consider it at all.

Why is this a restriction?  If authors want to do something with key events other than spat nav they can still register event listeners and preventDefault the key events before they reach spat nav.

Or is the intent that you don't want authors to be able to override spat nav? (this would seem like an odd design for apps designed specifically for spat-nav)

In addition, a JS library can provide authors with an API to allow overriding/listening key events if this is really an issue. 

Perhaps I don't fully understand the key event listener use cases you're worried about.

 

Authors can handle the key-based event handlers without any restriction, but I wanted to say that authors need to learn how to use the events handlers via JS SpatNav guideline. If you'd like to understand the guideline, please refer to the following link. (https://github.com/WICG/spatial-navigation#handling-browser-events)

BTW, I just want to get your opinion about the scroll alignment API. You can see the scroll alignment JS API at the following link. https://scroll-into-view-if-needed.netlify.com/ Is it enough as a JS API or could it be considered as a standard API to be supported by browser engines?

This link appears to be a polyfill of functionality already shipped in Blink in scrollIntoView and scrollIntoViewIfNeeded so this is already available.

Sorry for confusing you with unclear question. We have a strong requirement for element alignment after scrolling. It's triggered by focus() method or SpatNav with arrow keys (not scrollIntoView) so we need to manually align the element position with JS tweak. I though we define a new CSS property such as "scroll-offset: start end", "scroll-offset-block: nearest", "scroll-offset-inline: center" referring to the design of scroll-behavior CSS property. I just wanted to have your opinion in which it makes sense or not in terms of a clear problem definition, though it seems out of scope on this thread.

Thanks,

Hyojin

To unsubscribe from this group and all its topics, send an email to blin...@chromium.org.

[hu...@logikbyran.se]

>  A proposed approach (one we've discussed at BlinkOn and before) would be to have the renderer build a tree of spatially-accessible elements and their geometry information and pass it to the browser process (my understanding is Microsoft had some success prototyping this recently).

I also like the idea of running SpatNav on the browser side. All SpatNav needs is a tree of focusables and their rects in the root frame so I guess we could re-use the accessibility tree(s)? On the other hand, perhaps building a full a11y tree is unnecessary when only SpatNav is on (but there's no screen reader)?

> the focus ring can be clipped, overlayed, and hidden by other content on the page

Also, a browser-sided SpatNav could paint SpatNav's rect on-top of the web content without overriding the page or browser's :focus style. Overriding :focus can be problematic.


>  I think we could significantly reduce the problem with a simple switch (e.g. <meta name="simple-spatial-navigation">).

I also like the idea of exposing a switch for authors to enable and disable SpatNav; SpatNav is enabled by default on Android TV WebViews. preventDefault() works but I guess it could prevent other [wanted] actions too. And, as you say, controlling SpatNav through JS seems less ergonomic.

> It encourages apps to be built by describing semantics (as ARIA attributes, where appropriate, perhaps with some specific spatial attributes otherwise) which will be helpful not just to spat nav but all input modalities.

I agree. Instead of being controlled by JS, I think SpatNav should read more declarative hints. For example, perhaps SpatNav should create "SpatNav containers" around (all?) ARIA widgets (unless declared not to)?

/ Hugo