Intent to Ship: WebXR DOM Overlay

[Klaus Weidner]

Contact emails kla...@chromium.org,mlam...@chromium.org Explainer https://github.com/immersive-web/dom-overlays Spec https://immersive-web.github.io/dom-overlays/ TAG review https://github.com/w3ctag/design-reviews/issues/470 (in progress) Summary Allows WebXR applications using immersive-ar on handheld devices to optionally activate a DOM overlay mode where the 2D page content is shown as an interactive transparent layer on top of the application-drawn WebGL content and camera image. While the current implementation is limited to ARCore-based handheld devices, the specification is intended to also support implementation on AR headsets. Link to “Intent to Prototype” blink-dev discussion https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/QRbZ0ZUjhmI/kdS3S9OtAgAJ Risks

Interoperability and Compatibility Compatibility risks: this is a new feature requiring explicit opt-in by the application, and won't affect existing WebXR pages that aren't asking for the feature to be activated as part of the XR session request. Interoperability risks: the initial shipped version targets handheld AR on phones, but the API and specification were discussed with other browser vendors, and even though there are no official positions in response to https://github.com/immersive-web/dom-overlays/issues/16 they have been contributing feedback from the point of view of headset implementations, including Magic Leap in https://github.com/immersive-web/dom-overlays/issues/9#issuecomment-579021344. We had discussed a prototype implementation demonstrated on a VR headset at a community group face-to-face meeting, with generally positive feedback from participants. Firefox: No public signals Edge: No public signals Safari: No public signals Web developers: Positive (https://github.com/immersive-web/dom-overlays/issues/16#issuecomment-590517800) Ergonomics The feature will be presented as a part of WebXR Device API. The feature will be most likely used in tandem with WebGL, and is designed to work on top of the existing Fullscreen API. Overall the proposed implementation is very similar to the existing implementation of custom HTML controls for video elements which uses the same hardware composition mechanisms. Activation As this feature builds on the WebXR Device API, which is still under development, progress depends on the continuing maturity and enabling of that API. In addition, this feature also depends on still to be defined AR capabilities of WebXR. That said, developers have shown a willingness to experiment with early AR capabilities. A-Frame users have started experimenting with it, for example https://medium.com/@tessrUX/build-a-3d-augmented-reality-stock-data-visualization-on-the-web-with-javascript-part-1-40f59c3784f9 , and model-viewer is actively working on integrating it to support annotations. Polyfills would be challenging since that would require rendering arbitrary DOM content in WebGL. That works for limited subsets, but is hard to do in general, especially for complex interactions. Security Similar to Fullscreen API, users may be confused about what is going on. The current implementation intentionally follows the example of Fullscreen API. Activation requires user activation together with a pre-existing consent prompt for AR sessions, and fully exits fullscreen mode when the immersive AR session ends.

Debuggability Chrome's Inspector works for the DOM overlay content while in immersive-ar mode, including screencast, clickable elements, and interactive style changes. Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? No This feature relies on AR support in the platform. The initial focus is on Android devices that support ARCore. Is this feature fully tested by web-platform-tests? Yes https://wpt.fyi/results/webxr/dom-overlay?label=experimental&label=master&aligned https://github.com/web-platform-tests/wpt/tree/master/webxr/dom-overlay/ Tracking bug https://bugs.chromium.org/p/chromium/issues/detail?id=991747 Demo links https://klausw.github.io/three.js/examples/webvr_lorenzattractor.html https://klausw.github.io/a-frame-car-sample/index.html https://storage.googleapis.com/chromium-webxr-test/latest.html?target=proposals/index.html Link to entry on the Chrome Platform Status https://www.chromestatus.com/feature/6048666307526656

This intent message was generated by Chrome Platform Status.

[Chris Harrelson]

Hi,

Have you asked representatives from other browser vendors about whether they are supportive of this API or willing to ship this feature?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFU2V81aOb-j9mdgCpAus98Hjw8Jz2qA7j5wa73NRaihhMyhkg%40mail.gmail.com.

[sligh...@chromium.org]

Following on from Chris' question, if we are leading (as the browser signals list suggests), is there a reason not to have run an Origin Trial to get good evidence that this is solving an important problem and doesn't have large API shape risks?

As a nit, I'd expect the explainer to include code to show how the proposal addresses the usecases it lists.

Regards

On Thursday, March 5, 2020 at 2:08:36 PM UTC-8, Chris Harrelson wrote:

Hi,

Have you asked representatives from other browser vendors about whether they are supportive of this API or willing to ship this feature?

On Fri, Feb 28, 2020 at 5:25 PM 'Klaus Weidner' via blink-dev <blin...@chromium.org> wrote:

Contact emails kla...@chromium.org,mlamouri@chromium.org Explainer https://github.com/immersive-web/dom-overlays Spec https://immersive-web.github.io/dom-overlays/ TAG review https://github.com/w3ctag/design-reviews/issues/470 (in progress) Summary Allows WebXR applications using immersive-ar on handheld devices to optionally activate a DOM overlay mode where the 2D page content is shown as an interactive transparent layer on top of the application-drawn WebGL content and camera image. While the current implementation is limited to ARCore-based handheld devices, the specification is intended to also support implementation on AR headsets. Link to “Intent to Prototype” blink-dev discussion https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/QRbZ0ZUjhmI/kdS3S9OtAgAJ Risks

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Klaus Weidner]

Chris, I had asked for feedback in https://github.com/immersive-web/dom-overlays/issues/16, but didn't get specific responses on that issue.

Previously, I had been discussing this with raviramachandra (Magic Leap) in https://github.com/immersive-web/dom-overlays/issues/9#issuecomment-579021344 and RafaelCintron (Microsoft) in https://github.com/immersive-web/dom-overlays/issues/18. While I think I've resolved the issues raised by them, I wouldn't want to go out on a limb and interpret that as direct support.

slightlyoff@, the currently interested websites/developers include A-Frame and model-viewer which are libraries/frameworks, and we don't currently have the ability to do origin trials with them. Model Viewer is currently implementing a hotspot feature on top of this API. We'd prefer not to wait for this capability given the good feedback we got already.

I'll update the explainer to cover the use cases in more detail.

On Thu, Mar 5, 2020 at 2:13 PM <sligh...@chromium.org> wrote:

Following on from Chris' question, if we are leading (as the browser signals list suggests), is there a reason not to have run an Origin Trial to get good evidence that this is solving an important problem and doesn't have large API shape risks?

As a nit, I'd expect the explainer to include code to show how the proposal addresses the usecases it lists.

Regards

On Thursday, March 5, 2020 at 2:08:36 PM UTC-8, Chris Harrelson wrote:

Hi,

Have you asked representatives from other browser vendors about whether they are supportive of this API or willing to ship this feature?

On Fri, Feb 28, 2020 at 5:25 PM 'Klaus Weidner' via blink-dev <blin...@chromium.org> wrote:

Contact emails kla...@chromium.org,mlam...@chromium.org Explainer https://github.com/immersive-web/dom-overlays Spec https://immersive-web.github.io/dom-overlays/ TAG review https://github.com/w3ctag/design-reviews/issues/470 (in progress) Summary Allows WebXR applications using immersive-ar on handheld devices to optionally activate a DOM overlay mode where the 2D page content is shown as an interactive transparent layer on top of the application-drawn WebGL content and camera image. While the current implementation is limited to ARCore-based handheld devices, the specification is intended to also support implementation on AR headsets. Link to “Intent to Prototype” blink-dev discussion https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/QRbZ0ZUjhmI/kdS3S9OtAgAJ Risks

Interoperability and Compatibility Compatibility risks: this is a new feature requiring explicit opt-in by the application, and won't affect existing WebXR pages that aren't asking for the feature to be activated as part of the XR session request. Interoperability risks: the initial shipped version targets handheld AR on phones, but the API and specification were discussed with other browser vendors, and even though there are no official positions in response to https://github.com/immersive-web/dom-overlays/issues/16 they have been contributing feedback from the point of view of headset implementations, including Magic Leap in https://github.com/immersive-web/dom-overlays/issues/9#issuecomment-579021344. We had discussed a prototype implementation demonstrated on a VR headset at a community group face-to-face meeting, with generally positive feedback from participants. Firefox: No public signals Edge: No public signals Safari: No public signals Web developers: Positive (https://github.com/immersive-web/dom-overlays/issues/16#issuecomment-590517800) Ergonomics The feature will be presented as a part of WebXR Device API. The feature will be most likely used in tandem with WebGL, and is designed to work on top of the existing Fullscreen API. Overall the proposed implementation is very similar to the existing implementation of custom HTML controls for video elements which uses the same hardware composition mechanisms. Activation As this feature builds on the WebXR Device API, which is still under development, progress depends on the continuing maturity and enabling of that API. In addition, this feature also depends on still to be defined AR capabilities of WebXR. That said, developers have shown a willingness to experiment with early AR capabilities. A-Frame users have started experimenting with it, for example https://medium.com/@tessrUX/build-a-3d-augmented-reality-stock-data-visualization-on-the-web-with-javascript-part-1-40f59c3784f9 , and model-viewer is actively working on integrating it to support annotations. Polyfills would be challenging since that would require rendering arbitrary DOM content in WebGL. That works for limited subsets, but is hard to do in general, especially for complex interactions. Security Similar to Fullscreen API, users may be confused about what is going on. The current implementation intentionally follows the example of Fullscreen API. Activation requires user activation together with a pre-existing consent prompt for AR sessions, and fully exits fullscreen mode when the immersive AR session ends.

Debuggability Chrome's Inspector works for the DOM overlay content while in immersive-ar mode, including screencast, clickable elements, and interactive style changes. Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? No This feature relies on AR support in the platform. The initial focus is on Android devices that support ARCore. Is this feature fully tested by web-platform-tests? Yes https://wpt.fyi/results/webxr/dom-overlay?label=experimental&label=master&aligned https://github.com/web-platform-tests/wpt/tree/master/webxr/dom-overlay/ Tracking bug https://bugs.chromium.org/p/chromium/issues/detail?id=991747 Demo links https://klausw.github.io/three.js/examples/webvr_lorenzattractor.html https://klausw.github.io/a-frame-car-sample/index.html https://storage.googleapis.com/chromium-webxr-test/latest.html?target=proposals/index.html Link to entry on the Chrome Platform Status https://www.chromestatus.com/feature/6048666307526656

This intent message was generated by Chrome Platform Status.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Chris Joel]

To offer some details about <model-viewer>'s interest: we need a way for DOM to be layered on top of an "immersive-ar" session. In particular we want this so that our annotations/hotspots, which are defined by our users in terms of DOM and CSS (see https://lopsided-motion.glitch.me/bad-a11y-astronaut.html for an example), can continue to be displayed when our users transition from plain-old-inline rendering to "immersive-ar" rendering. We also intend for users to be able to decorate the "immersive-ar" session screen space with whatever DOM they wish, in case they want to add branding, UI controls or other features. It is not possible to do any of this with WebXR as it exists today, short of adding the capabilities of DOM overlays or something like it.

On Thursday, March 5, 2020 at 4:54:21 PM UTC-8 Klaus Weidner wrote:

Chris, I had asked for feedback in https://github.com/immersive-web/dom-overlays/issues/16, but didn't get specific responses on that issue.

Previously, I had been discussing this with raviramachandra (Magic Leap) in https://github.com/immersive-web/dom-overlays/issues/9#issuecomment-579021344 and RafaelCintron (Microsoft) in https://github.com/immersive-web/dom-overlays/issues/18. While I think I've resolved the issues raised by them, I wouldn't want to go out on a limb and interpret that as direct support.

slightlyoff@, the currently interested websites/developers include A-Frame and model-viewer which are libraries/frameworks, and we don't currently have the ability to do origin trials with them. Model Viewer is currently implementing a hotspot feature on top of this API. We'd prefer not to wait for this capability given the good feedback we got already.

I'll update the explainer to cover the use cases in more detail.

On Thu, Mar 5, 2020 at 2:13 PM <sligh...@chromium.org> wrote:

Following on from Chris' question, if we are leading (as the browser signals list suggests), is there a reason not to have run an Origin Trial to get good evidence that this is solving an important problem and doesn't have large API shape risks?

As a nit, I'd expect the explainer to include code to show how the proposal addresses the usecases it lists.

Regards

On Thursday, March 5, 2020 at 2:08:36 PM UTC-8, Chris Harrelson wrote:

Hi,

Have you asked representatives from other browser vendors about whether they are supportive of this API or willing to ship this feature?

On Fri, Feb 28, 2020 at 5:25 PM 'Klaus Weidner' via blink-dev <blin...@chromium.org> wrote:

Contact emails kla...@chromium.org,mlamouri@chromium.org Explainer https://github.com/immersive-web/dom-overlays Spec https://immersive-web.github.io/dom-overlays/ TAG review https://github.com/w3ctag/design-reviews/issues/470 (in progress) Summary Allows WebXR applications using immersive-ar on handheld devices to optionally activate a DOM overlay mode where the 2D page content is shown as an interactive transparent layer on top of the application-drawn WebGL content and camera image. While the current implementation is limited to ARCore-based handheld devices, the specification is intended to also support implementation on AR headsets. Link to “Intent to Prototype” blink-dev discussion https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/QRbZ0ZUjhmI/kdS3S9OtAgAJ Risks

Interoperability and Compatibility Compatibility risks: this is a new feature requiring explicit opt-in by the application, and won't affect existing WebXR pages that aren't asking for the feature to be activated as part of the XR session request. Interoperability risks: the initial shipped version targets handheld AR on phones, but the API and specification were discussed with other browser vendors, and even though there are no official positions in response to https://github.com/immersive-web/dom-overlays/issues/16 they have been contributing feedback from the point of view of headset implementations, including Magic Leap in https://github.com/immersive-web/dom-overlays/issues/9#issuecomment-579021344. We had discussed a prototype implementation demonstrated on a VR headset at a community group face-to-face meeting, with generally positive feedback from participants. Firefox: No public signals Edge: No public signals Safari: No public signals Web developers: Positive (https://github.com/immersive-web/dom-overlays/issues/16#issuecomment-590517800) Ergonomics The feature will be presented as a part of WebXR Device API. The feature will be most likely used in tandem with WebGL, and is designed to work on top of the existing Fullscreen API. Overall the proposed implementation is very similar to the existing implementation of custom HTML controls for video elements which uses the same hardware composition mechanisms. Activation As this feature builds on the WebXR Device API, which is still under development, progress depends on the continuing maturity and enabling of that API. In addition, this feature also depends on still to be defined AR capabilities of WebXR. That said, developers have shown a willingness to experiment with early AR capabilities. A-Frame users have started experimenting with it, for example https://medium.com/@tessrUX/build-a-3d-augmented-reality-stock-data-visualization-on-the-web-with-javascript-part-1-40f59c3784f9 , and model-viewer is actively working on integrating it to support annotations. Polyfills would be challenging since that would require rendering arbitrary DOM content in WebGL. That works for limited subsets, but is hard to do in general, especially for complex interactions. Security Similar to Fullscreen API, users may be confused about what is going on. The current implementation intentionally follows the example of Fullscreen API. Activation requires user activation together with a pre-existing consent prompt for AR sessions, and fully exits fullscreen mode when the immersive AR session ends.

Debuggability Chrome's Inspector works for the DOM overlay content while in immersive-ar mode, including screencast, clickable elements, and interactive style changes. Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? No This feature relies on AR support in the platform. The initial focus is on Android devices that support ARCore. Is this feature fully tested by web-platform-tests? Yes https://wpt.fyi/results/webxr/dom-overlay?label=experimental&label=master&aligned https://github.com/web-platform-tests/wpt/tree/master/webxr/dom-overlay/ Tracking bug https://bugs.chromium.org/p/chromium/issues/detail?id=991747 Demo links https://klausw.github.io/three.js/examples/webvr_lorenzattractor.html https://klausw.github.io/a-frame-car-sample/index.html https://storage.googleapis.com/chromium-webxr-test/latest.html?target=proposals/index.html Link to entry on the Chrome Platform Status https://www.chromestatus.com/feature/6048666307526656

This intent message was generated by Chrome Platform Status.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Chris Harrelson]

LGTM1

On Sun, Mar 8, 2020 at 2:15 PM 'Chris Joel' via blink-dev <blin...@chromium.org> wrote:

To offer some details about <model-viewer>'s interest: we need a way for DOM to be layered on top of an "immersive-ar" session. In particular we want this so that our annotations/hotspots, which are defined by our users in terms of DOM and CSS (see https://lopsided-motion.glitch.me/bad-a11y-astronaut.html for an example), can continue to be displayed when our users transition from plain-old-inline rendering to "immersive-ar" rendering. We also intend for users to be able to decorate the "immersive-ar" session screen space with whatever DOM they wish, in case they want to add branding, UI controls or other features. It is not possible to do any of this with WebXR as it exists today, short of adding the capabilities of DOM overlays or something like it.

On Thursday, March 5, 2020 at 4:54:21 PM UTC-8 Klaus Weidner wrote:

Chris, I had asked for feedback in https://github.com/immersive-web/dom-overlays/issues/16, but didn't get specific responses on that issue.

Previously, I had been discussing this with raviramachandra (Magic Leap) in https://github.com/immersive-web/dom-overlays/issues/9#issuecomment-579021344 and RafaelCintron (Microsoft) in https://github.com/immersive-web/dom-overlays/issues/18. While I think I've resolved the issues raised by them, I wouldn't want to go out on a limb and interpret that as direct support.

slightlyoff@, the currently interested websites/developers include A-Frame and model-viewer which are libraries/frameworks, and we don't currently have the ability to do origin trials with them. Model Viewer is currently implementing a hotspot feature on top of this API. We'd prefer not to wait for this capability given the good feedback we got already.

I'll update the explainer to cover the use cases in more detail.

On Thu, Mar 5, 2020 at 2:13 PM <sligh...@chromium.org> wrote:

Following on from Chris' question, if we are leading (as the browser signals list suggests), is there a reason not to have run an Origin Trial to get good evidence that this is solving an important problem and doesn't have large API shape risks?

As a nit, I'd expect the explainer to include code to show how the proposal addresses the usecases it lists.

Regards

On Thursday, March 5, 2020 at 2:08:36 PM UTC-8, Chris Harrelson wrote:

Hi,

Have you asked representatives from other browser vendors about whether they are supportive of this API or willing to ship this feature?

On Fri, Feb 28, 2020 at 5:25 PM 'Klaus Weidner' via blink-dev <blin...@chromium.org> wrote:

Contact emails kla...@chromium.org,mlam...@chromium.org Explainer https://github.com/immersive-web/dom-overlays Spec https://immersive-web.github.io/dom-overlays/ TAG review https://github.com/w3ctag/design-reviews/issues/470 (in progress) Summary Allows WebXR applications using immersive-ar on handheld devices to optionally activate a DOM overlay mode where the 2D page content is shown as an interactive transparent layer on top of the application-drawn WebGL content and camera image. While the current implementation is limited to ARCore-based handheld devices, the specification is intended to also support implementation on AR headsets. Link to “Intent to Prototype” blink-dev discussion https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/QRbZ0ZUjhmI/kdS3S9OtAgAJ Risks

Interoperability and Compatibility Compatibility risks: this is a new feature requiring explicit opt-in by the application, and won't affect existing WebXR pages that aren't asking for the feature to be activated as part of the XR session request. Interoperability risks: the initial shipped version targets handheld AR on phones, but the API and specification were discussed with other browser vendors, and even though there are no official positions in response to https://github.com/immersive-web/dom-overlays/issues/16 they have been contributing feedback from the point of view of headset implementations, including Magic Leap in https://github.com/immersive-web/dom-overlays/issues/9#issuecomment-579021344. We had discussed a prototype implementation demonstrated on a VR headset at a community group face-to-face meeting, with generally positive feedback from participants. Firefox: No public signals Edge: No public signals Safari: No public signals Web developers: Positive (https://github.com/immersive-web/dom-overlays/issues/16#issuecomment-590517800) Ergonomics The feature will be presented as a part of WebXR Device API. The feature will be most likely used in tandem with WebGL, and is designed to work on top of the existing Fullscreen API. Overall the proposed implementation is very similar to the existing implementation of custom HTML controls for video elements which uses the same hardware composition mechanisms. Activation As this feature builds on the WebXR Device API, which is still under development, progress depends on the continuing maturity and enabling of that API. In addition, this feature also depends on still to be defined AR capabilities of WebXR. That said, developers have shown a willingness to experiment with early AR capabilities. A-Frame users have started experimenting with it, for example https://medium.com/@tessrUX/build-a-3d-augmented-reality-stock-data-visualization-on-the-web-with-javascript-part-1-40f59c3784f9 , and model-viewer is actively working on integrating it to support annotations. Polyfills would be challenging since that would require rendering arbitrary DOM content in WebGL. That works for limited subsets, but is hard to do in general, especially for complex interactions. Security Similar to Fullscreen API, users may be confused about what is going on. The current implementation intentionally follows the example of Fullscreen API. Activation requires user activation together with a pre-existing consent prompt for AR sessions, and fully exits fullscreen mode when the immersive AR session ends.

Debuggability Chrome's Inspector works for the DOM overlay content while in immersive-ar mode, including screencast, clickable elements, and interactive style changes. Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? No This feature relies on AR support in the platform. The initial focus is on Android devices that support ARCore. Is this feature fully tested by web-platform-tests? Yes https://wpt.fyi/results/webxr/dom-overlay?label=experimental&label=master&aligned https://github.com/web-platform-tests/wpt/tree/master/webxr/dom-overlay/ Tracking bug https://bugs.chromium.org/p/chromium/issues/detail?id=991747 Demo links https://klausw.github.io/three.js/examples/webvr_lorenzattractor.html https://klausw.github.io/a-frame-car-sample/index.html https://storage.googleapis.com/chromium-webxr-test/latest.html?target=proposals/index.html Link to entry on the Chrome Platform Status https://www.chromestatus.com/feature/6048666307526656

This intent message was generated by Chrome Platform Status.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFU2V81aOb-j9mdgCpAus98Hjw8Jz2qA7j5wa73NRaihhMyhkg%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/f4816ac1-c072-4bff-8c1f-db47547b2905%40chromium.org.

[Alex Russell]

Chatted with @Mounir Lamouri about this out of band and requested some updates to the explainers to make it easier for non-XR-insiders to grok what the feature does. Thanks to @Klaus Weidner for making the changes!

LGTM2

Regards

[Yoav Weiss]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CANr5HFW_N2B%3D%2BS2TN%2B2L8wTH5C-89wMhf98X0HPueX78xuPh6w%40mail.gmail.com.