Intent to Ship: WebXR Gamepad Module

[Sam Drazin]

Contact emails

baj...@chromium.org, samd...@chromium.org, mlam...@chromium.org, alco...@chromium.org 

Explainer

https://github.com/immersive-web/webxr-gamepads-module/blob/master/gamepads-module-explainer.md

Spec

We are shipping the Gamepads Module https://immersive-web.github.io/webxr-gamepads-module/ along with the core WebXR Device API (https://immersive-web.github.io/webxr/)

TAG review for WebXR Device API: https://github.com/w3ctag/design-reviews/issues/403

All current feedback has been addressed in the spec, as well as in Chromium.

Summary

The WebXR Gamepad Module defines how the Gamepad API integrates with WebXR to communicate the state of  various physical inputs (such as buttons, touch pads, and triggers) of an XRInputSource.  

Previously the Gamepad API was used in conjunction with the deprecated WebVR API, and was adopted by WebXR with additional ergonomic improvements to support it's input needs. It is being shipped as a module rather than part of the WebXR core API to allow it to evolve more flexibly as the Gamepad API continues to add features.

Link to “Intent to Implement” blink-dev discussion

I2I for WebXR Device API (which contained the Gamepad API at time of this I2I): https://groups.google.com/a/chromium.org/d/topic/blink-dev/XMLgC_SR0dw/discussion

Note: The “Revised WebVR API” (aka WebVR 2.0) referenced in that thread was renamed WebXR Device API several months later.

Link to Origin Trial feedback summary

Find a summary of the WebXR M76-77 origin trial usage and developer feedback here.  There has been little specific feedback on the Gamepads API; usage metrics indicate its usage is mostly contained to Canary (likely experimentation by developers).

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Input from the Gamepad API is supported for immersive sessions only, which Chrome will support for Android and Windows.

Demo link

See the “Controller State” demo from this page: https://immersive-web.github.io/webxr-samples/

Risks

Interoperability and Compatibility

Edge: In development (within Chromium)

Firefox: In development

Safari: No signals

Web / Framework developers: Positive signals.  Oculus, Microsoft, Magic Leap and other hardware OEMs are part of the Working Group, and each represent VR systems with advanced controller inputs that leverage the Gamepad API

Ergonomics

The WebXR Gamepad Module reuses most of the API surface defined by the existing Gamepad API, primarily specifying areas where the API behavior differs due to the integration with the WebXR spec. New developments in the Gamepad API are expected to be adopted for this use case as well, and keeping pace with those developments is part of why this feature is being developed as a module rather than a part of the core WebXR spec.

Activation

The WebXR Polyfill exposes Gamepads objects as described by the WebXR Gamepad Module, ensuring that the inputs are mapped as described on most popular devices. Additionally, developers are already familiar with how the Gamepad API works for traditional controllers, and much of that knowledge applies directly to this use case as well. Finally, WebVR (the deprecated precursor to WebXR) also utilized Gamepads in a similar manner, which helped inform ergonomic improvements to this module. As a result developers that have worked with immersive content on the web previously will already be familiar with this input model.  

Is this feature fully tested by web-platform-tests? Link to test suite results from wpt.fyi.

WebXR Gamepad Module WPTs: https://wpt.fyi/results/webxr?label=master&label=experimental&aligned&q=gamepad

Entry on the feature dashboard

This module is being launched along with the WebXR Device API (whose entry is linked): https://www.chromestatus.com/feature/5659025263820800

[Yoav Weiss]

On Tue, Oct 1, 2019 at 12:52 AM Sam Drazin <samd...@chromium.org> wrote:

Contact emails

baj...@chromium.org, samd...@chromium.org, mlam...@chromium.org, alco...@chromium.org 

Explainer

https://github.com/immersive-web/webxr-gamepads-module/blob/master/gamepads-module-explainer.md

Spec

We are shipping the Gamepads Module https://immersive-web.github.io/webxr-gamepads-module/ along with the core WebXR Device API (https://immersive-web.github.io/webxr/)

TAG review for WebXR Device API: https://github.com/w3ctag/design-reviews/issues/403

Did the TAG review include the Gamepads module as well?

 

All current feedback has been addressed in the spec, as well as in Chromium.

Summary

The WebXR Gamepad Module defines how the Gamepad API integrates with WebXR to communicate the state of  various physical inputs (such as buttons, touch pads, and triggers) of an XRInputSource.  

Previously the Gamepad API was used in conjunction with the deprecated WebVR API, and was adopted by WebXR with additional ergonomic improvements to support it's input needs. It is being shipped as a module rather than part of the WebXR core API to allow it to evolve more flexibly as the Gamepad API continues to add features.

Link to “Intent to Implement” blink-dev discussion

I2I for WebXR Device API (which contained the Gamepad API at time of this I2I): https://groups.google.com/a/chromium.org/d/topic/blink-dev/XMLgC_SR0dw/discussion

Note: The “Revised WebVR API” (aka WebVR 2.0) referenced in that thread was renamed WebXR Device API several months later.

Link to Origin Trial feedback summary

Find a summary of the WebXR M76-77 origin trial usage and developer feedback here.  There has been little specific feedback on the Gamepads API; usage metrics indicate its usage is mostly contained to Canary (likely experimentation by developers).

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Input from the Gamepad API is supported for immersive sessions only, which Chrome will support for Android and Windows.

Demo link

See the “Controller State” demo from this page: https://immersive-web.github.io/webxr-samples/

Risks

Interoperability and Compatibility

Similar to the Device API feedback, do we have a sense of how stable the API is?

 

Edge: In development (within Chromium)

Firefox: In development

Safari: No signals

Web / Framework developers: Positive signals.  Oculus, Microsoft, Magic Leap and other hardware OEMs are part of the Working Group, and each represent VR systems with advanced controller inputs that leverage the Gamepad API

Ergonomics

The WebXR Gamepad Module reuses most of the API surface defined by the existing Gamepad API, primarily specifying areas where the API behavior differs due to the integration with the WebXR spec. New developments in the Gamepad API are expected to be adopted for this use case as well, and keeping pace with those developments is part of why this feature is being developed as a module rather than a part of the core WebXR spec.

Activation

The WebXR Polyfill exposes Gamepads objects as described by the WebXR Gamepad Module, ensuring that the inputs are mapped as described on most popular devices. Additionally, developers are already familiar with how the Gamepad API works for traditional controllers, and much of that knowledge applies directly to this use case as well. Finally, WebVR (the deprecated precursor to WebXR) also utilized Gamepads in a similar manner, which helped inform ergonomic improvements to this module. As a result developers that have worked with immersive content on the web previously will already be familiar with this input model.  

Is this feature fully tested by web-platform-tests? Link to test suite results from wpt.fyi.

WebXR Gamepad Module WPTs: https://wpt.fyi/results/webxr?label=master&label=experimental&aligned&q=gamepad

Entry on the feature dashboard

This module is being launched along with the WebXR Device API (whose entry is linked): https://www.chromestatus.com/feature/5659025263820800

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAD_pDDnGvZ%2BvgU6sovMRKsO3%3D5KzhktuLO7vpXXYi0tZ-5qcLg%40mail.gmail.com.

[rhal...@google.com]

Hi,

Tag review mentions that this module adds additional privacy considerations. Could you please ensure this gets a privacy review before launch?

[Mounir Lamouri]

Hi,

This feature went through a privacy and security review following the Chrome launch process. 

Furthermore, regarding the TAG review, the Gamepad Module was split from the core spec close to the TAG review request of the core spec. The gamepade module is very small: it mostly extends the Gamepad API and the core XR spec and was reviewed at TPAC with the Gamepad editors. It was obviously reviewed by all the Immersive Web group members. 

I think there was a slight confusion and the Chrome team was under the impression that the initial TAG review was covering this module too. I filed a formal request for the module to be reviewed: https://github.com/w3ctag/design-reviews/issues/430

-- Mounir

[Philip Jägenstedt]

Like on the device API, I wonder if anything in https://github.com/immersive-web/webxr-gamepads-module/issues needs to be resolved before shipping? The "potential breaking change" label is eye-catching, and so even if no change is made, making that explicit by closing the issue would be a good idea.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2B-LeH8oshpU5U4z-i6n3WacYNAAhZkqDOkpEazis0c9kBmeqw%40mail.gmail.com.

[Mounir Lamouri]

The only potentially breaking issue for the Gamepad module is  https://github.com/immersive-web/webxr-gamepads-module/issues/18

This issue is that the Gamepad object in the XR spec is live while the Gamepad API requires the object to be a snapshot (title of the issue is incorrect). However, the specification doesn't match reality and the Gamepad objects returned by the Gamepad API are actually not snapshots but a somewhat live objects (gets updated every call to getGamepads()). Microsoft, tried to implement the API per spec and had to change it because they discovered it wasn't compatible with content on the Web.

Therefore, Gamepad API doesn't match reality and we aren't sure whether the issue will resolve in the direction the Gamepad editors asked us to. We believe that we should launch the live version of it and when the Gamepad API decides to normalise UA behaviour on the spec that, we will deprecate the `gamepad` attribute for a `getGamepad()` method (which is the convention for snapshot objects). We believe this is a reasonable compromise.

-- Mounir

[Philip Jägenstedt]

Thanks Mounir, #18 is indeed the issue with the eye-catching "potential breaking change" label. Let's see if I can understand the current state of things.

navigator.getGamepads() doesn't really say if the same Gamepad objects are returned twice, and https://wpt.fyi/results/gamepad/ doesn't seem to have any tests around this, including manual tests. The implementation in Chromium seems to only create the objects once and update those objects in place, but I don't have a gamepad to test that with. https://github.com/w3c/gamepad/issues/8 has been open for a long time about this issue, I've just poked it.

https://immersive-web.github.io/webxr-gamepads-module/#xrinputsource-interface is assuming that a Gamepad object can change internally, so that one can hold on to an instance and check its state every animation frame.

Is there anything blocking resolving https://github.com/w3c/gamepad/issues/8 and https://github.com/immersive-web/webxr-gamepads-module/issues/18 by clearly saying that Gamepad objects are live, and testing it at least with manual tests for both specs?

[Brandon Jones]

Commented on https://github.com/w3c/gamepad/issues/8, but I'll restate some of it here for ease of reference.

I'm not an editor of the Gamepad API any longer, so I can't really comment from the perspective of that API, but I do agree that clarifying that spec to indicate that Gamepads are live seems like the cleanest and easiest way forward for everyone. If they disagree with that direction I think it's pretty important to first ensure that all UA vendors are onboard with making the backwards compatibility-breaking change to update getGamepads() before we make any changes to WebXR to match that direction. I'll admit that I'm somewhat skeptical of that happening at this point.

[Philip Jägenstedt]

+James Hollyer +Matt Reynolds I see that you're now listed as editors in https://w3c.github.io/gamepad/. How would you like to see https://github.com/w3c/gamepad/issues/8 resolved, and do you have preferences for https://github.com/immersive-web/webxr-gamepads-module/issues/18 as well?

[Matt Reynolds]

James and I recently took on editing duties for Gamepad API and are still discussing what to do on this issue.

My current thinking is that the spec should be amended to explicitly make Gamepad a live object, but we'll need to evaluate how this will affect existing apps before making this change. The pseudo-live behavior described above is how Chrome used to work but this is no longer accurate. In the current implementation, Chrome will not modify button or axis state in Gamepad objects that have been returned to script.

If implementing the gamepad property as a live object is useful for WebXR, that sounds good to me as long as we don't stray too far from existing "liveness" implementations.

Firefox treats both Gamepad and GamepadButton as live objects and its implementation will serve as a point of reference if we decide to change the spec. These objects update as soon as a button or axis is changed, no need to call navigator.getGamepads() again. As live objects, it's possible to attach new properties to Gamepad or GamepadButton instances that are preserved when the object is updated. This isn't currently possible in Chrome because new objects are created whenever the state changes, losing any modifications. In WebXR's live gamepad object we should make sure this works:

>> xrpad.buttons[0].myname = "Button 0";
// press and release button 0 to cause the live object to update
>> xrpad.buttons[0].myname;
"Button 0"

We'll keep the Immersive Web team in the loop as we figure out how to normalize UA behavior.

> we will deprecate the `gamepad` attribute for a `getGamepad()` method (which is the convention for snapshot objects).

This SGTM, but hopefully will not be necessary.

- Matt

[Philip Jägenstedt]

Thanks Matt, glad to hear there's a plan to address this. Right now WebXR refers to the Gamepad interface and makes only a non-normative remark about the object being "live", so the normative behavior here is governed by the Gamepad API spec.

Could this issue be prioritized so that we end up shipping something that's matching the spec(s), compatible with existing web content and on a path to becoming interoperable?

The proposal by @Manisheart makes sense to me. To expand on that, any time the underlying gamepad state changes (but not while script is running) the state on the script-exposed objects could also change. The FrozenArray<T> attributes would start returning new frozen arrays. If one wants to hold on to a previous button state, all one would need to do is hold on to the array object, a copy ought not be necessary.

Do you think this would be compatible with existing web content, or what do the constraints look like?

[Matt Reynolds]

Sure, we'll prioritize resolving this issue. We need to collect additional metrics before we can make a decision, currently we don't know to what extent this change would be compatible with existing content.

@Manisheart's suggestion is not fully compatible with the Firefox implementation, I would prefer that WebXR stay as close as possible to their implementation to avoid introducing more interop issues. That would mean the buttons array should not return a new FrozenArray<GamepadButton> when the button state changes, instead updating the GamepadButton objects in place. The axes array is not live and should return a new FrozenArray<double> when the axis state changes.

[Philip Jägenstedt]

Thanks Matt. Matching Firefox as closely as possible both for interop and to reduce compat risk. Hope that works out! I'll link this discussion in https://github.com/w3c/gamepad/issues/8.

Since https://immersive-web.github.io/webxr-gamepads-module/#xrinputsource-interface already assumes live objects and this is all heading in the direction of matching Firefox, I don't think WebXR gamepad support needs to be blocked on what can be observed from the navigator.getGamepads() entrypoint.

LGTM1

[Yoav Weiss]

LGTM2

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYdetFNb-waOVPpZGq7VVoYKRZUTZ2kbWiza_MaegC6YAQ%40mail.gmail.com.

[Chris Harrelson]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACj%3DBEgiiXuc7mFOV0ALksryBrOXMSLTm0ovrBYVedSyOum_wg%40mail.gmail.com.