Mocking API for Bluetooth Devices

[Giovanni]

Hi all,

Writing cross-browser tests for the Web Bluetooth API is difficult because it interacts with devices that live outside the browser. To that extent we are proposing a Test API to accompany the spec:

https://docs.google.com/document/d/1Nhv_oVDCodd1pEH_jj9k8gF4rPGb_84VYaZ9IG8M_WY/edit?usp=sharing

Any browser that implements the Web Bluetooth API could also implement this Test API and reuse all of our tests (which eventually will become Web Platform Tests).

Please have a look,

Gio

[Philip Jägenstedt]

It's awesome that you're trying to tackle this problem of test automation for Bluetooth.

It's a big document and a big API surface, but I see that it all hangs off of navigator.bluetooth.test. Although, none of the interfaces have [NoInterfaceObject], but I assume it still isn't expected that these interface objects would show up on window?

Do you intend to merge this into https://webbluetoothcg.github.io/web-bluetooth/? I think that developing the testing APIs in the main spec would make a lot of sense, and trying to keep it in sync as a separate project doesn't seem like much fun. My own pet idea is to have a testing namespace, and to have all specs but things in a "partial namespace testing", so that there's exactly one object to expose or note expose. But a Web IDL extended attribute like [Testing] could also work.

One key design choice: "The interface will be implemented in C++ and will only be available in content_shell."

I take it this means that the testing code would be behind a compile-time flag and wouldn't even ship in an official Chrome binary. This means that it will not be possible to https://wptdashboard.appspot.com/ that would show results from the actual shipping browsers. But even more importantly, it means that web developers would have no way to write regression tests for their integration with Web Bluetooth. It's tempting to solve these two problems at the same time, but it would come at a cost of binary size. What is your thinking on this?

[Giovanni Ortuño]

Hi Philip,

On Wed, Feb 15, 2017 at 3:28 PM Philip Jägenstedt <foo...@chromium.org> wrote:

It's awesome that you're trying to tackle this problem of test automation for Bluetooth.

It's a big document and a big API surface, but I see that it all hangs off of navigator.bluetooth.test. Although, none of the interfaces have [NoInterfaceObject], but I assume it still isn't expected that these interface objects would show up on window?

Correct, the interface shouldn't appear on window. I added [NoInterfaceObject] to indicate that.

 

Do you intend to merge this into https://webbluetoothcg.github.io/web-bluetooth/? I think that developing the testing APIs in the main spec would make a lot of sense, and trying to keep it in sync as a separate project doesn't seem like much fun. My own pet idea is to have a testing namespace, and to have all specs but things in a "partial namespace testing", so that there's exactly one object to expose or note expose. But a Web IDL extended attribute like [Testing] could also work.

The original plan was to have an accompanying spec that other browsers could implemented if they wanted to reuse our tests. Similar to the outdated on at https://webbluetoothcg.github.io/web-bluetooth/tests.html

That said we could totally make this part of the main spec.

 

One key design choice: "The interface will be implemented in C++ and will only be available in content_shell."

I take it this means that the testing code would be behind a compile-time flag and wouldn't even ship in an official Chrome binary. This means that it will not be possible to https://wptdashboard.appspot.com/ that would show results from the actual shipping browsers. But even more importantly, it means that web developers would have no way to write regression tests for their integration with Web Bluetooth. It's tempting to solve these two problems at the same time, but it would come at a cost of binary size. What is your thinking on this?

I touched upon this in "Alternative: Expose the test interface in regular Chrome":

Advantages:

• Web Platform Tests could directly use the test interface.

• Web Developers could reuse the interface’s implementation to fake devices and test their websites and write regression tests against our code.

• https://wptdashboard.appspot.com/ would show results from the actual shipping browsers for Web Bluetooth

Disadvantages:

• Harder implementation of the interface:

• Test functions no longer available e.g. FlushForTesting.

• Mojo in JS for any website is not yet ready (though Chromecast needs it), so we would need to use WebIDL in C++.

• Requires security review for all IPC changes.

• Increased binary size for a feature that very few will use.

Anyway, I think the decision is up for discussion and we could totally push for it if folks think the benefits outweigh the costs. 

Note that this is just a first step, we are just defining an API and writing one implementation for it. There are other alternatives for the implementation of the API e.g. Web Driver or, as discussed during BlinkOn, WebUSB, that would allow web developers to write regression tests. We are not taking those routes yet because there is no support for them in our test infrastructure. 

 

On Wed, Feb 15, 2017 at 4:12 AM Giovanni <ort...@chromium.org> wrote:

Hi all,

Writing cross-browser tests for the Web Bluetooth API is difficult because it interacts with devices that live outside the browser. To that extent we are proposing a Test API to accompany the spec:

https://docs.google.com/document/d/1Nhv_oVDCodd1pEH_jj9k8gF4rPGb_84VYaZ9IG8M_WY/edit?usp=sharing

Any browser that implements the Web Bluetooth API could also implement this Test API and reuse all of our tests (which eventually will become Web Platform Tests).

Please have a look,

Gio

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYd_TEJ%2BQg-3tLRY_XDyBDD-Ve_RVYK6PU18UkhAYmYZcg%40mail.gmail.com.

[Philip Jägenstedt]

On Thu, Feb 16, 2017 at 11:52 AM Giovanni Ortuño <ort...@chromium.org> wrote:

Hi Philip,

On Wed, Feb 15, 2017 at 3:28 PM Philip Jägenstedt <foo...@chromium.org> wrote:

It's awesome that you're trying to tackle this problem of test automation for Bluetooth.

It's a big document and a big API surface, but I see that it all hangs off of navigator.bluetooth.test. Although, none of the interfaces have [NoInterfaceObject], but I assume it still isn't expected that these interface objects would show up on window?

Correct, the interface shouldn't appear on window. I added [NoInterfaceObject] to indicate that.

 

Do you intend to merge this into https://webbluetoothcg.github.io/web-bluetooth/? I think that developing the testing APIs in the main spec would make a lot of sense, and trying to keep it in sync as a separate project doesn't seem like much fun. My own pet idea is to have a testing namespace, and to have all specs but things in a "partial namespace testing", so that there's exactly one object to expose or note expose. But a Web IDL extended attribute like [Testing] could also work.

The original plan was to have an accompanying spec that other browsers could implemented if they wanted to reuse our tests. Similar to the outdated on at https://webbluetoothcg.github.io/web-bluetooth/tests.html

That said we could totally make this part of the main spec.

I suppose that will depend on the eagerness of other implementers and it's ultimately up to the spec editor's discretion. I'll be interested to see how this develops :)

One key design choice: "The interface will be implemented in C++ and will only be available in content_shell."

I take it this means that the testing code would be behind a compile-time flag and wouldn't even ship in an official Chrome binary. This means that it will not be possible to https://wptdashboard.appspot.com/ that would show results from the actual shipping browsers. But even more importantly, it means that web developers would have no way to write regression tests for their integration with Web Bluetooth. It's tempting to solve these two problems at the same time, but it would come at a cost of binary size. What is your thinking on this?

I touched upon this in "Alternative: Expose the test interface in regular Chrome":

Advantages:

• Web Platform Tests could directly use the test interface.

• Web Developers could reuse the interface’s implementation to fake devices and test their websites and write regression tests against our code.

• https://wptdashboard.appspot.com/ would show results from the actual shipping browsers for Web Bluetooth

Disadvantages:

• Harder implementation of the interface:

• Test functions no longer available e.g. FlushForTesting.

• Mojo in JS for any website is not yet ready (though Chromecast needs it), so we would need to use WebIDL in C++.

• Requires security review for all IPC changes.

• Increased binary size for a feature that very few will use.

Anyway, I think the decision is up for discussion and we could totally push for it if folks think the benefits outweigh the costs. 

Note that this is just a first step, we are just defining an API and writing one implementation for it. There are other alternatives for the implementation of the API e.g. Web Driver or, as discussed during BlinkOn, WebUSB, that would allow web developers to write regression tests. We are not taking those routes yet because there is no support for them in our test infrastructure. 

Ah, thanks, I hadn't seen that doc before.

About using or not using Web IDL, what you have is already specified using Web IDL, do you mean that it will be implemented in some other way? Would the "bindings" code be written by hand in JavaScript and use Mojo in JS?

Regardless, I believe that the main trade-off here is that of binary size if we extrapolate to most new features doing the same thing. We have already accepted some increase in binary size to support WebDriver and DevTools, which are also features that very few will use, but which I think are clearly worth shipping so that people can actually develop and test effectively. Would it be possible for you to make Chromium release builds with and without the code in question to get some estimate of the added binary size?

[Giovanni Ortuño]

Right, that's the initial proposal: Hand written JavaScript that talks to a Mojo service directly. If we wanted to ship the API we would have to implement it in C++, like any other Web API.

 

Regardless, I believe that the main trade-off here is that of binary size if we extrapolate to most new features doing the same thing. We have already accepted some increase in binary size to support WebDriver and DevTools, which are also features that very few will use, but which I think are clearly worth shipping so that people can actually develop and test effectively. Would it be possible for you to make Chromium release builds with and without the code in question to get some estimate of the added binary size?

I made stub implementations of the proposed mojo service and the blink implementation. Below are the results:

args.gn:

symbol_level = 0

is_component_build = false

is_debug = false

enable_nacl = false

is_official_build = true

ninja -C out/Default chrome

Base Chrome: 166,369,992 bytes

Chrome with Mojo service: 166,483,880 bytes (+113.89 Kilobytes)

Chrome With Mojo service + Blink Implementation: 166,602,336 bytes (+232.34 Kilobytes in total, +118.45 Kilobytes vs Chrome with Mojo service)

Gio 

--

You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYeFk5VT5CkKrAuz%2BMpxFrPeOB5Nz8w79rouRLupXzfDjg%40mail.gmail.com.

[Philip Jägenstedt]

Those are pretty discouraging numbers. With either one of them, it extrapolates to at least megabytes and perhaps tens of megabytes of extra code shipping to users.

With the current approach, how will all of that JavaScript be loaded? Would it be possible to ship just the API for talking to Mojo, and then download the other support code when needed somehow? I'm looking for any way for developers to test using this on an official Chrome build that doesn't require us to ship much more code.

I'm also curious as to why IDL bindings are so much fatter. Are they doing the same amount of work as the hand-written JavaScript bindings?

[Giovanni Ortuño]

Hi Philip,

The current proposal assumed we were going to just expose this in content shell for layout tests. So the layout tests would just include the necessary JS to talk to the FakeBluetooth Mojo Interface implemented in the browser.

Here are all the options I could think of for how to ship this in official Chrome:

Option 1: Bluetooth Mojo Service + Web Bluetooth Test JS Impl:

Expose a Bluetooth Mojo Service from device/bluetooth. Implement Web Bluetooth Test in JS on top of a JS implementation of this new Bluetooth Mojo Service.

Binary size increase: 0 bytes. All testing code would live in test sites: Bluetooth Impl + Web Bluetooth Test Impl.

Option 2: Fake Bluetooth Service + Web Bluetooth Test JS Impl:

Expose a FakeBluetooth Mojo Interface to control the device/bluetooth C++ interface and implement it in the browser. Implement Web Bluetooth Test in JS on top of this interface.

Binary size increase: >100 Kilobytes based on http://crrev.com/2722473002. We would ship FakeBluetooth’s impl. Web Bluetooth Test Impl would live in test sites.

Option 3: Bluetooth Mojo Service + Web Bluetooth Test Blink Impl

Expose a Bluetooth Mojo Service from device/bluetooth. Implement Web Bluetooth Test using Blink’s IDL Bindings on top of the Bluetooth Mojo Service.

Binary size increase: >200 Kilobytes based on http://crrev.com/2722473002 + http://crrev.com/2716263002. Assuming a Bluetooth Mojo Service impl is as big as an impl of the current device/bluetooth interface.

Option 4: Fake Bluetooth Service + Web Bluetooth Test Blink Impl

Expose a FakeBluetooth Mojo Interface to control the device/bluetooth C++ interface and implement it in the browser. Implement Web Bluetooth Test using Blink’s IDL Bindings on top of this Mojo interface.

Binary size increase: >200 Kilobytes based on http://crrev.com/2722473002 + http://crrev.com/2716263002. We would ship both Fake Bluetooth Impl and Web Bluetooth Test Impl.

Analysis

Option 1 is very similar to what WebUSB, Geolocation, and probably more APIs, currently do in Layout Tests. It also has no binary size impact since all the test code would live in test websites.

Options 1 and 3 would require us to introduce a new Bluetooth Mojo Service and refactor device/bluetooth clients to use this new service. This is a pretty significant task that would require at least (optimistically speaking) a quarter of work.

Options 1 and 2 would require us to expose overriding Mojo service from JS and interacting with a Mojo Service from JS. These two features would probably be behind a flag as they are very dangerous. (--enable-web-platform-tests?). We will also have to check with Mojo and Blink owners to see if they are OK with this.

Option 4 is the least amount of work but has the biggest binary size impact.

Option 3 has no advantages over Option 4 and is here only for completeness.

Conclusion

In the long term Option 1 is the better of all options. It has no impact on binary size, works similarly to how Layout Tests currently work and could pave the way for future Test APIs. On the other hand, it’s the Option that results in the most work for the Web Bluetooth team as it would require a refactor of device/bluetooth. Option 2 could be a good compromise as it doesn’t increase the binary size as much and results in less work for the Web Bluetooth Team.

What do you think?

Gio

[Rick Byers]

+agrieve@ who is driving APK size efforts.  Any non-trivial increase (and certainly anything on the order of 100kb) to the Chrome APK size just for testing purposes is probably unacceptable.  But obviously anything specific to the content-shell binary should be fine.

--

You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAMDKGBUpHnTZmLFsJNsekNbFKEGxcOpZo1dEenko_6%3D9pPMnFA%40mail.gmail.com.

[Giovanni Ortuño]

I see. Then would it be possible to push our tests to Web Platform Tests even if they only work in content_shell? The alternative of refactoring device/bluetooth or making our own test Bluetooth Dongle would significantly increase the amount of work.

Gio

--

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

[Giovanni Ortuño]

Chatted with Philip offline. He agreed to upstreaming the tests even if the Test API is only available in ContentShell as long as it fails in an obvious way when ran outside of ContentShell.

Thanks!

Gio

[Philip Jägenstedt]

Giovanni and I had another chat today, and my takeaway is that we should do something that builds on the existing mojo mocking, to meet teams where they already are. Specs should define their own testing APIs and web-platform-tests should assume that those APIs exist, and just fail if they don't. Even if specs use IDL for this, we could implement it in JavaScript that put in testharnessreport.js or load from there, having the exact Web IDL semantics of APIs like this right actually isn't important I think.

There's a thread on public-test-infra where we need to sort this out, will follow up there as well.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAMDKGBWsFykcUEmFTOPA5zBCXdjGjZ1Z0K3NZTDcZEfmSB7OGg%40mail.gmail.com.