Let's talk about device API testing

[Philipp von Weitershausen]

We've mentioned several times now that we will be requiring tests for
all patches from now on. For a lot of "boring" parts of the code, we
can write unit tests with xpcshell tests or Mochitests. However, some
device capabilities are hard to test that way. Good news is,
Marionette + the emulator allow us to test some of our device APIs in
the context of the full Gonk stack. Folks have been asking for an
overview of what's possible and what isn't, so here it is:

Things supported by the emulator out of the box (see
http://developer.android.com/guide/developing/devices/emulator.html):

* battery (however, chargingTime and dischargingTime not supported)
* rudimentary RIL features such as calls, sms, network registration
state (unclear SIM card features, data calls, etc.)
* geolocation

Not supported. Might be possible by enhancing emulator capabilities
and/or using fake daemons:

* Wifi
* Bluetooth
* sensors (e.g. proximity, accelerometer, etc.)

Fundamentally impossible to automate (needs manual test framework):

* audio: microphone, speakerphone, mute, volume, etc.
* vibration
* camera (host machine's webcam is passed through, so not 100%
impossible, just very hard, especially in a CI environment)

I'm probably missing some APIs here. Please feel free to amend this list.


For infos on how to write emulator tests, please refer to the
Marionette section on MDN. Specifically:

* Marionette JS tests, especially runEmulatorCmd:
https://developer.mozilla.org/en/Marionette/Marionette_JavaScript_Tests#runEmulatorCmd%28%29
* We're in the process of rewriting the mixed Python/JS tests to pure
JS: https://bugzilla.mozilla.org/show_bug.cgi?id=756607

Or drop me a line.

[Chris Jones]

Thanks for hacking on this and writing this up!

----- Original Message -----
> From: "Philipp von Weitershausen" <phil...@googlemail.com>
> To: "Mozilla B2G mailing list" <dev...@lists.mozilla.org>
> Sent: Thursday, May 24, 2012 12:39:47 AM
> Subject: [b2g] Let's talk about device API testing
>
> * sensors (e.g. proximity, accelerometer, etc.)
>

We can test many (most?) of these by sending phony input events to qemu. There aren't nice helpers though; IIRC you have to raw |event send ...| these.

> * vibration

Shian-Yow added code to have qemu dump when the virtual vibrator motor was enabled a while ago. The test harness should be able to read this.

Cheers,
Chris

[Philipp von Weitershausen]

On Thu, May 24, 2012 at 12:59 AM, Chris Jones <cjo...@mozilla.com> wrote:
>> * sensors (e.g. proximity, accelerometer, etc.)
>
> We can test many (most?) of these by sending phony input events to qemu.  There aren't nice helpers though; IIRC you have to raw |event send ...| these.

Oh I didn't realize the "event" command did htat. WHen I poked at it
it seemed mostly about key and touch events. We can write helpers or
at least document the sensor commands. I'll do some digging.

>> * vibration
>
> Shian-Yow added code to have qemu dump when the virtual vibrator motor was enabled a while ago.  The test harness should be able to read this.

Awesome! I'll follow up with him.

[Geo Mealer]

On 2012-05-24 07:39:47 +0000, Philipp von Weitershausen said:
>
> * Wifi
> * Bluetooth
> * sensors (e.g. proximity, accelerometer, etc.)

Re: sensors, orientation looks like it's probably workable.

[telnet to the emulatior]

sensor get orientation
orientation = 0:0:0
OK
sensor set orientation 180:0:0
OK
sensor get orientation
orientation = 180:0:0

Haven't checked yet to see if the events show up in Gecko yet from changing it.

On this subject, I've started a QA Web API planning wiki tree here:

https://wiki.mozilla.org/B2G/QA/WebAPI_Test_Plan

…with Screen Orientation as the first to be implemented this week,
since it's small and (emulator willing) very achievable as a spin-up
section.

https://wiki.mozilla.org/B2G/QA/WebAPI_Test_Plan/Screen_Orientation

I'll be doing some proof of concept stuff tomorrow and will know then
if the orientation feature works in-emulator. If it doesn't, I'll plan
on doing Orientation with an on-device app and move on to planning some
other section that does work with the emulator.

Geo

[Philipp von Weitershausen]

On Wed, May 30, 2012 at 4:38 PM, Geo Mealer <gme...@mozilla.com> wrote:
> On 2012-05-24 07:39:47 +0000, Philipp von Weitershausen said:
>>
>>

>> * Wifi
>> * Bluetooth
>> * sensors (e.g. proximity, accelerometer, etc.)
>
>

> Re: sensors, orientation looks like it's probably workable.
>
> [telnet to the emulatior]
>
> sensor get orientation
> orientation = 0:0:0
> OK
> sensor set orientation 180:0:0
> OK
> sensor get orientation
> orientation = 180:0:0

Nice! Sad that this isn't documented at all on
http://developer.android.com/guide/developing/devices/emulator.html.

> Haven't checked yet to see if the events show up in Gecko yet from changing
> it.

Easiest way to verify is probably to just write a test ;). See

https://mxr.mozilla.org/mozilla-central/source/dom/sms/tests/marionette/test_incoming.js
https://mxr.mozilla.org/mozilla-central/source/dom/battery/test/marionette/test_battery.js
https://mxr.mozilla.org/mozilla-central/source/dom/telephony/test/marionette/

for some examples.