Testing hardware interaction
Chris Jones
tldr; add Litmus tests first, use qemu for android/gonk.
We've been discussing how to test hardware interfaces like battery status, where running exhaustive tests on real HW would be time consuming or difficult, and interfaces like vibrator where the effects of the API aren't visible to gecko. For the purposes of this discussion, there are two platforms: android/gonk and everything else. Most new APIs we're adding are only fully implementable on gonk, but most can be implemented on android, and there are some that can be implemented on several other platforms. So given The Other Platforms, we need a solution that lets us reuse tests there as much as possible.
That's where Litmus comes in. For an API like battery status, we can write tests that show system status (charging/discharging, battery level, critical) and then walk the tester through basic actions ("plug in your device", "unplug it", "press this button to run expensive JS for a few minutes") and check that the intended effect happens. Similarly, for something like vibrator, we can tell the user "we're turning the vibrator on for 2 seconds ... did you feel that?". These are also useful for spot-checking real hardware on which we can run OSes that we control, but on which full regression testing might be hard for average developers.
Note that these wouldn't the replace of testing API corner cases, like passing negative numbers where positive are expected; those should still be mochitests. For checking that sensitive, disabled-by-default APIs like SMS are indeed disabled, we can also use mochitests (mostly, I think).
We'd like to make the bar for landing new device-interaction APIs be these Litmus and API/security tests, initially. Does anyone see a problem with that?
For complete and fully-automated tests, we need some kind of mock devices that gecko can talk to, and that our tests can control. We've been discussing several ways to do this, but it turns out that for android/gonk, most of the problems have already been solved in android's qemu fork. For example, the emulator allows an external program to directly manipulate all the battery state we care about. So we can use a combination of test code that runs in the external test driver and test code in gecko to explore all states. This approach works for devices like vibrator too, because calls to enable the vibrator motor are forwarded to qemu, and we can dump that information to the test driver (this isn't implemented in Gingerbread qemu yet, but it would be trivial to add).
Even cooler, we can use qemu to do black-box testing of telephony and SMS. qemu includes a virtual modem that knows how to establish connections with other running qemu instances. That means that we can spin up two qemu instances, load the same test in both (test.html#foo in one, test.html#bar in the other, say), and have the two instances frob each other. So, for example, one instance could send an SMS message to the other, and the other would verify that the message is received. (The cross-emulator modem interface looks like it was designed more for the emulator UI than for automated testing, but it'll work well enough initially and it's something we can hack on later.)
Writing tests that have bits spread across one or two emulators and the test driver would be icky if done separately. Here's a way we could maybe cut down the ick: write all the code in the same HTML test file, and have the driver extract the parts it needs to run in the driver. For example
<html>
<head>
<!-- Runs in gecko -->
<script type="text/javascript">
window.navigator.vibrate([ 100 ]);
</script>
<!-- Runs in test driver -->
<script type="application/python">
import devices;
assert 100 == devices.vibrator.waitForEnable()
...
</script>
...
Or for the SMS example above,
<html>
<head>
<!-- Runs as "#sender" in one instance, and "#receiver" in the other -->
<script type="text/javascript">
text = "hello";
if ("#sender" == location.hash)
navigator.sms.send(/*magic number*/, text);
else
window.addEventListener("smsreceived",
function(m) { assert(text == m.body); });
...
</script>
...
We need to be careful about proper synchronization here, but it's not rocket science.
Let's get on the same page about our approach and then divvy up the work amongst ourselves.
Cheers,
Chris
John Hammink
although I am coming from a more novice place as developer (completely
new to JS and HTML); I was already of the opinion, at least, that the
best way to proceed was with test html pages which we in QA could
manually bang on until we get around to building some kind of
automation round it. These are the kind of examples I need!