Let QA write your Gaia/Marionette UI tests!

[Zac Campbell]

Are you tired? is the cat bothering you for its food? Forgotten to shave
for the last 6 days? Don't have time to write your UI tests?

Then let QA do it for you!

We are a professional automation team of staff and volunteers producing
Python UI tests that run on TBPL, Travis and against Hamachi devices
with nightly builds, all on m-c and b2g26_v1.2 which we review and file
bugs against. Our Hamachi/device tests cover many aspects of hardware
un-reachable by desktopb2g.

We already have 115+ tests running daily and are adding 3-4 new tests
every week.

We're open to writing test cases you need written for desktopb2g or for
Hamachi device. Caveats are that it can run on a 'vanilla' engineering
nightly build, eg without rebuilding Gaia, but switching prefs on and
off is OK! and we'd prefer to focus on core functionality rather than
crazy edge cases. Automation is 'expensive' so we focus on the most
painful/likely to regress parts of functionality.

Contact me with your test cases and I'll try to convert them into tasks
for automation and we'll get them done!

Zac

[Jonas Sicking]

On Nov 14, 2013 3:57 AM, "Zac Campbell" <zcam...@mozilla.com> wrote:
>
> Are you tired? is the cat bothering you for its food? Forgotten to shave
for the last 6 days? Don't have time to write your UI tests?
>
> Then let QA do it for you!

Hahaha, I love it! Zac++

/ Jonas

[Gareth Aye]

I will let you, but only if you use the js tooling so that I can review and
maintain the tests :)

> _______________________________________________
> dev-b2g mailing list
> dev...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-b2g
>



--
Best,
Gareth

[Zac Campbell]

Well I was trying to offer you a QA/automation team to do that for you!

but I am interested in new test cases for new functionality coming up
that QA haven't been let in on yet or if there's coverage that your team
needs for on-device testing to use hardware that desktopb2g doesn't have
access to. We can cover some space that Travis cannot.

Julien has requested some for Messaging app that we did and Rudy also
asked for a keyboard/TBPL test which we did last week.

On 18/11/13 16:58, Gareth Aye wrote:
> I will let you, but only if you use the js tooling so that I can
> review and maintain the tests :)
>
>
> On Sat, Nov 16, 2013 at 3:47 PM, Jonas Sicking <jo...@sicking.cc
> <mailto:jo...@sicking.cc>> wrote:
>
> On Nov 14, 2013 3:57 AM, "Zac Campbell" <zcam...@mozilla.com

> <mailto:zcam...@mozilla.com>> wrote:
> >
> > Are you tired? is the cat bothering you for its food? Forgotten
> to shave
> for the last 6 days? Don't have time to write your UI tests?
> >
> > Then let QA do it for you!
>
> Hahaha, I love it! Zac++
>
> / Jonas
> _______________________________________________
> dev-b2g mailing list

> dev...@lists.mozilla.org <mailto:dev...@lists.mozilla.org>

[Gareth Aye]

Ok well that's all well and good, but I think we should be careful about
how we do this. Will there be separate pull requests to Gaia? I posit that
the correct way to develop is to review and land test code alongside
feature/bug code. Let's prove this to ourselves by exploring other, wrong
ways in which we can do this.

Case 1: Suppose we review/land our tests before the feature/bug code.

- This breaks the build.
- It's possible that we learn things while writing the feature/bug code
about the constraints of the problem and that the feature changes. In this
case, our tests have gotten old and crufty before they ever passed!
- It makes our generous reviewers take two steps out of their busy days
instead of one.

Case 2: Suppose we review/land our tests after the feature/bug code.

- We check untested code into source control. Do we even know if it works?
- If the feature/bug code gets backed out and the test stays, then we break
the build.
- We waste our reviewer's time again making them review two patches.

Our proof is finished since the only options are to land our test code
before, alongside, or after our bug code. Therefore, if we're going to have
people other than feature/bug code writers developing automated test cases,
then everyone who's working on the bug must work together to submit a
single patch and/or pull request to Gaia. This workflow isn't what I've
observed so far with the Python test code, and I hope that we keep to it
for the future health of our project.

Personally this seems like a lot of hassle to me (which is why I prefer to
write my own tests), but I am happy for others to do this so long as they
do it thoughtfully (in a way that doesn't waste reviewer time, stop good
tests from being written, and break the build).

On Tue, Nov 19, 2013 at 3:54 AM, Zac Campbell <zcam...@mozilla.com> wrote:

> Well I was trying to offer you a QA/automation team to do that for you!
>
> but I am interested in new test cases for new functionality coming up that
> QA haven't been let in on yet or if there's coverage that your team needs
> for on-device testing to use hardware that desktopb2g doesn't have access
> to. We can cover some space that Travis cannot.
>
> Julien has requested some for Messaging app that we did and Rudy also
> asked for a keyboard/TBPL test which we did last week.
>
>
>
>
>
>
>
> On 18/11/13 16:58, Gareth Aye wrote:
>
> I will let you, but only if you use the js tooling so that I can review
> and maintain the tests :)
>
>

> On Sat, Nov 16, 2013 at 3:47 PM, Jonas Sicking <jo...@sicking.cc> wrote:

>
>> On Nov 14, 2013 3:57 AM, "Zac Campbell" <zcam...@mozilla.com> wrote:
>> >
>> > Are you tired? is the cat bothering you for its food? Forgotten to shave
>> for the last 6 days? Don't have time to write your UI tests?
>> >
>> > Then let QA do it for you!
>>
>> Hahaha, I love it! Zac++
>>
>> / Jonas
>> _______________________________________________
>> dev-b2g mailing list
>> dev...@lists.mozilla.org

>> https://lists.mozilla.org/listinfo/dev-b2g
>>
>
>
>
> --
> Best,
> Gareth
>
>
>

--
Best,
Gareth

[Zac Campbell]

Well geez i was just offering, kindly I thought, to have QA write some additional test coverage, especially against real device hardware. I thought it would be valuable.

But I can read between the lines; you want Python and QA's tests to sod off.

[Naoki Hirata]

They are valuable. Personally I see them as another tier to testing.

Js catches changes upon checkin; mochi tests capture changes during build time and python tests capture changes after build on real devices. This is before the smoke tests and manual testing so we can better our time on the manual testing. Just because it doesn't happen in the JS level or Mochi test level, it could happen on the actual device because of driver issues.

I think it would be good that people be aware of what we're trying to capture in each of the automation and why they are set up. :)

Note: that I state changes not bugs. Because a change in result of the test doesn't necessarily mean that it's a bug in the change in code; it could mean that it's the test that needs changing.

Regards,
Naoki

[Dale Harvey]

The JS tests can / do run on device though?

Having extra testing is always a good thing, but I think the point Gareth
was making is that we should be looking to avoid having competing suites
which is often duplicating effort.

The developers are using the js tests, they have the same capability
(afaik) as the python suite, it is part of our workflow (and a documented
requirement) that we test functionality we are landing, that will as far as
I know always be done in the js suite.

Therefore is seems that test effort should go into working on the test
suite that is part of the developers workflow, its going to mean the test
suite turns red less and there is far less baggage involved in learning the
dev workflow.

Personally I would have been happy (and tbh probably preferred) using the
existing python infrastructure, however as far as I can tell that ship has
sailed and 'make test-integration' is now burned into gaia devs brains and
theres huge value in having a single canonical test suite (as if we dont
have enough problems with multiple build configurations).

The offer to help is definitely very much appreciated, would be awesome to
get a common workflow agreed on here.

[Gareth Aye]

Zac - I didn't mean to offend! I got into this line of work by doing a lot
of theoretical math, so if I use proof-by-exhaustion to point out an issue
I have with your proposal, it doesn't mean that I don't want your help
making firefox os a quality product.

More test coverage is always good. I also know that b2g developers haven't
always been great about writing tests. For feature/bug code that landed
without tests, late is much better than never.

However, I think I made some valid points about why (for future
work/bugs/patches) we should strive to either have one person write the
feature/bugfix and the tests or have dev/qa work land in the same pull
request/patch.

It's also true (and I think you sensed from my response) that I want
developers to write and maintain tests. I believe this is the only scalable
way forward. That doesn't mean that, for the test coverage we already lack,
we couldn't use some help.

--
Best,
Gareth

[Zac Campbell]

It strikes me that with devs writing tests there's a huge potential for gaming your system of only landing tests with bugs and/or patches.

Why would a dev want to write a rigid and reliable test? It delays the feature/bug fix, it's harder and it might have to be maintained in the future if it fails. We all know it's easy to write a test that's more likely to stay green over time.

The reviewer on the other hand can take the green "OK to merge" of Travis far to literally and not thoroughly review the tests, particularly when under some time pressure.

With a 2nd party writing tests and defining quality there's a balance to the equation, like the opposing sides of the stock market that, in the long term, keep things in equilibrium.

[stephen...@gmail.com]

Zac -

Let's not assume intentions nor receptions to offered testing help -- I (we?) happen to know that both James and Gareth (among quite a few, growing number of others, now) are focused on improving the quality of B2G, across many aspects.

To assume otherwise is to (continue to?) promulgate an "Us (Python) vs. them (JS)" stance, which is neither an accurate, reflective view, nor a useful, productive, one.

Gareth has presented several constructive and thought-provoking questions and thoughts -- we'd do well to respond in a likewise manner. I think they would/do bear good discussion, and have been on many a mind, especially of late.

It's going to take cooperation to learn how to scale appropriately/sustainability -- let's not derail those already-in-place efforts.

- Stephen

Gaia/Gonk/Gecko are complex, in isolation -- and, certainly, moreso in

[Julien Wajsberg]

Le 20/11/2013 02:40, Zac Campbell a écrit :
> It strikes me that with devs writing tests there's a huge potential for gaming your system of only landing tests with bugs and/or patches.
>
> Why would a dev want to write a rigid and reliable test? It delays the feature/bug fix, it's harder and it might have to be maintained in the future if it fails. We all know it's easy to write a test that's more likely to stay green over time.
>
> The reviewer on the other hand can take the green "OK to merge" of Travis far to literally and not thoroughly review the tests, particularly when under some time pressure.

Well, I think we can rely on reliable engineers to write and review tests :)

>
> With a 2nd party writing tests and defining quality there's a balance to the equation, like the opposing sides of the stock market that, in the long term, keep things in equilibrium.
>

I don't really understand why both frameworks are in opposition.

* the JS framework is good for developers to write tests
* the Python framework is good for QA to write tests

When I look at the Gaia repository, I see some apps have JS-based tests,
and that's good. But a lot of apps don't have any (lack of time to write
some or learn, lack of "first test" in the app) and we still need some
integration tests, right? That's where zac and his team are valuable.

In the european timezone they've been very available to help.

As for red trees because a UI Test has not been updated, that should
have been caught at pull request time anyway.

In the end, I'd really prefer everyone to use the JS-based framework. As
everybody said, in the end, both frameworks are similar, and what's
difficult is to write a good UI Test. What I dislike with the
Python-based framework is that you end up writing both JS and Python,
this is not natural. But as long as someone maintains the Python-based
tests I won't say much.

--
Julien

[Gareth Aye]

On Tue, Nov 19, 2013 at 8:40 PM, Zac Campbell <zcam...@mozilla.com> wrote:


> Why would a dev want to write a rigid and reliable test? It delays the
> feature/bug fix, it's harder and it might have to be maintained in the
> future if it fails. We all know it's easy to write a test that's more
> likely to stay green over time.
>

Module owners and peers at Mozilla have *integrity* and put a high premium
on *quality*. I trust the developers with whom I work to do the right thing
when they're writing and reviewing code. I never r+ a patch until I have
read (and understand at least as well as the patch writer) every line in
the diff and my past Gaia reviewers have given my patches the same courtesy.

> The reviewer on the other hand can take the green "OK to merge" of Travis
> far to literally and not thoroughly review the tests, particularly when
> under some time pressure.
>

They could but they don't. If they do, they are not fit to be module
owners/peers.

>
> ----- Original Message -----
> From: Gareth Aye <garet...@gmail.com>
> To: Zac Campbell <zcam...@mozilla.com>
> Cc: Jonas Sicking <jo...@sicking.cc>, dev-...@lists.mozilla.org, dev-b2g
> <dev...@lists.mozilla.org>
> Sent: Tue, 19 Nov 2013 15:34:29 -0800 (PST)
> Subject: Re: [b2g] Let QA write your Gaia/Marionette UI tests!
>

> Zac - I didn't mean to offend! I got into this line of work by doing a lot
> of theoretical math, so if I use proof-by-exhaustion to point out an issue
> I have with your proposal, it doesn't mean that I don't want your help
> making firefox os a quality product.
>
> More test coverage is always good. I also know that b2g developers haven't
> always been great about writing tests. For feature/bug code that landed
> without tests, late is much better than never.
>
> However, I think I made some valid points about why (for future
> work/bugs/patches) we should strive to either have one person write the
> feature/bugfix and the tests or have dev/qa work land in the same pull
> request/patch.
>
> It's also true (and I think you sensed from my response) that I want
> developers to write and maintain tests. I believe this is the only scalable
> way forward. That doesn't mean that, for the test coverage we already lack,
> we couldn't use some help.
>
>

> On Tue, Nov 19, 2013 at 5:50 PM, Zac Campbell <zcam...@mozilla.com>