Marionette-based tests for the UI (Python, for now) -- where should they live, and what should the process be to get them into the repo(s)?

[stephen...@gmail.com]

Hey all -

My team, Web QA [1], is helping out the core Gaia QA team by spearheading Marionette-driven, Python-based smoketests, at least for the foreseeable future (I imagine this might change as testability and unit-test coverage itself, increases).

A couple immediate questions spring to mind, as we begin:

1. James Lal suggested they perhaps would live in apps/{APP_NAME}/test/integration
2. When we get to the point that we'd like to check them in, what is the review process for this not-part-of-build code, assuming it's OK to land it alongside the apps it's testing.

Thoughts?

Thanks,

- Stephen

[1] https://wiki.mozilla.org/QA/Execution/Web_Testing

[Stephen Donner]

Vivien, Chris, can both of you please weigh in on this? I'd like to know the way, going forward.

Thanks!

- Stephen

[Stephen Donner]

Vivien, Chris, can both of you please weigh in on this? I'd like to know the way, going forward.

Thanks!

- Stephen

On Tuesday, October 2, 2012 11:07:24 PM UTC-7, Stephen Donner wrote:

[jones....@gmail.com]

It's generally a good idea to get review on tests, even just light review.

I'm a bit concerned about using python tests for smoke-testing of gaia apps though ... if the tests are too "invasive", i.e. too white-box, then they can slow down code changes instead of speeding them up.

Can you link an example?

Cheers,
Chris

[jones....@gmail.com]

It's generally a good idea to get review on tests, even just light review.

I'm a bit concerned about using python tests for smoke-testing of gaia apps though ... if the tests are too "invasive", i.e. too white-box, then they can slow down code changes instead of speeding them up.

Can you link an example?

Cheers,
Chris

On Friday, October 5, 2012 9:48:47 AM UTC-7, Stephen Donner wrote:

[Stephen Donner]

We don't plan on proposing that these are tree/build-blocking tests (at least, initially); merely, that they help the QA team gain ground on testing, coverage, and easier reporting.

Since these won't be part of the build, and while we're proving the technology/coverage/maintainability, etc., what level of review would you feel is necessary+appropriate, for inclusion alongside the apps they test?

We'd like to get this off the ground quickly, so we can get a clearer sense of the benefit + potential "invasiveness" we'll get with these tests.

Here's an example of a sample test that Jonathan Griffin wrote to help guide Web QA to get started:

https://github.com/mozilla-b2g/gaia/blob/master/tests/marionette/gaiatest/tests/test_calculator.py

(And, while it's important to get coverage, another goal is to help drive the Marionette framework itself.)

- Stephen

[Stephen Donner]

We don't plan on proposing that these are tree/build-blocking tests (at least, initially); merely, that they help the QA team gain ground on testing, coverage, and easier reporting.

Since these won't be part of the build, and while we're proving the technology/coverage/maintainability, etc., what level of review would you feel is necessary+appropriate, for inclusion alongside the apps they test?

We'd like to get this off the ground quickly, so we can get a clearer sense of the benefit + potential "invasiveness" we'll get with these tests.

Here's an example of a sample test that Jonathan Griffin wrote to help guide Web QA to get started:

https://github.com/mozilla-b2g/gaia/blob/master/tests/marionette/gaiatest/tests/test_calculator.py

(And, while it's important to get coverage, another goal is to help drive the Marionette framework itself.)

- Stephen

On Monday, October 8, 2012 8:24:58 PM UTC-7, (unknown) wrote:

[Jonathan Griffin]

We've deliberately designed the tests to be very black-box. There's
only a little JS code used to manage the lock screen and app launching:

https://github.com/mozilla-b2g/gaia/blob/master/tests/marionette/gaiatest/gaia_apps.js
https://github.com/mozilla-b2g/gaia/blob/master/tests/marionette/gaiatest/gaia_test.py#L16

This code is shared in common among tests and so is easy to update
globally when changes are needed.

The actual tests should be very point-and-clicky, e.g.,
https://github.com/mozilla-b2g/gaia/blob/master/tests/marionette/gaiatest/tests/test_calculator.py.

Jonathan