Integration tests have landed (marionette-js-runner)

[James Lal]

Hey All-

Today we integrated the first version of marionette-js-runner (integration
tests) in gaia.
One important difference from anything we had before is this all runs on
node.js/npm infrastructure.

The really important features you should know about:

- Automatic setup of test environment (it will setup b2g-desktop,
profiles for you, etc..)
- Support for fully synchronous (the default) and asynchronous marionette
apis
- Full support modifying the profile during tests (if you want to test
the lockscreen on/off you can do that)
- Spawn multiple marionette backed hosts (send emails from one email
client to another)
- Really simple CLI interface to run test: `./bin/gaia-marionette
<path/to/test>`
- Fully integrated into travis CI (your tests will run!)
- Docs https://github.com/mozilla-b2g/gaia#integration-tests
- Each test runs inside of its own isolated profile environment (one test
can't affect another)

All this awesomeness is due to the following people (alphabetically):

- Gareth Aye <gaye>
- Ganesh Ghosh <gghosh>
- Jonathan Griffin <jgriffin>
- Michael Henretty <mhenretty>
- Ghislain "Aus" Lacroix <auswerk>
- James Lal <lightsofapollo>
- Mike Pennisi <jugglinmike>

Nothing is perfect on the first (second or third) try some of this work is
a product of a year of on/off hacking
other parts are brand new developed from scratch specifically to solve
problems we found while developing this.
Please reach out to one of these people (hard work is rewarded by more hard
work) for help!

File bugs here: http://mzl.la/15yMUMy

---

Future plans:

- Integrate firefox development environment into integration test suite
(work has already started)
- Faster profile builds (make tests run faster)
- Parallel test running [run 1 test per core]
- More docs / maybe doc site
- announce a official set of module owners/peers responsible for this code

P.S: If you have a build/custom-prefs.js like this:
https://gist.github.com/jugglinmike/db855d476a432b3b4ada you might run into
problems please ensure marionette is not turned off.

[Kevin Grandon]

This is huge. Thanks for the hard work on this! It will be amazing to have these test inside of gaia, and in javascript :)

-Kevin

_______________________________________________
dev-gaia mailing list
dev-...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-gaia

[Kevin Hu]

Hi, James,

This is awesome! Thank you very much!

Best regards,
Kevin Hu

[James Lal]

One person who I didn't mention above is Evan Tseng <evanxd> who
implemented the awesome MultiAction and Action apis for our marionette js
client
http://lightsofapollo.github.io/marionette_js_client/api-docs/classes/Marionette.Actions.html
.

evanxd++

[Yuren Ju]

Awesome! Thank you James and the team work on it!
it could decrease regression bugs if we write a lot integration tests :D

- Yuren

[Ben Kelly]

On 08/01/2013 05:33 PM, James Lal wrote:
> Today we integrated the first version of marionette-js-runner (integration
> tests) in gaia.
> One important difference from anything we had before is this all runs on
> node.js/npm infrastructure.

This is awesome! Thanks to you all for implementing it!

One question, though:

Is there an idiomatic example or template test case for gaia that
developers can use to get started quickly?

It would be great to have something showing best practices for gaia that
could be updated as the infrastructure changes.

Just a thought. Thanks again.

Ben

[Gareth Aye]

Hey Ben,

So we have the docs for the test harness here
https://github.com/mozilla-b2g/marionette-js-runner/blob/master/README.mdwhich
should be useful. Also, more general marionette js client docs live
here
http://lightsofapollo.github.io/marionette_js_client/api-docs/classes/Marionette.Client.htmlthough
I imagine we'll publish them to
mozilla-b2g.github.io soon.

So far we only have one test case in Gaia which landed here
https://github.com/mozilla-b2g/gaia/blob/master/apps/calendar/test/marionette/launch_test.js.

As far as best practices go, I'd say:

- Synchronous calls to the marionette server are easier to read than async
ones

- For anything having to do with installing, managing, launching, switching
between, or otherwise interacting with apps, please use (and contribute
to!) https://github.com/mozilla-b2g/marionette-apps

- If you're finding that you're writing a lot of code that runs in gecko to
do something like power management by talking to web apis, please consider
abstracting your work into a "plugin" so that the rest of B2G can leverage
your work (lightsofapollo, jugglinmike, and gaye would be happy to help
advise and review). See
https://developer.mozilla.org/en-US/docs/Marionette/Marionette_JavaScript_Tools#Marionette_JS_Client_Plugins

- Keep your test cases independent as always so that it's easier for us to
parallelize builds in the future (also to prevent others from getting
headaches when they change your code)

We're thinking also of creating a new module for the marionette js stuff,
so I'm sure there's more to come in terms of documentation and help for
newcomers. Stay tuned!

> ______________________________**_________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/**listinfo/dev-gaia<https://lists.mozilla.org/listinfo/dev-gaia>
>



--
Best,
Gareth

[Gareth Aye]

asuth kindly pointed out that my gmail + lists.mozilla.org issues with
links persisted in my last email, so please forgive this dupe written
with Thunderbird and more whitespace!

Hey Ben,

So we have the docs for the test harness here

https://github.com/mozilla-b2g/marionette-js-runner/blob/master/README.md <https://github.com/mozilla-b2g/marionette-js-runner/blob/master/README.md>

which should be useful. Also, more general marionette js client docs
live here

http://lightsofapollo.github.io/marionette_js_client/api-docs/classes/Marionette.Client.html

<http://lightsofapollo.github.io/marionette_js_client/api-docs/classes/Marionette.Client.html>

though I imagine we'll publish them to mozilla-b2g.github.io

<http://mozilla-b2g.github.io> soon.

So far we only have one test case in Gaia which landed here

https://github.com/mozilla-b2g/gaia/blob/master/apps/calendar/test/marionette/launch_test.js

<https://github.com/mozilla-b2g/gaia/blob/master/apps/calendar/test/marionette/launch_test.js>

As far as best practices go, I'd say:

- Synchronous calls to the marionette server are easier to read than
async ones

- For anything having to do with installing, managing, launching,
switching between, or otherwise interacting with apps, please use (and
contribute to!)

https://github.com/mozilla-b2g/marionette-apps
<https://github.com/mozilla-b2g/marionette-apps>

- If you're finding that you're writing a lot of code that runs in gecko
to do something like power management by talking to web apis, please
consider abstracting your work into a "plugin" so that the rest of B2G
can leverage your work (lightsofapollo, jugglinmike, and gaye would be
happy to help advise and review).

See
https://developer.mozilla.org/en-US/docs/Marionette/Marionette_JavaScript_Tools#Marionette_JS_Client_Plugins
<https://developer.mozilla.org/en-US/docs/Marionette/Marionette_JavaScript_Tools#Marionette_JS_Client_Plugins>


- Keep your test cases independent as always so that it's easier for us
to parallelize builds in the future (also to prevent others from getting
headaches when they change your code)

We're thinking also of creating a new module for the marionette js
stuff, so I'm sure there's more to come in terms of documentation and
help for newcomers. Stay tuned!

> _______________________________________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-gaia

[James Lal]

As far as patterns go I really like the pattern of abstracting away the
specific actions you app needs to do vs the tests that you run on those
actions.
This makes the tests less brittle as we change UX experiences.

The gaia-ui-tests repo has a great example:

https://github.com/mozilla/gaia-ui-tests/blob/master/gaiatest/apps/browser/app.py(app
specific abstractions)
https://github.com/mozilla/gaia-ui-tests/blob/master/gaiatest/tests/browser/test_browser_search.py(the
test)

I plan to write my tests in a similar way to this.

On Fri, Aug 2, 2013 at 10:32 AM, Gareth Aye <garet...@gmail.com> wrote:

> asuth kindly pointed out that my gmail + lists.mozilla.org issues with
> links persisted in my last email, so please forgive this dupe written with
> Thunderbird and more whitespace!
>
>
> Hey Ben,
>
> So we have the docs for the test harness here
>

> https://github.com/mozilla-**b2g/marionette-js-runner/blob/**
> master/README.md<https://github.com/mozilla-b2g/marionette-js-runner/blob/master/README.md><
> https://github.com/mozilla-**b2g/marionette-js-runner/blob/**
> master/README.md<https://github.com/mozilla-b2g/marionette-js-runner/blob/master/README.md>

> >
>
> which should be useful. Also, more general marionette js client docs live
> here
>

> http://lightsofapollo.github.**io/marionette_js_client/api-**
> docs/classes/Marionette.**Client.html<http://lightsofapollo.github.io/marionette_js_client/api-docs/classes/Marionette.Client.html><
> http://lightsofapollo.github.**io/marionette_js_client/api-**
> docs/classes/Marionette.**Client.html<http://lightsofapollo.github.io/marionette_js_client/api-docs/classes/Marionette.Client.html>

> >
>
> though I imagine we'll publish them to mozilla-b2g.github.io <
> http://mozilla-b2g.github.io> soon.
>
>
> So far we only have one test case in Gaia which landed here
>

> https://github.com/mozilla-**b2g/gaia/blob/master/apps/**
> calendar/test/marionette/**launch_test.js<https://github.com/mozilla-b2g/gaia/blob/master/apps/calendar/test/marionette/launch_test.js><
> https://github.com/mozilla-**b2g/gaia/blob/master/apps/**
> calendar/test/marionette/**launch_test.js<https://github.com/mozilla-b2g/gaia/blob/master/apps/calendar/test/marionette/launch_test.js>

> >
>
>
> As far as best practices go, I'd say:
>
> - Synchronous calls to the marionette server are easier to read than async
> ones
>
> - For anything having to do with installing, managing, launching,
> switching between, or otherwise interacting with apps, please use (and
> contribute to!)
>

> https://github.com/mozilla-**b2g/marionette-apps<https://github.com/mozilla-b2g/marionette-apps><
> https://github.com/mozilla-**b2g/marionette-apps<https://github.com/mozilla-b2g/marionette-apps>

> >
>
>
> - If you're finding that you're writing a lot of code that runs in gecko
> to do something like power management by talking to web apis, please
> consider abstracting your work into a "plugin" so that the rest of B2G can
> leverage your work (lightsofapollo, jugglinmike, and gaye would be happy to
> help advise and review).
>

> See https://developer.mozilla.org/**en-US/docs/Marionette/**
> Marionette_JavaScript_Tools#**Marionette_JS_Client_Plugins<https://developer.mozilla.org/en-US/docs/Marionette/Marionette_JavaScript_Tools#Marionette_JS_Client_Plugins><
> https://developer.mozilla.**org/en-US/docs/Marionette/**
> Marionette_JavaScript_Tools#**Marionette_JS_Client_Plugins<https://developer.mozilla.org/en-US/docs/Marionette/Marionette_JavaScript_Tools#Marionette_JS_Client_Plugins>>

>
>
> - Keep your test cases independent as always so that it's easier for us to
> parallelize builds in the future (also to prevent others from getting
> headaches when they change your code)
>
> We're thinking also of creating a new module for the marionette js stuff,
> so I'm sure there's more to come in terms of documentation and help for
> newcomers. Stay tuned!
>
> On 08/02/2013 08:45 AM, Ben Kelly wrote:
>
>> On 08/01/2013 05:33 PM, James Lal wrote:
>>
>>> Today we integrated the first version of marionette-js-runner
>>> (integration
>>> tests) in gaia.
>>> One important difference from anything we had before is this all runs on
>>> node.js/npm infrastructure.
>>>
>>
>> This is awesome! Thanks to you all for implementing it!
>>
>> One question, though:
>>
>> Is there an idiomatic example or template test case for gaia that
>> developers can use to get started quickly?
>>
>> It would be great to have something showing best practices for gaia that
>> could be updated as the infrastructure changes.
>>
>> Just a thought. Thanks again.
>>
>> Ben

>> ______________________________**_________________
>> dev-gaia mailing list
>> dev-...@lists.mozilla.org
>> https://lists.mozilla.org/**listinfo/dev-gaia<https://lists.mozilla.org/listinfo/dev-gaia>
>>
>
> ______________________________**_________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/**listinfo/dev-gaia<https://lists.mozilla.org/listinfo/dev-gaia>
>

[Punam Dahiya]

Hi,
I am trying to run the gaia marionette tests on my local environment connected to B2G device. For this, I am following the instructions here:
https://developer.mozilla.org/en-US/docs/Marionette/Running_Tests#From_Gaia

Specifically, here is my setup
1. I have a unagi device, so setup port forwarding like this: adb forward tcp:2828 tcp:2828

2. Next I tried to run gaia tests by going to gaia directory and doing this: 'make marionette' as suggested in the link, but am running into the following errors:
Finished processing dependencies for marionette-client==0.5.35
starting httpd
running webserver on http://192.168.1.121:54357/
TEST-START browser_a_test_framework.js
TEST-START browser_all_apps.js
TEST-START browser_clock_stopwatch.js
TEST-START browser_clock_timer.js
TEST-START browser_desktop_notifications.js
TEST-START browser_dialer_call_from_contacts.js
TEST-START browser_dialer_contact_intent.js
TEST-START browser_dialer_contacts_edition.js
TEST-START browser_dialer_contacts_search.js
TEST-START browser_dialer_history.js
TEST-START browser_dialer_keypad_and_dial.js
TEST-START browser_dialer_phone_number_sanitizing.js
TEST-START browser_sms_conversation.js
TEST-START browser_sms_new.js
TEST-START test_calculator.py
Traceback (most recent call last):
File "~/b2g/B2G/gecko/testing/marionette/client/marionette/runtests.py", line 795, in <module>
cli()
File "~/b2g/B2G/gecko/testing/marionette/client/marionette/runtests.py", line 790, in cli
runner = startTestRunner(runner_class, options, tests)
File "~/b2g/B2G/gecko/testing/marionette/client/marionette/runtests.py", line 782, in startTestRunner
runner.run_tests(tests)
File "~/b2g/B2G/gecko/testing/marionette/client/marionette/runtests.py", line 427, in run_tests
self.run_test(test)
File "~/b2g/B2G/gecko/testing/marionette/client/marionette/runtests.py", line 476, in run_test
self.run_test(filepath)
File "~/b2g/B2G/gecko/testing/marionette/client/marionette/runtests.py", line 530, in run_test
**self.test_kwargs)
File "~/b2g/B2G/gecko/testing/marionette/client/marionette/marionette_test.py", line 286, in add_tests_to_suite
test_mod = imp.load_source(mod_name, filepath)
File "~/source/gaia/tests/python/gaia-ui-tests/gaiatest/tests/test_calculator.py", line 5, in <module>
from gaiatest import GaiaTestCase
ImportError: No module named gaiatest
make: *** [marionette] Error 1

Any ideas, why the gaia tests are failing to run. I have the latest gaia code and I installed gaiatest manually as well.
/usr/local/bin/gaiatest

Next, I ran marionette tests of gecko, doing this: ./test.sh marionette --address=localhost:2828
This at least executes the test suite, there are some failures and some successes.
SUMMARY
-------
passed: 64
failed: 115
todo: 0


Just to see, if I can run the gaia tests from a gecko marionette client directly, I did try to run a gaia calendar test from '~/b2g/B2G/gecko/testing/marionette/client/marionette' directory:

Punams-MacBook-Pro:marionette $ python runtests.py --address localhost:2828 ~/source/gaia/apps/calendar/test/marionette/launch_test.js
starting httpd
running webserver on http://192.168.1.121:53539/
TEST-START launch_test.js

SUMMARY
-------
passed: 0
failed: 0
todo: 0



But that doesn't seem to help..

Thanks
Punam

----- Original Message -----
From: "James Lal" <ja...@lightsofapollo.com>
To: "Gareth Aye" <garet...@gmail.com>
Cc: dev-...@lists.mozilla.org
Sent: Friday, August 2, 2013 10:49:51 AM
Subject: Re: Integration tests have landed (marionette-js-runner)

_______________________________________________
dev-gaia mailing list
dev-...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-gaia

[Jonathan Griffin]

There are several different kinds of tests; the "launch_test.js" file
you were trying to run is a gaia unit test
(https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/Testing/Gaia_unit_tests),
not a Marionette test.

All of the browser_.js tests you are trying to run are old
mochitest-style tests; none of them work currently, and are not supported.

There are a set of Gaia UI tests at
https://github.com/mozilla/gaia-ui-tests; you can see instructions for
running them there. See also
https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/Testing/gaia-ui-tests/Gaia_UI_Tests_Installation.

James Lal and others on the Gaia team are working on a JS client for
Marionette that supports writing Gaia tests in JS; see
https://developer.mozilla.org/en-US/docs/Marionette/Marionette_JavaScript_Tools.

Jonathan

> _______________________________________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-gaia
> _______________________________________________
> dev-gaia mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-gaia

[Julien Wajsberg]

Le 24/08/2013 23:32, Jonathan Griffin a écrit :
> There are several different kinds of tests; the "launch_test.js" file
> you were trying to run is a gaia unit test
> (https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/Testing/Gaia_unit_tests),
> not a Marionette test.
>
> All of the browser_.js tests you are trying to run are old
> mochitest-style tests; none of them work currently, and are not
> supported.
>
> There are a set of Gaia UI tests at
> https://github.com/mozilla/gaia-ui-tests; you can see instructions for
> running them there. See also
> https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/Testing/gaia-ui-tests/Gaia_UI_Tests_Installation.
>
> James Lal and others on the Gaia team are working on a JS client for
> Marionette that supports writing Gaia tests in JS; see
> https://developer.mozilla.org/en-US/docs/Marionette/Marionette_JavaScript_Tools.

The launch_test.js file is actually using this new framework, it's not a
unit test.
hint: it's in the /test/marionette subdirectory, wheras unit test are in
/test/unit ;)

Therefore you can probably run this using the information on that page.
Don't hesitate to add things to the MDN page if necessary or not
accurate or not precise enough, this is a Wiki :)

--
Julien

[gio...@gmail.com]

Hi,

I want to run the simplest marionette js example in firefox, but I am getting an error. Here are all the details:

I have put --host marionette-firefox-host in marionette-mocha.opts and here's my code:

marionette('github.com', function() {
var github = 'http://github.com';

var client = marionette.client();

setup(function() {

client.goUrl(github);
});

test('logging into github', function() {
var tests = client.findElement('#iliketests');
assert.equal(tests.text(), 'false');
});

});

I am getting following error:

0 passing (29ms)
2 failing

1) github.com "before each" hook:


+ expected - actual


rror: .product must be given

2) github.com "after each" hook:

Uncaught
+ expected - actual


ypeError: Cannot read property 'send' of undefined

I have installed following:

npm install --save-dev marionette-client marionette-js-runner marionette-firefox-host

And the way I invoke my test is like this:

./node_modules/.bin/marionette-mocha test/demo_test.js


Could you help me figure out what's wrong here?

Thanks,
Panagiota