Using Mozmill for testing Firefox addon

[Stanislav Khotinok]

I have a Mozilla Firefox addon and wanna test it. I've found Mozmill and wrote the small Python script, which is just taking each Firefox version and run the command like this:

mozmill --binary=C:\browsers\firefox\38.0.1\firefox.exe 
--addon=C:\my_ext\ext 
--test=C:\my_ext\tests\mozmill\

Here is the script  

unit_test_runner_public.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from subprocess import check_output
import time

firefox_binary_path = 'C:\\browsers\\firefox\\'
ff_addon_location = 'C:\\my_ext\\ext'
tests_folder = 'C:\\my_ext\\tests\\mozmill\\'
ff_versions = ['38.0.1', '37.0.2', '36.0.4', '35.0.1', '34.0.5', '33.1.1',
               '32.0.3', '31.0', '30.0', '29.0.1', '28.0', '27.0.1', '26.0',
               '25.0.1', '24.0', '23.0.1', '22.0', '21.0', '20.0.1', '19.0.2',
               '18.0.2', '17.0.1', '16.0.2', '15.0.1', '14.0.1', '13.0.1']


def build_ff_path(ff_version):
    return "%s%s%s" % (firefox_binary_path, ff_version, "\\firefox.exe")


for item in ff_versions:
    print "##### Started unit tests for Mozilla Firefox %s #####" % item
    current_run = "mozmill --binary=%s --addon=%s --test=%s" % \
        (build_ff_path(item), ff_addon_location, tests_folder)
    test_run_result = check_output(current_run, shell=True)
    print "##### Finished unit tests for Mozilla Firefox %s #####" % item

    time.sleep(10)

So Mozmill is starting the browser, running the tests and then closing the browser and doing it for each Firefox version starting from 38.0.1 to 13.0.1

The problem is, that almost each time it hangs on some random Firefox version. So it opens the browser instance, run the tests, but then it's not closing the browser and the Firefox window hangs for some time and then I see such exception in the terminal:

##### Finished unit tests for Mozilla Firefox 16.0.2 #####
##### Started unit tests for Mozilla Firefox 15.0.1 #####
mozversion INFO | application_buildid: 20120905151427
mozversion INFO | application_changeset: 0b774a1067fe
mozversion INFO | application_display_name: Firefox
mozversion INFO | application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
mozversion INFO | application_name: Firefox
mozversion INFO | application_repository: http://hg.mozilla.org/releases/mozilla
-release
mozversion INFO | application_vendor: Mozilla
mozversion INFO | application_version: 15.0.1
mozversion INFO | platform_buildid: 20120905151427
mozversion INFO | platform_changeset: 0b774a1067fe
mozversion INFO | platform_repository: http://hg.mozilla.org/releases/mozilla-re
lease
mozversion INFO | platform_version: 15.0.1
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\mozmill\__init__.py", line 878, in run
    mozmill.run(tests, self.options.restart)
  File "C:\Python27\lib\site-packages\mozmill\__init__.py", line 473, in run
    self.stop_runner()
  File "C:\Python27\lib\site-packages\mozmill\__init__.py", line 595, in stop_ru
nner
    raise Exception('client process shutdown unsuccessful')
Exception: client process shutdown unsuccessful
Traceback (most recent call last):
  File "unit_test_runner_public.py", line 24, in <module>
    test_run_result = check_output(current_run, shell=True)
  File "C:\Python27\lib\subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'mozmill --binary=C:\browsers\firefox\15.
0.1\firefox.exe --addon=C:\my_ext\ext --test=C:\my_ext\tests\mozmill\' returned 
non-zero exit status 1

C:\my_ext>

And every time it happens with the random version of Firefox, so there is no pattern like issue with some specific Firefox version.

System details are:

• OS: Microsoft Windows 7 Enterprise SP1 x86
• Python: 2.7.9
• Mozmill: 2.0.10

And the output of pip list:

blessings (1.6)
jsbridge (3.0.3)
ManifestDestiny (0.5.7)
manifestparser (1.1)
mozcrash (0.14)
mozdevice (0.45)
mozfile (1.1)
mozinfo (0.7)
mozlog (2.11)
mozmill (2.0.10)
moznetwork (0.24)
mozprocess (0.22)
mozprofile (0.23)
mozrunner (5.35)
mozversion (1.0)
pip (1.5.6)
setuptools (7.0)

Does anyone had experience with such issues?

[Henrik Skupin]

Stanislav Khotinok wrote on 05/26/2015 01:51 PM:

Hi Stanislav,

> I have a Mozilla Firefox addon and wanna test it. I've found Mozmill and
> wrote the small Python script, which is just taking each Firefox version
> and run the command like this:

That is great, but please be aware that mozmill will be discontinued
really soon. This test framework has been replaced by Marionette, and
our specific Firefox related tests are currently getting implemented in
the firefox-ui-tests repository:

https://github.com/mozilla/firefox-ui-tests

So you may think about to continue with your tests in Mozmill, or better
switch over too.

> mozversion INFO | platform_version: 15.0.1
> Traceback (most recent call last):
> File "C:\Python27\lib\site-packages\mozmill\__init__.py", line 878, in run
> mozmill.run(tests, self.options.restart)
> File "C:\Python27\lib\site-packages\mozmill\__init__.py", line 473, in run
> self.stop_runner()
> File "C:\Python27\lib\site-packages\mozmill\__init__.py", line 595, in stop_ru
> nner
> raise Exception('client process shutdown unsuccessful')
> Exception: client process shutdown unsuccessful
> Traceback (most recent call last):
> File "unit_test_runner_public.py", line 24, in <module>
> test_run_result = check_output(current_run, shell=True)
> File "C:\Python27\lib\subprocess.py", line 573, in check_output
> raise CalledProcessError(retcode, cmd, output=output)
> subprocess.CalledProcessError: Command 'mozmill --binary=C:\browsers\firefox\15.
> 0.1\firefox.exe --addon=C:\my_ext\ext --test=C:\my_ext\tests\mozmill\' returned
> non-zero exit status 1

I can remember that we also had failures like those but very rarely and
hard to reproduce. Means we weren't able to fix them yet, and we won't
have the time to get them fixed either. Most likely you won't hit this
problem with Marionette.

Best,

--
Henrik Skupin
Senior Test Engineer
Mozilla Corporation

[Stanislav Khotinok]

Hi Henrik,

thanks a lot for quick reply!

If it's possible - just wanted to clarify few things about https://github.com/mozilla/firefox-ui-tests 

1. Some time ago I've seen something like:

pip install firefox-ui-tests

But as I see now you are suggesting to clone the GitHub repo and build from it. You are not using official PyPi repository anymore?

2. As I see after running 

firefox-ui-tests --help

There is no options to specify --addon= like it was done in case of Mozmill? Doest it have any possibility to specify Firefox plugin? Cause if not, then if I understand correct the only option will be to specify some prebuilt Firefox profile with already injected plugin.

Best Regards,

Stanislav Khotinok

[Henrik Skupin]

Stanislav Khotinok wrote on 05/26/2015 07:14 PM:

Hey,

> 1. Some time ago I've seen something like:
> |
> pip install firefox-ui-tests
> |
>
> But as I see now you are suggesting to clone the GitHub repo and build
> from it. You are not using official PyPi repository anymore?

Managing tests via a pypi package is not really doable. We would have to
release new versions each day, and what's harder for multiple branches
(versions of Firefox). Because of that you should clone the repository
and call "python setup.py install/develop" yourself.

> 2. As I see after running
>
> |
> firefox-ui-tests --help
> |
>
> There is no options to specify --addon= like it was done in case of
> Mozmill? Doest it have any possibility to specify Firefox plugin? Cause
> if not, then if I understand correct the only option will be to specify
> some prebuilt Firefox profile with already injected plugin.

Oh, it could be that this is not available yet. Would you mind to file a
bug in Bugzilla with your specific needs? It will be under Testing /
Marionette. We can then see to get this implemented. It shouldn't be
that hard given that also Marionette uses mozrunner, which supports this
option.

As workaround you could use an already created profile. Otherwise you
can join us on IRC in the #automation channel for a live discussion.