Accesskeys Mozmill results for Firefox 4 RC1

Henrik Skupin

unread,

Mar 4, 2011, 12:36:47 PM3/4/11

to

Hi,

Here you can find the results of the Mozmill l10n testrun for the RC1
candidate build:

http://mozmill-archive.brasstacks.mozilla.com/#/l10n/reports

http://people.mozilla.com/~hskupin/l10n-results/

--
Henrik

Ehsan Akhgari

unread,

Mar 4, 2011, 4:13:37 PM3/4/11

to Henrik Skupin, dev-...@lists.mozilla.org

I came across a problem in these reports. Take this report for example:
<http://mozmill-archive.brasstacks.mozilla.com/#/l10n/report/aa753950c09ba4cde29deade2791d336>

I really couldn't tell what this warning is about:

jum.assert(false) - accessKey: پ found in string's: [id: (id is
undefined), label: (label is undefined)], [id: (id is undefined), label:
(label is undefined)]

Can you look into it, please?

Thanks!
Ehsan

Henrik Skupin

unread,

Mar 4, 2011, 5:03:50 PM3/4/11

to Ehsan Akhgari, dev-...@lists.mozilla.org

Ehsan Akhgari wrote on 3/4/11 10:13 PM:

> I came across a problem in these reports. Take this report for example:
> <http://mozmill-archive.brasstacks.mozilla.com/#/l10n/report/aa753950c09ba4cde29deade2791d336>
>
> I really couldn't tell what this warning is about:
>
> jum.assert(false) - accessKey: پ found in string's: [id: (id is
> undefined), label: (label is undefined)], [id: (id is undefined), label:
> (label is undefined)]

It looks like it is somehow duplicated with:

jum.assert(false) - accessKey: پ found in string's: [id:

advancedJSButton, label: پیشرفته…], [id: (id is undefined), label:
(label is undefined)], [id: advancedFonts, label: پیشرفته…]

jum.assert(false) - accessKey: پ found in string's: [id: (id is
undefined), label: (label is undefined)], [id: (id is undefined), label:
(label is undefined)]

It's hard for me to tell exactly because I can't read that language and
the letters are hard to distinguish for me. Could you try to fix the
first problem locally and check if that also solves the second assert?
You can use the Mozmill Crowd extension for it if you don't want to deal
with the command line.

--
Henrik

Ehsan Akhgari

unread,

Mar 4, 2011, 6:23:15 PM3/4/11

to Henrik Skupin, dev-...@lists.mozilla.org, Axel Hecht

On 11-03-04 2:03 PM, Henrik Skupin wrote:
> It's hard for me to tell exactly because I can't read that language and
> the letters are hard to distinguish for me. Could you try to fix the
> first problem locally and check if that also solves the second assert?
> You can use the Mozmill Crowd extension for it if you don't want to deal
> with the command line.

I've actually fixed the rest of the accesskey problems, and have pushed
and signed off the revision. Will that cause a new report to be generated?

Ehsan

Axel Hecht

unread,

Mar 4, 2011, 6:56:04 PM3/4/11

to

At this point, Henrik is running the tests on occasion, apparently per
milestone.

Axel

João Miguel Neves

unread,

Mar 4, 2011, 7:04:35 PM3/4/11

to dev-...@lists.mozilla.org

But anyone can install the Mozmill Crowd add-on and run it when wanted.

João Miguel Neves

Henrik Skupin

unread,

Mar 5, 2011, 5:38:19 AM3/5/11

to

Ehsan Akhgari wrote on 3/5/11 12:23 AM:

> I've actually fixed the rest of the accesskey problems, and have pushed
> and signed off the revision. Will that cause a new report to be generated?

We don't have a way yet to run all those tests on a daily basis for
whatever kind of builds we have. Right now I only execute those tests
for candidate builds. I can't trigger a testrun for nightly builds
because that one is blocked on bug 628655. But feel free to use the
Mozmill Crowd extension and execute the l10n tests against your locale
build. You will get the same results, with screenshots. Only don't
forget to enable sending of reports in the preferences.

--
Henrik

Ehsan Akhgari

unread,

Mar 7, 2011, 3:59:23 PM3/7/11

to Henrik Skupin, dev-...@lists.mozilla.org

I gave MozMill Crowd a shot, and it worked beautifully! I just have two
questions:

1. Is there any way to get a formatted test output, perhaps as a result
of the test data that I submitted? The ERROR lines are a pain to read,
because they are very long and contain a json object.

2. Is there any way to correlate the screenshots with the error
messages? I'm not sure which screenshot belongs to which error.

Thanks!
Ehsan

Henrik Skupin

unread,

Mar 8, 2011, 6:15:05 AM3/8/11

to

Ehsan Akhgari wrote on 3/7/11 9:59 PM:

> I gave MozMill Crowd a shot, and it worked beautifully! I just have two
> questions:
>
> 1. Is there any way to get a formatted test output, perhaps as a result
> of the test data that I submitted? The ERROR lines are a pain to read,
> because they are very long and contain a json object.

The result you have sent is available on our mozmill-crowd dashboard.
Right now we don't point the user to that location but it's something I
will add in a later version. Also a better formatted output will be
implemented. For now just check the following report:

http://mozmill-crowd.brasstacks.mozilla.com/#/l10n/report/f179c276497eff82ec93f78b1e39261b

The plain json document you will see here:

http://mozmill-crowd.brasstacks.mozilla.com/db/f179c276497eff82ec93f78b1e39261b

You can format the output with the help of http://jsonviewer.stack.hu/
if necessary

> 2. Is there any way to correlate the screenshots with the error
> messages? I'm not sure which screenshot belongs to which error.

Each jum assert will cause a screenshot to be created. So you will have
to count up. We can improve that in the future.

--
Henrik

Axel Hecht

unread,

Mar 8, 2011, 7:42:15 AM3/8/11

to

I don't think that's true.
http://hg.mozilla.org/qa/mozmill-tests/file/1423d15983c5/shared-modules/localization.js#l101
does an assert per badbox, and then one screenshot per pane.

I think we should invert that in the end. Collect the bad rects and
messages, do the screenshot, then record the messages, either per pane
or per message, hooking the screenshot name to the message.

The question how many messages we should see is really up to
debate/bikeshed. It's one not accessible pane, but there are multiple
issues with it. What's the right count to be able to judge the quality
metric, not sure.

Axel

Ehsan Akhgari

unread,

Mar 8, 2011, 11:47:07 AM3/8/11

to Henrik Skupin, dev-...@lists.mozilla.org

On 11-03-08 6:15 AM, Henrik Skupin wrote:
> Ehsan Akhgari wrote on 3/7/11 9:59 PM:
>
>> I gave MozMill Crowd a shot, and it worked beautifully! I just have two
>> questions:
>>
>> 1. Is there any way to get a formatted test output, perhaps as a result
>> of the test data that I submitted? The ERROR lines are a pain to read,
>> because they are very long and contain a json object.
>
> The result you have sent is available on our mozmill-crowd dashboard.
> Right now we don't point the user to that location but it's something I
> will add in a later version. Also a better formatted output will be
> implemented. For now just check the following report:
>
> http://mozmill-crowd.brasstacks.mozilla.com/#/l10n/report/f179c276497eff82ec93f78b1e39261b
>
> The plain json document you will see here:
>
> http://mozmill-crowd.brasstacks.mozilla.com/db/f179c276497eff82ec93f78b1e39261b
>
> You can format the output with the help of http://jsonviewer.stack.hu/
> if necessary

Thanks. In the interest of learning how to fish, can you please tell me
how you retrieved that URL, so that I can do that on my own next time?

Also, I didn't get any testPrefWindowCroppedElements failures in the
reports that you had generated. Is there anything specific in my setup
which has caused this? Should I treat these as real errors?

Ehsan

Ehsan Akhgari

unread,

Mar 8, 2011, 11:48:25 AM3/8/11

to Axel Hecht, dev-...@lists.mozilla.org

On 11-03-08 7:42 AM, Axel Hecht wrote:
>>> 2. Is there any way to correlate the screenshots with the error
>>> messages? I'm not sure which screenshot belongs to which error.
>>
>> Each jum assert will cause a screenshot to be created. So you will have
>> to count up. We can improve that in the future.
>>
>
> I don't think that's true.
> http://hg.mozilla.org/qa/mozmill-tests/file/1423d15983c5/shared-modules/localization.js#l101
> does an assert per badbox, and then one screenshot per pane.

So, I still don't have any idea which screenshot corresponds to which
error in my case... :(

Ehsan

Henrik Skupin

unread,

Mar 8, 2011, 12:32:45 PM3/8/11

to

Axel Hecht wrote on 3/8/11 1:42 PM:

> I don't think that's true.
> http://hg.mozilla.org/qa/mozmill-tests/file/1423d15983c5/shared-modules/localization.js#l101
> does an assert per badbox, and then one screenshot per pane.

You are right, sorry. We combine that for panes. Can you file a bug for
the stuff you have mentioned?

--
Henrik

Henrik Skupin

unread,

Mar 8, 2011, 12:34:05 PM3/8/11

to

Ehsan Akhgari wrote on 3/8/11 5:48 PM:

> So, I still don't have any idea which screenshot corresponds to which
> error in my case... :(

Those are still in the right order. And the assert messages contain the
id or label of elements you can use to find the exact screenshot.

--
Henrik

Henrik Skupin

unread,

Mar 8, 2011, 12:35:41 PM3/8/11

to

Ehsan Akhgari wrote on 3/8/11 5:47 PM:

> Thanks. In the interest of learning how to fish, can you please tell me
> how you retrieved that URL, so that I can do that on my own next time?

Once a report has been uploaded it will appear on the l10n view:
http://mozmill-crowd.brasstacks.mozilla.com/#/l10n/reports

> Also, I didn't get any testPrefWindowCroppedElements failures in the
> reports that you had generated. Is there anything specific in my setup
> which has caused this? Should I treat these as real errors?

We have disabled those for official test-runs against releases because
those are faulty and produce false positives.

--
Henrik