Need widget/cocoa tests

[Håkan Waara]

We need some way to get testcases into widget/, and in particular
cocoa widgets. I was first about to file this as a bug, but let's see
if we can get some discussion on it first. What is it that we want to
do? What are the obstacles?

To make this easier, I'll try to break down the problem as I see it in
two major steps. (Josh and Steven can probably help me out here.)

We need something that...

1) Can generate events as real (to our system) as if they were
generated by a user
2) Can assess the state of the system after these events have
happened, to see if everything is as expected from the "native" side
of things (first responder, key window, etc).

For #1 we could probably use Quartz Events to synthesize all kinds of
events: clicks, drags, key events. See
http://developer.apple.com/documentation/Carbon/Reference/QuartzEventServicesRef/Reference/reference.html

Shaver suggested that we create an external app that our testing
system can request events from, that would wait a second or two to let
us unwind the event loop before hitting us with them. This should be
pretty easy to do.

For #2: Probably smichaud or someone else who has tried to create
regular testcases for focus (and other tricky-to-reproduce) bugs
should chime in on what is needed, and what is difficult.

Some ideas/thoughts off the top of my head for #2:

We could create helper methods that found out which widget is the
"first responder", "key window" etc to get such Cocoa event concepts
into the testcases. It's already simple to find out the Gecko side of
things ("which DOM element has focus?").

There's a lot of complicated code to make sure we redirect events to
the right nsChildView or nsCocoaWindow, because often the first
responder (the native event receiver) is not the widget (in gecko
terms) that really should handle those events. Cocoa widgets'
interaction with the the rest of gecko, and in which order we need to
pass down events (NS_GOTFOCUS, NS_LOSTFOCUS, etc) is also kind of
black magic.

Maybe we can "externalize" information on how such events travel to
the test environment -- for example, the user clicks Firefox, mouse
event goes to nsChildView A which redirects it to B which passes it
down to DOM object so-and-so. Then we could watch where things ended
up and how. Just an idea.

Ideas, comments? I think what needs to be done first is identify what
is difficult, and then we can get to work trying to address those
problems separated.

[smich...@gmail.com]

I'd add an item 3 to your list, Håkan:

3) Make sure these tests can be automated (from an existing test
harness or perhaps a new one).

Without this, 1) and 2) aren't much use.

(I'll add that 1) and 2) sound reasonable ... but I won't really have
time to look into this until after the Firefox 3 release.)

[Ray Kiddy]

Håkan Waara wrote:
> We need some way to get testcases into widget/, and in particular
> cocoa widgets. I was first about to file this as a bug, but let's see
> if we can get some discussion on it first. What is it that we want to
> do? What are the obstacles?
>
> To make this easier, I'll try to break down the problem as I see it in
> two major steps. (Josh and Steven can probably help me out here.)
>
> We need something that...
>
> 1) Can generate events as real (to our system) as if they were
> generated by a user
> 2) Can assess the state of the system after these events have
> happened, to see if everything is as expected from the "native" side
> of things (first responder, key window, etc).
>

Where this kind of thing has been done, it is using System Events. There
are different ways to generate System Events.

For example, see:
http://developer.apple.com/documentation/AppleScript/Conceptual/AppleScriptX/Concepts/as_related_apps.html

This points to documents on UI scripting and System Events accessible
from AppleScript.

The authors of a paper on fuzz testing (randomized event testing) used
System Events in the Mac OS X tests. The papers are at
http://pages.cs.wisc.edu/~bart/fuzz/fuzz.html.

You might also want to look at something in the newer versions of Xcode.
I cannot remember what the public name of this was, but you can record
actions and play them back for testing purposes.

- ray

[Clint Talbert]

I would tend to agree. I don't completely understand all of the
intricacies of the cocoa event model, but here's one idea I had while
reading this and thinking about option 3 above...

What if you create this "event generator" mechanism, and then write
mochitest-chrome tests to ensure that the events are captured properly
at the proper levels. I'm not sure how tightly the "event generator"
could be coupled with the mochitest-chrome test in order to keep them in
sync, but that might be one way to make these tests use a current test
harness that we already have in production.

Anyhow, just a thought....

Clint