using Firefox accessibility for modal dialogs automation

[Arkady Ioffe]

Hi,

I am rewriting our Firefox add-on to be conformant with web Extension
requirements.
All job done, but I stuck with modal dialogs like alert, confirm, prompt
, login ..
We decided to use Firefox accessibility for their automation.
OS is Windows 10 professional. Current Firefox version is 68ESR.
Cannot record user actions (button click, edit set Text, checkbox
check), as no notification happens, when modal dialog appears despite
of  MouseKeyboard Hook installed and initialized for all Firefox UI threads.

Also on playback I try the following in the assumption that only one
instance of Firefox runs and uses our profile and extension on temporary
base. Preventing easy hacking during production run.

1.  The code related to accessibility has been created as a part of
Native messaging host , written in C++, what provides an easy way to
exchange data between our extension and accessibility parsing. Native
Host runs in a separate process, which parent is a top level Firefox UI
process.
2.  Find top level UI Firefox window by enumerating all desktop top
level windows, until the window matching Criterion found. Here the
Criterion is :
window is visible and belongs to top level Firefox UI process; window
has Window class 'MozillaWindowClass'; window style doesn't include flag
WM_POPUP, but includes caption; a caption doesn't mention words
"console" or "toolbox". Comparison of result with results of such tools
like AccInsight, InspectWin and Spy++ shows that each time we found the
correct window.
3.  Knowing the handle to the top level UI Firefox window, we finding a
dialog by enumeration all windows belonging to the thread of top level
UI Firefox window, until Criterion2 met.
Here the Citerion2 is: window owner is the top level Firefox UI window,
window has window class "MozillaDialogClass" or "#32770". And again
comparison of our result with result of Microsoft tools showed that we
find a dialog window without errors.
4.  Now having a handle to a dialog window we can use MSAA, supported by
Firefox, in order to find accessible object inside a dialog (buttons,
edits, combos, static text , checkbox).
Here the mystery starts too.  I tried to use OBJID_ALERT,
OBJID_NATIVEOM,  OBJID_WINDOW and OBJID_CLIENT as a second argument in call
AccessibleObjectFromWindow(hDlgWin, OBJID, IID_IAccessible,
(void**)&iAcc). First two choices didn't work, while obtaining
IAccessible for OBJID_WINDOW and OBJID_CLIENT work.
Case OBJID_CLIENT returns accessible with no children and a role
SYSTEM_CLIENT. Case OBJID_WINDOW returns accessible object with the role
of SYSTEM_WINDOW.  In the last case I can traverse accessible tree
through IAccessible, IDispatch just to find that it is actually a top
level Firefox UI window object having children like system menu,
toolbar, menu bar, menu and a SYSTEM_CLIENT among  them.
Thus I concluded that for unknown to me reasons
AccessibleObjectFromWindow returns either owner of  modal dialog window
or dialog itself, but not sees inside this object telling that it has no
children and its role is just System_CLIENT. I also tried
AccessibleFromPoint by central point of modal dialog window, but it
returned different IAccessible but still for SYSTEM_CLIENT with no
children.

Obviously I am doing something wrong, as AccInsight and InspectWin have
no troubles to see that modal dialog with the same window handle as a
dialog, and see inside showing accessible edit, text and buttons. I
spent a week trying to find information on Internet , but fails. Could
you give me some help to correct what I am doing wrong. Sure if some
interest I can share code excerpts and answer other questions.

Regards, Arkady

[James Teh]

Hi Arkady,

On Wed, Jul 1, 2020 at 8:23 AM Arkady Ioffe <arkady...@oracle.com> wrote:

> All job done, but I stuck with modal dialogs like alert, confirm, prompt
> , login ..
>

Do you mean modal dialogs created by Firefox or custom modal dialogs
created by your application? The reason for my confusion is that you
mention a login dialog. Firefox does present a login dialog, but only for
HTTP authentication, which is rarely used now. Is that the one you mean?

We decided to use Firefox accessibility for their automation.
> OS is Windows 10 professional. Current Firefox version is 68ESR.
> Cannot record user actions (button click, edit set Text, checkbox
> check), as no notification happens, when modal dialog appears despite
> of MouseKeyboard Hook installed and initialized for all Firefox UI
> threads.
>

Exactly what Win32 hook is this and what notifications are you expecting?
Accessibility communicates events via win events; see SetWinEventHook:
https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setwineventhook

3. Knowing the handle to the top level UI Firefox window, we finding a
> dialog by enumeration all windows belonging to the thread of top level
> UI Firefox window, until Criterion2 met.
> Here the Citerion2 is: window owner is the top level Firefox UI window,
> window has window class "MozillaDialogClass" or "#32770". And again
> comparison of our result with result of Microsoft tools showed that we
> find a dialog window without errors.
>

Modals like alert(), prompt(), etc. don't use separate HWNDs. They use the
main Firefox HWND. Looking at the code, I do see some cases where
MozillaDialogClass is used, but I've actually never seen this; it may be
legacy or just a wrapper window.
#32770 is for system dialogs which are not managed by Firefox.

Finding a Firefox dialog would be pretty tricky. Instead, I'd suggest
watching for EVENT_OBJECT_FOCUS win events (one of which will be fired when
the dialog appears) and looking to see if one of the ancestors has
ROLE_SYSTEM_DIALOG. Alternatively, I guess you could use
AccessibleObjectFromPoint, but you would need to do this on the main
Firefox HWND.

Hope this helps.

jamie