Finding the coordinates of the browser alerts in Firefox

65 views
Skip to first unread message

Rajapriya Radhakrishnan

unread,
Apr 12, 2022, 9:01:01 AM4/12/22
to Henrik Skupin, dev-we...@mozilla.org, rav...@gmail.com, kana...@gmail.com
Hi Skupin,

I need to find the coordinates of the browser dialog boxes such as alert, confirm and prompt with respect to the window. Could you please check and help me with this. 

NOTE: Attached (firefox_confirm_dialog_box.png) is the image of the test alert page I use now.


With Regards,
Rajapriya R.
firefox_confirm_dialog_box.png

Henrik Skupin

unread,
Apr 19, 2022, 2:39:17 PM4/19/22
to dev-we...@mozilla.org
Rajapriya Radhakrishnan wrote on 12.04.22 15:00:

Hi,

> I need to find the coordinates of the browser dialog boxes such as
> alert, confirm and prompt with respect to the window. Could you please
> check and help me with this. 

Sorry but this isn't supported by the WebDriver Protocol. If you need to
know the coordinates you will have to use some other tools to retrieve
that information.

--
Henrik Skupin
Staff Software Engineer
Mozilla Corporation

Rajapriya Radhakrishnan

unread,
May 31, 2022, 8:17:58 AM5/31/22
to dev-we...@mozilla.org, hsk...@mozilla.com
switching to my new mail id rajapr...@gmail.com because of some log-in issues with my existing email id - rajapri...@gmail.com.

Rajapriya Radhakrishnan

unread,
Jun 1, 2022, 6:23:22 AM6/1/22
to dev-we...@mozilla.org, hsk...@mozilla.com, rav...@gmail.com
Hi Skupin,

Please check the attachments for
1) the overview of the use case with alert action (overview_of_the_use_case_with_alert_action.png)
2) the detailed view of the client system with two users (detailed_view_of_the_client_system_with_two_users)

Overview
User#1 is supposed to do any one of the below actions in order to interact with the Browser#1.
   1. Tap
   2. Double Tap
   3. Scroll Up/Down and Right/Left
   4. Pinch-Zoom in/out

> In parallel with User#1User#2 wants to view the same page (which is currently displayed on the Browser#1) in the Browser#2 without interacting with it. This Browser#2 screen is for viewing purposes only.

So the goal here is, we need to bring the same page content in Browser#2 although there is no user interaction in it.

Solution
1. First, the client application will capture the touch coordinates and action type once the user is done with their action on the browser touch screen.

   Assume, both users get context menu on their screen after a long press. User#1 can select the menu items by tapping it directly. But what about User#2' screen??? The same action must be done here somehow. So, when a user clicks an item from the context menu, the client application will fetch the touch coordinates (abs_x, abs_y) and the action type (say, single tap). 
   
2. Then this info will be sent to the host application network.

3. The host application will receive the data, form the web driver commands and send it to the browser in the following way (consider the above scenario mentioned in #1, action type is single tap on the context menu' items with abs_x, abs_y coordinates):
       3.1. First we need to check whether the click is intended for window elements / context menu items / alerts.
       
               How to pre-validate if the context menu/alert is active?
                   For the context menu, it is possible by reading the state.            
                       with driver1.using_context("chrome"):
                       cm_el = driver1.find_element(By.ID, "contentAreaContextMenu")
                       return cm_el.get_property("state")
           
                   For alerts, by reading the text (you mentioned this point in previous mail).
                       alert = driver1.switch_to_alert()
                       text = alert.text
           
       3.2. If it is on the context menu, we have to convert the window' absolute coordinates to relative coordinates w.r.t context menu origin. Thus, we can target the same location of the browser page where the user is touched on the Browser#1. Then will send the click action like below using the relative coordinates.
           
               As you know, the origin of the context menu can be retrieved by using "WebDriver:GetElementRect" command. By doing the calculation below, we will get the relative coordinates.
                       rel_x = abs_x - cm_x;
                       rel_y = abs_y - cm_y;
                            
                       where,
                                            abs_x, abs_y -  window' absolute coordinates sent by client application.
                                            cm_x, cm_y - origin coordinates of the context menu w.r.t the viewport.
                                            rel_x, rel_y - relative coordinates w.r.t context menu origin.         
       
           
               The pointer events sequence is formed like below,
                       cm_el = driver1.find_element(By.ID, "contentAreaContextMenu")
                       touchHandler.pointer_move(rel_x, rel_y, origin=cm_el).pointer_down(0).pause(1500).pointer_up(0).perform()

           
The above scenario is for the context menu. Now I have to handle the alerts as well. As I mentioned the logic in 3.2, I need a way of getting coordinates of alerts since we cant use the window coordinates directly inside the alert (If I'm wrong here, please correct me). 

Now I hope you are clear on why I'm not using the Alert' methods. I need to either accept/dismiss the alert using the received coordinates (Please note, the host application is unaware of the buttons the received coordinates point to, so I couldn't make use of accept() and dismiss() methods directly.)

If any point is not clear, please let me know. 


Regards,
Rajapriya
detailed_view_of_the_client_system_with_two_users.png
overview_of_the_use_case_with_alert_action.png

Rajapriya Radhakrishnan

unread,
Jun 1, 2022, 6:33:07 AM6/1/22
to dev-we...@mozilla.org, hsk...@mozilla.com, rav...@gmail.com
If the overview of my use case and proposed solution is valid and correct, I will move on to my queries now.

1. Can we send click action inside the alert?? Because, today I tried to click inside the alert after switching to it, action is done but not as expected (will continue this in my next query), and got "UnexpectedAlertOpen" error message. If the test code is wrong, please share with me the correct test code.

Test code

#!/usr/bin/env python3

import time

from marionette_driver.marionette import ActionSequence, Actions, Marionette
from marionette_driver.wait import Wait
from marionette_driver.by import By

driver1 = Marionette()
driver1.start_session()

touchHandler = ActionSequence(driver1, "pointer", "mouse0", {"pointerType": "mouse"})

## the sequence is to click a button so that alert will be shown
touchHandler.pointer_move(580, 240).pointer_down().pointer_up().perform()

## switch to alert
alert = driver1.switch_to_alert()

## reading the text
text = alert.text

## creating another handle to click inside the alert
touchHandler1 = ActionSequence(driver1, "pointer", "mouse0", {"pointerType": "mouse"})
touchHandler1.pointer_move(30, 40).pointer_down().pointer_up.perform()

Error Info
<Command id=5, name=WebDriver:PerformActions, params={'actions': [{'type': 'pointer', 'id': 'mouse0', 'actions': [{'type': 'pointerMove', 'x': 30, 'y': 40}, {'type': 'pointerDown', 'button': 0},  {'type': 'pointerUp', 'button': 0}], 'parameters': {'pointerType': 'mouse'}}]}>
Traceback (most recent call last):
  File "26_31mar2022_alerts.py", line 54, in <module>
    touchHandler1.pointer_move(30, 40).pointer_down().pointer_up().perform()
  File "/home/rar/.local/lib/python3.8/site-packages/marionette_driver/marionette.py", line 75, in perform
    self.marionette.actions.perform([self.dict])
  File "/home/rar/.local/lib/python3.8/site-packages/marionette_driver/marionette.py", line 188, in perform
    return self.marionette._send_message("WebDriver:PerformActions", body)
  File "/home/rar/.local/lib/python3.8/site-packages/marionette_driver/decorators.py", line 26, in _
    return func(*args, **kwargs)
  File "/home/rar/.local/lib/python3.8/site-packages/marionette_driver/marionette.py", line 607, in _send_message
    self._handle_error(err)
  File "/home/rar/.local/lib/python3.8/site-packages/marionette_driver/marionette.py", line 631, in _handle_error
    raise errors.lookup(error)(message, stacktrace=stacktrace)
marionette_driver.errors.UnexpectedAlertOpen: Dismissed user prompt dialog: I am a JS Confirm
stacktrace:
   WebDriverError@chrome://remote/content/shared/webdriver/Errors.jsm:183:5
   UnexpectedAlertOpenError@chrome://remote/content/shared/webdriver/Errors.jsm:488:5
   GeckoDriver.prototype._handleUserPrompts@chrome://remote/content/marionette/driver.js:2537:13
<Command id=6, name=WebDriver:DeleteSession, params=None>

2. After switching to the alert, If I send click action with any random coordinates (tested coordinates are (0,0) and (2000, 2000) and some random inputs), it always targets the first button present in the alert. Even If I pass some out of boundary values, it's not throwing any error like out of boundary, whereas in browser window, it throws an error when I passed some invalid range values. How to avoid it?

3. It's about setting the origin for the pointer_move() method.

python3.8/site-packages/marionette_driver/marionette.py/
class ActionSequence(object):
    ...
    def pointer_move(self, x, y, duration=None, origin=None):
        """Queue a pointerMove action.

        :param x: Destination x-axis coordinate of pointer in CSS pixels.
        :param y: Destination y-axis coordinate of pointer in CSS pixels.
        :param duration: Number of milliseconds over which to distribute the
                         move. If None, remote end defaults to 0.
        :param origin: Origin of coordinates, either "viewport", "pointer" or
                       an Element. If None, remote end defaults to "viewport".
        """

Case-1: For windows, the default origin is viewport (0, 0).
Case-2: For context menu, we query element handle and will set it as origin like below.
               cm_el = driver1.find_element(By.ID, "contentAreaContextMenu")
               touchHandler.pointer_move(rel_x, rel_y, origin=cm_el).pointer_down(0).pause(1500).pointer_up(0).perform()
Case-3: For alert, we don't have any way to get the handle of the alert, so I just simply passed only the (x,y) coordinates for my testing purpose.
               ## creating another handle to click inside the alert
               touchHandler1 = ActionSequence(driver1, "pointer", "mouse0", {"pointerType": "mouse"})
               touchHandler1.pointer_move(30, 40).pointer_down().pointer_up().perform()

               If I don't set any origin explicitly, the viewport will be set by default. Please confirm the viewport is set as origin for alerts too. If so, how the pointer_move() would be handled??

I request you to clarify my queries at the earliest.


Regards,
Rajapriya
Reply all
Reply to author
Forward
0 new messages