I try to use the CGEventCreateMouseEvent method to click on the element coordinates

[Hank Huang]

public static void main(String[] args) {
         AXUIElementRef app = CallApi.getElementRefByPid(31622);
         CGPoint mouseCursorPosition = app.get_ChildrenElements().get(0).get_CGPoint();
         System.out.println(mouseCursorPosition);

/* print this value

CGPoint(auto-allocated@0x7fd6b7d13c50 (16 bytes)) {

  double x@0x0=535.0

  double y@0x8=196.0

}

*/

          CGEventRef theEvent =             CoreGraphicsFunctions.INSTANCE.CGEventCreateMouseEvent(null, kCGEventMouseMoved, mouseCursorPosition, kCGMouseButtonLeft);
          CoreGraphicsFunctions.INSTANCE.CGEventPost(kCGHIDEventTap, theEvent);

    }
}

I try to use the CGEventCreateMouseEvent method to click on the element coordinates, but it always clicks on the (0,0) position of the upper left corner of my screen.

it's my code  for GCPoint

import com.sun.jna.Structure;

@Structure.FieldOrder({ "x", "y" })
public class CGPoint extends Structure {
/**
* A point with X and Y coordinates
*/
public double x;
public double y;

public int getX() {
return (int) Math.round(this.x);
}

public int getY() {
return (int) Math.round(this.y);
}
}

[Daniel Widdis]

1. (Not related to your problem) While the output indicates sane values for your CGPoint coordinates, the structure contains two CGFloat values, not double values.  This means they can be float values on a 32-bit OS.

2. From examples I've seen elsewhere, the mouseMoved event doesn't actually click the mouse.  You also need to follow up with leftMouseDown and leftMouseUp types.  Passing the mouse button as the last parameter looks like it's ignored for the left and right buttons, it's just a number 2 - 31 for a third (or more) button.

How/when are you detecting the click location?

3. Don't forget to release the EventRef. :) 

[Daniel B. Widdis]

CoreGraphicsFunctions CGF = CoreGraphicsFunctions.INSTANCE;

CGEventRef theEvent = CGF.CGEventCreateMouseEvent(null, kCGEventMouseMoved, mouseCursorPosition, 0);

CGF.CGEventPost(kCGHIDEventTap, theEvent);

CGF.CGEventSetType(theEvent, kCGEventLeftMouseDown);  

CGF.CGEventPost(kCGHIDEventTap, theEvent); // Click!

CGF.CGEventSetType(theEvent, kCGEventLeftMouseUp);  

CGF.CGEventPost(kCGHIDEventTap, theEvent);

CGF.CFRelease(theEvent);

--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/2a7485f0-9677-4d11-a22a-12b35e849cbfn%40googlegroups.com.

--

Dan Widdis

[Hank Huang]

public static void main(String[] args) {

//        AXUIElementRef app = CallApi.getElementRefByPid(39469);
        CoreGraphicsFunctions CGF = CoreGraphicsFunctions.INSTANCE;

        CGPoint mouseCursorPosition  = CGPoint.CGPointMake(900F,334F);
        System.out.println(mouseCursorPosition);

//

CGPoint(auto-allocated@0x7fdf04f13ae0 (8 bytes)) {
  float x@0x0=900.0
  float y@0x4=334.0
}

        CGEventRef theEvent = CoreGraphicsFunctions.INSTANCE.CGEventCreateMouseEvent(null, kCGEventMouseMoved, mouseCursorPosition, kCGMouseButtonLeft);

        CGF.CGEventPost(kCGHIDEventTap, theEvent);
        CGF.CGEventSetType(theEvent, kCGEventLeftMouseDown);
        CGF.CGEventPost(kCGHIDEventTap, theEvent); // Click!
        CGF.CGEventSetType(theEvent, kCGEventLeftMouseUp);
        CGF.CGEventPost(kCGHIDEventTap, theEvent);
        CGF.CFRelease(theEvent);
    }

It still only clicks on the upper left corner, I see that the menu bar in the upper left corner is opened, and the mouse moves to that position at the same time

---

From: jna-...@googlegroups.com <jna-...@googlegroups.com> on behalf of Daniel B. Widdis <wid...@gmail.com>
Sent: Wednesday, July 14, 2021 1:50 AM
To: Java Native Access <jna-...@googlegroups.com>
Subject: Re: I try to use the CGEventCreateMouseEvent method to click on the element coordinates

 

[EXTERNAL]

To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/CANb1Va6_fmk97g6Dgi-M7swBQbbqhoB1Yu5DyXfHj4iujrWPPw%40mail.gmail.com.

[Daniel Widdis]

Beginning in macOS 10.14, security permissions were changed to require the user to explicitly provide permission for apps to use keyboard and mouse events. So:

 - this won't work if your app is sandboxed

 - the first time you try to do this you should get a prompt asking for permission, and/or you can update security and privacy settings to allow it

See https://developer.apple.com/forums/thread/103992?answerId=315370022#315370022

[Daniel Widdis]

Only other suggestion I might try is seeing if you need a source other than null.  Whatever other source is controlling the mouse on the screen may be maintaining that state and losing yours.

Try using CGEventSourceCreate with the constant  kCGEventSourceStateHIDSystemState.

[Hank Huang]

I tried these searches that were correct for me. The same code is a clickable position in swift, so I guess it’s the wrong CGPoint.

[Hank Huang]

        CoreGraphicsFunctions CGF = CoreGraphicsFunctions.INSTANCE;

        CGPoint mouseCursorPosition  = CGPoint.CGPointMake(900F,334F);
        System.out.println(mouseCursorPosition);

        CGEventRef theEvent = CoreGraphicsFunctions.INSTANCE.CGEventCreateMouseEvent(null, kCGEventMouseMoved, null, kCGMouseButtonLeft);

        CGF.CGEventPost(kCGHIDEventTap, theEvent);
        CGF.CGEventSetType(theEvent, kCGEventLeftMouseDown);
        CGF.CGEventPost(kCGHIDEventTap, theEvent); // Click!
        CGF.CGEventSetType(theEvent, kCGEventLeftMouseUp);
        CGF.CGEventPost(kCGHIDEventTap, theEvent);
        CGF.CFRelease(theEvent);

And I found one thing about other things. Even if I can pass Null or other values, he still clicks on the screen. It still proves a problem. It should still be related to the value of CGPoint. I don’t know how to get the correct one. CGPoint value

2021年7月15日 上午2:52，'Hank Huang' via Java Native Access <jna-...@googlegroups.com> 写道：

[EXTERNAL]

I tried these searches that were correct for me. The same code is a clickable position in swift, so I guess it’s the wrong CGPoint.

2021年7月15日 上午1:14，Daniel Widdis <wid...@gmail.com> 写道：

 - this won't work if your app is sandboxed

--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/2A8E5EBA-7DB2-4425-A0AA-9AE66BB8CC77%40ringcentral.com.

[Daniel B. Widdis]

I think your original use of double for CGFloat would have been correct on 64-bit.  I only mentioned it as a caution if you were supporting older 32-bit platforms. 

To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/41987C9C-2CA2-4802-B89A-9D74747E7444%40ringcentral.com.

--

Dan Widdis

[Hank Huang]

I’ m so excited to tell you that I have already resolved this problem. At last, I used NSPoint rather than CGPoint. I ended up referencing this https://developer.apple.com/documentation/foundation/nspoint?language=objc.

2021年7月15日 上午8:34，Daniel B. Widdis <wid...@gmail.com> 写道：

[EXTERNAL]

To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/CANb1Va4ZuT7KN4Kr5t3dxyMw6DHjG%2B6JUvcfDbnTD30w691yKA%40mail.gmail.com.

[Daniel B. Widdis]

Glad you got it working, but the definitions are the same -- both are typedef'd as a struct with two floating-point values, float on 32-bit or double on 64-bit. 

I suspect your original use of double was correct for your 64-bit use case and you changed two things at once, introducing confusion.

To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/90B5FE4E-55BC-4847-B347-A1930D9406BC%40ringcentral.com.

--

Dan Widdis