UIAutomation COM via JNA

[Mark Humphreys]

Hi

I am trying to encapsulate the MS UIAutomation library via JNA - this is UIAutomationClient.dll - some Documentation. This is a series complex (to me at least) COM interfaces, and I was wondering whether anyone had any suggestions as to where I should start ? Or any examples of doing the same sort of thing.

I have previously got this working with COM4J, but there are limitations of the extracted interfaces that mean that a lot of the library is unavailable, hence trying it do it all in JNA itself.

Thanks in advance.

Mark H

[L Will Ahonen]

Hi,

We're doing pretty many things via this. For historical reasons, we're not using most of the JNA COM support, and are just doing it the old-and-ugly way, which is reading the vtable out of a com object and then banging those function pointers directly.

Do you have a specific goal you want to achieve? If you're not trying to create good bindings for the library, and only want to get some specific task done, I can hook you up with our ancient-but-works interface definitions for  IUIAutomationElement, ILegacyIAccessible and IUIAutomationElementArray.

Have a good one,

Will

[Mark Humphreys]

I have a library that wraps up the UIAutomationClient library and  uses that to drive a UI in tests. Currently I have the wrapper code and need to plug the actual definitions into the back, as it were. What the actual code looks like to call into the interface isn't really an issue as that isn't going to get surfaced anywhere. I have this morning got some stuff working, and can get at the root IUIAutomationElement, and gets it's name and classname, which is a big step. It looks like a lot of boiler plate 

Thanks for the offer of your bindings, it would be great to see how someone else is approaching this.

[L Will Ahonen]

Hi,

This is not the modern, recommended way, but it works reliably and has been in production for a number of years. Mostly copy-pasted from different points in our code base, there might be some missing imports but I tried to make sure all the batteries are included.

BR,

Will

[Mark Humphreys]

Great, I really appreciate this - I've post back here anything useful that I end up with.

Thanks

[Mark Humphreys]

Not sure if this is the modern, recommended way either but here are the editted highlights. I have partial success.

*  I looked at the examples posted on here for COM Mapping of ITaskbarList3

* Create a handler that descends from Unknown.

public static UIAutomationHandler create() {
    Ole32.INSTANCE.CoInitializeEx(Pointer.NULL, Ole32.COINIT_APARTMENTTHREADED);

    PointerByReference pbr = new PointerByReference();

    WinNT.HRESULT hr = Ole32.INSTANCE.CoCreateInstance(
            CLSID_CUIAutomation,
            null,
            WTypes.CLSCTX_SERVER,
            IID_IUIAutomation,
            pbr);

    COMUtils.checkRC(hr);

    UIAutomationHandler tb = new UIAutomationHandler(pbr.getValue());

    return tb;
}

* I can then write wrappers around each of the COM methods (for example)

 

 public void GetRootElement(PointerByReference elt) {

    int result = this._invokeNativeInt(5, new Object[]{this.getPointer(), elt});
    COMUtils.checkRC(new WinNT.HRESULT(result));
}

* I can see that this works as I have also written wrappers for IAutomationElement, and can get the name and classname of the root element itself.

However, when I try to create a property condition (IUIAutomationAndCondition), which requires 2 other property conditions, then I get an Invalid memory exception.

My call to CreateAndCondition looks like this..

(basic is takes 2 conditions and gives you back a condition based on them)

public void CreateAndCondition(PointerByReference condition0, PointerByReference condition1, PointerByReference condition) {
    int result = this._invokeNativeInt(25, new Object[]{this.getPointer(), condition0, condition1, condition});
    COMUtils.checkRC(new WinNT.HRESULT(result));
}

I suspect that it is something I need to do when I create the input Conditions (wrapper looks like this).

public void CreatePropertyCondition(int propertyId, Variant.VARIANT value, PointerByReference elt) {
    int result = this._invokeNativeInt(23, new Object[]{this.getPointer(), propertyId, value, elt});
    COMUtils.checkRC(new WinNT.HRESULT(result));
}

Thanks, your examples don't look too much different from mine.

Mark H

 

[Mark Humphreys]

Well, I spotted by first mistake, the CreateAndCondition method should look like this:

public void CreateAndCondition(Pointer condition0, Pointer condition1, PointerByReference condition) {

    int result = this._invokeNativeInt(25, new Object[]{this.getPointer(), condition0, condition1, condition});
    COMUtils.checkRC(new WinNT.HRESULT(result));
}

Although it still doesn't help with my actual issue.

On Monday, 27 June 2016 16:04:54 UTC+1, Mark Humphreys wrote:

[L Will Ahonen]

Hi,

Did you notice that the variant is passed as value, not as reference on https://msdn.microsoft.com/en-us/library/windows/desktop/ee671529(v=vs.85).aspx

Does the CreatePropertyCondition go throught without error?

Cheers,

Will

[Mark Humphreys]

It does complete without an error, but I suspect that it is still wrong, hence when it is used in the CreateAndCondition, then it's 'wrongness' causes the issue. I'll have another think about this, and I've rewritten it to be much more like your examples (although still loading via Com), and it still happens, so I am clearly misunderstanding something.

[Mark Humphreys]

OK, I have got this working (or at least working better), I was passing in the values to the CreateAndCondition incorrectly, like this ..

int resultAA = ia.CreateAndCondition(pCondition1.getPointer(), pCondition2.getPointer(), pCondition);

instead it should have been

int resultAA = ia.CreateAndCondition(pCondition1.getValue(), pCondition2.getValue(), pCondition);

I can now get to the findAll call, and get values back. So that is all working as expected now. 

Thanks for help and patience

Mark H

[Elwin Wildschut]

@Mark: Would be nice if you can share more on a full working example. 

Qusetions I have

1. Do you have a fully wrapped UIAutomationClient? any logic rules you applied for the part you did. can you share it?
2. Did you use the windows SDK include files and available IDL files to create the wrapper file or did you use JNAerator to generate (https://en.wikipedia.org/wiki/JNAerator) basic things
   

When I read UIAutomationClient.idl and count I can see an "easy way" to just count and get / generate the stuff based on IDL but no full clue on how the wrapper Java code should look likt

 

[4400...@gmail.com]

Hi, Is there any way to get missing imports?