Re: [jni4net] Generics (again) and workarounds

[Pavel Šavara]

Hi Mihail,

I did some prototype of generic proxies about 2 yeas ago. There are lot of issues with the API it has to generate. And Java type erasure makes it even more challenging.

That said, it's almost impossible to fix current proxygen to generate any such code.

I started working on rewrite of proxygen in branch 0.9. It's very much work in progress. There are no generics either, but at least, the code is clean enough, so it could be extended in the future.

I'm not sure when I return my attention to that, I'm busy with daily job and family.

Pavel

On Tue, Nov 6, 2012 at 8:03 AM, Mihai Cadariu <mit...@gmail.com> wrote:

Hi Pavel,

First off, great work! Thank you so much. jni4net and proxygen are simply so cool!

I don't want to beat a dead horse, but my question is about generics... again. I'm exposing a .net library (500+ classes) to java and some classes do use generics. I read the posts before and I know they are not supported and there are some possible workarounds (adapters, wrappers, facades, etc).

However, given that I have 500+ classes, I wouldn't want to create an adapter for each! Also having some classes in the original namespace and some in my own wrapper doesn't sound right or nice.

Currently the Java code calling a .net method that returns an IDictionary<string, string> yields the following exception:

Exception in thread "main" System.ArgumentNullException: Value cannot be null.

Parameter name: key

   at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)

   at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)

   at net.sf.jni4net.utils.Registry.ResolveNew(Type type) in C:\Customers\jni4net\jni4net.n\src\utils\Registry.resolve.cs:line 101

   at net.sf.jni4net.utils.Registry.GetCLRRecord(Type iface) in C:\Customers\jni4net\jni4net.n\src\utils\Registry.cs:line 80

   at net.sf.jni4net.utils.Convertor.FullC2J[TBoth](JNIEnv env, TBoth obj) in C:\Customers\jni4net\jni4net.n\src\utils\Convertor.C2J.cs:line 98

Debugging the code, I found that Registry.resolve.cs is trying to lookup the proxy class for the IDictionary<string, string> and it tries to lookup "IDictionary`2" type from knownCLR. This fails and blows up with the exception above. The cause is that I don't have a IDictionary`2 proxy.

So my question is, how can I get a proxy for IDictionary`2? Manually is also fine (I'm still new to JNI, but I'm digging in very quickly). Is this even an option? Or simply jni4net can't handle the generic proxy class?

Thank you!

--
You received this message because you are subscribed to jni...@googlegroups.com
http://groups.google.com/group/jni4net?hl=en-GB?hl=en-GB
http://jni4net.sf.net/

[Mihai Cadariu]

I see what you mean. Busy life, I know ;)

My current approach I'm about to test is to actually modify (by hand so far) the proxy method signatures generated by proxygen so that they don't actually use/return any generics. Instead I'm attempting to pass to them non-generic variants and return non-generics. This means that during the actual proxy method call to the 'real' object I will have to do some converting myself.

For example, if my return in the proxy method is something like this:

@__return = global::net.sf.jni4net.utils.Convertor.FullC2J<global::System.Collections.Generic.IList<MyObject>>(@__env, @__real.GetMyObject());

then I would modify it to non-generic, to look like this (pseudo-code for now):

global::SystemCollections.IList theObject = MyConvertor.GenericToIList(@__real.GetMyObject());

@__return = global::net.sf.jni4net.utils.Convertor.FullC2J<global::System.Collections.IList>(@__env, theObject);

where MyConvertor.GenericToIList(IList<MyObject>) is some method  I would write.

Let's see what gives.

Mihai

[Mihai Cadariu]

I did it... and it works!

Modified generated CS to look like this:

private static global::net.sf.jni4net.utils.JniHandle MyPropertiesList10(global::System.IntPtr @__envp, global::net.sf.jni4net.utils.JniLocalHandle @__obj) {

    global::net.sf.jni4net.jni.JNIEnv @__env = global::net.sf.jni4net.jni.JNIEnv.Wrap(@__envp);

    global::net.sf.jni4net.utils.JniHandle @__return = default(global::net.sf.jni4net.utils.JniHandle);

    try {

        global::MyExample @__real = global::net.sf.jni4net.utils.Convertor.StrongJp2C<global::MyExample>(@__env, @__obj);

        global::System.Collections.IList myPropertiesList = new global::System.Collections.ArrayList((global::System.Collections.Generic.List<SomeObject>)@__real.MyPropertiesList);

        @__return = global::net.sf.jni4net.utils.Convertor.FullC2J<global::System.Collections.IList>(@__env, myPropertiesList);

    } catch (global::System.Exception __ex) {

        @__env.ThrowExisting(__ex);

    }

    return @__return;

}

Then modified the java generated signature to:

@net.sf.jni4net.attributes.ClrMethod("()[[LSystem/Collections/IList;")

public native system.collections.IList getMyPropertiesList();

Finally in my java client I can make the calls like this:

MyExample example = new MyExample();

system.collections.IList list = example.getMyPropertiesList();

System.out.println("Count: " + list.getCount());

for (int i = 0; i < list.getCount(); i++) {

    SomeObject so = (SomeObject) list.getItem(i);

    System.out.println("Some Object: " + so);

}

for (IEnumerator enumerator = list.GetEnumerator(); enumerator.MoveNext();) {

    SomeObject cp = (SomeObject) enumerator.getCurrent();

    System.out.println("Enumerated: " + so);

}

Now I need to see how easy is to apply this patch to proxygen (if at all possible) or otherwise I will just patch the proxies manually.

Also on Java side (in the calling client code) it would be nice to have real java collections. I guess that's not a big deal, could write some convertors for it (maybe they are already there in jni4net? didn't check).

Mihai

[Pavel Šavara]

The fact that List<T> implements IList<T> and IList at same time is helping you now. But there could be another class which implements IList<T> and doesn't implement IList.

I was looking for general solution to the problem, but maybe a partial solution would be better than no solution ....  ;-)

Ideally you should be able to implement IList<T> on java side and pass it back to C#

That means that proxygen must be able to fully implement IList<T> proxy.

Also finally how you plan to construct instance of List<Something> from Java side?

Something like new List<Something>(Something.class) would be necessary and special handling of the constructor.

Also presenting Java generics in both forms (generic and 1.4 type-erased) as 2 different classes in C#, which are implicitly cast to each other is challenging. 

Mostly for interfaces, which don't have cast operator in C#.

And finally generic constraints, covariance & contravariance are just nice to have I guess.

Back to your simplified use case: if you are just shuffling C# instances on Java side, maybe you could hack the 0.8 proxygen reflection/substitution layer for the specific cases of IList and IDictionary.

Keep us posted

Cheers

Pavel

Mihai

[Mihai Cadariu]

I did some work in the past week and I now have a strong (yet partial) implementation of generics with ProxyGen. At least it suits my needs with the 500+ class library that I converted to proxies. It's still partial support because as you said it is not solving the general case. Instead it provides generics for the major collection types (List, Map, Iterable, Collection and Set).

What a learning curve with ProxyGen -- I never used CodeDom before, but it's very cool. Powerful. My addition to ProxyGen is a big hack :) and I'm sure the stuff I wrote can be done much much neater, more generic or more robust, but again, it suits my needs. I am willing to share the changed/hacked sources... Let me know if interested.

So this is the approach I took (the stuff below is mainly to expose C# generics to Java, but a very similar approach can be done for Java to C# as well):

1. I wrote C# convertor classes that take a generic collection type and return the same elements only inside a non-generic collection type. Example System.Collections.Generic.IList<T> to System.Collection.IList. I made these convertors for each collection type mentioned above. They return the same elements, but inside a different container;

        public static IList ToIList<T>(IList<T> source) {

            return new ArrayList((List<T>)source);

        }

2. Then I changed ProxyGen to wrap the call to the __real object into these convertors and changed the type argument of the FullC2J to the non-generic collection:

        @__return = global::net.sf.jni4net.utils.Convertor.FullC2J<global::System.Collections.IList>(@__env, Mitza.Jni4Net.Convertor.ToIList(@__real.ComponentPresentations));

3. On the Java side, modified ProxyGen to change the signature of the native method to use the non-generic collection type:

        public native system.collections.IList getComponentPresentations();

4. Added a new delegate (the generic) that actually calls the non-generic native. The idea here is to be able to get the generics directly from the proxy class without me having to wrap them:

    public java.util.List<tridion.contentmanager.communicationmanagement.ComponentPresentation> getComponentPresentationsList() {

        return new mitza.jni4net.wrapper.List<tridion.contentmanager.communicationmanagement.ComponentPresentation>(getComponentPresentations());

    }

5. And of course I wrote the Java wrappers (for each collection type) to 'convert' the non-generic List (as per above) to the generic List<T>. All operations on these collections happen in fact on the wrapped non-generic proxy, so the collection elements are 'live' (modifications will show up immediately on the C# side):

        public class List<E extends system.Object> implements java.util.List<E> {

            private IList iList;

            public boolean add(E element) {

                iList.Add(element);

                return true;

            } //... etc

6. For the Java to C# generic support, I wrote the 'same' wrappers only this time on the C# side and they wrap the java.util.* collection proxy objects:

    public class List<T> : System.Collections.Generic.IList<T> where T : java.lang.Object {

        private java.util.List list;

        public int IndexOf(T item) {

            return list.indexOf(item);

        }

There are some limitations, as you were already pointing out

- I currently don't create new collections. That might be tricky. Creating new items in a collection it's not a problem

- Primitive wrapper type arguments are not supported for generics in ProxyGen. In this case I simply skip generating the Java delegate

- The wrappers have an erasure to the 'proxy' class. So in Java wrapper you have to use a C# proxy object that inherits from system.Object (the same in C# wrapper, have to use something from under java.lang.Object)

Overall I'm pretty pleased with the setup. It works nicely. But what a nightmare it was with the KeyValuePair<K, V> for types such as List<KeyValurPair<K, V>>. It is very difficult to nail a completely general solution.

I can do Java-style loops like this:

for (ComponentPresentation cp : page.getComponentPresentationsList()) {

    Component component = cp.getComponent();

    ComponentTemplate componentTemplate = cp.getComponentTemplate();

    System.out.println(String.format("\tCP: Component: '%s' %s + Component Template: '%s' %s",

            component.getTitle(), component.getId(), componentTemplate.getTitle(), componentTemplate.getId()));

}

Thanks again for creating jni4net and proxygen, they are such cool tools!

[Pavel Šavara]

Hi Mihai,

very nice achievement! 

Does your brain hurt from switching 2 type systems 10x per minute ?

:-D

I guess that lot of people would appreciate to use the code you created. 

I suggest that I will create new branch this and make you contributor.

Could you confirm that you are willing to publish it under GPLv3 license ? 

And also runtime parts (your  Mitza.Jni4Net namespace) under LGPLv3.

Regarding Microsoft code DOM is actually pretty poor. It doesn't support lot of C# features and Java generation is pure hack, also ms code dom is not easily extensible.

To address that for jni4net 0.9 I started new project here: 

http://code.google.com/p/polyglottos/source/browse/#svn%2Ftrunk

This is jni4net unrelated samples.

http://zamboch.blogspot.cz/2011/12/fluentator-generate-fluent-api-for-your.html

http://code.google.com/p/polyglottos/source/browse/trunk/polyglottos.test/src/CSharpCodeDomTest.cs

No  comments for rest of your points.

Thanks!

Pavel

[Mihai Cadariu]

Hi Pavel,

I was going crazy with understanding what gets executed where and in which context. Also type erasure is a nasty beast -- I had a hard time making those Java wrappers work. I was even looking at parameterizing type references (for generic arguments) using reflection (http://gafter.blogspot.com/2006/12/super-type-tokens.html), but luckily erasures worked for me here and I didn't have to do it myself.

I'm fine with the licensing aspects. Let me know where to check it in.


Cheers,

Mihai

2012/11/14 Pavel Šavara <pavel....@gmail.com>

[Pavel Šavara]

This is the branch for that code

https://jni4net.googlecode.com/svn/branches/0-8-generics/

I added your gmail address as project contributor, see your googlecode.com profile for svn password.

[Mihai Cadariu]

Done. Checked in.

[Oleg Tikhonov]

[Stephen Mobley]

Hey guys,

So, I wanted to use your 0.8-genetics version of proxygen with Mihai's code for the generic.  I was able to set everything up and compile the code to produce the proxygen.exe and config files.  But it seems that proxygen is still working the same way.  Has Mihai's code been incorporated to the build path for proxygen?  Or do I need to move things around to get it included? 

I'm using some .dll files for a C# api that I have no way to access the source code and I keep getting the "System.NullReferenceException: Object reference not set to an instance of an object" error.  Let me know what happened.  Thanks!

Stephen

On Thursday, November 15, 2012 1:22:24 AM UTC-5, Mihai Cadariu wrote:

Done. Checked in.

[Mihai Cadariu]

I added the distributables to svn. You can check http://jni4net.googlecode.com/svn/branches/0-8-generics/distributable/

[christian Jensen]

Hey all,

I have successfully compiled and been able to run a project with the 0.8.8.0 version of jni4net. But the code obviously fails with with the errors related to generics.

Then I discovered Mihai's branch on github.

So I cloned the repo and ran proxygen on my .dll the usual way from the /distributable/ dir and it completed successfully. Proxygen says it had fixed an IEnumerable for me.

But when I run build.cmd it says "error CS0103: The name 'Mitza' does not exist in the current context".

What obvious thing am I missing here? 

The signature looks like this: IEnumerable<SensorValue> Values { get; }

[Neha N]

Hi Mihai and Pavel,

Can I please have access to the jni4net code which includes generics. The link above doesnot exist anymore.

Thanks in advance!

regards,

Neha

This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). 

If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully;  (ii) contact Northgate Public Services immediately on +44(0)1442 768445 quoting the name of the sender and the addressee then delete it from your system.

Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted.  You should scan attachments (if any) for viruses.

Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW.  Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 400001.

[Pavel Šavara]

https://github.com/jni4net/jni4net/tree/0-8-generics

--

--
You received this message because you are subscribed to jni...@googlegroups.com
http://groups.google.com/group/jni4net?hl=en-GB?hl=en-GB
http://jni4net.sf.net/

---
You received this message because you are subscribed to the Google Groups "jni4net" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jni4net+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.