uima/uimafit classloading issues

[jimpi...@gmail.com]

Hi all, 

I'm trying to write a thin uima wrapper for Clojure. As soon as I discovered uimafit I was literally ecstatic as I too, cannot stand writing XML descriptors. MY problems is the following:

UIMA uses its own classloader (which wraps URLClassloader if I'm not mistaken) and doesn't allow injection of pre-exisiting object instances. This basically means that in order to call createPrimitive() the actual class file must be on the classpath on disk.  However, I am generating classes from Clojure that reside on memory and not on disk! URLClassloader cannot find them even though the classes do exist! 

I tried fiddling with Classloaders but no success...More specifically I tried forcing UIMA to use the DynamicClassLoader that Clojure uses...Then I tried using the experimental org.uimafit.util.SimpleNamedResourceManager but again no success...This is stating to drive me up the wall...

any classloader gurus out there? any help/insight will be greatly appreciated.... :)

Jim

[Richard Eckart de Castilho]

Hi,

> I'm trying to write a thin uima wrapper for Clojure. As soon as I discovered uimafit I was literally ecstatic as I too, cannot stand writing XML descriptors. MY problems is the following:
> UIMA uses its own classloader (which wraps URLClassloader if I'm not mistaken) and doesn't allow injection of pre-exisiting object instances.

UIMA doesn't allow injection of pre-existing instances at all, only the experimental stuff you mention below hacks that in.

> This basically means that in order to call createPrimitive() the actual class file must be on the classpath on disk. However, I am generating classes from Clojure that reside on memory and not on disk! URLClassloader cannot find them even though the classes do exist!

In particular, the class must be resolvable from the classloader that was used to load the UIMA framework - or (what should also work) it must be resolvable from the extension loader set in the resource manager (see below). The UIMAClassloader extends the URLClassloader, but it can defer loading classes to a "paren" classloader as well. If that "parent" can resolve your Clojure classes, so should UIMA be able to do it.

I never tracked this down, but unless you are using PEARs, UIMA should not fiddle around with classloading a lot. To resolve the classes, it should use the classloader that the UIMA framework classes are instantiated with. If you can make sure that this classloader has access to the Clojure classes, all should be fine.

> I tried fiddling with Classloaders but no success...More specifically I tried forcing UIMA to use the DynamicClassLoader that Clojure uses...Then I tried using the experimental org.uimafit.util.SimpleNamedResourceManager but again no success...This is stating to drive me up the wall…

Classloading issues tend to have that effect…

> any classloader gurus out there? any help/insight will be greatly appreciated.... :)

If you already got as far as configuring your own ResourceManager, try using the setExtensionClassPath(parent, classpath, resolveResource) method to reconifgure the UIMA classloader used for your components and set the Clojure classloader instance you need as "parent".

That's all for UIMA. No, uimaFIT doesn't allow any setting of classloaders so far. That means, that uimaFIT can only work with classes if all their static dependencies are resolvable from the same classloader that is used to resolve the uimaFIT classes.

I suppose the best bet you have to trying to make sure that your Clojure Classloader is very close to the root of your classloader hierarchy and that all classloaders used to instantiate UIMA/uimaFIT stuff have that as a direct or indirect parent.

Cheers,

-- Richard

[Richard Eckart de Castilho]

>> I'm trying to write a thin uima wrapper for Clojure. As soon as I discovered uimafit I was literally ecstatic as I too, cannot stand writing XML descriptors. MY problems is the following:
>> UIMA uses its own classloader (which wraps URLClassloader if I'm not mistaken) and doesn't allow injection of pre-exisiting object instances.

Btw. what direction are you trying?

- Call Clojure from UIMA?
- Call UIMA from Clojure?
- Both?

-- Richard

[jimpi...@gmail.com]

Wow that was a prompt reply! Thanks a lot for your time Richard :)

Your answer almost covers me 100% so I won't bother you for long...1 last question though:

Would the following work or do I have to write my own manager?   

UIMAFRamework.newDefaultResourceManager().setExtensionClassPath(dynamicClassloader, "", true); 

if yes, what exactly would go as the argument in the middle (the classpath string)? what classpath do I want to pass in? I'm slightly confused...

again, thanks a lot... :)

Jim

[jimpi...@gmail.com]

Wow that was a prompt reply! Thanks a lot for your time Richard :)

Your answer almost covers me 100% so I won't bother you for long...1 last question though:

Would the following work or do I have to write my own manager?   

UIMAFRamework.newDefaultResourceManager().setExtensionClassPath(dynamicClassloader, "", true); 

if yes, what exactly would go as the argument in the middle (the classpath string)? what classpath do I want to pass in? I'm slightly confused...

again, thanks a lot... :)

Jim

[jimpi...@gmail.com]

Calling UIMA from Clojure mainly ...

Jim

[jimpi...@gmail.com]

Basically I want to provide a concise and idiomatic way to be able to make any class/functiom/component uima-compatible. I can easily do that using somehting like this:

(defn uima-compatible "Generate an UIMA-friendly class that delegates to  the specified component." 

 [component jcas-input-extractor]

 (proxy [org.uimafit.component.JCasAnnotator_ImplBase][] ;;extend the base class

   (process [^JCas jc] 

    (let [xs (jcas-input-extractor jc)] 

      (component xs))))) ;;assuming component is a function for now

But then I'm stuck, because the generated class (from 'proxy') resides on memory...After I resolve this, the next step would be to start constructing pipelines and so on...So yes it is Clojure calling UIMA....

Jim

[Richard Eckart de Castilho]

> Wow that was a prompt reply! Thanks a lot for your time Richard :)
>
> Your answer almost covers me 100% so I won't bother you for long...1 last question though:
> Would the following work or do I have to write my own manager?
> UIMAFRamework.newDefaultResourceManager().setExtensionClassPath(dynamicClassloader, "", true);
> if yes, what exactly would go as the argument in the middle (the classpath string)? what classpath do I want to pass in? I'm slightly confused...

I'd try passing in an empty string (as you do). After all, you don't want to add any JARs from the file-system, you only want to make sure that the dynamicClassLoader is the parent.

If that didn't work for you so far, it'd require a more in-depth analysis of the classloader hierarchy. Mind, that its not only important which classloader is used to resolve a class, but also which classloader within the hierarchy is eventually used to define/instantiate the class. E.g. consider this:

CL1 - root loader
CL2 - loader used for UIMA
CL3 - clojure loader

So if you call UIMA from a class that's been instantiated by CL3, it works. But UIMA classes that are instantiated by CL2 would not be able to "see" any classes that CL3 provides.

If you want to call UIMA from Clojure, you may get somewhere setting the extensionClassPath, but you may also run into a dead end.

One way out of that may be to implement your own classloader CL something along this:

- Clojure DynamicClassLoader must use CL as a parent, so that it can access classes loaded by it
- CL knows about the Clojure DynamicClassLoader and tries to load classes by it *first*
- if not found, tries to locate the *class file*! (not the class) as a resource and
define the class itself
- if no class file was found, try to defer to the parent classloader

If I didn't miss anything, that should more-or-less create a situation where all Clojure and Java classes loaded via that CL can see each other. Of course there must be some (probably thread-safe) recursion cancellation in that.

In the end, you'd need a boots-trapping class which you manually load via your new CL (to make sure it's not loaded via the system classloader), and which contains then calls your actual logic.

I'd be interested in how you eventually resolved the setup.

Cheers,

-- Richard

[Richard Eckart de Castilho]

Am 14.04.2013 um 13:12 schrieb jimpi...@gmail.com:

> Basically I want to provide a concise and idiomatic way to be able to make any class/functiom/component uima-compatible. I can easily do that using somehting like this:
>
> (defn uima-compatible "Generate an UIMA-friendly class that delegates to the specified component."
> [component jcas-input-extractor]
> (proxy [org.uimafit.component.JCasAnnotator_ImplBase][] ;;extend the base class
> (process [^JCas jc]
> (let [xs (jcas-input-extractor jc)]
> (component xs))))) ;;assuming component is a function for now
>
> But then I'm stuck, because the generated class (from 'proxy') resides on memory...After I resolve this, the next step would be to start constructing pipelines and so on...So yes it is Clojure calling UIMA....

At that point it's Clojure calling UIMA. But later, when you want to use that component, it's probably UIMA instantiating your class by name. At that point, if you set the extensionloader, it may work.

-- Richard

[Philip Ogren]

Jim,

If you get this working, it would be nice to have a concise summary of what you did along with an example that would be suitable for the wiki.

I'm pretty sure that Steve has done something similar with Scala so maybe we could squeeze out the same from him and have a section in the documentation called "calling UIMA from JVM-based languages.

Thanks,
Philip

Sent from my iPhone

> --
> You received this message because you are subscribed to the Google Groups "uimafit-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to uimafit-user...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Richard Eckart de Castilho]

I've already done UIMA + Groovy, which worked just fine. I think Clojure is special here because of classes being dynamically compiled at run-time. Groovy classes get compiled during a regular build. No idea about Scala, but I think it's also compiled.

-- Richard

[Philip Ogren]

Yeah - I don't think Scala has the same class loader issues.  I was trying to make a broader comparison related to JVM-based languages.  

[jimpi...@gmail.com]

the eclipse debugger gives me more detailed stack trace caused by the exception the terminal was showing to me :

Exception in thread "main" org.apache.uima.resource.ResourceInitializationException: Annotator class "hotel_nlp.externals.uima.proxy$org.uimafit.component.JCasAnnotator_ImplBase$0" was not found. (Descriptor: <unknown>)

at org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.initializeAnalysisComponent(PrimitiveAnalysisEngine_impl.java:207)

at org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.initialize(PrimitiveAnalysisEngine_impl.java:156)

at org.apache.uima.impl.AnalysisEngineFactory_impl.produceResource(AnalysisEngineFactory_impl.java:94)

at org.apache.uima.impl.CompositeResourceFactory_impl.produceResource(CompositeResourceFactory_impl.java:62)

at org.apache.uima.UIMAFramework.produceResource(UIMAFramework.java:269)

at org.apache.uima.UIMAFramework.produceAnalysisEngine(UIMAFramework.java:387)

at hotel_nlp.externals.uima$produce.doInvoke(uima.clj:110)

at clojure.lang.RestFn.invoke(RestFn.java:464)

at hotel_nlp.externals.uima$eval3811.invoke(uima.clj:146)

at clojure.lang.Compiler.eval(Compiler.java:6619)

at clojure.lang.Compiler.load(Compiler.java:7064)

at clojure.lang.RT.loadResourceScript(RT.java:370)

at clojure.lang.RT.loadResourceScript(RT.java:357)

at clojure.lang.RT.loadResourceScript(RT.java:349)

at clojure.main$load_script.invoke(main.clj:293)

at clojure.main$init_opt.invoke(main.clj:299)

at clojure.main$initialize.invoke(main.clj:327)

at clojure.main$null_opt.invoke(main.clj:362)

at clojure.main$main.doInvoke(main.clj:440)

at clojure.lang.RestFn.invoke(RestFn.java:512)

at clojure.lang.Var.invoke(Var.java:435)

at clojure.lang.AFn.applyToHelper(AFn.java:185)

at clojure.lang.Var.applyTo(Var.java:532)

at clojure.main.main(main.java:37)

Caused by: java.lang.ClassNotFoundException: hotel_nlp.externals.uima.proxy$org.uimafit.component.JCasAnnotator_ImplBase$0

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:423)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:356)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:186)

at org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.initializeAnalysisComponent(PrimitiveAnalysisEngine_impl.java:204)

... 23 more

[jimpi...@gmail.com]

Finally some progress! not much though... it turns out the string parameters were malformed (e.g "PARAM_RESOURCE_MANAGER" instead of "RESOURCE_MANAGER")...what a stupid mistake on my part...HOwever now things are not better...look at the following which is even more confusing than the previous stacktrace:

ClassNotFoundException org.springframework.beans.PropertyAccessorFactory

java.net.URLClassLoader$1.run (URLClassLoader.java:366)

java.net.URLClassLoader$1.run (URLClassLoader.java:355)

java.security.AccessController.doPrivileged (AccessController.java:-2)

java.net.URLClassLoader.findClass (URLClassLoader.java:354)

java.lang.ClassLoader.loadClass (ClassLoader.java:423)

sun.misc.Launcher$AppClassLoader.loadClass (Launcher.java:308)

java.lang.ClassLoader.loadClass (ClassLoader.java:356)

org.springframework.validation.DirectFieldBindingResult.createDirectFieldAccessor (DirectFieldBindingResult.java:81)

org.springframework.validation.DirectFieldBindingResult.getPropertyAccessor (DirectFieldBindingResult.java:69)

org.springframework.validation.DataBinder.getPropertyEditorRegistry (DataBinder.java:288)

org.springframework.validation.DataBinder.registerCustomEditor (DataBinder.java:535)

org.uimafit.propertyeditors.PropertyEditorUtil.registerUimaFITEditors (PropertyEditorUtil.java:38)

Where on earth did SPRING come from ?! IT seems the classloading issue for my proxied class dissapeared but something else broke!

many many thanks... 

Jim

[Richard Eckart de Castilho]

Well, uimaFIT uses Spring to convert parameter values and to scan the classpath for descriptors. It's just much more convenient to use their stuff than to implement all again. However, that looks like a normal "JAR missing on classpath" issue.

Cheers,

-- Richard

[Jim - FooBar();]

On 14/04/13 17:30, Richard Eckart de Castilho wrote:
> However, that looks like a normal "JAR missing on classpath" issue.

well, my maven dependencies only point to uima-core 2.4.0 & uimafit
1.4.0...should I point somewhere else as well?

Jim

[Richard Eckart de Castilho]

> On 14/04/13 17:30, Richard Eckart de Castilho wrote:
>> However, that looks like a normal "JAR missing on classpath" issue.
>
> well, my maven dependencies only point to uima-core 2.4.0 & uimafit 1.4.0...should I point somewhere else as well?

uimaFIT 1.4.0 depends on spring-core and spring-context. spring-context depends on spring-beans. The class "org.springframework.beans.PropertyAccessorFactory" is in spring-beans. Maybe I should add spring-beans as a direct dependency in the next uimaFIT release…

So far I have heard of no problems of Maven resolving that. Maybe you have some other dependencies that "downgrade" Spring below 3.1.x?

-- Richard

[Jim - FooBar();]

On 14/04/13 17:51, Richard Eckart de Castilho wrote:
> uimaFIT 1.4.0 depends on spring-core and spring-context. spring-context depends on spring-beans. The class "org.springframework.beans.PropertyAccessorFactory" is in spring-beans. Maybe I should add spring-beans as a direct dependency in the next uimaFIT release�

>
> So far I have heard of no problems of Maven resolving that. Maybe you have some other dependencies that "downgrade" Spring below 3.1.x?
>
> -- Richard

Nice catch RIchard, you're the man! :)

gate-core depends on the arcane spring-beans 2.0.8...added an exclusion,
the new dependency was fetched and everything worked! I finally got my
primitive engine that delegates to my own class...thank you thank you
thank you... :) let me know what you need from me for that wiki example
Philip talked about...


Jim

[Richard Eckart de Castilho]

Am 14.04.2013 um 18:59 schrieb "Jim - FooBar();" <jimpi...@gmail.com>:

> On 14/04/13 17:51, Richard Eckart de Castilho wrote:

>> uimaFIT 1.4.0 depends on spring-core and spring-context. spring-context depends on spring-beans. The class "org.springframework.beans.PropertyAccessorFactory" is in spring-beans. Maybe I should add spring-beans as a direct dependency in the next uimaFIT release…

>>
>> So far I have heard of no problems of Maven resolving that. Maybe you have some other dependencies that "downgrade" Spring below 3.1.x?
>>
>> -- Richard
>
> Nice catch RIchard, you're the man! :)
>
> gate-core depends on the arcane spring-beans 2.0.8...added an exclusion, the new dependency was fetched and everything worked! I finally got my primitive engine that delegates to my own class...thank you thank you thank you... :) let me know what you need from me for that wiki example Philip talked about...

A working example of how you created an UIMA component in Clojure and how you ran it would be great. Since the migration of uimaFIT to Apache, the wiki is kind of phasing out. I have already converted most of the stuff to the DocBook format that Apache UIMA uses for it's documentation.

If you compile a little Clojure/uimaFIT example/tutorial, it would be great if you could post it to the Apache Jira and contribute it to Apache uimaFIT, so I can include it in the documentation. Depending on when you manage to write something up, it could also make sense to publish it before the next uimaFIT release on the Google Code page.

Cheers,

-- Richard

[Richard Eckart de Castilho]

>> gate-core depends on the arcane spring-beans 2.0.8...added an exclusion, the new dependency was fetched and everything worked! I finally got my primitive engine that delegates to my own class...thank you thank you thank you... :) let me know what you need from me for that wiki example Philip talked about...
>
> A working example of how you created an UIMA component in Clojure and how you ran it would be great. Since the migration of uimaFIT to Apache, the wiki is kind of phasing out. I have already converted most of the stuff to the DocBook format that Apache UIMA uses for it's documentation.
>
> If you compile a little Clojure/uimaFIT example/tutorial, it would be great if you could post it to the Apache Jira and contribute it to Apache uimaFIT, so I can include it in the documentation. Depending on when you manage to write something up, it could also make sense to publish it before the next uimaFIT release on the Google Code page.

You shouldn't worry about the format. Just any format would be fine, I'll format it appropriately, depending on where it's included.

-- Richard