How does the classloader for plugins differentiate from the standard java classloader

[Martin Thurau]

Hello everyone

I'm currently developing a custom transformation plugin. This code
uses Apache CXF to do some webservice calls. The code works if I
execute it outside of Kettle but dies with a NullPointerException as a
plugin. I have debugged the CXF code that gets executed and to me it
looks like it loads "some" classes from "somewhere" "somehow".
I have no detailed information as the code in question isn't
documented. I have already asked at the CXF mailinglist if a custom
classloader could somehow break the CXF code but haven't got a answer
some far.

To my understanding a Kettle plugin is executed with a custom
classloader in it's context, so I suspect, that this may the root
cause of the problem. So my question is: does the custom classloader
of Kettle does something different from the standard java classloader?
Does it have some restrictions to be aware of?

I understand that this is a very unspecific question but I think if I
could provide the CXF developers with some more informations what the
context of the bug (and to me, this is a bug, as code shouldn't
simeply die with a NullPointerExecption) I can somehow solve my
problem. So any assumption, no matter how vague, would help.

Regards
Martin Thurau

[Gabriele]

Hi Martin,

did you try to catch the "Throwable" excpetion instead of the normal "Exception"? The Throwable exception is able to catch even the NoClassDefFound exception that I think may be your problem!

regards.

Gabriele.

---

Da: Martin Thurau <martin...@gmail.com>
A: kettle-developers <kettle-d...@googlegroups.com>
Inviato: Mercoledì 11 Maggio 2011 16:57
Oggetto: How does the classloader for plugins differentiate from the standard java classloader

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-d...@googlegroups.com.
To unsubscribe from this group, send email to kettle-developers+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

[Nick Baker]

Kettle's Plugin Classloaders are different in that they're "inverted" or self-first. A plugin's classloader will first try to find a class within itself and look up to the parent classloader only when it's not found. This allows us to supply a different version of a particular library by placing it in the plugin's local lib folder. 

I'm suspect that this would cause an issue with CXF as it is very similar to how most application servers isolate webapps. Though if it is you may have luck setting the Thread's context classloader temporarily to your plugin classloader:

Classloader originalClassloader = Thread.currentThread().getContextClassLoader();

Thread.currentThread().setContextClassLoader(MyPluginClass.class.getClassLoader());

...//Execute library code

Thread.currentThread().setContextClassLoader(originalClassloader);

-Nick B

To unsubscribe from this group, send email to kettle-develop...@googlegroups.com.

[Martin Thurau]

Yes, this works for me. I have added to following statement before
creating my WS-Client and the error is gone:

Thread.currentThread().setContextClassLoader(getClass().getClassLoader());

This seams to be an issue with Apache CXF 2.4 since other users had
similar problems when using CXF in a context where the classloader was
something non-default. So it wasn't Kettles fault ;-)
Thank you for the information.

Regards
Martin

On May 12, 6:57 pm, Nick Baker <codeoncof...@gmail.com> wrote:
> Kettle's Plugin Classloaders are different in that they're "inverted" or
> self-first. A plugin's classloader will first try to find a class within
> itself and look up to the parent classloader only when it's not found. This
> allows us to supply a different version of a particular library by placing
> it in the plugin's local lib folder.
>
> I'm suspect that this would cause an issue with CXF as it is very similar to
> how most application servers isolate webapps. Though if it is you may have
> luck setting the Thread's context classloader temporarily to your plugin
> classloader:
>
> Classloader originalClassloader =
> Thread.currentThread().getContextClassLoader();
> Thread.currentThread().setContextClassLoader(MyPluginClass.class.getClassLo ader());
> ...//Execute library code
> Thread.currentThread().setContextClassLoader(originalClassloader);
>
> -Nick B
>
>
>
>
>
>
>
> On Wed, May 11, 2011 at 6:16 PM, Gabriele <ic...@yahoo.it> wrote:
> > Hi Martin,
>
> > did you try to catch the "Throwable" excpetion instead of the normal
> > "Exception"? The Throwable exception is able to catch even the
> > NoClassDefFound exception that I think may be your problem!
>
> > regards.
>
> > Gabriele.
>

> > ------------------------------
> > *Da:* Martin Thurau <martin.thu...@gmail.com>
> > *A:* kettle-developers <kettle-d...@googlegroups.com>
> > *Inviato:* Mercoledì 11 Maggio 2011 16:57
> > *Oggetto:* How does the classloader for plugins differentiate from the

[kepha]

Hi,
I found this thread and I see it's old but I'm experiencing similar problems now so I thought better not making separate thread.

I'm doing some coding inside Kettle 4.4.0 but for this coding I need to use and access pentaho-big-data-plugin.

Inside the Kettle I have put big data plugin's jar to its class path since I need to have access to it's classes and the plugin itself I put in the kettle-home/plugins directory. Now when I do the following casting to get the Mapper's ktr filename from Pentaho MapReduce entry:

JobEntryCopy jge = jobMeta.getJobEntry(i);
String mapperFilename = ((JobEntryHadoopTransJobExecutor)jge.getEntry()).getMapTrans();

the second line throws the exception
java.lang.ClassCastException: org.pentaho.di.job.entries.hadooptransjobexecutor.JobEntryHadoopTransJobExecutor cannot be cast to org.pentaho.di.job.entries.hadooptransjobexecutor.JobEntryHadoopTransJobExecutor

Which is caused because the JobEntryHadoopTransJobExecutor is loaded from the current context ClassLoader and the getEntry will return JobEntryHadoopTransJobExecutor loaded from the plugin's custom ClassLoader.

I tried changing the context class loader  with the code you provided in the following manener:

Thread.currentThread().setContextClassLoader(jge.getEntry().getClass().getClassLoader());

But this didn't help, JobEntryHadoopTransJobExecutor  is again loaded from the default ClassLoader.

Do you have any suggestion how can I solve this issue?
How can I force casting with the class loaded from plugin's classLoader?


Thanks!

[Matt Casters]

You can get the class loader for any given plugin class with

PluginRegistry.getInstance().getClassLoader(jge.getEntry());

I hope that helps you to set the context class loader.

Matt

2012/8/29 kepha <petar...@gmail.com>

To view this discussion on the web visit https://groups.google.com/d/msg/kettle-developers/-/rRL_1KiQNrAJ.

To post to this group, send email to kettle-d...@googlegroups.com.
To unsubscribe from this group, send email to kettle-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

--
Matt Casters <mcas...@pentaho.org>
Chief Data Integration, Kettle founder, Author of Pentaho Kettle Solutions (Wiley)
Fonteinstraat 70 - 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37

Pentaho  -  Powerful Analytics Made Easy

[Petar Jovanovic]

This gives the following compile-time error:
The method getClassLoader(PluginInterface) in the type PluginRegistry is not applicable for the arguments (JobEntryInterface)

And I think with jge.getEntry().getClass().getClassLoader() I also get the correct ClassLoader but when I set the context one to that it it seems it still doesn't use it for loading the class
JobEntryHadoopTransJobExecutor.

 

2012/8/28 Matt Casters <mcas...@pentaho.org>

[Jordan Ganoff]

You can retrieve the PluginInterface for the Pentaho MapReduce job entry from the PluginRegistry with:

PluginRegistry.getInstance().findPluginWithId(JobEntryPluginType.class, "HadoopTransJobExecutorPlugin");

You can then use PluginRegistry.getInstance().getClassLoader(..) to look up the class loader for that job entry plugin.

Now, not knowing what you're doing, could your changes be made in the big data plugin itself? I also recommend leaving the Big Data plugin jars in the plugin folder. If there is specific functionality you need in another plugin perhaps you can refactor that into a separate library.

- Jordan

[Petar Jovanovic]

Well even if I refactor specific functionality it will be loaded differently from the plugin's classes.
I need to use some functionalities of big data plugin so I include plugin's jar on the class path of the kettle's project but the plugin itself is loaded by it's own classloader so the clases loaded in kettle (from jar) even though they are the same are considered different since they are loaded by different class loaders.
What I want to do for the parts where I'm mixing plugins and kettles loaded classes to is reload all used classes with only one classloader (plugins).

Do you have some other suggestion? How can I access plugin's classes from kettle if I don't include it in the class path?

Thanks

2012/8/28 Jordan Ganoff <jga...@pentaho.com>

[Matt Casters]

The fact that job entries represent the discreet and isolated execution of a task is rather an important concept in Kettle.  I would hate for another job entry to "magically" execute functionality in another part of a job.  That is why I would insist on keeping the plugin classes loaded in a separate class loader.  

Since we're talking about the re-use of the execution engine (finding metadata is easy) the important question becomes: what "certain functionalities" would you want to re-use?

Matt

2012/8/30 Petar Jovanovic <petar...@gmail.com>

[Petar Jovanovic]

Sorry maybe I didn't explain good. By functionality I meant functionality of the plugin's classes not execution functionality of it, what indeed I need is to get the specific metadata information from plugin's entries (e.g., mapper/reducer transformation, cluster information) but without getting the complete xml, just some parts. I know I can get the complete xml but it would be better to get just the parts I need (in order not to parse xml). 
How can I get these?

Thanks

[Matt Casters]

Thanks for clarifying.  In that case I think you need to indeed set the correct context class loader.

Try using the class loader from parentJob.  That should be the one that loaded the metadata.

Thread.currentThread().setContextClassLoader(parentJob.getClass().getClassLoader());

JobEntryCopy jobEntryCopy = parentJob.getJobMeta().findJobEntry("Name", 0, false);

JobEntryHadoopTransJobExecutor = (JobEntryHadoopTransJobExecutor)jobEntryCopy.getEntry();

Sorry to say I haven't tried it myself.

Beste of luck,

Matt

[Petar Jovanovic]

Yes that is what I've been trying but it's not working.
For bigdata job entries jobEntryCopy.getEntry() returns JobEntryHadoopTransJobExecutor loaded by plugin's loader, so I was trying to set context loader to the one of the plugin hoping that in that case casting class
(JobEntryHadoopTransJobExecutor) will be loaded from plugin's loader but it doesn't work, I guess because this class is already loaded with the kettle's loader (from pentaho-big-data-plugin.jar).
So what I might need to do is to reload that class after changing the context loader but cannot find a way to do it.

2012/8/30 Matt Casters <mcas...@pentaho.org>

[Matt Casters]

OK, so in je2 from classloader2 you want to reference je1 from classloader1.

If we do the following it fails:

Classloader2.JobExecutor je2 = (ClassLoader2.JobExecutor)je1

This makes sense actually since je1 is of class ClassLoader1.JobExecutor.

So perhaps you need to set the context loader to the first class loader.

Thread.currentThread().setContextClassLoader(je1.getClass().getClassLoader());  After that the class cast would become

Classloader1.JobExecutor je2 = (ClassLoader1.JobExecutor)je1

At least, in theory :-)  

Or perhaps you could clone() the metadata and work with that.

Best of luck,

Matt

[Matt Casters]

A a more general piece of advice I would recommend using variables to indicate the metadata pieces that are needed in multiple locations.  That way you wouldn't even have to look up the information in the other job entry.

Matt

2012/8/30 Matt Casters <mcas...@pentaho.org>

[Petar Jovanovic]

Exactly what I tried but it seems even after switching the context loader the class JobExecutor, since it is already loaded from the kettle's loader, is not reloaded from plugin's loader.

What I do is accessing the complete Job metadata and for each job entry some specific parts of its metadata. So for the standard kettle entries I can do the cast but for big data ones since they come form the plugin I cannot.

I'm trying to figure out how to dynamically reload the class (that is previously loaded from one class loader) with different class loader.

Thanks for help and support!

2012/8/30 Matt Casters <mcas...@pentaho.org>

[Matt Casters]

I still don't know what you're exactly trying to do.  It sounds pretty strange for a job entry to access other job entry metadata.

2012/8/30 Petar Jovanovic <petar...@gmail.com>

[Petar Jovanovic]

No no I do not access from one job entry another job entry metadata .

I first access  JobMeta and get the entries of that job. Then for each entry I want to get specific metadata of that job entry. So without casting I can only access common metadata defined by JobEntryInterface but I would like to access some specific parts of each entry.

2012/8/30 Matt Casters <mcas...@pentaho.org>

[Matt Casters]

Ah, that explains what you're not doing.  Are you building some form of metadata system then?

2012/8/30 Petar Jovanovic <petar...@gmail.com>

[Petar Jovanovic]

Yes, something like that. I'm trying to enable translation from Kettle's metadata into another format.

2012/8/30 Matt Casters <mcas...@pentaho.org>

[Matt Casters]

Well we created the Repository plugin system for that.  You could use something like that.

2012/8/30 Petar Jovanovic <petar...@gmail.com>