JPMML-SparkML-XGBoost

[Shihgian Lee]

Hi Villu,

I am trying to use JPMML-SparkML-XGBoost to convert XGBoost4J-Spark model using xgboost4j-spark version 0.9.0. I am learning XGBoost4J by following this example.

Initially, I received Transformer class ml.dmlc.xgboost4j.scala.spark.XGBoostClassificationModel is not supported error. Then, I found this issue that explains the error. I built the JPMML-SparkML-XGBoost locally and make sure the following is added to my project pom.xml:

<dependency>

    <groupId>org.jpmml</groupId>

    <artifactId>jpmml-sparkml-xgboost</artifactId>

    <version>1.0-SNAPSHOT</version>

</dependency>

Now, I am getting the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/jpmml/xgboost/HasXGBoostOptions

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:756)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)

at java.net.URLClassLoader.access$100(URLClassLoader.java:74)

at java.net.URLClassLoader$1.run(URLClassLoader.java:369)

at java.net.URLClassLoader$1.run(URLClassLoader.java:363)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:362)

at java.lang.ClassLoader.loadClass(ClassLoader.java:418)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)

at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

at org.jpmml.sparkml.ConverterFactory.init(ConverterFactory.java:198)

at org.jpmml.sparkml.ConverterFactory.init(ConverterFactory.java:161)

at org.jpmml.sparkml.ConverterFactory.<clinit>(ConverterFactory.java:237)

at org.jpmml.sparkml.PMMLBuilder.init(PMMLBuilder.java:469)

at org.jpmml.sparkml.PMMLBuilder.<clinit>(PMMLBuilder.java:533)

at com.porch.science.dataproc.SparkXGBoost4JTraining$.main(SparkXGBoost4JTraining.scala:114)

at com.porch.science.dataproc.SparkXGBoost4JTraining.main(SparkXGBoost4JTraining.scala)

Caused by: java.lang.ClassNotFoundException: org.jpmml.xgboost.HasXGBoostOptions

at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

at java.lang.ClassLoader.loadClass(ClassLoader.java:418)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)

at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

... 19 more

I suspect I am not using the plugin correctly. Can you please help to point out what I did wrong?

Thank you in advance for your help!

Thanks,

Shihgian

[Shihgian Lee]

Hi ViIllu,

I would like to clarify that I am running the code in IntelliJ and not through spark-shell so that I can debug my code.

Thanks!

[Villu Ruusmann]

Hi Shihgian,

>
> I would like to clarify that I am running the code in IntelliJ and
> not through spark-shell so that I can debug my code.
>

If you rely on IDEs, then you'll never learn to set up Java
application classpaths correctly and reproducibly.

> Exception in thread "main" java.lang.NoClassDefFoundError: org/jpmml/xgboost/HasXGBoostOption
>

The JPMML-SparkML-XGBoost library depends on the JPMML-XGBoost library:
https://github.com/jpmml/jpmml-sparkml-xgboost/blob/master/pom.xml#L48-L53

A NoClassDefFoundError that references org.jpmml.xgboost.* class(es)
indicates that this library is currently missing from your Java
application classpath.

The JPMML library ecosystem is heavily modularized for maximum
reusability. It's very tedious to identify and keep track of all those
JPMML library interdependencies manually. Be sure to switch to a
proper build tool such as Apache Maven.

For information about JPMML library dependencies, you may run Apache
Maven command "mvn dependency:tree" in JPMML-SparkML-XGBoost root
directory. It shows that you need to include JPMML-XGBoost,
JPMML-Converter and JPMML-Model libraries at minimum.


VR